Elasticsearch User Guide

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. At its core, it stores JSON documents in an index, makes them searchable in near real-time, and lets you run aggregations across billions of records in milliseconds. Version 9.x introduces ES|QL (a pipe-based query language), expanded generative AI inference endpoints, and a unified data stream and lifecycle management model.

This guide covers everything you need to run Elasticsearch day-to-day: installing it, configuring it, indexing data, querying it in every supported style, and operating it reliably.

Table of Contents

1. Quick Start
2. Installation
3. Configuration
4. Core Usage
5. Common Tasks
6. Troubleshooting
7. FAQ

1. Quick Start

Get a single-node Elasticsearch cluster running in under five minutes.

1.1 Docker (fastest path)

docker pull docker.elastic.co/elasticsearch/elasticsearch:9.5.0

docker run -d \
  --name es01 \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  docker.elastic.co/elasticsearch/elasticsearch:9.5.0

Wait about 20 seconds, then verify:

curl http://localhost:9200/

Expected output:

{
  "name" : "es01",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "abc123xyz",
  "version" : {
    "number" : "9.5.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "lucene_version" : "10.1.0",
    "minimum_wire_compatibility_version" : "8.17.0",
    "minimum_index_compatibility_version" : "8.0.0"
  },
  "tagline" : "You Know, for Search"
}

1.2 Index your first document

curl -X PUT "localhost:9200/products/_doc/1" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Wireless Headphones",
    "brand": "SoundMax",
    "price": 79.99,
    "in_stock": true,
    "description": "Over-ear noise-cancelling bluetooth headphones with 30-hour battery"
  }'

1.3 Search it

curl "localhost:9200/products/_search?q=noise-cancelling&pretty"

curl -X GET "localhost:9200/products/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "match": {
        "description": "noise cancelling"
      }
    }
  }'

Expected output:

{
  "took" : 3,
  "timed_out" : false,
  "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 },
  "hits" : {
    "total" : { "value" : 1, "relation" : "eq" },
    "max_score" : 0.8630463,
    "hits" : [
      {
        "_index" : "products",
        "_id" : "1",
        "_score" : 0.8630463,
        "_source" : {
          "name" : "Wireless Headphones",
          "brand" : "SoundMax",
          "price" : 79.99,
          "in_stock" : true,
          "description" : "Over-ear noise-cancelling bluetooth headphones with 30-hour battery"
        }
      }
    ]
  }
}

Tip

Append ?pretty to any API call to get human-readable JSON. Use ?human to format byte counts and durations as readable strings (e.g., "1.2gb" instead of 1289748480). These parameters are accepted on every Elasticsearch endpoint.

1.4 Five-minute tour

# Check cluster health
curl "localhost:9200/_cluster/health?pretty"

# List all indices
curl "localhost:9200/_cat/indices?v"

# Bulk index three documents
curl -X POST "localhost:9200/products/_bulk?pretty" \
  -H "Content-Type: application/x-ndjson" \
  -d '
{"index":{"_id":"2"}}
{"name":"USB-C Hub","brand":"TechLink","price":39.99,"in_stock":true,"description":"7-port hub with 4K HDMI and 100W PD"}
{"index":{"_id":"3"}}
{"name":"Mechanical Keyboard","brand":"KeyCraft","price":129.00,"in_stock":false,"description":"TKL layout with Cherry MX Blue switches"}
{"index":{"_id":"4"}}
{"name":"Webcam HD","brand":"VisionPro","price":59.99,"in_stock":true,"description":"1080p 30fps autofocus USB camera"}
'

# Range query + aggregation
curl -X GET "localhost:9200/products/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "range": { "price": { "lte": 80 } } },
    "aggs": {
      "avg_price": { "avg": { "field": "price" } },
      "by_brand": { "terms": { "field": "brand.keyword" } }
    }
  }'

2. Installation

2.1 System requirements

Important

Elasticsearch ships with a bundled JDK. Do not set JAVA_HOME to point to a system Java. Using a mismatched JDK version can silently corrupt data or cause startup failures.

2.2 Install on Linux (tar.gz)

# Download
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-9.5.0-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-9.5.0-linux-x86_64.tar.gz.sha512
shasum -a 512 -c elasticsearch-9.5.0-linux-x86_64.tar.gz.sha512

# Extract
tar -xzf elasticsearch-9.5.0-linux-x86_64.tar.gz
cd elasticsearch-9.5.0/

# Start
./bin/elasticsearch

On first start with security enabled (default), Elasticsearch prints the elastic superuser password and enrollment tokens to stdout — save them.

2.3 Install on Debian/Ubuntu (apt)

# Import GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
  sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

# Add repository
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] \
  https://artifacts.elastic.co/packages/9.x/apt stable main" | \
  sudo tee /etc/apt/sources.list.d/elastic-9.x.list

# Install
sudo apt-get update && sudo apt-get install elasticsearch

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

2.4 Install on RHEL/CentOS (rpm)

# Import GPG key
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# Add repository /etc/yum.repos.d/elasticsearch.repo
cat <<EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 9.x packages
baseurl=https://artifacts.elastic.co/packages/9.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
EOF

sudo yum install --enablerepo=elasticsearch elasticsearch
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

2.5 Docker

# Single node, security disabled (development only)
docker run -d \
  --name es01 \
  --net elastic \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  -v esdata01:/usr/share/elasticsearch/data \
  docker.elastic.co/elasticsearch/elasticsearch:9.5.0

# Single node, security enabled (stores certs + password in volumes)
docker run -d \
  --name es01 \
  --net elastic \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  -v esdata01:/usr/share/elasticsearch/data \
  -v escerts01:/usr/share/elasticsearch/config/certs \
  docker.elastic.co/elasticsearch/elasticsearch:9.5.0

Warning

Always mount a named volume for /usr/share/elasticsearch/data. Without a persistent volume, all indexed data is lost when the container stops.

2.6 Docker Compose (multi-node cluster)

# docker-compose.yml
version: '3.8'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.0
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.0
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock: { soft: -1, hard: -1 }
    volumes:
      - esdata02:/usr/share/elasticsearch/data
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.0
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock: { soft: -1, hard: -1 }
    volumes:
      - esdata03:/usr/share/elasticsearch/data
    networks:
      - elastic

  kibana:
    image: docker.elastic.co/kibana/kibana:9.5.0
    ports:
      - 5601:5601
    environment:
      - ELASTICSEARCH_HOSTS=http://es01:9200
    networks:
      - elastic

volumes:
  esdata01:
  esdata02:
  esdata03:

networks:
  elastic:
    driver: bridge

docker compose up -d

2.7 Verify the installation

# Without security
curl http://localhost:9200/_cluster/health?pretty

# With security (replace PASSWORD with the one printed at first start)
curl -u elastic:PASSWORD https://localhost:9200/_cluster/health?pretty --cacert config/certs/http_ca.crt

Expected healthy output:

{
  "cluster_name" : "my-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 10,
  "active_shards" : 20,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

2.8 Plugin management

Elasticsearch plugins are managed declaratively via elasticsearch-plugins.yml in the config directory, or imperatively via the elasticsearch-plugin tool.

Declarative (recommended for production):

# elasticsearch-plugins.yml
plugins:
  - id: analysis-icu
  - id: analysis-kuromoji
  - id: mapper-murmur3
  - id: repository-s3
  - id: repository-gcs

Imperative:

# Install from Elastic's registry
bin/elasticsearch-plugin install analysis-icu

# Install from URL
bin/elasticsearch-plugin install https://example.com/my-plugin-9.5.0.zip

# List installed plugins
bin/elasticsearch-plugin list

# Remove a plugin
bin/elasticsearch-plugin remove analysis-icu

Note

Plugin changes require a full node restart. In a rolling upgrade scenario, restart nodes one at a time while the cluster remains operational.

3. Configuration

3.1 elasticsearch.yml

The primary configuration file. Located at config/elasticsearch.yml (tar/zip installs) or /etc/elasticsearch/elasticsearch.yml (package installs).

# Cluster identity
cluster.name: my-production-cluster
node.name: node-1

# Paths
path.data: /var/data/elasticsearch
path.logs: /var/log/elasticsearch

# Memory
bootstrap.memory_lock: true

# Network
network.host: 192.168.1.10
http.port: 9200

# Discovery (list of seed node hostnames or IPs)
discovery.seed_hosts:
  - 192.168.1.10
  - 192.168.1.11
  - 192.168.1.12

# Bootstrap: list master-eligible nodes for first cluster formation only
cluster.initial_master_nodes:
  - node-1
  - node-2
  - node-3

# Safety
action.destructive_requires_name: true

Important

cluster.initial_master_nodes is used only on the very first cluster bootstrap. Remove it from elasticsearch.yml after the cluster is formed to prevent accidental re-bootstrap on future restarts.

3.2 Key configuration settings

3.3 JVM heap size

Set in config/jvm.options or via the ES_JAVA_OPTS environment variable:

# jvm.options (recommended — edit the file)
-Xms4g
-Xmx4g

# Or environment variable (Docker / systemd override)
export ES_JAVA_OPTS="-Xms4g -Xmx4g"

Rules:

• Set min (-Xms) equal to max (-Xmx) to avoid GC overhead from heap resizing.
• Do not exceed 31 GB — above 32 GB the JVM switches to 64-bit object pointers, wasting memory.
• Leave at least 50% of system RAM for the OS page cache (Lucene relies on it heavily).

3.4 Docker environment variable configuration

When running in Docker, every elasticsearch.yml setting can be set via environment variables using the ES_SETTING_ prefix. Dots become double underscores:

Example:

docker run -d \
  -e "ES_SETTING_CLUSTER__NAME=my-cluster" \
  -e "ES_SETTING_NODE__NAME=node-1" \
  -e "ES_SETTING_NETWORK__HOST=0.0.0.0" \
  docker.elastic.co/elasticsearch/elasticsearch:9.5.0

3.5 Secure settings (keystore)

Sensitive values (passwords, API keys, S3 credentials) should never go in elasticsearch.yml in plaintext. Store them in the Elasticsearch keystore:

# Create the keystore (first time)
bin/elasticsearch-keystore create

# Add a secret
bin/elasticsearch-keystore add s3.client.default.access_key

# Add a file-based secret (e.g., GCS service account key)
bin/elasticsearch-keystore add-file gcs.client.default.credentials_file /path/to/key.json

# List all keys
bin/elasticsearch-keystore list

# Remove a key
bin/elasticsearch-keystore remove s3.client.default.access_key

# Reload secure settings without restart (some settings support this)
curl -X POST "localhost:9200/_nodes/reload_secure_settings"

3.6 Security configuration

TLS and authentication are enabled by default in 8.x+. To generate certificates:

# Generate CA + node certificates
bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

# Generate certificates for multiple nodes at once
bin/elasticsearch-certutil cert \
  --ca elastic-stack-ca.p12 \
  --in instances.yml \
  --out certs.zip

instances.yml example:

instances:
  - name: node-1
    dns:
      - node-1.example.com
    ip:
      - 192.168.1.10
  - name: node-2
    dns:
      - node-2.example.com
    ip:
      - 192.168.1.11

3.7 User management (native realm)

# Add a new user (interactive password prompt)
bin/elasticsearch-users useradd alice -r superuser

# Add with explicit role
bin/elasticsearch-users useradd bob -r kibana_admin,viewer

# List users
bin/elasticsearch-users list

# Delete a user
bin/elasticsearch-users userdel alice

# Change password
bin/elasticsearch-users passwd alice

4. Core Usage

4.1 REST API fundamentals

Elasticsearch exposes its entire functionality through a RESTful HTTP API. The base URL is http[s]://{host}:{port}.

Common query parameters accepted on every endpoint:

# Use filter_path to return only what you need
curl "localhost:9200/_cluster/health?filter_path=status,number_of_nodes"
# {"status":"green","number_of_nodes":3}

4.2 Indices

Create an index with explicit mapping

curl -X PUT "localhost:9200/articles?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "analysis": {
        "analyzer": {
          "english_with_stop": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase", "english_stop", "english_stemmer"]
          }
        },
        "filter": {
          "english_stop": {
            "type": "stop",
            "stopwords": "_english_"
          },
          "english_stemmer": {
            "type": "stemmer",
            "language": "english"
          }
        }
      }
    },
    "mappings": {
      "properties": {
        "title": {
          "type": "text",
          "analyzer": "english_with_stop",
          "fields": {
            "keyword": { "type": "keyword" }
          }
        },
        "author": { "type": "keyword" },
        "published_at": { "type": "date", "format": "strict_date_optional_time" },
        "views": { "type": "long" },
        "tags": { "type": "keyword" },
        "body": { "type": "text", "analyzer": "english_with_stop" },
        "location": { "type": "geo_point" },
        "embedding": {
          "type": "dense_vector",
          "dims": 768,
          "index": true,
          "similarity": "cosine"
        }
      }
    }
  }'

Get, open, close, delete

# Get index info
curl "localhost:9200/articles?pretty"

# Get just mappings
curl "localhost:9200/articles/_mapping?pretty"

# Get just settings
curl "localhost:9200/articles/_settings?pretty"

# Close (suspend I/O, keeps data)
curl -X POST "localhost:9200/articles/_close"

# Re-open
curl -X POST "localhost:9200/articles/_open"

# Delete (irreversible — requires exact name when action.destructive_requires_name=true)
curl -X DELETE "localhost:9200/articles"

Refresh, flush, forcemerge

# Refresh — make recent writes searchable immediately
curl -X POST "localhost:9200/articles/_refresh"

# Flush — fsync segments to disk + clear translog
curl -X POST "localhost:9200/articles/_flush"

# Force merge — reduce segment count (use on read-only/completed indices)
curl -X POST "localhost:9200/articles/_forcemerge?max_num_segments=1"

4.3 Indexing documents

Index (create or replace)

# With explicit ID
curl -X PUT "localhost:9200/articles/_doc/42?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Getting Started with Elasticsearch",
    "author": "alice",
    "published_at": "2026-03-01T10:00:00Z",
    "views": 1500,
    "tags": ["elasticsearch", "tutorial", "search"],
    "body": "Elasticsearch is a distributed search engine..."
  }'

# Auto-generated ID (POST)
curl -X POST "localhost:9200/articles/_doc?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "title": "Advanced Aggregations", "author": "bob", "views": 800 }'

Create only (fail if exists)

curl -X PUT "localhost:9200/articles/_create/42?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "title": "New Article" }'

# Or using _doc with op_type=create
curl -X PUT "localhost:9200/articles/_doc/42?op_type=create" \
  -H "Content-Type: application/json" \
  -d '{ "title": "New Article" }'

Update (partial)

# Partial update — only specified fields change
curl -X POST "localhost:9200/articles/_update/42?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "doc": {
      "views": 2000,
      "tags": ["elasticsearch", "tutorial", "search", "updated"]
    }
  }'

# Scripted update
curl -X POST "localhost:9200/articles/_update/42?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "script": {
      "source": "ctx._source.views += params.increment",
      "params": { "increment": 100 }
    }
  }'

# Upsert — insert if not found, update if found
curl -X POST "localhost:9200/articles/_update/99?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "script": {
      "source": "ctx._source.views += params.increment",
      "params": { "increment": 1 }
    },
    "upsert": {
      "title": "New Article",
      "views": 1
    }
  }'

Get and delete

# Get document
curl "localhost:9200/articles/_doc/42?pretty"

# Get source only
curl "localhost:9200/articles/_source/42"

# Check existence (HEAD — returns 200 or 404, no body)
curl -I "localhost:9200/articles/_doc/42"

# Delete
curl -X DELETE "localhost:9200/articles/_doc/42"

Bulk API

The Bulk API processes multiple operations in a single request. Each operation is two lines: an action line + optional document line.

curl -X POST "localhost:9200/_bulk?pretty" \
  -H "Content-Type: application/x-ndjson" \
  -d '
{"index":{"_index":"articles","_id":"1"}}
{"title":"Article One","author":"alice","views":100}
{"index":{"_index":"articles","_id":"2"}}
{"title":"Article Two","author":"bob","views":200}
{"update":{"_index":"articles","_id":"1"}}
{"doc":{"views":150}}
{"delete":{"_index":"articles","_id":"3"}}
'

Tip

Keep bulk request sizes between 5 MB and 15 MB per batch. Very large batches increase GC pressure; very small batches increase HTTP overhead. Index-specific bulk: POST /articles/_bulk.

Multi-get (mget)

curl -X GET "localhost:9200/_mget?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "docs": [
      { "_index": "articles", "_id": "1" },
      { "_index": "articles", "_id": "2" },
      { "_index": "products", "_id": "1" }
    ]
  }'

4.4 Searching

Match query (full-text)

curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "match": {
        "body": {
          "query": "distributed search engine",
          "operator": "and"
        }
      }
    }
  }'

Multi-match

curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "multi_match": {
        "query": "elasticsearch tutorial",
        "fields": ["title^3", "body", "tags"],
        "type": "best_fields"
      }
    }
  }'

Boolean query

curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "bool": {
        "must": [
          { "match": { "body": "elasticsearch" } }
        ],
        "filter": [
          { "term": { "author": "alice" } },
          { "range": { "published_at": { "gte": "2026-01-01" } } }
        ],
        "must_not": [
          { "term": { "tags": "deprecated" } }
        ],
        "should": [
          { "term": { "tags": "featured" } }
        ],
        "minimum_should_match": 0
      }
    }
  }'

Term-level queries (exact values)

# Term — exact keyword match
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "term": { "author": "alice" } } }'

# Terms — multiple values (OR)
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "terms": { "tags": ["elasticsearch", "lucene"] } } }'

# Range
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "range": { "views": { "gte": 500, "lt": 5000 } } } }'

# Exists — documents that have a field
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "exists": { "field": "tags" } } }'

# Prefix
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "prefix": { "author": "ali" } } }'

# Wildcard
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "wildcard": { "title.keyword": "Getting*" } } }'

# Fuzzy — handles typos
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "fuzzy": { "author": { "value": "alise", "fuzziness": "AUTO" } } } }'

# IDs
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "ids": { "values": ["1", "2", "42"] } } }'

Full-text: match_phrase, query_string

# Phrase match — terms in order, adjacent
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "match_phrase": { "body": "distributed search engine" } } }'

# Phrase prefix (autocomplete)
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "match_phrase_prefix": { "title": "Getting St" } } }'

# Query string (Lucene syntax)
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "query_string": {
        "query": "(title:elasticsearch OR tags:search) AND views:[100 TO *]",
        "default_field": "body"
      }
    }
  }'

Pagination

# From/size (offset-based, avoid deep pagination past 10,000)
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "from": 20,
    "size": 10,
    "query": { "match_all": {} },
    "sort": [{ "published_at": "desc" }]
  }'

# Search after (efficient deep pagination)
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 10,
    "query": { "match_all": {} },
    "sort": [{ "published_at": "desc" }, { "_id": "asc" }],
    "search_after": ["2026-01-15T12:00:00Z", "article-id-here"]
  }'

Sorting and source filtering

curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "match_all": {} },
    "sort": [
      { "views": { "order": "desc" } },
      { "published_at": { "order": "desc" } },
      "_score"
    ],
    "_source": ["title", "author", "views"],
    "size": 5
  }'

Highlighting

curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "match": { "body": "distributed search" } },
    "highlight": {
      "fields": {
        "body": {
          "fragment_size": 150,
          "number_of_fragments": 3,
          "pre_tags": ["<em>"],
          "post_tags": ["</em>"]
        },
        "title": {}
      }
    }
  }'

4.5 Aggregations

Bucket aggregations

curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 0,
    "aggs": {
      "by_author": {
        "terms": {
          "field": "author",
          "size": 10
        },
        "aggs": {
          "avg_views": { "avg": { "field": "views" } },
          "max_views": { "max": { "field": "views" } }
        }
      },
      "views_histogram": {
        "histogram": {
          "field": "views",
          "interval": 500
        }
      },
      "published_over_time": {
        "date_histogram": {
          "field": "published_at",
          "calendar_interval": "month"
        }
      },
      "views_range": {
        "range": {
          "field": "views",
          "ranges": [
            { "to": 100 },
            { "from": 100, "to": 1000 },
            { "from": 1000 }
          ]
        }
      }
    }
  }'

Metric aggregations

curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 0,
    "aggs": {
      "total_views": { "sum": { "field": "views" } },
      "avg_views": { "avg": { "field": "views" } },
      "min_views": { "min": { "field": "views" } },
      "max_views": { "max": { "field": "views" } },
      "views_stats": { "stats": { "field": "views" } },
      "views_percentiles": {
        "percentiles": {
          "field": "views",
          "percents": [25, 50, 75, 90, 95, 99]
        }
      },
      "unique_authors": { "cardinality": { "field": "author" } }
    }
  }'

Pipeline aggregations

curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 0,
    "aggs": {
      "monthly": {
        "date_histogram": {
          "field": "published_at",
          "calendar_interval": "month"
        },
        "aggs": {
          "total_views": { "sum": { "field": "views" } },
          "cumulative_views": {
            "cumulative_sum": { "buckets_path": "total_views" }
          },
          "avg_views": { "avg": { "field": "views" } }
        }
      },
      "best_month": {
        "max_bucket": {
          "buckets_path": "monthly>total_views"
        }
      }
    }
  }'

4.6 KNN and vector search

# kNN search with a pre-computed vector
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "knn": {
      "field": "embedding",
      "query_vector": [0.12, -0.34, 0.56],
      "k": 10,
      "num_candidates": 100
    },
    "_source": ["title", "author"]
  }'

# Hybrid: kNN + keyword filter
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "knn": {
      "field": "embedding",
      "query_vector": [0.12, -0.34, 0.56],
      "k": 10,
      "num_candidates": 100,
      "filter": {
        "term": { "tags": "elasticsearch" }
      }
    },
    "query": {
      "match": { "title": "vector search" }
    }
  }'

Tip

For semantic search use cases, combine kNN vector search (semantic similarity) with a keyword bool query (exact matching) and use Reciprocal Rank Fusion (RRF) to merge result sets. This hybrid approach consistently outperforms either method alone on real-world datasets.

4.7 ES|QL — pipe-based query language

ES|QL is Elasticsearch's modern pipe-based query language, similar to Splunk SPL or Kusto KQL.

curl -X POST "localhost:9200/_query?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "FROM articles | WHERE views > 500 | STATS avg_views = AVG(views), total = COUNT(*) BY author | SORT avg_views DESC | LIMIT 10"
  }'

# ES|QL with date filtering
curl -X POST "localhost:9200/_query?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "FROM articles | WHERE published_at >= \"2026-01-01\" | STATS monthly_views = SUM(views) BY BUCKET(published_at, 1 month) | SORT monthly_views DESC"
  }'

# Async ES|QL (for long-running queries)
curl -X POST "localhost:9200/_query/async?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "FROM articles | STATS total = COUNT(*) BY author | SORT total DESC",
    "wait_for_completion_timeout": "10s"
  }'

Note

ES|QL is ideal for log analytics and operational data. It operates on fields directly (not analyzed text), making it fast for structured data. For full-text search on analyzed fields, use the Search API with match or query_string queries.

4.8 EQL — Event Query Language

EQL is optimized for time-series event data and threat hunting.

# Basic EQL sequence search
curl -X GET "localhost:9200/logs/_eql/search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "sequence by host.name [process where process.name == \"cmd.exe\"] [network where destination.port == 443]",
    "size": 10
  }'

# EQL with filter
curl -X GET "localhost:9200/logs/_eql/search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "process where process.name : (\"*.exe\", \"*.bat\")",
    "filter": {
      "range": { "@timestamp": { "gte": "now-1d" } }
    },
    "size": 20
  }'

4.9 SQL interface

# SQL query
curl -X POST "localhost:9200/_sql?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT author, COUNT(*) as article_count, AVG(views) as avg_views FROM articles GROUP BY author ORDER BY article_count DESC LIMIT 10"
  }'

# Translate SQL to DSL (for learning/debugging)
curl -X POST "localhost:9200/_sql/translate?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT * FROM articles WHERE views > 1000"
  }'

# Pagination via cursor
curl -X POST "localhost:9200/_sql?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "cursor": "sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAA..."
  }'

5. Common Tasks

5.1 Data streams and time-series data

Data streams are the recommended way to manage append-only time-series data (logs, metrics, events).

Set up a data stream

# 1. Create an index lifecycle policy
curl -X PUT "localhost:9200/_ilm/policy/logs-policy?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "policy": {
      "phases": {
        "hot": {
          "actions": {
            "rollover": {
              "max_primary_shard_size": "50gb",
              "max_age": "7d"
            }
          }
        },
        "warm": {
          "min_age": "7d",
          "actions": {
            "shrink": { "number_of_shards": 1 },
            "forcemerge": { "max_num_segments": 1 }
          }
        },
        "cold": {
          "min_age": "30d",
          "actions": {
            "freeze": {}
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": { "delete": {} }
        }
      }
    }
  }'

# 2. Create a component template for mappings
curl -X PUT "localhost:9200/_component_template/logs-mappings?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "template": {
      "mappings": {
        "properties": {
          "@timestamp": { "type": "date" },
          "log.level": { "type": "keyword" },
          "message": { "type": "text" },
          "host.name": { "type": "keyword" },
          "service.name": { "type": "keyword" }
        }
      }
    }
  }'

# 3. Create an index template that targets the data stream
curl -X PUT "localhost:9200/_index_template/logs-template?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "index_patterns": ["logs-*-*"],
    "data_stream": {},
    "composed_of": ["logs-mappings"],
    "template": {
      "settings": {
        "number_of_shards": 1,
        "index.lifecycle.name": "logs-policy"
      }
    },
    "priority": 200
  }'

# 4. Create the data stream
curl -X PUT "localhost:9200/_data_stream/logs-app-prod"

# 5. Index documents (must include @timestamp)
curl -X POST "localhost:9200/logs-app-prod/_doc?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "@timestamp": "2026-05-13T10:00:00Z",
    "log.level": "error",
    "message": "Connection timeout to database",
    "host.name": "web-01",
    "service.name": "payment-service"
  }'

Manage data streams

# List data streams
curl "localhost:9200/_data_stream?pretty"

# Get data stream stats
curl "localhost:9200/_data_stream/logs-app-prod/_stats?pretty"

# Rollover (force a new backing index)
curl -X POST "localhost:9200/logs-app-prod/_rollover?pretty"

# Delete a data stream (removes all backing indices)
curl -X DELETE "localhost:9200/_data_stream/logs-app-prod"

5.2 Ingest pipelines

Ingest pipelines transform documents before they are indexed. They run on ingest nodes.

Create a pipeline

curl -X PUT "localhost:9200/_ingest/pipeline/parse-apache-log?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Parse Apache Combined Log Format",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": ["%{COMBINEDAPACHELOG}"]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": ["dd/MMM/yyyy:HH:mm:ss Z"],
          "target_field": "@timestamp"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "long",
          "ignore_missing": true
        }
      },
      {
        "convert": {
          "field": "response",
          "type": "integer",
          "ignore_missing": true
        }
      },
      {
        "geoip": {
          "field": "clientip",
          "target_field": "geoip",
          "ignore_missing": true
        }
      },
      {
        "remove": {
          "field": ["timestamp"],
          "ignore_missing": true
        }
      }
    ],
    "on_failure": [
      {
        "set": {
          "field": "_index",
          "value": "failed-{{ _index }}"
        }
      }
    ]
  }'

Available ingest processors

Simulate and test a pipeline

curl -X POST "localhost:9200/_ingest/pipeline/parse-apache-log/_simulate?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "docs": [
      {
        "_source": {
          "message": "127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326"
        }
      }
    ]
  }'

Use a pipeline on indexing

# Single document
curl -X PUT "localhost:9200/logs/_doc/1?pipeline=parse-apache-log" \
  -H "Content-Type: application/json" \
  -d '{ "message": "127.0.0.1 - - [01/Jan/2026:00:00:01 +0000] \"GET / HTTP/1.1\" 200 512" }'

# Set pipeline as the index default
curl -X PUT "localhost:9200/logs/_settings" \
  -H "Content-Type: application/json" \
  -d '{ "index.default_pipeline": "parse-apache-log" }'

5.3 Reindex

Copy documents from one index to another, optionally transforming them:

# Basic reindex
curl -X POST "localhost:9200/_reindex?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": { "index": "articles-v1" },
    "dest": { "index": "articles-v2" }
  }'

# Reindex with query filter
curl -X POST "localhost:9200/_reindex?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": {
      "index": "articles-v1",
      "query": { "range": { "views": { "gte": 100 } } }
    },
    "dest": { "index": "articles-popular" }
  }'

# Reindex with script transformation
curl -X POST "localhost:9200/_reindex?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": { "index": "articles-v1" },
    "dest": {
      "index": "articles-v2",
      "pipeline": "my-transform-pipeline"
    },
    "script": {
      "source": "ctx._source.migrated_at = params.now",
      "params": { "now": "2026-05-13" }
    }
  }'

# Reindex from remote cluster
curl -X POST "localhost:9200/_reindex?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": {
      "remote": {
        "host": "http://old-cluster:9200",
        "username": "user",
        "password": "pass"
      },
      "index": "articles"
    },
    "dest": { "index": "articles" }
  }'

# Async reindex (for large indices)
curl -X POST "localhost:9200/_reindex?wait_for_completion=false&pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": { "index": "large-index" },
    "dest": { "index": "large-index-v2" }
  }'
# Returns: {"task":"node-id:task-number"}

# Monitor task progress
curl "localhost:9200/_tasks/node-id:task-number?pretty"

5.4 Index aliases

Aliases let you point a logical name at one or more physical indices, enabling zero-downtime reindexing:

# Create alias
curl -X POST "localhost:9200/_aliases?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      { "add": { "index": "articles-v2", "alias": "articles" } }
    ]
  }'

# Atomic alias swap (zero downtime)
curl -X POST "localhost:9200/_aliases?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      { "remove": { "index": "articles-v1", "alias": "articles" } },
      { "add": { "index": "articles-v2", "alias": "articles" } }
    ]
  }'

# Filtered alias (a virtual view)
curl -X POST "localhost:9200/_aliases?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      {
        "add": {
          "index": "articles",
          "alias": "alice-articles",
          "filter": { "term": { "author": "alice" } }
        }
      }
    ]
  }'

# Write alias (controls which index receives writes in multi-index aliases)
curl -X POST "localhost:9200/_aliases?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      { "add": { "index": "articles-v2", "alias": "articles", "is_write_index": true } },
      { "add": { "index": "articles-v1", "alias": "articles" } }
    ]
  }'

# List aliases
curl "localhost:9200/_aliases?pretty"
curl "localhost:9200/articles/_alias?pretty"

5.5 Index lifecycle management (ILM)

ILM automates the progression of indices through hot → warm → cold → frozen → delete phases.

# View an index's current ILM status
curl "localhost:9200/my-index/_ilm/explain?pretty"

# Manually move to next phase
curl -X POST "localhost:9200/_ilm/move/my-index?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "current_step": { "phase": "hot", "action": "rollover", "name": "check-rollover-ready" },
    "next_step": { "phase": "warm" }
  }'

# Retry failed ILM step
curl -X POST "localhost:9200/my-index/_ilm/retry?pretty"

# Start/stop ILM globally
curl -X POST "localhost:9200/_ilm/start"
curl -X POST "localhost:9200/_ilm/stop"
curl "localhost:9200/_ilm/status"

5.6 Snapshots and backups

# Register a repository (filesystem)
curl -X PUT "localhost:9200/_snapshot/my-backup?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "fs",
    "settings": {
      "location": "/mnt/backups/elasticsearch"
    }
  }'

# Register S3 repository (requires repository-s3 plugin)
curl -X PUT "localhost:9200/_snapshot/s3-backup?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "s3",
    "settings": {
      "bucket": "my-es-backups",
      "region": "us-east-1",
      "base_path": "prod/elasticsearch"
    }
  }'

# Take a snapshot
curl -X PUT "localhost:9200/_snapshot/my-backup/snapshot-2026-05-13?wait_for_completion=true&pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "indices": "articles,products",
    "ignore_unavailable": true,
    "include_global_state": false
  }'

# List snapshots
curl "localhost:9200/_snapshot/my-backup/_all?pretty"
curl "localhost:9200/_snapshot/my-backup/snapshot-2026-05-13?pretty"

# Restore a snapshot
curl -X POST "localhost:9200/_snapshot/my-backup/snapshot-2026-05-13/_restore?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "indices": "articles",
    "rename_pattern": "(.+)",
    "rename_replacement": "$1_restored"
  }'

# Delete a snapshot
curl -X DELETE "localhost:9200/_snapshot/my-backup/snapshot-2026-05-13"

Snapshot Lifecycle Management (SLM)

curl -X PUT "localhost:9200/_slm/policy/daily-snapshots?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "schedule": "0 30 1 * * ?",
    "name": "<daily-snap-{now/d}>",
    "repository": "my-backup",
    "config": {
      "indices": ["*"],
      "ignore_unavailable": false,
      "include_global_state": true
    },
    "retention": {
      "expire_after": "30d",
      "min_count": 5,
      "max_count": 50
    }
  }'

# Execute immediately
curl -X POST "localhost:9200/_slm/policy/daily-snapshots/_execute"

# View SLM stats
curl "localhost:9200/_slm/stats?pretty"

5.7 Multi-search (msearch)

curl -X GET "localhost:9200/_msearch?pretty" \
  -H "Content-Type: application/x-ndjson" \
  -d '
{"index": "articles"}
{"query": {"match": {"title": "elasticsearch"}}, "size": 5}
{"index": "products"}
{"query": {"range": {"price": {"lte": 50}}}, "sort": [{"price": "asc"}], "size": 5}
{"index": "articles"}
{"query": {"term": {"author": "alice"}}, "aggs": {"avg_views": {"avg": {"field": "views"}}}}
'

5.8 Async search

For queries that may take more than a few seconds:

# Submit
curl -X POST "localhost:9200/articles/_async_search?wait_for_completion_timeout=1s&pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "match_all": {} },
    "aggs": {
      "by_author": { "terms": { "field": "author", "size": 100 } }
    }
  }'
# Returns {"id":"FmRldE8zREVEUzA...","is_partial":true,...}

# Poll for results
curl "localhost:9200/_async_search/FmRldE8zREVEUzA...?pretty"

# Delete when done
curl -X DELETE "localhost:9200/_async_search/FmRldE8zREVEUzA..."

5.9 Search templates

# Store a search template
curl -X PUT "localhost:9200/_scripts/article-search-template?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "script": {
      "lang": "mustache",
      "source": {
        "query": {
          "bool": {
            "must": [
              { "match": { "body": "{{query_text}}" } }
            ],
            "filter": [
              { "term": { "author": "{{author}}" } }
            ]
          }
        },
        "from": "{{from}}{{^from}}0{{/from}}",
        "size": "{{size}}{{^size}}10{{/size}}"
      }
    }
  }'

# Use the template
curl -X GET "localhost:9200/articles/_search/template?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "article-search-template",
    "params": {
      "query_text": "elasticsearch tutorial",
      "author": "alice",
      "size": 5
    }
  }'

5.10 Scroll API (batch export)

For exporting all documents from a large index. Prefer search_after for interactive pagination:

# Open scroll context (keep alive for 5 minutes)
curl -X GET "localhost:9200/articles/_search?scroll=5m&pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 1000,
    "query": { "match_all": {} },
    "sort": ["_doc"]
  }'
# Returns {"_scroll_id":"DXF1ZXJ5QW5kRmV0Y2gB...","hits":...}

# Get next batch
curl -X GET "localhost:9200/_search/scroll?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "scroll": "5m",
    "scroll_id": "DXF1ZXJ5QW5kRmV0Y2gB..."
  }'

# Clean up scroll context
curl -X DELETE "localhost:9200/_search/scroll" \
  -H "Content-Type: application/json" \
  -d '{ "scroll_id": "DXF1ZXJ5QW5kRmV0Y2gB..." }'

5.11 Update by query / Delete by query

# Update all matching documents
curl -X POST "localhost:9200/articles/_update_by_query?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "term": { "author": "alice" } },
    "script": {
      "source": "ctx._source.verified_author = true"
    }
  }'

# Delete all matching documents
curl -X POST "localhost:9200/articles/_delete_by_query?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "range": {
        "published_at": { "lt": "2020-01-01" }
      }
    }
  }'

# Async (large operations)
curl -X POST "localhost:9200/articles/_delete_by_query?wait_for_completion=false&pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "term": { "tags": "spam" } } }'

5.12 Cluster and node APIs

# Cluster health (per-index detail)
curl "localhost:9200/_cluster/health/articles,products?level=indices&pretty"

# Cluster state (filtered to routing table)
curl "localhost:9200/_cluster/state/routing_table/articles?pretty"

# Cluster stats (aggregate)
curl "localhost:9200/_cluster/stats?pretty"

# Pending tasks
curl "localhost:9200/_cluster/pending_tasks?pretty"

# Explain shard allocation (why is a shard unassigned?)
curl -X GET "localhost:9200/_cluster/allocation/explain?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "index": "articles",
    "shard": 0,
    "primary": true
  }'

# Node info
curl "localhost:9200/_nodes?pretty"
curl "localhost:9200/_nodes/es01?pretty"
curl "localhost:9200/_nodes/es01/stats?pretty"

# Hot threads (what is each node doing right now?)
curl "localhost:9200/_nodes/hot_threads"

# Reroute a shard manually
curl -X POST "localhost:9200/_cluster/reroute?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "commands": [
      {
        "move": {
          "index": "articles",
          "shard": 0,
          "from_node": "es01",
          "to_node": "es02"
        }
      }
    ]
  }'

5.13 Cat APIs (human-readable cluster info)

# All cat APIs support: ?v (verbose, with headers), ?h=col1,col2 (select columns)
# ?s=col (sort), ?format=json (JSON output)

curl "localhost:9200/_cat/health?v"
curl "localhost:9200/_cat/nodes?v&h=name,ip,heap.percent,ram.percent,cpu,load_1m,disk.used_percent,role"
curl "localhost:9200/_cat/indices?v&s=store.size:desc"
curl "localhost:9200/_cat/shards?v&s=index,shard,prirep"
curl "localhost:9200/_cat/aliases?v"
curl "localhost:9200/_cat/allocation?v"
curl "localhost:9200/_cat/master?v"
curl "localhost:9200/_cat/plugins?v"
curl "localhost:9200/_cat/recovery?v&active_only=true"
curl "localhost:9200/_cat/segments/articles?v"
curl "localhost:9200/_cat/thread_pool?v&h=node_name,name,active,rejected,completed"
curl "localhost:9200/_cat/tasks?v"
curl "localhost:9200/_cat/templates?v"

5.14 Security: users, roles, and API keys

# Create a role
curl -X PUT "localhost:9200/_security/role/articles-reader?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "cluster": ["monitor"],
    "indices": [
      {
        "names": ["articles*"],
        "privileges": ["read", "view_index_metadata"]
      }
    ]
  }'

# Create a user
curl -X PUT "localhost:9200/_security/user/reader-bob?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "password": "securepassword123",
    "roles": ["articles-reader"],
    "full_name": "Bob the Reader",
    "email": "[email removed]"
  }'

# Create an API key
curl -X PUT "localhost:9200/_security/api_key?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-app-key",
    "expiration": "30d",
    "role_descriptors": {
      "articles-role": {
        "cluster": ["monitor"],
        "indices": [
          {
            "names": ["articles*"],
            "privileges": ["read"]
          }
        ]
      }
    }
  }'

# List API keys
curl "localhost:9200/_security/api_key?pretty"

# Invalidate API keys
curl -X DELETE "localhost:9200/_security/api_key?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "ids": ["VuaCfGcBCdbkIjaNgtVk"] }'

5.15 Machine learning — anomaly detection

# Create a single-metric anomaly detection job
curl -X PUT "localhost:9200/_ml/anomaly_detectors/high-error-rate?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Detect unusually high error rates",
    "analysis_config": {
      "bucket_span": "15m",
      "detectors": [
        {
          "function": "high_count",
          "over_field_name": "service.name",
          "partition_field_name": "host.name"
        }
      ]
    },
    "data_description": {
      "time_field": "@timestamp"
    }
  }'

# Create a datafeed
curl -X PUT "localhost:9200/_ml/datafeeds/datafeed-high-error-rate?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": "high-error-rate",
    "indices": ["logs-*"],
    "query": {
      "term": { "log.level": "error" }
    }
  }'

# Open job
curl -X POST "localhost:9200/_ml/anomaly_detectors/high-error-rate/_open"

# Start datafeed
curl -X POST "localhost:9200/_ml/datafeeds/datafeed-high-error-rate/_start" \
  -H "Content-Type: application/json" \
  -d '{ "start": "2026-01-01T00:00:00Z" }'

# Get anomaly records
curl "localhost:9200/_ml/anomaly_detectors/high-error-rate/results/records?pretty"

# Stop and close
curl -X POST "localhost:9200/_ml/datafeeds/datafeed-high-error-rate/_stop"
curl -X POST "localhost:9200/_ml/anomaly_detectors/high-error-rate/_close"

5.16 AI inference endpoints

Elasticsearch 9.x includes built-in inference endpoints that connect to external AI models for text embeddings, completions, and reranking.

# Create an inference endpoint (OpenAI embeddings)
curl -X PUT "localhost:9200/_inference/text_embedding/my-openai-embeddings?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "openai",
    "service_settings": {
      "model_id": "text-embedding-3-small",
      "api_key": "sk-..."
    }
  }'

# Create an inference endpoint (Anthropic completion)
curl -X PUT "localhost:9200/_inference/completion/my-claude?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "anthropic",
    "service_settings": {
      "model_id": "claude-3-5-sonnet-20241022",
      "api_key": "sk-ant-..."
    }
  }'

# Create an inference endpoint (Cohere reranking)
curl -X PUT "localhost:9200/_inference/rerank/my-cohere-rerank?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "cohere",
    "service_settings": {
      "model_id": "rerank-english-v3.0",
      "api_key": "..."
    }
  }'

# Run inference directly
curl -X POST "localhost:9200/_inference/text_embedding/my-openai-embeddings?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "input": "Elasticsearch vector search tutorial" }'

# Supported providers: openai, anthropic, cohere, amazon_bedrock, google_vertex_ai,
# google_ai_studio, azure_openai, azure_ai_studio, mistral, hugging_face, elastic, elser

5.17 Transforms

Transforms pivot or group-by data from one index into a summarized destination index:

# Create a pivot transform
curl -X PUT "localhost:9200/_transform/articles-by-author?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": { "index": "articles" },
    "pivot": {
      "group_by": {
        "author": { "terms": { "field": "author" } }
      },
      "aggregations": {
        "article_count": { "value_count": { "field": "_id" } },
        "total_views": { "sum": { "field": "views" } },
        "avg_views": { "avg": { "field": "views" } },
        "latest_published": { "max": { "field": "published_at" } }
      }
    },
    "dest": { "index": "author-stats" },
    "sync": {
      "time": {
        "field": "published_at",
        "delay": "60s"
      }
    }
  }'

# Start the transform
curl -X POST "localhost:9200/_transform/articles-by-author/_start"

# Get stats
curl "localhost:9200/_transform/articles-by-author/_stats?pretty"

# Stop
curl -X POST "localhost:9200/_transform/articles-by-author/_stop"

5.18 Cross-cluster replication (CCR)

CCR replicates indices from a leader cluster to one or more follower clusters for disaster recovery or geo-distribution:

# On the follower cluster: configure remote cluster
curl -X PUT "localhost:9200/_cluster/settings?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "persistent": {
      "cluster.remote.leader-cluster.seeds": ["leader-node:9300"]
    }
  }'

# Create a follower index
curl -X PUT "localhost:9200/articles-replica/_ccr/follow?wait_for_active_shards=1&pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "remote_cluster": "leader-cluster",
    "leader_index": "articles"
  }'

# Get CCR stats
curl "localhost:9200/_ccr/stats?pretty"
curl "localhost:9200/articles-replica/_ccr/info?pretty"

# Pause and resume replication
curl -X POST "localhost:9200/articles-replica/_ccr/pause_follow"
curl -X POST "localhost:9200/articles-replica/_ccr/resume_follow"

5.19 Enrich policies

Enrich processors look up data from an "enrich index" and embed it into incoming documents during ingest:

# Create the source index
curl -X PUT "localhost:9200/user-profiles/_doc/alice" \
  -H "Content-Type: application/json" \
  -d '{ "email": "[email removed]", "full_name": "Alice Smith", "department": "Engineering" }'

# Create an enrich policy
curl -X PUT "localhost:9200/_enrich/policy/user-lookup?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "match": {
      "indices": "user-profiles",
      "match_field": "email",
      "enrich_fields": ["full_name", "department"]
    }
  }'

# Execute the policy (builds the enrich index)
curl -X POST "localhost:9200/_enrich/policy/user-lookup/_execute"

# Add enrich processor to a pipeline
curl -X PUT "localhost:9200/_ingest/pipeline/enrich-users?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "processors": [
      {
        "enrich": {
          "policy_name": "user-lookup",
          "field": "user_email",
          "target_field": "user"
        }
      }
    ]
  }'

5.20 Query rules

Query rules promote specific documents to the top of search results based on matching conditions:

# Create a ruleset
curl -X PUT "localhost:9200/_query_rules/my-promotions?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "rules": [
      {
        "rule_id": "promoted-elasticsearch-guide",
        "type": "pinned",
        "criteria": [
          { "type": "contains", "metadata": "query", "values": ["elasticsearch"] }
        ],
        "actions": {
          "ids": ["42"]
        }
      }
    ]
  }'

# Use rules in a search
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "rule": {
        "match_criteria": { "query": "elasticsearch tutorial" },
        "ruleset_ids": ["my-promotions"],
        "organic": {
          "match": { "body": "elasticsearch tutorial" }
        }
      }
    }
  }'

5.21 Synonyms

# Create a synonym set
curl -X PUT "localhost:9200/_synonyms/my-synonyms?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "synonyms_set": [
      { "id": "syn1", "synonyms": "elasticsearch, es, elastic search" },
      { "id": "syn2", "synonyms": "k8s => kubernetes" },
      { "id": "syn3", "synonyms": "js, javascript, ecmascript" }
    ]
  }'

# Reference from an index analyzer
curl -X PUT "localhost:9200/my-index?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "analysis": {
        "filter": {
          "my_synonyms": {
            "type": "synonym_graph",
            "synonyms_set": "my-synonyms",
            "updateable": true
          }
        },
        "analyzer": {
          "my_analyzer": {
            "tokenizer": "standard",
            "filter": ["lowercase", "my_synonyms"]
          }
        }
      }
    }
  }'

5.22 Connectors (data source integration)

Connectors crawl external data sources (SharePoint, Confluence, MySQL, etc.) and index content into Elasticsearch:

# List available connector types
curl "localhost:9200/_connector/_list?pretty"

# Create a connector
curl -X PUT "localhost:9200/_connector/my-sharepoint?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "SharePoint Online Connector",
    "service_type": "sharepoint_online",
    "index_name": "sharepoint-content"
  }'

# Update configuration
curl -X PUT "localhost:9200/_connector/my-sharepoint/_configuration?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "values": {
      "tenant_id": "your-tenant-id",
      "client_id": "your-client-id"
    }
  }'

# Check sync jobs
curl "localhost:9200/_connector/_sync_job?connector_id=my-sharepoint&pretty"

5.23 GeoIP and geo queries

# Geo distance filter
curl -X GET "localhost:9200/stores/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "geo_distance": {
        "distance": "10km",
        "location": {
          "lat": 40.715,
          "lon": -74.011
        }
      }
    },
    "sort": [
      {
        "_geo_distance": {
          "location": { "lat": 40.715, "lon": -74.011 },
          "order": "asc",
          "unit": "km"
        }
      }
    ]
  }'

# Geo bounding box
curl -X GET "localhost:9200/stores/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "geo_bounding_box": {
        "location": {
          "top_left": { "lat": 41.0, "lon": -74.5 },
          "bottom_right": { "lat": 40.5, "lon": -73.5 }
        }
      }
    }
  }'

# Geohash grid aggregation
curl -X GET "localhost:9200/stores/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 0,
    "aggs": {
      "geo_grid": {
        "geohash_grid": {
          "field": "location",
          "precision": 5
        }
      }
    }
  }'

5.24 Text analysis and custom analyzers

# Test an analyzer
curl -X GET "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "analyzer": "english",
    "text": "The quick brown foxes are jumping over the lazy dogs"
  }'

# Test index-specific analyzer
curl -X GET "localhost:9200/articles/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "field": "body",
    "text": "Elasticsearch distributed search tutorial"
  }'

# Test ad-hoc analyzer
curl -X GET "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "tokenizer": "standard",
    "filter": ["lowercase", "stop", "snowball"],
    "text": "Running elasticsearch queries effectively"
  }'

5.25 Profile API (query debugging)

curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "profile": true,
    "query": {
      "bool": {
        "must": [{ "match": { "body": "elasticsearch" } }],
        "filter": [{ "term": { "author": "alice" } }]
      }
    }
  }'

The profile object shows exact timing for each query clause and Lucene collector, helping you identify slow query components.

5.26 Explain API (scoring debug)

# Why did document 42 score the way it did?
curl -X GET "localhost:9200/articles/_explain/42?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "match": { "body": "elasticsearch tutorial" } } }'

5.27 Field capabilities

# What fields exist across multiple indices and what are their types?
curl -X GET "localhost:9200/articles,products/_field_caps?fields=*&pretty"

# Filter to specific fields
curl -X GET "localhost:9200/_field_caps?fields=author,views,price&pretty"

5.28 Index shrink, split, clone

# Shrink — reduce shard count (index must be read-only, all shards on one node)
curl -X POST "localhost:9200/articles/_shrink/articles-shrunk?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "index.number_of_shards": 1,
      "index.number_of_replicas": 1,
      "index.codec": "best_compression"
    }
  }'

# Split — increase shard count
curl -X POST "localhost:9200/articles/_split/articles-split?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "settings": { "index.number_of_shards": 6 } }'

# Clone — exact copy with same shard count
curl -X POST "localhost:9200/articles/_clone/articles-backup?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "settings": { "index.number_of_replicas": 0 } }'

5.29 Watcher (alerting)

# Create a watch that alerts on high error count
curl -X PUT "localhost:9200/_watcher/watch/high-errors?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "trigger": {
      "schedule": { "interval": "5m" }
    },
    "input": {
      "search": {
        "request": {
          "indices": ["logs-*"],
          "body": {
            "query": {
              "bool": {
                "filter": [
                  { "term": { "log.level": "error" } },
                  { "range": { "@timestamp": { "gte": "now-5m" } } }
                ]
              }
            }
          }
        }
      }
    },
    "condition": {
      "compare": {
        "ctx.payload.hits.total.value": { "gt": 100 }
      }
    },
    "actions": {
      "send-email": {
        "email": {
          "to": ["[email removed]"],
          "subject": "High error rate detected",
          "body": "{{ctx.payload.hits.total.value}} errors in the last 5 minutes"
        }
      }
    }
  }'

# Execute a watch manually
curl -X POST "localhost:9200/_watcher/watch/high-errors/_execute?pretty"

# Get watch status
curl "localhost:9200/_watcher/watch/high-errors?pretty"

5.30 Graph API

# Find connections between tags co-occurring in articles
curl -X GET "localhost:9200/articles/_graph/explore?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "match": { "body": "elasticsearch" } },
    "vertices": [
      { "field": "tags", "size": 10 },
      { "field": "author", "size": 5 }
    ],
    "connections": {
      "query": { "match_all": {} }
    }
  }'

6. Troubleshooting

Note

For quick cluster diagnostics, always start with /_cluster/health?pretty, /_cat/nodes?v, and /_cat/shards?v. These three commands reveal the majority of operational issues.

6.1 Cluster status red

Symptom: _cluster/health returns "status": "red".

Red means at least one primary shard is unassigned — writes to that shard are failing.

# Identify the unassigned shards
curl "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason,unassigned.details" | grep UNASSIGNED

# Get detailed explanation
curl -X GET "localhost:9200/_cluster/allocation/explain?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "index": "my-index", "shard": 0, "primary": true }'

Common causes:

• NODE_LEFT — the node that had the primary shard left the cluster. Check if it will come back; if not, ensure replicas are promoted: POST /_cluster/reroute?retry_failed=true.
• ALLOCATION_FAILED — disk too full, or index settings conflict. Check /_cat/allocation?v for disk usage.
• NEW_INDEX_RESTORED — fresh cluster, index.number_of_replicas too high for available nodes. Reduce replicas temporarily: PUT /my-index/_settings {"index.number_of_replicas": 0}.

6.2 Cluster status yellow

Symptom: "status": "yellow" — all primary shards assigned, but some replicas are not.

This is normal on single-node clusters (no other node to place replicas on). On multi-node clusters, check disk usage and shard count limits:

# Check disk usage per node
curl "localhost:9200/_cat/allocation?v"

# Check total shard count (default limit: 1000 shards per node)
curl "localhost:9200/_cluster/settings?pretty&include_defaults=true" | grep max_shards_per_node

# Temporarily increase limit (use with caution)
curl -X PUT "localhost:9200/_cluster/settings" \
  -H "Content-Type: application/json" \
  -d '{ "persistent": { "cluster.max_shards_per_node": 2000 } }'

6.3 High heap usage / GC pressure

# Check heap usage per node
curl "localhost:9200/_cat/nodes?v&h=name,heap.percent,heap.current,heap.max"

Remedies:

• Force merge old read-only indices: POST /my-index/_forcemerge?max_num_segments=1
• Reduce indices.queries.cache.size (default 10% of heap) if query cache is large
• Run fewer concurrent searches or add more nodes
• Ensure bootstrap.memory_lock: true to prevent heap swapping

6.4 Slow queries

# Enable slow log threshold
curl -X PUT "localhost:9200/articles/_settings" \
  -H "Content-Type: application/json" \
  -d '{
    "index.search.slowlog.threshold.query.warn": "10s",
    "index.search.slowlog.threshold.query.info": "5s",
    "index.search.slowlog.threshold.fetch.warn": "1s",
    "index.search.slowlog.level": "info"
  }'

Slow query causes and fixes:

• Wildcard leading * — forces a full term scan. Use ngram or edge_ngram analyzers instead.
• Deep pagination (from: 10000) — use search_after with a sort on a unique field.
• Script-heavy aggregations — consider precomputing via transforms.
• Large size — retrieve only the fields you need with _source: ["field1","field2"].
• Expensive query_string with many clauses — rewrite to bool with filter clauses.

6.5 Disk space issues

# Check disk usage
curl "localhost:9200/_cat/allocation?v&h=node,disk.used,disk.avail,disk.percent"

# Which indices are largest?
curl "localhost:9200/_cat/indices?v&s=store.size:desc&h=index,store.size,pri,rep"

Remedies:

• Force merge to reduce segment overhead: POST /old-index/_forcemerge?max_num_segments=1
• Delete old snapshots
• Implement ILM delete phase
• Temporarily raise watermarks (emergency only): PUT /_cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.flood_stage": "99%" } }

Warning

Elasticsearch sets write blocks on indices when disk usage reaches the flood-stage watermark (default 95%). When this happens, index.blocks.read_only_allow_delete is set to true. You must free disk space AND clear the block: PUT /my-index/_settings {"index.blocks.read_only_allow_delete": null}.

6.6 Indexing rejected (queue full)

# Check thread pool queue depths
curl "localhost:9200/_cat/thread_pool?v&h=node_name,name,active,queue,rejected"

If write queue shows rejections: slow down the indexing client, increase thread_pool.write.queue_size in elasticsearch.yml, or add more nodes.

6.7 Mapping conflicts

Symptom: mapper_parsing_exception or illegal_argument_exception about field type conflict.

# Check current mapping
curl "localhost:9200/my-index/_mapping?pretty"

# If the field type is wrong, you must reindex:
# 1. Create new index with correct mapping
# 2. Reindex into it
# 3. Swap aliases

Warning

You cannot change the type of an existing mapped field in-place. The only solution is to create a new index with the correct mapping, reindex the data, and update any aliases pointing to the index.

6.8 Version conflicts (optimistic concurrency)

# Use retry_on_conflict for update
curl -X POST "localhost:9200/articles/_update/42?retry_on_conflict=3&pretty" \
  -H "Content-Type: application/json" \
  -d '{ "script": { "source": "ctx._source.views += 1" } }'

# Or use if_seq_no / if_primary_term for explicit CAS
curl -X PUT "localhost:9200/articles/_doc/42?if_seq_no=10&if_primary_term=1&pretty" \
  -H "Content-Type: application/json" \
  -d '{ "title": "Updated Title", "views": 1601 }'

6.9 Node not joining cluster

Checklist:

1. cluster.name must match across all nodes
2. discovery.seed_hosts must include at least one reachable master-eligible node
3. Firewall must allow port 9300 (transport) between nodes
4. Clock skew must be < 5 seconds between nodes (timedatectl status)
5. cluster.initial_master_nodes should only be set during first bootstrap

6.10 Certificate errors (TLS)

# Verify certificate
openssl s_client -connect localhost:9200 -CAfile config/certs/http_ca.crt

# Skip verification temporarily (development only — never production)
curl -k https://localhost:9200/

# Regenerate expired certificates
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --days 1095

6.11 Authentication failures

# Test credentials
curl -u elastic:mypassword https://localhost:9200/

# Reset elastic password
bin/elasticsearch-reset-password -u elastic

# Check who am I
curl -u elastic:mypassword "localhost:9200/_security/_authenticate?pretty"

6.12 Bulk indexing performance

# Check bulk rejection rate
curl "localhost:9200/_cat/thread_pool?v&h=node_name,name,active,queue,rejected" | grep write

Tuning checklist for high-throughput bulk indexing:

• Use the Bulk API (never index one document at a time)
• Batch size 5–15 MB per request
• Use multiple parallel clients (one per CPU core is a reasonable start)
• Set "refresh_interval": "30s" or "-1" during initial load, restore to "1s" after
• Reduce replica count to 0 during initial load, set to target after
• Disable _source compression if disk is not a constraint

6.13 max_result_window exceeded

Symptom: Result window is too large, from + size must be less than or equal to: [10000]

# Increase the limit (prefer search_after for deep pagination)
curl -X PUT "localhost:9200/my-index/_settings" \
  -H "Content-Type: application/json" \
  -d '{ "index.max_result_window": 50000 }'

6.14 Index write-blocked

Symptom: cluster_block_exception on write operations.

# Check for blocks
curl "localhost:9200/_cluster/state/blocks?pretty"

# Remove a read-only block (usually set by disk watermark)
curl -X PUT "localhost:9200/my-index/_settings" \
  -H "Content-Type: application/json" \
  -d '{ "index.blocks.read_only_allow_delete": null }'

# Remove all blocks on all indices
curl -X PUT "localhost:9200/_all/_settings" \
  -H "Content-Type: application/json" \
  -d '{ "index.blocks.read_only_allow_delete": null }'

6.15 Log locations

# Follow logs in real time (package install)
tail -f /var/log/elasticsearch/my-cluster.log

# Docker
docker logs -f es01

# Get recent deprecation warnings (critical before major version upgrades)
tail -100 /var/log/elasticsearch/my-cluster_deprecation.log

Warning

Review deprecation logs thoroughly before any major version upgrade. Each deprecated feature your application relies on must be migrated before the next major version drops support for it.

6.16 Task management

# List running tasks
curl "localhost:9200/_tasks?detailed=true&pretty"

# List tasks for a specific action
curl "localhost:9200/_tasks?actions=indices:data/write/reindex&detailed=true&pretty"

# Cancel a running task
curl -X POST "localhost:9200/_tasks/node-id:task-number/_cancel"

# Get task result
curl "localhost:9200/_tasks/node-id:task-number?wait_for_completion=true&pretty"

6.17 Recovery monitoring

# Monitor shard recovery
curl "localhost:9200/_cat/recovery?v&active_only=true&h=index,shard,type,stage,source_node,target_node,files_percent,bytes_percent,time"

# Detailed recovery info
curl "localhost:9200/_recovery?pretty&active_only=true"

6.18 Hot threads analysis

# What is every thread in the cluster doing right now?
curl "localhost:9200/_nodes/hot_threads?threads=5&interval=1s"

# Specific node
curl "localhost:9200/_nodes/es01/hot_threads"

6.19 Searchable snapshots

Searchable snapshots allow querying data stored in a snapshot repository without fully restoring it:

# Mount a snapshot as a searchable index
curl -X POST "localhost:9200/_snapshot/my-backup/snapshot-2026-05-13/_mount?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "index": "articles",
    "renamed_index": "articles-archive",
    "storage": "shared_cache"
  }'

# Query just like a regular index
curl "localhost:9200/articles-archive/_count?pretty"

6.20 Common error codes

6.21 Shard count best practices

Caution

Avoid "over-sharding" — having too many small shards is a leading cause of Elasticsearch performance problems. Each shard has overhead in memory and CPU. Target 20–50 GB per shard. A 100 GB index with 3 primary shards is healthier than a 10 GB index with 30 primary shards.

# Check shard sizes
curl "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,store" | sort -k5 -h

6.22 Pending cluster tasks

curl "localhost:9200/_cluster/pending_tasks?pretty"

Many pending tasks (> 10) often indicate the master node is under stress. Check master node CPU/heap with _cat/nodes.

6.23 Dangling indices

# List dangling indices (shards found on disk but not in cluster state)
curl "localhost:9200/_dangling?pretty"

# Import a dangling index
curl -X POST "localhost:9200/_dangling/AE3A6uGgR7mGM..." \
  -H "Content-Type: application/json" \
  -d '{ "accept_data_loss": true }'

6.24 Migration assistance

# Check for deprecated features before upgrading
curl "localhost:9200/_migration/deprecations?pretty"

# Get feature upgrade assistance
curl "localhost:9200/_migration/system_features?pretty"

6.25 Performance profiling with the Search Profiler

# Profile aggregations
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "profile": true,
    "size": 0,
    "aggs": {
      "by_author": { "terms": { "field": "author" } }
    }
  }'

Look for high "time_in_nanos" values in the profile output — these point to expensive operations worth optimizing.

6.26 Watcher troubleshooting

# Check watcher status
curl "localhost:9200/_watcher/stats?pretty"

# Activate a watch
curl -X POST "localhost:9200/_watcher/watch/high-errors/_activate"

# Start/stop watcher
curl -X POST "localhost:9200/_watcher/_start"
curl -X POST "localhost:9200/_watcher/_stop"

7. FAQ

General

Q: What is the difference between an index and a shard in Elasticsearch?

An index is a logical namespace — like a database table — that you interact with via the API. A shard is the physical unit of storage and search. Every index is divided into one or more primary shards, each of which is a self-contained Lucene index. Shards are distributed across nodes, enabling both parallelism (search queries fan out to all shards) and redundancy (replica shards).

Q: How many shards should my index have?

For most indices under 100 GB, start with 1–3 primary shards. Rule of thumb: target 20–50 GB per shard. If you are using data streams (logs/metrics), let ILM handle rollover based on size or time. Over-sharding (many small shards) hurts performance more than under-sharding.

Q: What is the difference between near real-time (NRT) and real-time search?

After indexing a document, it is not immediately searchable. By default, Elasticsearch refreshes the in-memory index every 1 second, making newly indexed documents searchable within about 1 second (NRT). Use POST /index/_refresh to make documents searchable immediately. The GET /index/_doc/{id} endpoint is real-time (reads from the translog) and does not require a refresh.

Q: What is the difference between filter and must in a bool query?

must clauses contribute to the relevance _score. filter clauses are binary — they include or exclude documents but do not affect scoring. Filters are also cached by default, making them significantly faster for repeated queries. Use filter for exact-match conditions (date ranges, terms, keywords) and must for full-text queries where scoring matters.

Q: Does Elasticsearch support transactions?

No. Elasticsearch does not support ACID transactions across multiple documents. Each document-level operation is atomic, but there is no multi-document transaction primitive. For consistency needs, use the if_seq_no / if_primary_term parameters for optimistic concurrency control on single documents.

Q: What is the _source field and can I disable it?

_source is the original JSON document as you sent it. It is stored separately from the indexed fields and is what gets returned in search results. You can disable it ("_source": {"enabled": false}) to save disk space, but then you cannot update or reindex documents and you get no source in search results. Disabling is only appropriate for specific write-heavy, read-rarely use cases.

Q: What is a dynamic mapping and when should I use explicit mappings?

Dynamic mapping lets Elasticsearch automatically detect and add new fields as they appear in documents. It is convenient for development but can cause problems in production: a new field with an unexpected type (e.g., a number that sometimes contains a string) will cause a mapping conflict. Use explicit mappings in production, and set "dynamic": "strict" to reject unknown fields.

curl -X PUT "localhost:9200/my-index/_mapping" \
  -H "Content-Type: application/json" \
  -d '{ "dynamic": "strict" }'

Q: What is the keyword vs text field type?

• text — analyzed (tokenized, lowercased, stemmed). Use for full-text search where you want partial matches.
• keyword — stored as-is, not analyzed. Use for exact matching, sorting, aggregations, and filtering.

Best practice: use "fields": {"keyword": {"type": "keyword"}} on text fields when you need both full-text search and aggregations/sorting on the same field.

Q: Can I run Elasticsearch on Windows?

Yes. The tar/zip distribution and Docker both work on Windows. For production workloads, Linux is strongly preferred — Linux allows mlockall for memory locking, and Windows imposes additional JVM overhead. For development on Windows, Docker Desktop with WSL2 backend is the recommended approach.

Q: What is the maximum document size?

There is no hard-coded document size limit, but the default http.max_content_length is 100 MB for HTTP requests. Individual field values of type text are limited to 32,766 bytes by Lucene. Documents close to or exceeding 1 MB should be evaluated for splitting or using binary attachments via the ingest-attachment plugin.

Indexing

Q: What is the best way to do a bulk load of millions of documents?

1. Disable replicas temporarily: PUT /my-index/_settings {"index.number_of_replicas": 0}
2. Set refresh_interval: -1 during load
3. Use the Bulk API with 5–15 MB batches
4. Use multiple parallel threads (start with 2× CPU cores)
5. Restore replicas and refresh interval after load: {"index.number_of_replicas": 1, "index.refresh_interval": "1s"}
6. Call POST /my-index/_forcemerge?max_num_segments=5 after load

Q: How do I prevent duplicate documents when indexing?

Provide explicit _id values and use PUT /index/_doc/{id} or "op_type": "create" to prevent overwrites. For content-based deduplication, compute a hash of the content and use it as the document ID. Elasticsearch has no built-in upsert-dedup; that logic lives in your application.

Q: What happens to in-flight writes if a node crashes?

Documents are written to the primary shard's translog (write-ahead log) before being acknowledged. If the node crashes before fsync, the translog replays on restart. As long as at least one replica is available and wait_for_active_shards is configured appropriately, data loss is bounded to the current translog segment.

Q: Can I change an existing field's mapping type?

No. Once a field is mapped, its type is immutable. To change a field type, you must: create a new index with the correct mapping, reindex the data using _reindex, and swap aliases.

Q: What is wait_for_active_shards?

This parameter on write operations controls how many shard copies (primary + replicas) must confirm the write before Elasticsearch acknowledges it. The default is 1 (only the primary). Setting it to all provides stronger durability guarantees but increases write latency.

Searching

Q: My search returns documents that shouldn't match — why?

The most common cause is using must when you should use filter. The match query uses full-text analysis (tokenization, stemming, lowercase), so a search for "run" may match "running". Use term queries on keyword fields for exact matching. Check what the analyzer produces with GET /index/_analyze.

Q: How do I search across multiple indices?

Specify a comma-separated list, a wildcard, or _all:

curl "localhost:9200/articles,products/_search?pretty" ...
curl "localhost:9200/logs-*/_search?pretty" ...
curl "localhost:9200/_all/_search?pretty" ...

Q: What is relevance scoring and how is _score calculated?

By default, Elasticsearch uses BM25 (Best Match 25), which considers term frequency (how often the term appears in the document), inverse document frequency (how rare the term is across all documents), and field length normalization. Use the Explain API (_explain) to see the exact score breakdown for a specific document.

Q: How do I do an exact case-sensitive search?

Index the field as keyword (not text) and use a term query. Text fields are analyzed (lowercased by default), making case-sensitive search impossible without a dedicated keyword sub-field.

Q: What is the difference between match_phrase and match?

match splits the query into tokens and searches for any token (or all tokens with "operator": "and"). match_phrase requires all tokens to appear in order with no gap between them (configurable with slop).

Q: Can I do aggregations on analyzed text fields?

No — analyzed text fields are not usable for aggregations, sorting, or scripting by default. Use .keyword sub-fields or keyword type fields instead. Aggregating on text requires the fielddata parameter enabled, which is memory-intensive and not recommended.

Q: What is a nested field and when do I need it?

When you index an array of objects, Elasticsearch flattens them, losing the relationship between fields within each object. If you need to query "products where price > 100 AND brand = 'Acme' in the same object", you need nested type mapping. Nested documents are stored as hidden documents and have higher indexing/query cost.

{
  "mappings": {
    "properties": {
      "variants": {
        "type": "nested",
        "properties": {
          "color": { "type": "keyword" },
          "price": { "type": "float" }
        }
      }
    }
  }
}

Q: How do I implement autocomplete?

Three approaches, in order of flexibility:

1. match_phrase_prefix — easiest, works on analyzed fields, no special mapping needed.
2. edge_ngram token filter — indexes prefixes at index time; faster queries, more disk.
3. completion suggester — purpose-built for autocomplete, fastest, requires dedicated field type.

# Edge ngram approach — analyzer definition
curl -X PUT "localhost:9200/my-index?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "analysis": {
        "filter": {
          "autocomplete_filter": {
            "type": "edge_ngram",
            "min_gram": 2,
            "max_gram": 20
          }
        },
        "analyzer": {
          "autocomplete": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase", "autocomplete_filter"]
          }
        }
      }
    }
  }'

Q: How do I do a "not" query?

Use must_not in a bool query:

{
  "query": {
    "bool": {
      "must": [{ "match": { "body": "elasticsearch" } }],
      "must_not": [{ "term": { "tags": "deprecated" } }]
    }
  }
}

Q: What is the difference between _search and _count?

_count returns only the count of matching documents — it skips fetching source and highlighting, making it significantly cheaper than _search?size=0. Use _count when you only need to know how many documents match.

Q: How does ES|QL differ from the Search DSL?

ES|QL uses a pipe-based syntax (FROM index | WHERE condition | STATS ... BY field) optimized for tabular result sets and operational analytics. The Search DSL (JSON query objects) is the primary API for full-text search, scoring, and aggregations on analyzed fields. ES|QL is generally faster for structured filters and group-by operations; the Search DSL is richer for relevance-ranked full-text search.

Q: When should I use EQL vs ES|QL vs the Search DSL?

Performance

Q: How much RAM should I give Elasticsearch?

Rule: half of available RAM to the JVM heap (up to 31 GB), the other half for the OS page cache. On a 64 GB server: -Xms31g -Xmx31g, leaving 33 GB for the OS.

Q: Why is Elasticsearch using so much disk?

• Lucene segments: each refresh creates new segment files; segments are merged over time
• _source storage doubles your data size (raw JSON stored alongside indexed fields)
• Replicas multiply disk usage by (1 + number_of_replicas)
• Force merge old indices to reduce segment overhead: POST /my-index/_forcemerge?max_num_segments=1

Q: How do I speed up aggregations on large datasets?

1. Use filter to narrow the document set before aggregating
2. Use transforms to precompute summary statistics
3. Use doc_values (enabled by default on keyword and numeric types) — avoid fielddata
4. Consider using sampler or diversified_sampler aggregations on very large datasets
5. Use execution_hint: map for terms aggregations on low-cardinality fields

Q: What is the refresh interval and how does it affect performance?

The refresh interval (default: 1s) controls how often Elasticsearch creates a new Lucene segment from the in-memory buffer, making documents searchable. Each refresh is I/O-intensive. For write-heavy workloads, increase to 30s or use refresh_interval: -1 (no automatic refresh). For real-time search requirements, keep at 1s or call _refresh after each batch.

Q: Should I use filter or query context in bool queries?

Use filter context for all exact-match conditions where scoring is irrelevant (date ranges, terms, keywords, exists). Filters are cached in the filter cache and reused across queries. Use must/should (query context) only for full-text fields where relevance scoring matters.

Operations

Q: How do I perform a rolling upgrade?

1. Disable shard allocation: PUT /_cluster/settings {"persistent": {"cluster.routing.allocation.enable": "primaries"}}
2. Stop node, upgrade, start node
3. Wait for node to rejoin and shards to be allocated: GET /_cat/health?v
4. Re-enable allocation: PUT /_cluster/settings {"persistent": {"cluster.routing.allocation.enable": null}}
5. Repeat for each node

Q: Can I downgrade Elasticsearch after upgrading?

No. Elasticsearch data directories are not backward-compatible. Once a node has started with a newer version, that node's data cannot be read by an older version. Always take a snapshot before upgrading.

Q: How do I monitor Elasticsearch in production?

Key metrics to monitor:

• Cluster health (green/yellow/red)
• JVM heap usage (alert at > 75%)
• GC duration (alert if GC takes > 500 ms frequently)
• Search/index latency (p50, p99)
• Rejected thread pool tasks (write, search queues)
• Disk usage (alert at > 80%)
• Unassigned shards

Use Kibana + Elastic APM for native monitoring, or export via the Metrics/Stats APIs to Prometheus/Grafana.

Q: What is the difference between a component template and an index template?

An index template is the top-level object that matches index patterns and controls settings/mappings. A component template is a reusable building block that index templates composed_of. Component templates allow sharing mapping or settings definitions across many index templates without duplication.

Q: How do I safely delete an index?

1. Set action.destructive_requires_name: true in elasticsearch.yml (prevents DELETE /_all)
2. Take a snapshot first
3. Verify the index name exactly before deleting
4. DELETE /exact-index-name

Q: What is the difference between cluster.initial_master_nodes and discovery.seed_hosts?

discovery.seed_hosts is a list of nodes used to seed the discovery process on every startup — it helps nodes find each other. cluster.initial_master_nodes is a bootstrap-only setting used on the very first cluster formation to prevent split-brain during initial startup. Once the cluster is formed and the cluster state is persisted, cluster.initial_master_nodes is no longer used and should ideally be removed.

Q: How do I add a new node to an existing cluster?

1. Install Elasticsearch on the new node
2. Set cluster.name to match the existing cluster
3. Set discovery.seed_hosts to point to existing master-eligible nodes
4. Do not set cluster.initial_master_nodes (that is for new clusters only)
5. Start the new node — it will automatically join and receive shard allocations

Q: Can Elasticsearch be used without Kibana?

Absolutely. Kibana is a UI companion but Elasticsearch is fully functional via its REST API. Many deployments use Elasticsearch with custom application frontends, Grafana, Superset, or direct API access.

Q: What is the difference between GET and POST for searches?

Both work for _search. GET with a body is technically non-standard HTTP but is widely supported. POST is unambiguous. Elasticsearch accepts both. For caching purposes, some HTTP proxies and load balancers may not cache POST requests — use GET when query caching at the HTTP layer is desirable.

Q: How do I use Elasticsearch with Python?

The official client is elasticsearch-py:

pip install elasticsearch

from elasticsearch import Elasticsearch

es = Elasticsearch("http://localhost:9200")

# Index a document
es.index(index="articles", id=1, document={
    "title": "Hello Elasticsearch",
    "body": "Getting started with the Python client"
})

# Search
response = es.search(index="articles", body={
    "query": {"match": {"body": "Python client"}}
})
for hit in response["hits"]["hits"]:
    print(hit["_source"]["title"], hit["_score"])

Q: How do I use Elasticsearch with JavaScript/Node.js?

npm install @elastic/elasticsearch

const { Client } = require('@elastic/elasticsearch')
const client = new Client({ node: 'http://localhost:9200' })

// Index
await client.index({
  index: 'articles',
  id: '1',
  document: { title: 'Hello', body: 'Getting started' }
})

// Search
const result = await client.search({
  index: 'articles',
  query: { match: { body: 'getting started' } }
})
console.log(result.hits.hits)

Q: What are data tiers and how do they affect storage?

Elasticsearch supports tiered storage with node roles:

• Hot tier (data_hot) — fast NVMe SSD, receives new writes and handles active searches
• Warm tier (data_warm) — cheaper SSD or HDD, for recent but less-queried data
• Cold tier (data_cold) — cold storage (HDD), low query frequency, reduced replicas
• Frozen tier (data_frozen) — minimal local cache, data served from snapshot repository

ILM moves indices down tiers automatically based on age or size. Configure node tiers with the node.roles setting in elasticsearch.yml.

Q: How do I integrate Elasticsearch with generative AI / RAG pipelines?

The standard pattern for Retrieval-Augmented Generation (RAG):

1. Use an inference endpoint to generate embeddings from your documents at ingest time (via an ingest pipeline with a sparse_embedding or text_embedding inference processor)
2. Store the resulting vectors in a dense_vector field
3. At query time, embed the user's question with the same model
4. Use knn search to retrieve semantically similar documents
5. Optionally combine with a keyword bool query (hybrid search)
6. Pass the retrieved documents as context to your LLM

Elasticsearch's native inference API (section 5.16) supports OpenAI, Anthropic, Cohere, Google, AWS Bedrock, Azure, and Mistral directly — no separate embedding service is required.