# Elasticsearch User Guide

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. At its core, it stores JSON documents in an index, makes them searchable in near real-time, and lets you run aggregations across billions of records in milliseconds. Version 9.x introduces ES|QL (a pipe-based query language), expanded generative AI inference endpoints, and a unified data stream and lifecycle management model.

This guide covers everything you need to run Elasticsearch day-to-day: installing it, configuring it, indexing data, querying it in every supported style, and operating it reliably.

---

## Table of Contents

1. [Quick Start](#1-quick-start)
2. [Installation](#2-installation)
3. [Configuration](#3-configuration)
4. [Core Usage](#4-core-usage)
5. [Common Tasks](#5-common-tasks)
6. [Troubleshooting](#6-troubleshooting)
7. [FAQ](#7-faq)

---

## 1. Quick Start

Get a single-node Elasticsearch cluster running in under five minutes.

### 1.1 Docker (fastest path)

```bash
docker pull docker.elastic.co/elasticsearch/elasticsearch:9.5.0

docker run -d \
  --name es01 \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  docker.elastic.co/elasticsearch/elasticsearch:9.5.0
```

Wait about 20 seconds, then verify:

```bash
curl http://localhost:9200/
```

Expected output:

```json
{
  "name" : "es01",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "abc123xyz",
  "version" : {
    "number" : "9.5.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "lucene_version" : "10.1.0",
    "minimum_wire_compatibility_version" : "8.17.0",
    "minimum_index_compatibility_version" : "8.0.0"
  },
  "tagline" : "You Know, for Search"
}
```

### 1.2 Index your first document

```bash
curl -X PUT "localhost:9200/products/_doc/1" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Wireless Headphones",
    "brand": "SoundMax",
    "price": 79.99,
    "in_stock": true,
    "description": "Over-ear noise-cancelling bluetooth headphones with 30-hour battery"
  }'
```

### 1.3 Search it

```bash
curl "localhost:9200/products/_search?q=noise-cancelling&pretty"
```

```bash
curl -X GET "localhost:9200/products/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "match": {
        "description": "noise cancelling"
      }
    }
  }'
```

Expected output:

```json
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 },
  "hits" : {
    "total" : { "value" : 1, "relation" : "eq" },
    "max_score" : 0.8630463,
    "hits" : [
      {
        "_index" : "products",
        "_id" : "1",
        "_score" : 0.8630463,
        "_source" : {
          "name" : "Wireless Headphones",
          "brand" : "SoundMax",
          "price" : 79.99,
          "in_stock" : true,
          "description" : "Over-ear noise-cancelling bluetooth headphones with 30-hour battery"
        }
      }
    ]
  }
}
```

> [!TIP]
> Append `?pretty` to any API call to get human-readable JSON. Use `?human` to format byte counts and durations as readable strings (e.g., `"1.2gb"` instead of `1289748480`). These parameters are accepted on every Elasticsearch endpoint.

### 1.4 Five-minute tour

```bash
# Check cluster health
curl "localhost:9200/_cluster/health?pretty"

# List all indices
curl "localhost:9200/_cat/indices?v"

# Bulk index three documents
curl -X POST "localhost:9200/products/_bulk?pretty" \
  -H "Content-Type: application/x-ndjson" \
  -d '
{"index":{"_id":"2"}}
{"name":"USB-C Hub","brand":"TechLink","price":39.99,"in_stock":true,"description":"7-port hub with 4K HDMI and 100W PD"}
{"index":{"_id":"3"}}
{"name":"Mechanical Keyboard","brand":"KeyCraft","price":129.00,"in_stock":false,"description":"TKL layout with Cherry MX Blue switches"}
{"index":{"_id":"4"}}
{"name":"Webcam HD","brand":"VisionPro","price":59.99,"in_stock":true,"description":"1080p 30fps autofocus USB camera"}
'

# Range query + aggregation
curl -X GET "localhost:9200/products/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "range": { "price": { "lte": 80 } } },
    "aggs": {
      "avg_price": { "avg": { "field": "price" } },
      "by_brand": { "terms": { "field": "brand.keyword" } }
    }
  }'
```

---

## 2. Installation

### 2.1 System requirements

| Component | Minimum | Recommended |
|-----------|---------|-------------|
| Java | Bundled JDK (no separate install needed) | — |
| RAM | 2 GB (JVM heap) | 32 GB per node (heap ≤ 31 GB) |
| Disk | 20 GB | SSD, 3× expected data size |
| OS | Linux (kernel 3.10+), macOS, Windows | Linux RHEL/Ubuntu/Debian 64-bit |
| CPU | 2 cores | 8+ cores for high-throughput clusters |

> [!IMPORTANT]
> Elasticsearch ships with a bundled JDK. Do **not** set `JAVA_HOME` to point to a system Java. Using a mismatched JDK version can silently corrupt data or cause startup failures.

### 2.2 Install on Linux (tar.gz)

```bash
# Download
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-9.5.0-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-9.5.0-linux-x86_64.tar.gz.sha512
shasum -a 512 -c elasticsearch-9.5.0-linux-x86_64.tar.gz.sha512

# Extract
tar -xzf elasticsearch-9.5.0-linux-x86_64.tar.gz
cd elasticsearch-9.5.0/

# Start
./bin/elasticsearch
```

On first start with security enabled (default), Elasticsearch prints the `elastic` superuser password and enrollment tokens to stdout — save them.

### 2.3 Install on Debian/Ubuntu (apt)

```bash
# Import GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
  sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

# Add repository
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] \
  https://artifacts.elastic.co/packages/9.x/apt stable main" | \
  sudo tee /etc/apt/sources.list.d/elastic-9.x.list

# Install
sudo apt-get update && sudo apt-get install elasticsearch

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
```

### 2.4 Install on RHEL/CentOS (rpm)

```bash
# Import GPG key
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# Add repository /etc/yum.repos.d/elasticsearch.repo
cat <<EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 9.x packages
baseurl=https://artifacts.elastic.co/packages/9.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md
EOF

sudo yum install --enablerepo=elasticsearch elasticsearch
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
```

### 2.5 Docker

```bash
# Single node, security disabled (development only)
docker run -d \
  --name es01 \
  --net elastic \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  -e "xpack.security.enabled=false" \
  -v esdata01:/usr/share/elasticsearch/data \
  docker.elastic.co/elasticsearch/elasticsearch:9.5.0

# Single node, security enabled (stores certs + password in volumes)
docker run -d \
  --name es01 \
  --net elastic \
  -p 9200:9200 \
  -e "discovery.type=single-node" \
  -v esdata01:/usr/share/elasticsearch/data \
  -v escerts01:/usr/share/elasticsearch/config/certs \
  docker.elastic.co/elasticsearch/elasticsearch:9.5.0
```

> [!WARNING]
> Always mount a named volume for `/usr/share/elasticsearch/data`. Without a persistent volume, all indexed data is lost when the container stops.

### 2.6 Docker Compose (multi-node cluster)

```yaml
# docker-compose.yml
version: '3.8'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.0
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.0
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock: { soft: -1, hard: -1 }
    volumes:
      - esdata02:/usr/share/elasticsearch/data
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.0
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock: { soft: -1, hard: -1 }
    volumes:
      - esdata03:/usr/share/elasticsearch/data
    networks:
      - elastic

  kibana:
    image: docker.elastic.co/kibana/kibana:9.5.0
    ports:
      - 5601:5601
    environment:
      - ELASTICSEARCH_HOSTS=http://es01:9200
    networks:
      - elastic

volumes:
  esdata01:
  esdata02:
  esdata03:

networks:
  elastic:
    driver: bridge
```

```bash
docker compose up -d
```

### 2.7 Verify the installation

```bash
# Without security
curl http://localhost:9200/_cluster/health?pretty

# With security (replace PASSWORD with the one printed at first start)
curl -u elastic:PASSWORD https://localhost:9200/_cluster/health?pretty --cacert config/certs/http_ca.crt
```

Expected healthy output:

```json
{
  "cluster_name" : "my-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 10,
  "active_shards" : 20,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
```

### 2.8 Plugin management

Elasticsearch plugins are managed declaratively via `elasticsearch-plugins.yml` in the config directory, or imperatively via the `elasticsearch-plugin` tool.

**Declarative (recommended for production):**

```yaml
# elasticsearch-plugins.yml
plugins:
  - id: analysis-icu
  - id: analysis-kuromoji
  - id: mapper-murmur3
  - id: repository-s3
  - id: repository-gcs
```

**Imperative:**

```bash
# Install from Elastic's registry
bin/elasticsearch-plugin install analysis-icu

# Install from URL
bin/elasticsearch-plugin install https://example.com/my-plugin-9.5.0.zip

# List installed plugins
bin/elasticsearch-plugin list

# Remove a plugin
bin/elasticsearch-plugin remove analysis-icu
```

> [!NOTE]
> Plugin changes require a full node restart. In a rolling upgrade scenario, restart nodes one at a time while the cluster remains operational.

---

## 3. Configuration

### 3.1 elasticsearch.yml

The primary configuration file. Located at `config/elasticsearch.yml` (tar/zip installs) or `/etc/elasticsearch/elasticsearch.yml` (package installs).

```yaml
# Cluster identity
cluster.name: my-production-cluster
node.name: node-1

# Paths
path.data: /var/data/elasticsearch
path.logs: /var/log/elasticsearch

# Memory
bootstrap.memory_lock: true

# Network
network.host: 192.168.1.10
http.port: 9200

# Discovery (list of seed node hostnames or IPs)
discovery.seed_hosts:
  - 192.168.1.10
  - 192.168.1.11
  - 192.168.1.12

# Bootstrap: list master-eligible nodes for first cluster formation only
cluster.initial_master_nodes:
  - node-1
  - node-2
  - node-3

# Safety
action.destructive_requires_name: true
```

> [!IMPORTANT]
> `cluster.initial_master_nodes` is used **only** on the very first cluster bootstrap. Remove it from `elasticsearch.yml` after the cluster is formed to prevent accidental re-bootstrap on future restarts.

### 3.2 Key configuration settings

| Setting | Default | Description |
|---------|---------|-------------|
| `cluster.name` | `elasticsearch` | Logical cluster name; nodes join only clusters with a matching name |
| `node.name` | Hostname | Human-readable node identifier; appears in logs and API responses |
| `path.data` | `$ES_HOME/data` | One or more paths where shard data is stored |
| `path.logs` | `$ES_HOME/logs` | Where log files are written |
| `bootstrap.memory_lock` | `false` | Prevents the JVM heap from being swapped to disk — set `true` in production |
| `network.host` | `_local_` | The address Elasticsearch binds to; use `0.0.0.0` to bind all interfaces |
| `http.port` | `9200` | REST API port |
| `discovery.seed_hosts` | `["127.0.0.1","[::1]"]` | List of known nodes used to seed cluster formation |
| `cluster.initial_master_nodes` | — | Bootstrap-only: names/IPs of master-eligible nodes for a brand-new cluster |
| `action.destructive_requires_name` | `false` | When `true`, wildcard (`*`) and `_all` are rejected on destructive APIs |

### 3.3 JVM heap size

Set in `config/jvm.options` or via the `ES_JAVA_OPTS` environment variable:

```bash
# jvm.options (recommended — edit the file)
-Xms4g
-Xmx4g

# Or environment variable (Docker / systemd override)
export ES_JAVA_OPTS="-Xms4g -Xmx4g"
```

Rules:
- Set min (`-Xms`) equal to max (`-Xmx`) to avoid GC overhead from heap resizing.
- Do **not** exceed 31 GB — above 32 GB the JVM switches to 64-bit object pointers, wasting memory.
- Leave at least 50% of system RAM for the OS page cache (Lucene relies on it heavily).

### 3.4 Docker environment variable configuration

When running in Docker, every `elasticsearch.yml` setting can be set via environment variables using the `ES_SETTING_` prefix. Dots become double underscores:

| `elasticsearch.yml` | Docker env var |
|---------------------|---------------|
| `cluster.name` | `ES_SETTING_CLUSTER__NAME` |
| `node.name` | `ES_SETTING_NODE__NAME` |
| `network.host` | `ES_SETTING_NETWORK__HOST` |
| `http.port` | `ES_SETTING_HTTP__PORT` |
| `bootstrap.memory_lock` | `ES_SETTING_BOOTSTRAP__MEMORY__LOCK` |
| `path.data` | `ES_SETTING_PATH__DATA` |
| `path.logs` | `ES_SETTING_PATH__LOGS` |

Example:

```bash
docker run -d \
  -e "ES_SETTING_CLUSTER__NAME=my-cluster" \
  -e "ES_SETTING_NODE__NAME=node-1" \
  -e "ES_SETTING_NETWORK__HOST=0.0.0.0" \
  docker.elastic.co/elasticsearch/elasticsearch:9.5.0
```

### 3.5 Secure settings (keystore)

Sensitive values (passwords, API keys, S3 credentials) should never go in `elasticsearch.yml` in plaintext. Store them in the Elasticsearch keystore:

```bash
# Create the keystore (first time)
bin/elasticsearch-keystore create

# Add a secret
bin/elasticsearch-keystore add s3.client.default.access_key

# Add a file-based secret (e.g., GCS service account key)
bin/elasticsearch-keystore add-file gcs.client.default.credentials_file /path/to/key.json

# List all keys
bin/elasticsearch-keystore list

# Remove a key
bin/elasticsearch-keystore remove s3.client.default.access_key

# Reload secure settings without restart (some settings support this)
curl -X POST "localhost:9200/_nodes/reload_secure_settings"
```

### 3.6 Security configuration

TLS and authentication are enabled by default in 8.x+. To generate certificates:

```bash
# Generate CA + node certificates
bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

# Generate certificates for multiple nodes at once
bin/elasticsearch-certutil cert \
  --ca elastic-stack-ca.p12 \
  --in instances.yml \
  --out certs.zip
```

`instances.yml` example:

```yaml
instances:
  - name: node-1
    dns:
      - node-1.example.com
    ip:
      - 192.168.1.10
  - name: node-2
    dns:
      - node-2.example.com
    ip:
      - 192.168.1.11
```

### 3.7 User management (native realm)

```bash
# Add a new user (interactive password prompt)
bin/elasticsearch-users useradd alice -r superuser

# Add with explicit role
bin/elasticsearch-users useradd bob -r kibana_admin,viewer

# List users
bin/elasticsearch-users list

# Delete a user
bin/elasticsearch-users userdel alice

# Change password
bin/elasticsearch-users passwd alice
```

---

## 4. Core Usage

### 4.1 REST API fundamentals

Elasticsearch exposes its entire functionality through a RESTful HTTP API. The base URL is `http[s]://{host}:{port}`.

**Common query parameters** accepted on every endpoint:

| Parameter | Description |
|-----------|-------------|
| `pretty` | Format JSON response with indentation |
| `human` | Display sizes/durations as human-readable strings |
| `error_trace` | Include Java stack trace in error responses (debugging) |
| `source` | Pass request body as a URL-encoded query parameter |
| `filter_path` | Comma-separated dot-notation paths to return (reduces response size) |

```bash
# Use filter_path to return only what you need
curl "localhost:9200/_cluster/health?filter_path=status,number_of_nodes"
# {"status":"green","number_of_nodes":3}
```

### 4.2 Indices

#### Create an index with explicit mapping

```bash
curl -X PUT "localhost:9200/articles?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "analysis": {
        "analyzer": {
          "english_with_stop": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase", "english_stop", "english_stemmer"]
          }
        },
        "filter": {
          "english_stop": {
            "type": "stop",
            "stopwords": "_english_"
          },
          "english_stemmer": {
            "type": "stemmer",
            "language": "english"
          }
        }
      }
    },
    "mappings": {
      "properties": {
        "title": {
          "type": "text",
          "analyzer": "english_with_stop",
          "fields": {
            "keyword": { "type": "keyword" }
          }
        },
        "author": { "type": "keyword" },
        "published_at": { "type": "date", "format": "strict_date_optional_time" },
        "views": { "type": "long" },
        "tags": { "type": "keyword" },
        "body": { "type": "text", "analyzer": "english_with_stop" },
        "location": { "type": "geo_point" },
        "embedding": {
          "type": "dense_vector",
          "dims": 768,
          "index": true,
          "similarity": "cosine"
        }
      }
    }
  }'
```

#### Get, open, close, delete

```bash
# Get index info
curl "localhost:9200/articles?pretty"

# Get just mappings
curl "localhost:9200/articles/_mapping?pretty"

# Get just settings
curl "localhost:9200/articles/_settings?pretty"

# Close (suspend I/O, keeps data)
curl -X POST "localhost:9200/articles/_close"

# Re-open
curl -X POST "localhost:9200/articles/_open"

# Delete (irreversible — requires exact name when action.destructive_requires_name=true)
curl -X DELETE "localhost:9200/articles"
```

#### Refresh, flush, forcemerge

```bash
# Refresh — make recent writes searchable immediately
curl -X POST "localhost:9200/articles/_refresh"

# Flush — fsync segments to disk + clear translog
curl -X POST "localhost:9200/articles/_flush"

# Force merge — reduce segment count (use on read-only/completed indices)
curl -X POST "localhost:9200/articles/_forcemerge?max_num_segments=1"
```

### 4.3 Indexing documents

#### Index (create or replace)

```bash
# With explicit ID
curl -X PUT "localhost:9200/articles/_doc/42?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Getting Started with Elasticsearch",
    "author": "alice",
    "published_at": "2026-03-01T10:00:00Z",
    "views": 1500,
    "tags": ["elasticsearch", "tutorial", "search"],
    "body": "Elasticsearch is a distributed search engine..."
  }'

# Auto-generated ID (POST)
curl -X POST "localhost:9200/articles/_doc?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "title": "Advanced Aggregations", "author": "bob", "views": 800 }'
```

#### Create only (fail if exists)

```bash
curl -X PUT "localhost:9200/articles/_create/42?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "title": "New Article" }'

# Or using _doc with op_type=create
curl -X PUT "localhost:9200/articles/_doc/42?op_type=create" \
  -H "Content-Type: application/json" \
  -d '{ "title": "New Article" }'
```

#### Update (partial)

```bash
# Partial update — only specified fields change
curl -X POST "localhost:9200/articles/_update/42?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "doc": {
      "views": 2000,
      "tags": ["elasticsearch", "tutorial", "search", "updated"]
    }
  }'

# Scripted update
curl -X POST "localhost:9200/articles/_update/42?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "script": {
      "source": "ctx._source.views += params.increment",
      "params": { "increment": 100 }
    }
  }'

# Upsert — insert if not found, update if found
curl -X POST "localhost:9200/articles/_update/99?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "script": {
      "source": "ctx._source.views += params.increment",
      "params": { "increment": 1 }
    },
    "upsert": {
      "title": "New Article",
      "views": 1
    }
  }'
```

#### Get and delete

```bash
# Get document
curl "localhost:9200/articles/_doc/42?pretty"

# Get source only
curl "localhost:9200/articles/_source/42"

# Check existence (HEAD — returns 200 or 404, no body)
curl -I "localhost:9200/articles/_doc/42"

# Delete
curl -X DELETE "localhost:9200/articles/_doc/42"
```

#### Bulk API

The Bulk API processes multiple operations in a single request. Each operation is two lines: an action line + optional document line.

```bash
curl -X POST "localhost:9200/_bulk?pretty" \
  -H "Content-Type: application/x-ndjson" \
  -d '
{"index":{"_index":"articles","_id":"1"}}
{"title":"Article One","author":"alice","views":100}
{"index":{"_index":"articles","_id":"2"}}
{"title":"Article Two","author":"bob","views":200}
{"update":{"_index":"articles","_id":"1"}}
{"doc":{"views":150}}
{"delete":{"_index":"articles","_id":"3"}}
'
```

> [!TIP]
> Keep bulk request sizes between 5 MB and 15 MB per batch. Very large batches increase GC pressure; very small batches increase HTTP overhead. Index-specific bulk: `POST /articles/_bulk`.

#### Multi-get (mget)

```bash
curl -X GET "localhost:9200/_mget?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "docs": [
      { "_index": "articles", "_id": "1" },
      { "_index": "articles", "_id": "2" },
      { "_index": "products", "_id": "1" }
    ]
  }'
```

### 4.4 Searching

#### Match query (full-text)

```bash
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "match": {
        "body": {
          "query": "distributed search engine",
          "operator": "and"
        }
      }
    }
  }'
```

#### Multi-match

```bash
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "multi_match": {
        "query": "elasticsearch tutorial",
        "fields": ["title^3", "body", "tags"],
        "type": "best_fields"
      }
    }
  }'
```

#### Boolean query

```bash
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "bool": {
        "must": [
          { "match": { "body": "elasticsearch" } }
        ],
        "filter": [
          { "term": { "author": "alice" } },
          { "range": { "published_at": { "gte": "2026-01-01" } } }
        ],
        "must_not": [
          { "term": { "tags": "deprecated" } }
        ],
        "should": [
          { "term": { "tags": "featured" } }
        ],
        "minimum_should_match": 0
      }
    }
  }'
```

#### Term-level queries (exact values)

```bash
# Term — exact keyword match
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "term": { "author": "alice" } } }'

# Terms — multiple values (OR)
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "terms": { "tags": ["elasticsearch", "lucene"] } } }'

# Range
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "range": { "views": { "gte": 500, "lt": 5000 } } } }'

# Exists — documents that have a field
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "exists": { "field": "tags" } } }'

# Prefix
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "prefix": { "author": "ali" } } }'

# Wildcard
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "wildcard": { "title.keyword": "Getting*" } } }'

# Fuzzy — handles typos
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "fuzzy": { "author": { "value": "alise", "fuzziness": "AUTO" } } } }'

# IDs
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "ids": { "values": ["1", "2", "42"] } } }'
```

#### Full-text: match_phrase, query_string

```bash
# Phrase match — terms in order, adjacent
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "match_phrase": { "body": "distributed search engine" } } }'

# Phrase prefix (autocomplete)
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "match_phrase_prefix": { "title": "Getting St" } } }'

# Query string (Lucene syntax)
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "query_string": {
        "query": "(title:elasticsearch OR tags:search) AND views:[100 TO *]",
        "default_field": "body"
      }
    }
  }'
```

#### Pagination

```bash
# From/size (offset-based, avoid deep pagination past 10,000)
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "from": 20,
    "size": 10,
    "query": { "match_all": {} },
    "sort": [{ "published_at": "desc" }]
  }'

# Search after (efficient deep pagination)
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 10,
    "query": { "match_all": {} },
    "sort": [{ "published_at": "desc" }, { "_id": "asc" }],
    "search_after": ["2026-01-15T12:00:00Z", "article-id-here"]
  }'
```

#### Sorting and source filtering

```bash
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "match_all": {} },
    "sort": [
      { "views": { "order": "desc" } },
      { "published_at": { "order": "desc" } },
      "_score"
    ],
    "_source": ["title", "author", "views"],
    "size": 5
  }'
```

#### Highlighting

```bash
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "match": { "body": "distributed search" } },
    "highlight": {
      "fields": {
        "body": {
          "fragment_size": 150,
          "number_of_fragments": 3,
          "pre_tags": ["<em>"],
          "post_tags": ["</em>"]
        },
        "title": {}
      }
    }
  }'
```

### 4.5 Aggregations

#### Bucket aggregations

```bash
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 0,
    "aggs": {
      "by_author": {
        "terms": {
          "field": "author",
          "size": 10
        },
        "aggs": {
          "avg_views": { "avg": { "field": "views" } },
          "max_views": { "max": { "field": "views" } }
        }
      },
      "views_histogram": {
        "histogram": {
          "field": "views",
          "interval": 500
        }
      },
      "published_over_time": {
        "date_histogram": {
          "field": "published_at",
          "calendar_interval": "month"
        }
      },
      "views_range": {
        "range": {
          "field": "views",
          "ranges": [
            { "to": 100 },
            { "from": 100, "to": 1000 },
            { "from": 1000 }
          ]
        }
      }
    }
  }'
```

#### Metric aggregations

```bash
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 0,
    "aggs": {
      "total_views": { "sum": { "field": "views" } },
      "avg_views": { "avg": { "field": "views" } },
      "min_views": { "min": { "field": "views" } },
      "max_views": { "max": { "field": "views" } },
      "views_stats": { "stats": { "field": "views" } },
      "views_percentiles": {
        "percentiles": {
          "field": "views",
          "percents": [25, 50, 75, 90, 95, 99]
        }
      },
      "unique_authors": { "cardinality": { "field": "author" } }
    }
  }'
```

#### Pipeline aggregations

```bash
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 0,
    "aggs": {
      "monthly": {
        "date_histogram": {
          "field": "published_at",
          "calendar_interval": "month"
        },
        "aggs": {
          "total_views": { "sum": { "field": "views" } },
          "cumulative_views": {
            "cumulative_sum": { "buckets_path": "total_views" }
          },
          "avg_views": { "avg": { "field": "views" } }
        }
      },
      "best_month": {
        "max_bucket": {
          "buckets_path": "monthly>total_views"
        }
      }
    }
  }'
```

### 4.6 KNN and vector search

```bash
# kNN search with a pre-computed vector
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "knn": {
      "field": "embedding",
      "query_vector": [0.12, -0.34, 0.56],
      "k": 10,
      "num_candidates": 100
    },
    "_source": ["title", "author"]
  }'

# Hybrid: kNN + keyword filter
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "knn": {
      "field": "embedding",
      "query_vector": [0.12, -0.34, 0.56],
      "k": 10,
      "num_candidates": 100,
      "filter": {
        "term": { "tags": "elasticsearch" }
      }
    },
    "query": {
      "match": { "title": "vector search" }
    }
  }'
```

> [!TIP]
> For semantic search use cases, combine kNN vector search (semantic similarity) with a keyword `bool` query (exact matching) and use Reciprocal Rank Fusion (RRF) to merge result sets. This hybrid approach consistently outperforms either method alone on real-world datasets.

### 4.7 ES|QL — pipe-based query language

ES|QL is Elasticsearch's modern pipe-based query language, similar to Splunk SPL or Kusto KQL.

```bash
curl -X POST "localhost:9200/_query?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "FROM articles | WHERE views > 500 | STATS avg_views = AVG(views), total = COUNT(*) BY author | SORT avg_views DESC | LIMIT 10"
  }'

# ES|QL with date filtering
curl -X POST "localhost:9200/_query?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "FROM articles | WHERE published_at >= \"2026-01-01\" | STATS monthly_views = SUM(views) BY BUCKET(published_at, 1 month) | SORT monthly_views DESC"
  }'

# Async ES|QL (for long-running queries)
curl -X POST "localhost:9200/_query/async?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "FROM articles | STATS total = COUNT(*) BY author | SORT total DESC",
    "wait_for_completion_timeout": "10s"
  }'
```

> [!NOTE]
> ES|QL is ideal for log analytics and operational data. It operates on fields directly (not analyzed text), making it fast for structured data. For full-text search on analyzed fields, use the Search API with `match` or `query_string` queries.

### 4.8 EQL — Event Query Language

EQL is optimized for time-series event data and threat hunting.

```bash
# Basic EQL sequence search
curl -X GET "localhost:9200/logs/_eql/search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "sequence by host.name [process where process.name == \"cmd.exe\"] [network where destination.port == 443]",
    "size": 10
  }'

# EQL with filter
curl -X GET "localhost:9200/logs/_eql/search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "process where process.name : (\"*.exe\", \"*.bat\")",
    "filter": {
      "range": { "@timestamp": { "gte": "now-1d" } }
    },
    "size": 20
  }'
```

### 4.9 SQL interface

```bash
# SQL query
curl -X POST "localhost:9200/_sql?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT author, COUNT(*) as article_count, AVG(views) as avg_views FROM articles GROUP BY author ORDER BY article_count DESC LIMIT 10"
  }'

# Translate SQL to DSL (for learning/debugging)
curl -X POST "localhost:9200/_sql/translate?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT * FROM articles WHERE views > 1000"
  }'

# Pagination via cursor
curl -X POST "localhost:9200/_sql?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "cursor": "sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAA..."
  }'
```

---

## 5. Common Tasks

### 5.1 Data streams and time-series data

Data streams are the recommended way to manage append-only time-series data (logs, metrics, events).

#### Set up a data stream

```bash
# 1. Create an index lifecycle policy
curl -X PUT "localhost:9200/_ilm/policy/logs-policy?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "policy": {
      "phases": {
        "hot": {
          "actions": {
            "rollover": {
              "max_primary_shard_size": "50gb",
              "max_age": "7d"
            }
          }
        },
        "warm": {
          "min_age": "7d",
          "actions": {
            "shrink": { "number_of_shards": 1 },
            "forcemerge": { "max_num_segments": 1 }
          }
        },
        "cold": {
          "min_age": "30d",
          "actions": {
            "freeze": {}
          }
        },
        "delete": {
          "min_age": "90d",
          "actions": { "delete": {} }
        }
      }
    }
  }'

# 2. Create a component template for mappings
curl -X PUT "localhost:9200/_component_template/logs-mappings?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "template": {
      "mappings": {
        "properties": {
          "@timestamp": { "type": "date" },
          "log.level": { "type": "keyword" },
          "message": { "type": "text" },
          "host.name": { "type": "keyword" },
          "service.name": { "type": "keyword" }
        }
      }
    }
  }'

# 3. Create an index template that targets the data stream
curl -X PUT "localhost:9200/_index_template/logs-template?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "index_patterns": ["logs-*-*"],
    "data_stream": {},
    "composed_of": ["logs-mappings"],
    "template": {
      "settings": {
        "number_of_shards": 1,
        "index.lifecycle.name": "logs-policy"
      }
    },
    "priority": 200
  }'

# 4. Create the data stream
curl -X PUT "localhost:9200/_data_stream/logs-app-prod"

# 5. Index documents (must include @timestamp)
curl -X POST "localhost:9200/logs-app-prod/_doc?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "@timestamp": "2026-05-13T10:00:00Z",
    "log.level": "error",
    "message": "Connection timeout to database",
    "host.name": "web-01",
    "service.name": "payment-service"
  }'
```

#### Manage data streams

```bash
# List data streams
curl "localhost:9200/_data_stream?pretty"

# Get data stream stats
curl "localhost:9200/_data_stream/logs-app-prod/_stats?pretty"

# Rollover (force a new backing index)
curl -X POST "localhost:9200/logs-app-prod/_rollover?pretty"

# Delete a data stream (removes all backing indices)
curl -X DELETE "localhost:9200/_data_stream/logs-app-prod"
```

### 5.2 Ingest pipelines

Ingest pipelines transform documents before they are indexed. They run on ingest nodes.

#### Create a pipeline

```bash
curl -X PUT "localhost:9200/_ingest/pipeline/parse-apache-log?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Parse Apache Combined Log Format",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": ["%{COMBINEDAPACHELOG}"]
        }
      },
      {
        "date": {
          "field": "timestamp",
          "formats": ["dd/MMM/yyyy:HH:mm:ss Z"],
          "target_field": "@timestamp"
        }
      },
      {
        "convert": {
          "field": "bytes",
          "type": "long",
          "ignore_missing": true
        }
      },
      {
        "convert": {
          "field": "response",
          "type": "integer",
          "ignore_missing": true
        }
      },
      {
        "geoip": {
          "field": "clientip",
          "target_field": "geoip",
          "ignore_missing": true
        }
      },
      {
        "remove": {
          "field": ["timestamp"],
          "ignore_missing": true
        }
      }
    ],
    "on_failure": [
      {
        "set": {
          "field": "_index",
          "value": "failed-{{ _index }}"
        }
      }
    ]
  }'
```

#### Available ingest processors

| Processor | Purpose |
|-----------|---------|
| `append` | Append values to an array field |
| `bytes` | Convert human-readable bytes to integer |
| `community_id` | Generate network community ID hash |
| `convert` | Convert field type (string → integer, etc.) |
| `csv` | Parse CSV row into named fields |
| `date` | Parse date string into `@timestamp` |
| `date_index_name` | Route to date-based index |
| `dissect` | Pattern-based field extraction (faster than grok) |
| `dot_expander` | Expand `a.b` key into nested object |
| `fail` | Raise error with message |
| `fingerprint` | Generate hash fingerprint from fields |
| `foreach` | Apply processor to each element of array |
| `grok` | Regex-based field extraction (named patterns) |
| `gsub` | Regex substitution on string field |
| `html_strip` | Remove HTML tags from string |
| `join` | Join array into delimited string |
| `json` | Parse JSON string into object |
| `kv` | Parse key=value pairs |
| `lowercase` | Lowercase a string field |
| `network_direction` | Determine inbound/outbound from CIDR |
| `pipeline` | Call another ingest pipeline |
| `redact` | Redact patterns (PII) from string fields |
| `registered_domain` | Extract registered domain from URL |
| `remove` | Remove one or more fields |
| `rename` | Rename a field |
| `reroute` | Change target data stream for the document |
| `script` | Run a Painless script |
| `set` | Set a field value |
| `set_security_user` | Populate fields from authenticated user context |
| `sort` | Sort array field |
| `split` | Split string into array by delimiter |
| `trim` | Trim leading/trailing whitespace |
| `uppercase` | Uppercase a string field |
| `uri_parts` | Parse URI into components |
| `url_decode` | URL-decode a string field |

#### Simulate and test a pipeline

```bash
curl -X POST "localhost:9200/_ingest/pipeline/parse-apache-log/_simulate?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "docs": [
      {
        "_source": {
          "message": "127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] \"GET /apache_pb.gif HTTP/1.0\" 200 2326"
        }
      }
    ]
  }'
```

#### Use a pipeline on indexing

```bash
# Single document
curl -X PUT "localhost:9200/logs/_doc/1?pipeline=parse-apache-log" \
  -H "Content-Type: application/json" \
  -d '{ "message": "127.0.0.1 - - [01/Jan/2026:00:00:01 +0000] \"GET / HTTP/1.1\" 200 512" }'

# Set pipeline as the index default
curl -X PUT "localhost:9200/logs/_settings" \
  -H "Content-Type: application/json" \
  -d '{ "index.default_pipeline": "parse-apache-log" }'
```

### 5.3 Reindex

Copy documents from one index to another, optionally transforming them:

```bash
# Basic reindex
curl -X POST "localhost:9200/_reindex?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": { "index": "articles-v1" },
    "dest": { "index": "articles-v2" }
  }'

# Reindex with query filter
curl -X POST "localhost:9200/_reindex?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": {
      "index": "articles-v1",
      "query": { "range": { "views": { "gte": 100 } } }
    },
    "dest": { "index": "articles-popular" }
  }'

# Reindex with script transformation
curl -X POST "localhost:9200/_reindex?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": { "index": "articles-v1" },
    "dest": {
      "index": "articles-v2",
      "pipeline": "my-transform-pipeline"
    },
    "script": {
      "source": "ctx._source.migrated_at = params.now",
      "params": { "now": "2026-05-13" }
    }
  }'

# Reindex from remote cluster
curl -X POST "localhost:9200/_reindex?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": {
      "remote": {
        "host": "http://old-cluster:9200",
        "username": "user",
        "password": "pass"
      },
      "index": "articles"
    },
    "dest": { "index": "articles" }
  }'

# Async reindex (for large indices)
curl -X POST "localhost:9200/_reindex?wait_for_completion=false&pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": { "index": "large-index" },
    "dest": { "index": "large-index-v2" }
  }'
# Returns: {"task":"node-id:task-number"}

# Monitor task progress
curl "localhost:9200/_tasks/node-id:task-number?pretty"
```

### 5.4 Index aliases

Aliases let you point a logical name at one or more physical indices, enabling zero-downtime reindexing:

```bash
# Create alias
curl -X POST "localhost:9200/_aliases?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      { "add": { "index": "articles-v2", "alias": "articles" } }
    ]
  }'

# Atomic alias swap (zero downtime)
curl -X POST "localhost:9200/_aliases?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      { "remove": { "index": "articles-v1", "alias": "articles" } },
      { "add": { "index": "articles-v2", "alias": "articles" } }
    ]
  }'

# Filtered alias (a virtual view)
curl -X POST "localhost:9200/_aliases?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      {
        "add": {
          "index": "articles",
          "alias": "alice-articles",
          "filter": { "term": { "author": "alice" } }
        }
      }
    ]
  }'

# Write alias (controls which index receives writes in multi-index aliases)
curl -X POST "localhost:9200/_aliases?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "actions": [
      { "add": { "index": "articles-v2", "alias": "articles", "is_write_index": true } },
      { "add": { "index": "articles-v1", "alias": "articles" } }
    ]
  }'

# List aliases
curl "localhost:9200/_aliases?pretty"
curl "localhost:9200/articles/_alias?pretty"
```

### 5.5 Index lifecycle management (ILM)

ILM automates the progression of indices through hot → warm → cold → frozen → delete phases.

```bash
# View an index's current ILM status
curl "localhost:9200/my-index/_ilm/explain?pretty"

# Manually move to next phase
curl -X POST "localhost:9200/_ilm/move/my-index?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "current_step": { "phase": "hot", "action": "rollover", "name": "check-rollover-ready" },
    "next_step": { "phase": "warm" }
  }'

# Retry failed ILM step
curl -X POST "localhost:9200/my-index/_ilm/retry?pretty"

# Start/stop ILM globally
curl -X POST "localhost:9200/_ilm/start"
curl -X POST "localhost:9200/_ilm/stop"
curl "localhost:9200/_ilm/status"
```

### 5.6 Snapshots and backups

```bash
# Register a repository (filesystem)
curl -X PUT "localhost:9200/_snapshot/my-backup?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "fs",
    "settings": {
      "location": "/mnt/backups/elasticsearch"
    }
  }'

# Register S3 repository (requires repository-s3 plugin)
curl -X PUT "localhost:9200/_snapshot/s3-backup?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "s3",
    "settings": {
      "bucket": "my-es-backups",
      "region": "us-east-1",
      "base_path": "prod/elasticsearch"
    }
  }'

# Take a snapshot
curl -X PUT "localhost:9200/_snapshot/my-backup/snapshot-2026-05-13?wait_for_completion=true&pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "indices": "articles,products",
    "ignore_unavailable": true,
    "include_global_state": false
  }'

# List snapshots
curl "localhost:9200/_snapshot/my-backup/_all?pretty"
curl "localhost:9200/_snapshot/my-backup/snapshot-2026-05-13?pretty"

# Restore a snapshot
curl -X POST "localhost:9200/_snapshot/my-backup/snapshot-2026-05-13/_restore?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "indices": "articles",
    "rename_pattern": "(.+)",
    "rename_replacement": "$1_restored"
  }'

# Delete a snapshot
curl -X DELETE "localhost:9200/_snapshot/my-backup/snapshot-2026-05-13"
```

#### Snapshot Lifecycle Management (SLM)

```bash
curl -X PUT "localhost:9200/_slm/policy/daily-snapshots?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "schedule": "0 30 1 * * ?",
    "name": "<daily-snap-{now/d}>",
    "repository": "my-backup",
    "config": {
      "indices": ["*"],
      "ignore_unavailable": false,
      "include_global_state": true
    },
    "retention": {
      "expire_after": "30d",
      "min_count": 5,
      "max_count": 50
    }
  }'

# Execute immediately
curl -X POST "localhost:9200/_slm/policy/daily-snapshots/_execute"

# View SLM stats
curl "localhost:9200/_slm/stats?pretty"
```

### 5.7 Multi-search (msearch)

```bash
curl -X GET "localhost:9200/_msearch?pretty" \
  -H "Content-Type: application/x-ndjson" \
  -d '
{"index": "articles"}
{"query": {"match": {"title": "elasticsearch"}}, "size": 5}
{"index": "products"}
{"query": {"range": {"price": {"lte": 50}}}, "sort": [{"price": "asc"}], "size": 5}
{"index": "articles"}
{"query": {"term": {"author": "alice"}}, "aggs": {"avg_views": {"avg": {"field": "views"}}}}
'
```

### 5.8 Async search

For queries that may take more than a few seconds:

```bash
# Submit
curl -X POST "localhost:9200/articles/_async_search?wait_for_completion_timeout=1s&pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "match_all": {} },
    "aggs": {
      "by_author": { "terms": { "field": "author", "size": 100 } }
    }
  }'
# Returns {"id":"FmRldE8zREVEUzA...","is_partial":true,...}

# Poll for results
curl "localhost:9200/_async_search/FmRldE8zREVEUzA...?pretty"

# Delete when done
curl -X DELETE "localhost:9200/_async_search/FmRldE8zREVEUzA..."
```

### 5.9 Search templates

```bash
# Store a search template
curl -X PUT "localhost:9200/_scripts/article-search-template?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "script": {
      "lang": "mustache",
      "source": {
        "query": {
          "bool": {
            "must": [
              { "match": { "body": "{{query_text}}" } }
            ],
            "filter": [
              { "term": { "author": "{{author}}" } }
            ]
          }
        },
        "from": "{{from}}{{^from}}0{{/from}}",
        "size": "{{size}}{{^size}}10{{/size}}"
      }
    }
  }'

# Use the template
curl -X GET "localhost:9200/articles/_search/template?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "article-search-template",
    "params": {
      "query_text": "elasticsearch tutorial",
      "author": "alice",
      "size": 5
    }
  }'
```

### 5.10 Scroll API (batch export)

For exporting all documents from a large index. Prefer `search_after` for interactive pagination:

```bash
# Open scroll context (keep alive for 5 minutes)
curl -X GET "localhost:9200/articles/_search?scroll=5m&pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 1000,
    "query": { "match_all": {} },
    "sort": ["_doc"]
  }'
# Returns {"_scroll_id":"DXF1ZXJ5QW5kRmV0Y2gB...","hits":...}

# Get next batch
curl -X GET "localhost:9200/_search/scroll?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "scroll": "5m",
    "scroll_id": "DXF1ZXJ5QW5kRmV0Y2gB..."
  }'

# Clean up scroll context
curl -X DELETE "localhost:9200/_search/scroll" \
  -H "Content-Type: application/json" \
  -d '{ "scroll_id": "DXF1ZXJ5QW5kRmV0Y2gB..." }'
```

### 5.11 Update by query / Delete by query

```bash
# Update all matching documents
curl -X POST "localhost:9200/articles/_update_by_query?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "term": { "author": "alice" } },
    "script": {
      "source": "ctx._source.verified_author = true"
    }
  }'

# Delete all matching documents
curl -X POST "localhost:9200/articles/_delete_by_query?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "range": {
        "published_at": { "lt": "2020-01-01" }
      }
    }
  }'

# Async (large operations)
curl -X POST "localhost:9200/articles/_delete_by_query?wait_for_completion=false&pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "term": { "tags": "spam" } } }'
```

### 5.12 Cluster and node APIs

```bash
# Cluster health (per-index detail)
curl "localhost:9200/_cluster/health/articles,products?level=indices&pretty"

# Cluster state (filtered to routing table)
curl "localhost:9200/_cluster/state/routing_table/articles?pretty"

# Cluster stats (aggregate)
curl "localhost:9200/_cluster/stats?pretty"

# Pending tasks
curl "localhost:9200/_cluster/pending_tasks?pretty"

# Explain shard allocation (why is a shard unassigned?)
curl -X GET "localhost:9200/_cluster/allocation/explain?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "index": "articles",
    "shard": 0,
    "primary": true
  }'

# Node info
curl "localhost:9200/_nodes?pretty"
curl "localhost:9200/_nodes/es01?pretty"
curl "localhost:9200/_nodes/es01/stats?pretty"

# Hot threads (what is each node doing right now?)
curl "localhost:9200/_nodes/hot_threads"

# Reroute a shard manually
curl -X POST "localhost:9200/_cluster/reroute?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "commands": [
      {
        "move": {
          "index": "articles",
          "shard": 0,
          "from_node": "es01",
          "to_node": "es02"
        }
      }
    ]
  }'
```

### 5.13 Cat APIs (human-readable cluster info)

```bash
# All cat APIs support: ?v (verbose, with headers), ?h=col1,col2 (select columns)
# ?s=col (sort), ?format=json (JSON output)

curl "localhost:9200/_cat/health?v"
curl "localhost:9200/_cat/nodes?v&h=name,ip,heap.percent,ram.percent,cpu,load_1m,disk.used_percent,role"
curl "localhost:9200/_cat/indices?v&s=store.size:desc"
curl "localhost:9200/_cat/shards?v&s=index,shard,prirep"
curl "localhost:9200/_cat/aliases?v"
curl "localhost:9200/_cat/allocation?v"
curl "localhost:9200/_cat/master?v"
curl "localhost:9200/_cat/plugins?v"
curl "localhost:9200/_cat/recovery?v&active_only=true"
curl "localhost:9200/_cat/segments/articles?v"
curl "localhost:9200/_cat/thread_pool?v&h=node_name,name,active,rejected,completed"
curl "localhost:9200/_cat/tasks?v"
curl "localhost:9200/_cat/templates?v"
```

### 5.14 Security: users, roles, and API keys

```bash
# Create a role
curl -X PUT "localhost:9200/_security/role/articles-reader?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "cluster": ["monitor"],
    "indices": [
      {
        "names": ["articles*"],
        "privileges": ["read", "view_index_metadata"]
      }
    ]
  }'

# Create a user
curl -X PUT "localhost:9200/_security/user/reader-bob?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "password": "securepassword123",
    "roles": ["articles-reader"],
    "full_name": "Bob the Reader",
    "email": "[email removed]"
  }'

# Create an API key
curl -X PUT "localhost:9200/_security/api_key?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-app-key",
    "expiration": "30d",
    "role_descriptors": {
      "articles-role": {
        "cluster": ["monitor"],
        "indices": [
          {
            "names": ["articles*"],
            "privileges": ["read"]
          }
        ]
      }
    }
  }'

# List API keys
curl "localhost:9200/_security/api_key?pretty"

# Invalidate API keys
curl -X DELETE "localhost:9200/_security/api_key?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "ids": ["VuaCfGcBCdbkIjaNgtVk"] }'
```

### 5.15 Machine learning — anomaly detection

```bash
# Create a single-metric anomaly detection job
curl -X PUT "localhost:9200/_ml/anomaly_detectors/high-error-rate?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "description": "Detect unusually high error rates",
    "analysis_config": {
      "bucket_span": "15m",
      "detectors": [
        {
          "function": "high_count",
          "over_field_name": "service.name",
          "partition_field_name": "host.name"
        }
      ]
    },
    "data_description": {
      "time_field": "@timestamp"
    }
  }'

# Create a datafeed
curl -X PUT "localhost:9200/_ml/datafeeds/datafeed-high-error-rate?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": "high-error-rate",
    "indices": ["logs-*"],
    "query": {
      "term": { "log.level": "error" }
    }
  }'

# Open job
curl -X POST "localhost:9200/_ml/anomaly_detectors/high-error-rate/_open"

# Start datafeed
curl -X POST "localhost:9200/_ml/datafeeds/datafeed-high-error-rate/_start" \
  -H "Content-Type: application/json" \
  -d '{ "start": "2026-01-01T00:00:00Z" }'

# Get anomaly records
curl "localhost:9200/_ml/anomaly_detectors/high-error-rate/results/records?pretty"

# Stop and close
curl -X POST "localhost:9200/_ml/datafeeds/datafeed-high-error-rate/_stop"
curl -X POST "localhost:9200/_ml/anomaly_detectors/high-error-rate/_close"
```

### 5.16 AI inference endpoints

Elasticsearch 9.x includes built-in inference endpoints that connect to external AI models for text embeddings, completions, and reranking.

```bash
# Create an inference endpoint (OpenAI embeddings)
curl -X PUT "localhost:9200/_inference/text_embedding/my-openai-embeddings?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "openai",
    "service_settings": {
      "model_id": "text-embedding-3-small",
      "api_key": "sk-..."
    }
  }'

# Create an inference endpoint (Anthropic completion)
curl -X PUT "localhost:9200/_inference/completion/my-claude?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "anthropic",
    "service_settings": {
      "model_id": "claude-3-5-sonnet-20241022",
      "api_key": "sk-ant-..."
    }
  }'

# Create an inference endpoint (Cohere reranking)
curl -X PUT "localhost:9200/_inference/rerank/my-cohere-rerank?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "service": "cohere",
    "service_settings": {
      "model_id": "rerank-english-v3.0",
      "api_key": "..."
    }
  }'

# Run inference directly
curl -X POST "localhost:9200/_inference/text_embedding/my-openai-embeddings?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "input": "Elasticsearch vector search tutorial" }'

# Supported providers: openai, anthropic, cohere, amazon_bedrock, google_vertex_ai,
# google_ai_studio, azure_openai, azure_ai_studio, mistral, hugging_face, elastic, elser
```

### 5.17 Transforms

Transforms pivot or group-by data from one index into a summarized destination index:

```bash
# Create a pivot transform
curl -X PUT "localhost:9200/_transform/articles-by-author?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "source": { "index": "articles" },
    "pivot": {
      "group_by": {
        "author": { "terms": { "field": "author" } }
      },
      "aggregations": {
        "article_count": { "value_count": { "field": "_id" } },
        "total_views": { "sum": { "field": "views" } },
        "avg_views": { "avg": { "field": "views" } },
        "latest_published": { "max": { "field": "published_at" } }
      }
    },
    "dest": { "index": "author-stats" },
    "sync": {
      "time": {
        "field": "published_at",
        "delay": "60s"
      }
    }
  }'

# Start the transform
curl -X POST "localhost:9200/_transform/articles-by-author/_start"

# Get stats
curl "localhost:9200/_transform/articles-by-author/_stats?pretty"

# Stop
curl -X POST "localhost:9200/_transform/articles-by-author/_stop"
```

### 5.18 Cross-cluster replication (CCR)

CCR replicates indices from a leader cluster to one or more follower clusters for disaster recovery or geo-distribution:

```bash
# On the follower cluster: configure remote cluster
curl -X PUT "localhost:9200/_cluster/settings?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "persistent": {
      "cluster.remote.leader-cluster.seeds": ["leader-node:9300"]
    }
  }'

# Create a follower index
curl -X PUT "localhost:9200/articles-replica/_ccr/follow?wait_for_active_shards=1&pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "remote_cluster": "leader-cluster",
    "leader_index": "articles"
  }'

# Get CCR stats
curl "localhost:9200/_ccr/stats?pretty"
curl "localhost:9200/articles-replica/_ccr/info?pretty"

# Pause and resume replication
curl -X POST "localhost:9200/articles-replica/_ccr/pause_follow"
curl -X POST "localhost:9200/articles-replica/_ccr/resume_follow"
```

### 5.19 Enrich policies

Enrich processors look up data from an "enrich index" and embed it into incoming documents during ingest:

```bash
# Create the source index
curl -X PUT "localhost:9200/user-profiles/_doc/alice" \
  -H "Content-Type: application/json" \
  -d '{ "email": "[email removed]", "full_name": "Alice Smith", "department": "Engineering" }'

# Create an enrich policy
curl -X PUT "localhost:9200/_enrich/policy/user-lookup?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "match": {
      "indices": "user-profiles",
      "match_field": "email",
      "enrich_fields": ["full_name", "department"]
    }
  }'

# Execute the policy (builds the enrich index)
curl -X POST "localhost:9200/_enrich/policy/user-lookup/_execute"

# Add enrich processor to a pipeline
curl -X PUT "localhost:9200/_ingest/pipeline/enrich-users?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "processors": [
      {
        "enrich": {
          "policy_name": "user-lookup",
          "field": "user_email",
          "target_field": "user"
        }
      }
    ]
  }'
```

### 5.20 Query rules

Query rules promote specific documents to the top of search results based on matching conditions:

```bash
# Create a ruleset
curl -X PUT "localhost:9200/_query_rules/my-promotions?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "rules": [
      {
        "rule_id": "promoted-elasticsearch-guide",
        "type": "pinned",
        "criteria": [
          { "type": "contains", "metadata": "query", "values": ["elasticsearch"] }
        ],
        "actions": {
          "ids": ["42"]
        }
      }
    ]
  }'

# Use rules in a search
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "rule": {
        "match_criteria": { "query": "elasticsearch tutorial" },
        "ruleset_ids": ["my-promotions"],
        "organic": {
          "match": { "body": "elasticsearch tutorial" }
        }
      }
    }
  }'
```

### 5.21 Synonyms

```bash
# Create a synonym set
curl -X PUT "localhost:9200/_synonyms/my-synonyms?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "synonyms_set": [
      { "id": "syn1", "synonyms": "elasticsearch, es, elastic search" },
      { "id": "syn2", "synonyms": "k8s => kubernetes" },
      { "id": "syn3", "synonyms": "js, javascript, ecmascript" }
    ]
  }'

# Reference from an index analyzer
curl -X PUT "localhost:9200/my-index?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "analysis": {
        "filter": {
          "my_synonyms": {
            "type": "synonym_graph",
            "synonyms_set": "my-synonyms",
            "updateable": true
          }
        },
        "analyzer": {
          "my_analyzer": {
            "tokenizer": "standard",
            "filter": ["lowercase", "my_synonyms"]
          }
        }
      }
    }
  }'
```

### 5.22 Connectors (data source integration)

Connectors crawl external data sources (SharePoint, Confluence, MySQL, etc.) and index content into Elasticsearch:

```bash
# List available connector types
curl "localhost:9200/_connector/_list?pretty"

# Create a connector
curl -X PUT "localhost:9200/_connector/my-sharepoint?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "SharePoint Online Connector",
    "service_type": "sharepoint_online",
    "index_name": "sharepoint-content"
  }'

# Update configuration
curl -X PUT "localhost:9200/_connector/my-sharepoint/_configuration?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "values": {
      "tenant_id": "your-tenant-id",
      "client_id": "your-client-id"
    }
  }'

# Check sync jobs
curl "localhost:9200/_connector/_sync_job?connector_id=my-sharepoint&pretty"
```

### 5.23 GeoIP and geo queries

```bash
# Geo distance filter
curl -X GET "localhost:9200/stores/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "geo_distance": {
        "distance": "10km",
        "location": {
          "lat": 40.715,
          "lon": -74.011
        }
      }
    },
    "sort": [
      {
        "_geo_distance": {
          "location": { "lat": 40.715, "lon": -74.011 },
          "order": "asc",
          "unit": "km"
        }
      }
    ]
  }'

# Geo bounding box
curl -X GET "localhost:9200/stores/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "geo_bounding_box": {
        "location": {
          "top_left": { "lat": 41.0, "lon": -74.5 },
          "bottom_right": { "lat": 40.5, "lon": -73.5 }
        }
      }
    }
  }'

# Geohash grid aggregation
curl -X GET "localhost:9200/stores/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "size": 0,
    "aggs": {
      "geo_grid": {
        "geohash_grid": {
          "field": "location",
          "precision": 5
        }
      }
    }
  }'
```

### 5.24 Text analysis and custom analyzers

```bash
# Test an analyzer
curl -X GET "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "analyzer": "english",
    "text": "The quick brown foxes are jumping over the lazy dogs"
  }'

# Test index-specific analyzer
curl -X GET "localhost:9200/articles/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "field": "body",
    "text": "Elasticsearch distributed search tutorial"
  }'

# Test ad-hoc analyzer
curl -X GET "localhost:9200/_analyze?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "tokenizer": "standard",
    "filter": ["lowercase", "stop", "snowball"],
    "text": "Running elasticsearch queries effectively"
  }'
```

### 5.25 Profile API (query debugging)

```bash
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "profile": true,
    "query": {
      "bool": {
        "must": [{ "match": { "body": "elasticsearch" } }],
        "filter": [{ "term": { "author": "alice" } }]
      }
    }
  }'
```

The `profile` object shows exact timing for each query clause and Lucene collector, helping you identify slow query components.

### 5.26 Explain API (scoring debug)

```bash
# Why did document 42 score the way it did?
curl -X GET "localhost:9200/articles/_explain/42?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "query": { "match": { "body": "elasticsearch tutorial" } } }'
```

### 5.27 Field capabilities

```bash
# What fields exist across multiple indices and what are their types?
curl -X GET "localhost:9200/articles,products/_field_caps?fields=*&pretty"

# Filter to specific fields
curl -X GET "localhost:9200/_field_caps?fields=author,views,price&pretty"
```

### 5.28 Index shrink, split, clone

```bash
# Shrink — reduce shard count (index must be read-only, all shards on one node)
curl -X POST "localhost:9200/articles/_shrink/articles-shrunk?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "index.number_of_shards": 1,
      "index.number_of_replicas": 1,
      "index.codec": "best_compression"
    }
  }'

# Split — increase shard count
curl -X POST "localhost:9200/articles/_split/articles-split?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "settings": { "index.number_of_shards": 6 } }'

# Clone — exact copy with same shard count
curl -X POST "localhost:9200/articles/_clone/articles-backup?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "settings": { "index.number_of_replicas": 0 } }'
```

### 5.29 Watcher (alerting)

```bash
# Create a watch that alerts on high error count
curl -X PUT "localhost:9200/_watcher/watch/high-errors?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "trigger": {
      "schedule": { "interval": "5m" }
    },
    "input": {
      "search": {
        "request": {
          "indices": ["logs-*"],
          "body": {
            "query": {
              "bool": {
                "filter": [
                  { "term": { "log.level": "error" } },
                  { "range": { "@timestamp": { "gte": "now-5m" } } }
                ]
              }
            }
          }
        }
      }
    },
    "condition": {
      "compare": {
        "ctx.payload.hits.total.value": { "gt": 100 }
      }
    },
    "actions": {
      "send-email": {
        "email": {
          "to": ["[email removed]"],
          "subject": "High error rate detected",
          "body": "{{ctx.payload.hits.total.value}} errors in the last 5 minutes"
        }
      }
    }
  }'

# Execute a watch manually
curl -X POST "localhost:9200/_watcher/watch/high-errors/_execute?pretty"

# Get watch status
curl "localhost:9200/_watcher/watch/high-errors?pretty"
```

### 5.30 Graph API

```bash
# Find connections between tags co-occurring in articles
curl -X GET "localhost:9200/articles/_graph/explore?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "query": { "match": { "body": "elasticsearch" } },
    "vertices": [
      { "field": "tags", "size": 10 },
      { "field": "author", "size": 5 }
    ],
    "connections": {
      "query": { "match_all": {} }
    }
  }'
```

---

## 6. Troubleshooting

> [!NOTE]
> For quick cluster diagnostics, always start with `/_cluster/health?pretty`, `/_cat/nodes?v`, and `/_cat/shards?v`. These three commands reveal the majority of operational issues.

### 6.1 Cluster status red

**Symptom:** `_cluster/health` returns `"status": "red"`.

Red means at least one primary shard is unassigned — writes to that shard are failing.

```bash
# Identify the unassigned shards
curl "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason,unassigned.details" | grep UNASSIGNED

# Get detailed explanation
curl -X GET "localhost:9200/_cluster/allocation/explain?pretty" \
  -H "Content-Type: application/json" \
  -d '{ "index": "my-index", "shard": 0, "primary": true }'
```

Common causes:
- **NODE_LEFT** — the node that had the primary shard left the cluster. Check if it will come back; if not, ensure replicas are promoted: `POST /_cluster/reroute?retry_failed=true`.
- **ALLOCATION_FAILED** — disk too full, or index settings conflict. Check `/_cat/allocation?v` for disk usage.
- **NEW_INDEX_RESTORED** — fresh cluster, `index.number_of_replicas` too high for available nodes. Reduce replicas temporarily: `PUT /my-index/_settings {"index.number_of_replicas": 0}`.

### 6.2 Cluster status yellow

**Symptom:** `"status": "yellow"` — all primary shards assigned, but some replicas are not.

This is normal on single-node clusters (no other node to place replicas on). On multi-node clusters, check disk usage and shard count limits:

```bash
# Check disk usage per node
curl "localhost:9200/_cat/allocation?v"

# Check total shard count (default limit: 1000 shards per node)
curl "localhost:9200/_cluster/settings?pretty&include_defaults=true" | grep max_shards_per_node

# Temporarily increase limit (use with caution)
curl -X PUT "localhost:9200/_cluster/settings" \
  -H "Content-Type: application/json" \
  -d '{ "persistent": { "cluster.max_shards_per_node": 2000 } }'
```

### 6.3 High heap usage / GC pressure

```bash
# Check heap usage per node
curl "localhost:9200/_cat/nodes?v&h=name,heap.percent,heap.current,heap.max"
```

**Remedies:**
- Force merge old read-only indices: `POST /my-index/_forcemerge?max_num_segments=1`
- Reduce `indices.queries.cache.size` (default 10% of heap) if query cache is large
- Run fewer concurrent searches or add more nodes
- Ensure `bootstrap.memory_lock: true` to prevent heap swapping

### 6.4 Slow queries

```bash
# Enable slow log threshold
curl -X PUT "localhost:9200/articles/_settings" \
  -H "Content-Type: application/json" \
  -d '{
    "index.search.slowlog.threshold.query.warn": "10s",
    "index.search.slowlog.threshold.query.info": "5s",
    "index.search.slowlog.threshold.fetch.warn": "1s",
    "index.search.slowlog.level": "info"
  }'
```

Slow query causes and fixes:
- **Wildcard leading `*`** — forces a full term scan. Use `ngram` or `edge_ngram` analyzers instead.
- **Deep pagination** (`from: 10000`) — use `search_after` with a sort on a unique field.
- **Script-heavy aggregations** — consider precomputing via transforms.
- **Large `size`** — retrieve only the fields you need with `_source: ["field1","field2"]`.
- **Expensive `query_string` with many clauses** — rewrite to `bool` with `filter` clauses.

### 6.5 Disk space issues

```bash
# Check disk usage
curl "localhost:9200/_cat/allocation?v&h=node,disk.used,disk.avail,disk.percent"

# Which indices are largest?
curl "localhost:9200/_cat/indices?v&s=store.size:desc&h=index,store.size,pri,rep"
```

**Remedies:**
- Force merge to reduce segment overhead: `POST /old-index/_forcemerge?max_num_segments=1`
- Delete old snapshots
- Implement ILM delete phase
- Temporarily raise watermarks (emergency only): `PUT /_cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.flood_stage": "99%" } }`

> [!WARNING]
> Elasticsearch sets write blocks on indices when disk usage reaches the flood-stage watermark (default 95%). When this happens, `index.blocks.read_only_allow_delete` is set to `true`. You must free disk space AND clear the block: `PUT /my-index/_settings {"index.blocks.read_only_allow_delete": null}`.

### 6.6 Indexing rejected (queue full)

```bash
# Check thread pool queue depths
curl "localhost:9200/_cat/thread_pool?v&h=node_name,name,active,queue,rejected"
```

If `write` queue shows rejections: slow down the indexing client, increase `thread_pool.write.queue_size` in `elasticsearch.yml`, or add more nodes.

### 6.7 Mapping conflicts

**Symptom:** `mapper_parsing_exception` or `illegal_argument_exception` about field type conflict.

```bash
# Check current mapping
curl "localhost:9200/my-index/_mapping?pretty"

# If the field type is wrong, you must reindex:
# 1. Create new index with correct mapping
# 2. Reindex into it
# 3. Swap aliases
```

> [!WARNING]
> You cannot change the type of an existing mapped field in-place. The only solution is to create a new index with the correct mapping, reindex the data, and update any aliases pointing to the index.

### 6.8 Version conflicts (optimistic concurrency)

```bash
# Use retry_on_conflict for update
curl -X POST "localhost:9200/articles/_update/42?retry_on_conflict=3&pretty" \
  -H "Content-Type: application/json" \
  -d '{ "script": { "source": "ctx._source.views += 1" } }'

# Or use if_seq_no / if_primary_term for explicit CAS
curl -X PUT "localhost:9200/articles/_doc/42?if_seq_no=10&if_primary_term=1&pretty" \
  -H "Content-Type: application/json" \
  -d '{ "title": "Updated Title", "views": 1601 }'
```

### 6.9 Node not joining cluster

Checklist:
1. `cluster.name` must match across all nodes
2. `discovery.seed_hosts` must include at least one reachable master-eligible node
3. Firewall must allow port 9300 (transport) between nodes
4. Clock skew must be < 5 seconds between nodes (`timedatectl status`)
5. `cluster.initial_master_nodes` should only be set during first bootstrap

### 6.10 Certificate errors (TLS)

```bash
# Verify certificate
openssl s_client -connect localhost:9200 -CAfile config/certs/http_ca.crt

# Skip verification temporarily (development only — never production)
curl -k https://localhost:9200/

# Regenerate expired certificates
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --days 1095
```

### 6.11 Authentication failures

```bash
# Test credentials
curl -u elastic:mypassword https://localhost:9200/

# Reset elastic password
bin/elasticsearch-reset-password -u elastic

# Check who am I
curl -u elastic:mypassword "localhost:9200/_security/_authenticate?pretty"
```

### 6.12 Bulk indexing performance

```bash
# Check bulk rejection rate
curl "localhost:9200/_cat/thread_pool?v&h=node_name,name,active,queue,rejected" | grep write
```

Tuning checklist for high-throughput bulk indexing:
- Use the Bulk API (never index one document at a time)
- Batch size 5–15 MB per request
- Use multiple parallel clients (one per CPU core is a reasonable start)
- Set `"refresh_interval": "30s"` or `"-1"` during initial load, restore to `"1s"` after
- Reduce replica count to 0 during initial load, set to target after
- Disable `_source` compression if disk is not a constraint

### 6.13 `max_result_window` exceeded

**Symptom:** `Result window is too large, from + size must be less than or equal to: [10000]`

```bash
# Increase the limit (prefer search_after for deep pagination)
curl -X PUT "localhost:9200/my-index/_settings" \
  -H "Content-Type: application/json" \
  -d '{ "index.max_result_window": 50000 }'
```

### 6.14 Index write-blocked

**Symptom:** `cluster_block_exception` on write operations.

```bash
# Check for blocks
curl "localhost:9200/_cluster/state/blocks?pretty"

# Remove a read-only block (usually set by disk watermark)
curl -X PUT "localhost:9200/my-index/_settings" \
  -H "Content-Type: application/json" \
  -d '{ "index.blocks.read_only_allow_delete": null }'

# Remove all blocks on all indices
curl -X PUT "localhost:9200/_all/_settings" \
  -H "Content-Type: application/json" \
  -d '{ "index.blocks.read_only_allow_delete": null }'
```

### 6.15 Log locations

| Install type | Log location |
|---|---|
| tar/zip | `$ES_HOME/logs/{cluster.name}.log` |
| Debian/Ubuntu | `/var/log/elasticsearch/{cluster.name}.log` |
| RHEL/CentOS | `/var/log/elasticsearch/{cluster.name}.log` |
| Docker | `docker logs es01` |

```bash
# Follow logs in real time (package install)
tail -f /var/log/elasticsearch/my-cluster.log

# Docker
docker logs -f es01

# Get recent deprecation warnings (critical before major version upgrades)
tail -100 /var/log/elasticsearch/my-cluster_deprecation.log
```

> [!WARNING]
> Review deprecation logs thoroughly before any major version upgrade. Each deprecated feature your application relies on must be migrated before the next major version drops support for it.

### 6.16 Task management

```bash
# List running tasks
curl "localhost:9200/_tasks?detailed=true&pretty"

# List tasks for a specific action
curl "localhost:9200/_tasks?actions=indices:data/write/reindex&detailed=true&pretty"

# Cancel a running task
curl -X POST "localhost:9200/_tasks/node-id:task-number/_cancel"

# Get task result
curl "localhost:9200/_tasks/node-id:task-number?wait_for_completion=true&pretty"
```

### 6.17 Recovery monitoring

```bash
# Monitor shard recovery
curl "localhost:9200/_cat/recovery?v&active_only=true&h=index,shard,type,stage,source_node,target_node,files_percent,bytes_percent,time"

# Detailed recovery info
curl "localhost:9200/_recovery?pretty&active_only=true"
```

### 6.18 Hot threads analysis

```bash
# What is every thread in the cluster doing right now?
curl "localhost:9200/_nodes/hot_threads?threads=5&interval=1s"

# Specific node
curl "localhost:9200/_nodes/es01/hot_threads"
```

### 6.19 Searchable snapshots

Searchable snapshots allow querying data stored in a snapshot repository without fully restoring it:

```bash
# Mount a snapshot as a searchable index
curl -X POST "localhost:9200/_snapshot/my-backup/snapshot-2026-05-13/_mount?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "index": "articles",
    "renamed_index": "articles-archive",
    "storage": "shared_cache"
  }'

# Query just like a regular index
curl "localhost:9200/articles-archive/_count?pretty"
```

### 6.20 Common error codes

| HTTP Status | Error Type | Typical Cause |
|-------------|-----------|---------------|
| 400 | `parsing_exception` | Malformed JSON or invalid query DSL |
| 400 | `mapper_parsing_exception` | Document field type doesn't match mapping |
| 401 | `security_exception` | Bad credentials or missing `Authorization` header |
| 403 | `security_exception` | Authenticated but no permission for this resource/action |
| 404 | `index_not_found_exception` | Index doesn't exist |
| 404 | `resource_not_found_exception` | Document not found |
| 409 | `version_conflict_engine_exception` | Optimistic locking conflict |
| 429 | `es_rejected_execution_exception` | Thread pool queue full |
| 503 | `master_not_discovered_exception` | No elected master node |

### 6.21 Shard count best practices

> [!CAUTION]
> Avoid "over-sharding" — having too many small shards is a leading cause of Elasticsearch performance problems. Each shard has overhead in memory and CPU. Target 20–50 GB per shard. A 100 GB index with 3 primary shards is healthier than a 10 GB index with 30 primary shards.

```bash
# Check shard sizes
curl "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,store" | sort -k5 -h
```

### 6.22 Pending cluster tasks

```bash
curl "localhost:9200/_cluster/pending_tasks?pretty"
```

Many pending tasks (> 10) often indicate the master node is under stress. Check master node CPU/heap with `_cat/nodes`.

### 6.23 Dangling indices

```bash
# List dangling indices (shards found on disk but not in cluster state)
curl "localhost:9200/_dangling?pretty"

# Import a dangling index
curl -X POST "localhost:9200/_dangling/AE3A6uGgR7mGM..." \
  -H "Content-Type: application/json" \
  -d '{ "accept_data_loss": true }'
```

### 6.24 Migration assistance

```bash
# Check for deprecated features before upgrading
curl "localhost:9200/_migration/deprecations?pretty"

# Get feature upgrade assistance
curl "localhost:9200/_migration/system_features?pretty"
```

### 6.25 Performance profiling with the Search Profiler

```bash
# Profile aggregations
curl -X GET "localhost:9200/articles/_search?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "profile": true,
    "size": 0,
    "aggs": {
      "by_author": { "terms": { "field": "author" } }
    }
  }'
```

Look for high `"time_in_nanos"` values in the profile output — these point to expensive operations worth optimizing.

### 6.26 Watcher troubleshooting

```bash
# Check watcher status
curl "localhost:9200/_watcher/stats?pretty"

# Activate a watch
curl -X POST "localhost:9200/_watcher/watch/high-errors/_activate"

# Start/stop watcher
curl -X POST "localhost:9200/_watcher/_start"
curl -X POST "localhost:9200/_watcher/_stop"
```

---

## 7. FAQ

### General

**Q: What is the difference between an index and a shard in Elasticsearch?**

An **index** is a logical namespace — like a database table — that you interact with via the API. A **shard** is the physical unit of storage and search. Every index is divided into one or more primary shards, each of which is a self-contained Lucene index. Shards are distributed across nodes, enabling both parallelism (search queries fan out to all shards) and redundancy (replica shards).

**Q: How many shards should my index have?**

For most indices under 100 GB, start with 1–3 primary shards. Rule of thumb: target 20–50 GB per shard. If you are using data streams (logs/metrics), let ILM handle rollover based on size or time. Over-sharding (many small shards) hurts performance more than under-sharding.

**Q: What is the difference between near real-time (NRT) and real-time search?**

After indexing a document, it is not immediately searchable. By default, Elasticsearch refreshes the in-memory index every 1 second, making newly indexed documents searchable within about 1 second (NRT). Use `POST /index/_refresh` to make documents searchable immediately. The `GET /index/_doc/{id}` endpoint is real-time (reads from the translog) and does not require a refresh.

**Q: What is the difference between `filter` and `must` in a bool query?**

`must` clauses contribute to the relevance `_score`. `filter` clauses are binary — they include or exclude documents but do **not** affect scoring. Filters are also cached by default, making them significantly faster for repeated queries. Use `filter` for exact-match conditions (date ranges, terms, keywords) and `must` for full-text queries where scoring matters.

**Q: Does Elasticsearch support transactions?**

No. Elasticsearch does not support ACID transactions across multiple documents. Each document-level operation is atomic, but there is no multi-document transaction primitive. For consistency needs, use the `if_seq_no` / `if_primary_term` parameters for optimistic concurrency control on single documents.

**Q: What is the `_source` field and can I disable it?**

`_source` is the original JSON document as you sent it. It is stored separately from the indexed fields and is what gets returned in search results. You can disable it (`"_source": {"enabled": false}`) to save disk space, but then you cannot update or reindex documents and you get no source in search results. Disabling is only appropriate for specific write-heavy, read-rarely use cases.

**Q: What is a dynamic mapping and when should I use explicit mappings?**

Dynamic mapping lets Elasticsearch automatically detect and add new fields as they appear in documents. It is convenient for development but can cause problems in production: a new field with an unexpected type (e.g., a number that sometimes contains a string) will cause a mapping conflict. Use explicit mappings in production, and set `"dynamic": "strict"` to reject unknown fields.

```bash
curl -X PUT "localhost:9200/my-index/_mapping" \
  -H "Content-Type: application/json" \
  -d '{ "dynamic": "strict" }'
```

**Q: What is the `keyword` vs `text` field type?**

- `text` — analyzed (tokenized, lowercased, stemmed). Use for full-text search where you want partial matches.
- `keyword` — stored as-is, not analyzed. Use for exact matching, sorting, aggregations, and filtering.

Best practice: use `"fields": {"keyword": {"type": "keyword"}}` on text fields when you need both full-text search and aggregations/sorting on the same field.

**Q: Can I run Elasticsearch on Windows?**

Yes. The tar/zip distribution and Docker both work on Windows. For production workloads, Linux is strongly preferred — Linux allows `mlockall` for memory locking, and Windows imposes additional JVM overhead. For development on Windows, Docker Desktop with WSL2 backend is the recommended approach.

**Q: What is the maximum document size?**

There is no hard-coded document size limit, but the default `http.max_content_length` is 100 MB for HTTP requests. Individual field values of type `text` are limited to 32,766 bytes by Lucene. Documents close to or exceeding 1 MB should be evaluated for splitting or using binary attachments via the `ingest-attachment` plugin.

### Indexing

**Q: What is the best way to do a bulk load of millions of documents?**

1. Disable replicas temporarily: `PUT /my-index/_settings {"index.number_of_replicas": 0}`
2. Set `refresh_interval: -1` during load
3. Use the Bulk API with 5–15 MB batches
4. Use multiple parallel threads (start with 2× CPU cores)
5. Restore replicas and refresh interval after load: `{"index.number_of_replicas": 1, "index.refresh_interval": "1s"}`
6. Call `POST /my-index/_forcemerge?max_num_segments=5` after load

**Q: How do I prevent duplicate documents when indexing?**

Provide explicit `_id` values and use `PUT /index/_doc/{id}` or `"op_type": "create"` to prevent overwrites. For content-based deduplication, compute a hash of the content and use it as the document ID. Elasticsearch has no built-in upsert-dedup; that logic lives in your application.

**Q: What happens to in-flight writes if a node crashes?**

Documents are written to the primary shard's translog (write-ahead log) before being acknowledged. If the node crashes before fsync, the translog replays on restart. As long as at least one replica is available and `wait_for_active_shards` is configured appropriately, data loss is bounded to the current translog segment.

**Q: Can I change an existing field's mapping type?**

No. Once a field is mapped, its type is immutable. To change a field type, you must: create a new index with the correct mapping, reindex the data using `_reindex`, and swap aliases.

**Q: What is `wait_for_active_shards`?**

This parameter on write operations controls how many shard copies (primary + replicas) must confirm the write before Elasticsearch acknowledges it. The default is `1` (only the primary). Setting it to `all` provides stronger durability guarantees but increases write latency.

### Searching

**Q: My search returns documents that shouldn't match — why?**

The most common cause is using `must` when you should use `filter`. The `match` query uses full-text analysis (tokenization, stemming, lowercase), so a search for "run" may match "running". Use `term` queries on `keyword` fields for exact matching. Check what the analyzer produces with `GET /index/_analyze`.

**Q: How do I search across multiple indices?**

Specify a comma-separated list, a wildcard, or `_all`:

```bash
curl "localhost:9200/articles,products/_search?pretty" ...
curl "localhost:9200/logs-*/_search?pretty" ...
curl "localhost:9200/_all/_search?pretty" ...
```

**Q: What is relevance scoring and how is `_score` calculated?**

By default, Elasticsearch uses BM25 (Best Match 25), which considers term frequency (how often the term appears in the document), inverse document frequency (how rare the term is across all documents), and field length normalization. Use the Explain API (`_explain`) to see the exact score breakdown for a specific document.

**Q: How do I do an exact case-sensitive search?**

Index the field as `keyword` (not `text`) and use a `term` query. Text fields are analyzed (lowercased by default), making case-sensitive search impossible without a dedicated `keyword` sub-field.

**Q: What is the difference between `match_phrase` and `match`?**

`match` splits the query into tokens and searches for any token (or all tokens with `"operator": "and"`). `match_phrase` requires all tokens to appear **in order** with no gap between them (configurable with `slop`).

**Q: Can I do aggregations on analyzed text fields?**

No — analyzed `text` fields are not usable for aggregations, sorting, or scripting by default. Use `.keyword` sub-fields or `keyword` type fields instead. Aggregating on text requires the `fielddata` parameter enabled, which is memory-intensive and not recommended.

**Q: What is a nested field and when do I need it?**

When you index an array of objects, Elasticsearch flattens them, losing the relationship between fields within each object. If you need to query "products where price > 100 AND brand = 'Acme' in the same object", you need `nested` type mapping. Nested documents are stored as hidden documents and have higher indexing/query cost.

```json
{
  "mappings": {
    "properties": {
      "variants": {
        "type": "nested",
        "properties": {
          "color": { "type": "keyword" },
          "price": { "type": "float" }
        }
      }
    }
  }
}
```

**Q: How do I implement autocomplete?**

Three approaches, in order of flexibility:

1. `match_phrase_prefix` — easiest, works on analyzed fields, no special mapping needed.
2. `edge_ngram` token filter — indexes prefixes at index time; faster queries, more disk.
3. `completion` suggester — purpose-built for autocomplete, fastest, requires dedicated field type.

```bash
# Edge ngram approach — analyzer definition
curl -X PUT "localhost:9200/my-index?pretty" \
  -H "Content-Type: application/json" \
  -d '{
    "settings": {
      "analysis": {
        "filter": {
          "autocomplete_filter": {
            "type": "edge_ngram",
            "min_gram": 2,
            "max_gram": 20
          }
        },
        "analyzer": {
          "autocomplete": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase", "autocomplete_filter"]
          }
        }
      }
    }
  }'
```

**Q: How do I do a "not" query?**

Use `must_not` in a `bool` query:

```json
{
  "query": {
    "bool": {
      "must": [{ "match": { "body": "elasticsearch" } }],
      "must_not": [{ "term": { "tags": "deprecated" } }]
    }
  }
}
```

**Q: What is the difference between `_search` and `_count`?**

`_count` returns only the count of matching documents — it skips fetching source and highlighting, making it significantly cheaper than `_search?size=0`. Use `_count` when you only need to know how many documents match.

**Q: How does ES|QL differ from the Search DSL?**

ES|QL uses a pipe-based syntax (`FROM index | WHERE condition | STATS ... BY field`) optimized for tabular result sets and operational analytics. The Search DSL (JSON query objects) is the primary API for full-text search, scoring, and aggregations on analyzed fields. ES|QL is generally faster for structured filters and group-by operations; the Search DSL is richer for relevance-ranked full-text search.

**Q: When should I use EQL vs ES|QL vs the Search DSL?**

| Use case | Best choice |
|----------|------------|
| Full-text search with ranking | Search DSL (`match`, `bool`) |
| Time-series log analytics, tabular output | ES|QL |
| Threat hunting, event sequences, SIEM | EQL |
| SQL-familiar users doing ad hoc analysis | SQL API |
| Vector/semantic similarity | Search DSL with `knn` |

### Performance

**Q: How much RAM should I give Elasticsearch?**

Rule: half of available RAM to the JVM heap (up to 31 GB), the other half for the OS page cache. On a 64 GB server: `-Xms31g -Xmx31g`, leaving 33 GB for the OS.

**Q: Why is Elasticsearch using so much disk?**

- Lucene segments: each refresh creates new segment files; segments are merged over time
- `_source` storage doubles your data size (raw JSON stored alongside indexed fields)
- Replicas multiply disk usage by (1 + `number_of_replicas`)
- Force merge old indices to reduce segment overhead: `POST /my-index/_forcemerge?max_num_segments=1`

**Q: How do I speed up aggregations on large datasets?**

1. Use `filter` to narrow the document set before aggregating
2. Use transforms to precompute summary statistics
3. Use `doc_values` (enabled by default on `keyword` and numeric types) — avoid `fielddata`
4. Consider using `sampler` or `diversified_sampler` aggregations on very large datasets
5. Use `execution_hint: map` for terms aggregations on low-cardinality fields

**Q: What is the refresh interval and how does it affect performance?**

The refresh interval (default: `1s`) controls how often Elasticsearch creates a new Lucene segment from the in-memory buffer, making documents searchable. Each refresh is I/O-intensive. For write-heavy workloads, increase to `30s` or use `refresh_interval: -1` (no automatic refresh). For real-time search requirements, keep at `1s` or call `_refresh` after each batch.

**Q: Should I use `filter` or `query` context in bool queries?**

Use `filter` context for all exact-match conditions where scoring is irrelevant (date ranges, terms, keywords, exists). Filters are cached in the filter cache and reused across queries. Use `must`/`should` (query context) only for full-text fields where relevance scoring matters.

### Operations

**Q: How do I perform a rolling upgrade?**

1. Disable shard allocation: `PUT /_cluster/settings {"persistent": {"cluster.routing.allocation.enable": "primaries"}}`
2. Stop node, upgrade, start node
3. Wait for node to rejoin and shards to be allocated: `GET /_cat/health?v`
4. Re-enable allocation: `PUT /_cluster/settings {"persistent": {"cluster.routing.allocation.enable": null}}`
5. Repeat for each node

**Q: Can I downgrade Elasticsearch after upgrading?**

No. Elasticsearch data directories are not backward-compatible. Once a node has started with a newer version, that node's data cannot be read by an older version. Always take a snapshot before upgrading.

**Q: How do I monitor Elasticsearch in production?**

Key metrics to monitor:
- Cluster health (green/yellow/red)
- JVM heap usage (alert at > 75%)
- GC duration (alert if GC takes > 500 ms frequently)
- Search/index latency (p50, p99)
- Rejected thread pool tasks (write, search queues)
- Disk usage (alert at > 80%)
- Unassigned shards

Use Kibana + Elastic APM for native monitoring, or export via the Metrics/Stats APIs to Prometheus/Grafana.

**Q: What is the difference between a component template and an index template?**

An **index template** is the top-level object that matches index patterns and controls settings/mappings. A **component template** is a reusable building block that index templates `composed_of`. Component templates allow sharing mapping or settings definitions across many index templates without duplication.

**Q: How do I safely delete an index?**

1. Set `action.destructive_requires_name: true` in `elasticsearch.yml` (prevents `DELETE /_all`)
2. Take a snapshot first
3. Verify the index name exactly before deleting
4. `DELETE /exact-index-name`

**Q: What is the difference between `cluster.initial_master_nodes` and `discovery.seed_hosts`?**

`discovery.seed_hosts` is a list of nodes used to seed the discovery process on every startup — it helps nodes find each other. `cluster.initial_master_nodes` is a bootstrap-only setting used on the very first cluster formation to prevent split-brain during initial startup. Once the cluster is formed and the cluster state is persisted, `cluster.initial_master_nodes` is no longer used and should ideally be removed.

**Q: How do I add a new node to an existing cluster?**

1. Install Elasticsearch on the new node
2. Set `cluster.name` to match the existing cluster
3. Set `discovery.seed_hosts` to point to existing master-eligible nodes
4. Do **not** set `cluster.initial_master_nodes` (that is for new clusters only)
5. Start the new node — it will automatically join and receive shard allocations

**Q: Can Elasticsearch be used without Kibana?**

Absolutely. Kibana is a UI companion but Elasticsearch is fully functional via its REST API. Many deployments use Elasticsearch with custom application frontends, Grafana, Superset, or direct API access.

**Q: What is the difference between `GET` and `POST` for searches?**

Both work for `_search`. `GET` with a body is technically non-standard HTTP but is widely supported. `POST` is unambiguous. Elasticsearch accepts both. For caching purposes, some HTTP proxies and load balancers may not cache `POST` requests — use `GET` when query caching at the HTTP layer is desirable.

**Q: How do I use Elasticsearch with Python?**

The official client is `elasticsearch-py`:

```bash
pip install elasticsearch
```

```python
from elasticsearch import Elasticsearch

es = Elasticsearch("http://localhost:9200")

# Index a document
es.index(index="articles", id=1, document={
    "title": "Hello Elasticsearch",
    "body": "Getting started with the Python client"
})

# Search
response = es.search(index="articles", body={
    "query": {"match": {"body": "Python client"}}
})
for hit in response["hits"]["hits"]:
    print(hit["_source"]["title"], hit["_score"])
```

**Q: How do I use Elasticsearch with JavaScript/Node.js?**

```bash
npm install @elastic/elasticsearch
```

```javascript
const { Client } = require('@elastic/elasticsearch')
const client = new Client({ node: 'http://localhost:9200' })

// Index
await client.index({
  index: 'articles',
  id: '1',
  document: { title: 'Hello', body: 'Getting started' }
})

// Search
const result = await client.search({
  index: 'articles',
  query: { match: { body: 'getting started' } }
})
console.log(result.hits.hits)
```

**Q: What are data tiers and how do they affect storage?**

Elasticsearch supports tiered storage with node roles:
- **Hot tier** (`data_hot`) — fast NVMe SSD, receives new writes and handles active searches
- **Warm tier** (`data_warm`) — cheaper SSD or HDD, for recent but less-queried data
- **Cold tier** (`data_cold`) — cold storage (HDD), low query frequency, reduced replicas
- **Frozen tier** (`data_frozen`) — minimal local cache, data served from snapshot repository

ILM moves indices down tiers automatically based on age or size. Configure node tiers with the `node.roles` setting in `elasticsearch.yml`.

**Q: How do I integrate Elasticsearch with generative AI / RAG pipelines?**

The standard pattern for Retrieval-Augmented Generation (RAG):

1. Use an inference endpoint to generate embeddings from your documents at ingest time (via an ingest pipeline with a `sparse_embedding` or `text_embedding` inference processor)
2. Store the resulting vectors in a `dense_vector` field
3. At query time, embed the user's question with the same model
4. Use `knn` search to retrieve semantically similar documents
5. Optionally combine with a keyword `bool` query (hybrid search)
6. Pass the retrieved documents as context to your LLM

Elasticsearch's native inference API (section 5.16) supports OpenAI, Anthropic, Cohere, Google, AWS Bedrock, Azure, and Mistral directly — no separate embedding service is required.

---

_Generated May 2026 from elastic/elasticsearch at 7ccc603. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._