# Milvus Admin Guide

This guide is the operational and architectural reference for **Milvus**, the open-source
distributed vector database. It targets administrators, SREs, platform engineers, and
architects who deploy, configure, secure, monitor, and extend Milvus in production. It assumes
familiarity with the user-facing concepts (collections, vectors, indexes, search) covered in
the companion **User Guide**.

Milvus is written in **Go** (the distributed control and data planes) and **C++** (the
`internal/core` vector search and storage engine, "Segcore" and the Knowhere index library).
It is designed to be **cloud-native and K8s-native**, separating compute from storage so that
read and write capacity scale independently. Source references in this guide point at the
Milvus codebase at commit `d9bbbc1`.

---

## 1. Executive Summary

| Aspect | Summary |
|---|---|
| **What it is** | A distributed vector database for similarity search over billions of embeddings |
| **Languages** | Go (control/data plane), C++ (Segcore search engine), Python (test client) |
| **Architecture** | Disaggregated: a coordinator, stateless workers, a streaming/WAL layer, and external storage |
| **External dependencies** | `etcd` (metadata + service discovery), object storage (MinIO/S3/GCS/Azure), a message queue / WAL (Woodpecker, Pulsar, Kafka, or RocksMQ) |
| **Deployment modes** | Milvus Lite (embedded), Standalone (one container), Distributed (Kubernetes) |
| **Client interfaces** | gRPC (`19530`), REST v2 (`/v2/vectordb/*`), SDKs (Python/Go/Java/Node) |
| **Security** | User auth, RBAC (Casbin), TLS (external + internal), quotas & rate limits |
| **Observability** | Prometheus metrics + health on `9091`, OpenTelemetry tracing, built-in Web UI |
| **License** | Apache 2.0, under the LF AI & Data Foundation |

The defining design choice is **storage/compute disaggregation built on a shared log**. All
durable state lives in external systems (etcd for metadata, object storage for data, a WAL for
the recent mutation stream). Milvus's own processes are stateless and restartable, which is
what enables horizontal scaling and fast failure recovery.

> [!IMPORTANT]
> As of the 2.6 line (this codebase, image `milvusdb/milvus:v3.0-beta`), the **streaming
> service is enabled by default** and the three legacy coordinators (root/data/query) are
> unified into a single **MixCoord** process. Plan deployments around MixCoord + workers +
> StreamingNode, not the older per-coordinator model.

---

## 2. Architecture

### 2.1 Component model

A Milvus deployment is composed of these roles, each started by the `milvus` binary via
`milvus run <role>` (`cmd/roles/roles.go`, `cmd/components/*`):

```
                         ┌──────────────────────────────┐
   SDK / REST clients ──▶│            Proxy             │  stateless gateway (gRPC+REST :19530)
                         └──────────────┬───────────────┘
                                        │
              ┌─────────────────────────┼─────────────────────────┐
              ▼                         ▼                         ▼
      ┌───────────────┐        ┌────────────────┐        ┌────────────────┐
      │   MixCoord    │        │  QueryNode(s)  │        │  DataNode(s)   │
      │ root+data+    │        │  load segments │        │  consume WAL,  │
      │ query coord   │        │  run searches  │        │  flush, compact│
      └───────┬───────┘        └───────┬────────┘        └───────┬────────┘
              │                        │                         │
              │                ┌───────┴─────────┐               │
              │                │  StreamingNode  │  WAL append/scan, time-tick
              │                └───────┬─────────┘               │
   ┌──────────┴───────────┬───────────┴─────────────┬───────────┴────────┐
   ▼                      ▼                          ▼                    ▼
 etcd                 Message queue / WAL        Object storage      (optional GPU)
 metadata +           Woodpecker | Pulsar |      MinIO | S3 | GCS |
 service discovery    Kafka | RocksMQ            Azure Blob
```

| Role | Responsibility | Stateful? |
|---|---|---|
| **Proxy** | Client gateway: validates requests, enforces auth/RBAC and quotas, routes to coordinators and nodes, reduces results | No |
| **MixCoord** | Unified coordinator — DDL/DCL (collections, partitions, schema), data lifecycle (segments, compaction, indexing, GC), query topology and load balancing | No (state in etcd) |
| **QueryNode** | Loads sealed/growing segments into memory, executes vector + scalar search and query | No (data in object storage) |
| **DataNode** | Consumes the write stream, buffers and flushes segments to object storage, runs compaction and index building | No |
| **StreamingNode** | Owns the write-ahead log (WAL): appends mutations, serves scans, drives time-tick and recovery | No (log is durable) |
| **CDC** | Optional change-data-capture role for cross-cluster replication | No |

> [!NOTE]
> MixCoord internally still contains the **RootCoord** (DDL/DCL), **DataCoord** (data/segment
> management), and **QueryCoord** (query topology) subsystems — their configuration sections
> (`rootCoord.*`, `dataCoord.*`, `queryCoord.*`) remain separate even though they run in one
> process. In a fully disaggregated cluster you may still run them as separate processes via
> the `mixture` server type with role flags.

### 2.2 The `milvus run` role matrix

`cmd/milvus/util.go:GetMilvusRoles` maps a server type to the set of enabled components:

| Server type | Components enabled |
|---|---|
| `standalone` | MixCoord + Proxy + QueryNode + DataNode + StreamingNode (single process, `Local=true`, may embed etcd/RocksMQ) |
| `mixcoord` | MixCoord only |
| `mixture` | Any combination via flags: `-rootcoord -querycoord -datacoord -querynode -datanode -proxy -streamingnode` |
| `proxy` | Proxy only |
| `querynode` | QueryNode only |
| `datanode` | DataNode only |
| `streamingnode` | StreamingNode (also enables an embedded QueryNode label) |
| `cdc` | CDC only |
| `embedded` | Like standalone but for Milvus Lite (embedded, config from `embedded-milvus.yaml`) |

Roles are also selectable by environment variable (`ENABLE_PROXY`, `ENABLE_QUERY_NODE`,
`ENABLE_DATA_NODE`, `ENABLE_STREAMING_NODE`, `ENABLE_ROOT_COORD`, `ENABLE_QUERY_COORD`,
`ENABLE_DATA_COORD`, `ENABLE_CDC`).

### 2.3 The storage layers

Milvus separates three kinds of state into three external systems:

1. **Metadata & coordination → etcd.** Collection schemas, segment metadata, index metadata,
   node sessions, and service discovery live in etcd under `etcd.rootPath` (default `by-dev`).
   TiKV is an alternative metadata store for very large metadata (`metastore.type: tikv`), but
   etcd is still required for service discovery.
2. **Durable data → object storage.** Sealed segment binlogs, delta logs, statistics, and
   index files are written to MinIO/S3/GCS/Azure under `minio.*`. This is the system of record
   for data.
3. **Recent mutations → WAL / message queue.** New inserts and deletes flow through a
   write-ahead log before they are flushed to object storage. The WAL implementation is
   pluggable: **Woodpecker** (native, recommended for new installs), **Pulsar**, **Kafka**, or
   **RocksMQ** (standalone only).

> [!TIP]
> For new deployments the project recommends **Woodpecker** as the WAL/MQ for better
> performance, simpler operations, and lower cost. RocksMQ is only valid in standalone mode;
> cluster mode defaults to Pulsar when `mq.type` is left at `default`.

### 2.4 Data lifecycle and segments

Data moves through **growing → sealed → indexed** segments:

- Writes land in **growing segments** (in memory, also mmappable) via the WAL.
- DataCoord seals a segment when it reaches size/age thresholds (`dataCoord.segment.maxSize`,
  `sealProportion`, `maxLife`, `maxIdleTime`, or the `sealPolicy.channel.*` watermarks).
- Sealed segments are flushed to object storage and **indexed** (vector and scalar indexes
  are built by DataNode/index tasks).
- **Compaction** merges small segments, applies deletes, and runs clustering compaction
  (`dataCoord.compaction.*`). **Garbage collection** removes orphaned files
  (`dataCoord.gc.*`).
- QueryNodes **load** sealed+indexed segments to serve search; a delegator forwards streaming
  deletes and growing data.

### 2.5 The C++ core (Segcore + Knowhere)

The `internal/core/src` tree is the C++ engine, reached from Go via cgo wrappers
(`internal/util/segcore`, `indexcgowrapper`, `analyzecgowrapper`):

- **Segcore** — in-segment storage, chunked columnar layout, expression evaluation, and the
  search/query operators.
- **Knowhere** — the vector index library (`knowhere.*` config) implementing FLAT, IVF*,
  HNSW, SCANN, DISKANN, and GPU (CAGRA) indexes with CPU SIMD acceleration
  (`common.simdType`: auto/avx512/avx2/avx/sse4_2).
- **Tiered storage** — `queryNode.segcore.tieredStorage.*` controls memory/disk cache warmup
  and eviction watermarks for hot/cold data.

---

## 3. Requirements

### 3.1 Build requirements

From `README.md` / `DEVELOPMENT.md`, building Milvus from source needs (Linux/Ubuntu 20.04+):

```text
Go:     >= 1.21   (go.mod pins go 1.25.x)
CMake:  >= 3.26.4 && < 4
GCC:    >= 11     (Linux)  /  llvm >= 15 (macOS)
Python: > 3.8 and <= 3.11
```

```bash
git clone https://github.com/milvus-io/milvus.git
cd milvus
./scripts/install_deps.sh
make
```

### 3.2 Runtime dependencies

| Dependency | Version (dev compose) | Role |
|---|---|---|
| etcd | v3.5.x | Metadata + service discovery |
| MinIO | RELEASE.2024-05-28 | Object storage (S3-compatible) |
| Pulsar | 2.8.x | Optional WAL/MQ (cluster default) |
| Kafka | — | Optional WAL/MQ |
| Woodpecker | built-in | Native WAL (recommended) |

Object storage may instead be AWS S3, Google Cloud Storage (`gcpnative`), Azure Blob, Aliyun
OSS, Tencent COS, or Huawei OBS (see `minio.cloudProvider` and `pkg/objectstorage/*`).

### 3.3 Sizing guidance

- **Memory** is the primary constraint for QueryNodes — loaded segments and indexes are held
  in RAM (or mmap/tiered cache). Size query nodes to your loaded dataset plus index overhead.
- **DataNodes** are CPU/memory bound during flush, compaction, and index building.
- **Object storage** must hold all segment, index, delta, and stats files; plan for
  several times the raw vector size after indexing and replication.

---

## 4. Configuration

Milvus is configured by `configs/milvus.yaml` (1,500+ keys). Configuration is layered:
YAML file → environment variables (a subset, e.g. `ETCD_ENDPOINTS`, `MINIO_ADDRESS`,
`MINIO_ACCESS_KEY_ID`) → runtime overrides via etcd (some keys are hot-reloadable; the
`paramtable` watches etcd and applies changes live). Some keys are **immutable** and persisted
at first startup (e.g. the selected WAL name).

> [!NOTE]
> Many keys preferentially read an environment variable when present — `etcd.endpoints` reads
> `ETCD_ENDPOINTS`, `minio.address` reads `MINIO_ADDRESS`, and the MinIO keys read
> `MINIO_ACCESS_KEY_ID` / `MINIO_SECRET_ACCESS_KEY`. This is how the Docker Compose files
> inject infrastructure addresses.

### 4.1 Metadata store — `etcd` / `metastore` / `tikv`

| Key | Default | Purpose |
|---|---|---|
| `etcd.endpoints` | `localhost:2379` | etcd endpoints (env `ETCD_ENDPOINTS`) |
| `etcd.rootPath` | `by-dev` | Root prefix for all Milvus keys — **change per cluster sharing one etcd** |
| `etcd.metaSubPath` / `kvSubPath` | `meta` / `kv` | Sub-prefixes for metadata and KV |
| `etcd.ssl.enabled` | `false` | TLS to etcd (`tlsCert`/`tlsKey`/`tlsCACert`, `tlsMinVersion` 1.3) |
| `etcd.use.embed` | `false` | Embed an in-process etcd (standalone/testing only) |
| `etcd.auth.enabled` | `true` | etcd auth (`userName`/`password`) |
| `etcd.requestTimeout` | `10000` ms | etcd op timeout |
| `metastore.type` | `etcd` | `etcd` or `tikv` |
| `metastore.maxEtcdTxnNum` | `64` | Max ops per etcd transaction |
| `tikv.endpoints` | `127.0.0.1:2389` | TiKV PD endpoint (when `metastore.type: tikv`) |

> [!CAUTION]
> Changing `etcd.rootPath`, `metaSubPath`, `kvSubPath`, or the `msgChannel` prefixes **after**
> a cluster has data will make existing data unreadable. Set them once, before first start.

### 4.2 Object storage — `minio` / `localStorage`

| Key | Default | Purpose |
|---|---|---|
| `minio.address` / `port` | `localhost` / `9000` | Storage endpoint (env `MINIO_ADDRESS`) |
| `minio.accessKeyID` / `secretAccessKey` | `minioadmin` | Credentials (env `MINIO_ACCESS_KEY_ID` / `MINIO_SECRET_ACCESS_KEY`) |
| `minio.useSSL` | `false` | TLS to storage (`ssl.tlsCACert`, `ssl.tlsMinVersion`) |
| `minio.bucketName` | `a-bucket` | Bucket (created if absent) |
| `minio.rootPath` | `files` | Key prefix in the bucket |
| `minio.useIAM` | `false` | Use IAM role instead of static keys |
| `minio.cloudProvider` | `aws` | `aws`, `gcp`, `gcpnative`, `aliyun`, `tencent`, `huawei`, `azure` |
| `minio.region` | — | Storage region |
| `localStorage.path` | `/var/lib/milvus/data/` | Local cache for query/search to avoid re-fetching from object storage |
| `common.storageType` | `remote` | `remote`, `local`, or `opendal` |

> [!TIP]
> In production prefer `useIAM: true` with instance/workload identity over static
> `accessKeyID`/`secretAccessKey`. Static keys in `milvus.yaml` are a credential-leak risk.

### 4.3 Message queue / WAL — `mq` / `woodpecker` / `pulsar` / `kafka` / `rocksmq`

| Key | Default | Purpose |
|---|---|---|
| `mq.type` | `default` | `default`, `pulsar`, `kafka`, `rocksmq`, `woodpecker` |
| `mq.enablePursuitMode` | `true` | Catch-up mode for lagging consumers |
| `woodpecker.storage.type` | `minio` | Native WAL storage backend (`minio` or `local`) |
| `woodpecker.client.segmentRollingPolicy.maxSize` | `256M` | WAL segment roll size |
| `pulsar.address` / `port` | `localhost` / `6650` | Pulsar broker (env `PULSAR_ADDRESS`) |
| `pulsar.maxMessageSize` | `2097152` | Max Pulsar message (2 MB) |
| `pulsar.tenant` / `namespace` | `public` / `default` | Pulsar multi-tenancy |
| `kafka.brokerList` | (commented) | Kafka brokers + SASL/SSL block |
| `rocksmq.path` | `/var/lib/milvus/rdb_data` | RocksMQ data dir (standalone only) |
| `rocksmq.retentionTimeInMinutes` | `4320` | RocksMQ retention (3 days) |

Priority when `mq.type: default`: standalone → `rocksmq > Pulsar > Kafka > Woodpecker`;
cluster → `Pulsar > Kafka > Woodpecker` (RocksMQ unsupported in cluster).

### 4.4 Coordinator and node configuration

Each role has its own section. Key knobs (selected — see `configs/milvus.yaml` for the full
set):

**`rootCoord`** (DDL/DCL): `maxPartitionNum` (1024), `maxDatabaseNum` (64), `dmlChannelNum`
(16), `minSegmentSizeToEnableIndex` (1024), `enableActiveStandby` (false), `port` 22125.

**`proxy`** (gateway): `maxNameLength` (255), `maxFieldNum` (64), `maxVectorFieldNum` (10),
`maxShardNum` (16), `maxDimension` (32768), `maxTaskNum` (1024), `maxConnectionNum` (10000),
`http.enabled` (true), `http.enableWebUI` (true), `accessLog.*`, `port` 19530,
`internalPort` 19529, `grpc.serverMaxRecvSize` (64 MB), `grpc.serverMaxSendSize` (256 MB).

**`queryCoord`** (topology/balance): `autoBalance` (true), `balancer` (`ScoreBasedBalancer`),
`overloadedMemoryThresholdPercentage` (90), `balanceIntervalSeconds` (60),
`loadTimeoutSeconds` (600), `enableStoppingBalance` (true), `port` 19531.

**`queryNode`** (search execution):
- Segcore: `segcore.chunkRows` (128), `knowhereThreadPoolNumRatio` (4),
  `interimIndex.enableIndex` (true), `interimIndex.nlist`/`nprobe`.
- Memory/mmap: `mmap.vectorField` (true), `mmap.vectorIndex` (false), `cache.readAheadPolicy`
  (`willneed`), `loadMemoryUsageFactor` (1), `enableDisk` (false), `maxDiskUsagePercentage`
  (95).
- Tiered storage: `segcore.tieredStorage.warmup.*` (sync/async/disable),
  `memoryLowWatermarkRatio` (0.75) / `memoryHighWatermarkRatio` (0.8), `evictionEnabled`
  (false).
- Scheduler: `scheduler.maxReadConcurrentRatio`, `scheduler.scheduleReadPolicy.name` (`fifo`
  or `user-task-polling`). `port` 21123.

**`dataCoord`** (data lifecycle):
- Segments: `segment.maxSize` (1024 MB), `diskSegmentMaxSize` (2048 MB), `sealProportion`
  (0.12), `maxLife` (86400 s), `maxIdleTime` (600 s).
- Compaction: `enableCompaction` (true), `compaction.enableAutoCompaction` (true),
  `compaction.clustering.enable` (true), `compaction.taskPrioritizer` (`level`),
  L0/mix/single/clustering trigger intervals.
- GC: `enableGarbageCollection` (true), `gc.interval` (3600 s), `gc.missingTolerance`
  (86400 s), `gc.dropTolerance` (10800 s). `port` 13333.

**`dataNode`** (flush/compaction):
- Sync: `dataSync.flowGraph.maxQueueLength` (16), `memory.forceSyncEnable` (true),
  `memory.forceSyncWatermark` (0.5).
- Segment buffer: `segment.insertBufSize` (16 MB), `segment.deleteBufBytes` (16 MB),
  `segment.syncPeriod` (600 s).
- Slots: `slot.slotCap` (16). Storage format: `storage.format` (`parquet`),
  `storage.deltalog` (`json`). `gracefulStopTimeout` (1800 s). `port` 21124.

**`streamingNode`** / **`streaming`** (WAL service): `streamingNode.port` 22222;
`streaming.walBalancer.*` (balance policy `vchannelFair`), `streaming.flush.*`
(memory/growing-segment watermarks), `streaming.walRecovery.*`, `streaming.txn.*`,
`streaming.replication.*`.

### 4.5 Common, GPU, Knowhere

**`common`** holds cross-cutting settings: `common.security.*` (see Security),
`common.gracefulStopTimeout` (1800 s), `common.simdType` (`auto`), `common.storageType`
(`remote`), thread-pool coefficients (`threadCoreCoefficient.*`), DiskIndex tuning
(`common.DiskIndex.*`), bloom-filter settings, and JSON-shredding toggles.

**`gpu`** (GPU builds only): `gpu.initMemSize` (2048 MB), `gpu.maxMemSize` (4096 MB),
`gpu.overloadedMemoryThresholdPercentage` (95).

**`knowhere`**: per-index-type default parameters (`AISAQ`, `DISKANN` build/search params)
auto-populated when `knowhere.enable` is true.

### 4.6 Logging, gRPC, tracing

| Section | Key | Default |
|---|---|---|
| `log` | `level` | `info` (trace/debug/info/warn/error/panic/fatal) |
| `log` | `file.rootPath` | empty (stdout) |
| `log` | `format` | `text` (or `JSON`) |
| `grpc` | `client.dialTimeout` / `keepAliveTime` | `200` / `10000` ms |
| `grpc` | `gracefulStopTimeout` | `3` s |
| `trace` | `exporter` | `noop` (or `stdout`, `jaeger`, `otlp`) |
| `trace` | `sampleFraction` | `0` |
| `trace` | `otlp.endpoint` / `jaeger.url` | — |

Log level is **hot-reloadable** — the server watches `log.level` in etcd and applies changes
without restart (`cmd/roles/roles.go:setupLogger`). Tracer config is likewise re-applied on
change.

---

## 5. Security

> [!CAUTION]
> A default Milvus install is **insecure by design for ease of first use**: authorization is
> off (`common.security.authorizationEnabled: false`) and the `root` user has the well-known
> password `Milvus`. Before exposing Milvus to any shared or untrusted network, enable auth,
> change the root password, and enable TLS.

### 5.1 Authentication

- `common.security.authorizationEnabled` — set `true` to require credentials on every request.
- `common.security.defaultRootPassword` (`Milvus`) — change this; the root account is the
  bootstrap admin. (Max length 72 chars.)
- `common.security.superUsers` — users that bypass some checks (e.g. the old-password check on
  credential update).
- `common.security.rootShouldBindRole` — require root to bind a role when auth is on.
- `common.security.enablePublicPrivilege` (`true`) — whether a default public privilege exists.

Clients authenticate with username/password (sent base64-encoded in the `authorization`
metadata header) or an API key (`client/milvusclient/client.go:parseAuthentication`).

### 5.2 RBAC

Milvus implements role-based access control backed by **Casbin** (`github.com/casbin/casbin`).
Administrators manage users, roles, privileges, and privilege groups (SDK methods in
`client/milvusclient/rbac.go` and `rbac_v2.go`):

- **Users** — `CreateUser`, `UpdatePassword`, `DropUser`, `ListUsers`, `DescribeUser`.
- **Roles** — `CreateRole`, `DropRole`, `GrantRole`, `RevokeRole`, `DescribeRole`,
  `ListRoles`.
- **Privileges (v2)** — `GrantPrivilegeV2`, `RevokePrivilegeV2` grant a privilege on an object
  (collection/database/etc.) to a role.
- **Privilege groups** — `CreatePrivilegeGroup`, `AddPrivilegesToGroup`,
  `RemovePrivilegesFromGroup`, `ListPrivilegeGroups` bundle privileges for reuse.
- **Backup/restore** — `BackupRBAC` / `RestoreRBAC` export and import the entire RBAC model
  (users, roles, grants, privilege groups) — use these before upgrades or for DR.

`proxy.resolveAliasForPrivilege` (true) controls whether aliases resolve to real collection
names during privilege checks.

### 5.3 TLS

Two independent TLS scopes:

- **External TLS** (`tls.*`) secures client↔Proxy traffic: `serverPemPath`, `serverKeyPath`,
  `caPemPath`. Enable on the client with `EnableTLSAuth`.
- **Internal TLS** (`internaltls.*` + `common.security.internaltlsEnabled`) secures
  intra-cluster gRPC between components: `serverPemPath`, `serverKeyPath`, `caPemPath`, and an
  `sni` matching the certificate. `common.security.tlsMode` selects the TLS mode (0/1/2 for
  off/one-way/two-way).
- etcd, MinIO, and Kafka each have their own TLS blocks (`etcd.ssl`, `minio.ssl`, `kafka.ssl`).

> [!NOTE]
> A FIPS-validated build is available: the binary can enable OpenSSL FIPS / Go BoringCrypto
> (`cmd/milvus/boring_*.go`, `pkg/util/fips`). The startup banner reports
> `Milvus FIPS in Go: BoringCrypto <bool>`.

### 5.4 The `/expr` debug endpoint

`common.security.exprEnabled` (default `false`) gates the `/expr` HTTP endpoint used for live
expression debugging. When enabled it is restricted to the `root` user via HTTP Basic Auth on
Proxy nodes. **Leave it disabled in production.**

### 5.5 Quotas and rate limits

`quotaAndLimits.*` is both a stability mechanism and a multi-tenant fairness control. By
default Milvus enables **time-tick protection**, **memory protection**, and **disk-quota
protection**; you can additionally enable DML/DDL/DQL rate limits.

| Protection | Key | Default |
|---|---|---|
| Master switch | `quotaAndLimits.enabled` | `true` |
| Memory protection | `limitWriting.memProtection.enabled` | `true` (low 0.85 / high 0.95 water marks for data & query nodes) |
| Disk quota | `limitWriting.diskProtection.enabled` | `true` (`diskQuota` -1 = unlimited) |
| Time-tick protection | `limitWriting.ttProtection.enabled` | `false` (maxDelay 1200 s) |
| DDL throttle | `ddl.enabled` | `false` (`collectionRate`/`partitionRate` -1) |
| DML throttle | `dml.enabled` | `false` (`insertRate`/`deleteRate` per db/collection/partition) |
| DQL throttle | `dql.enabled` | `false` (`searchRate`/`queryRate`) |
| Flush throttle | `flushRate.enabled` | `true` (collection 0.1 qps) |
| Force-deny writes/reads | `limitWriting.forceDeny` / `limitReading.forceDeny` | `false` |
| Max result entries | `proxy.maxResultEntries` | `-1` (disabled) |

> [!WARNING]
> When memory or disk water marks are crossed, Milvus **throttles or rejects DML** to protect
> the cluster. "Quota exceeded" / write-denied errors usually mean a node is near its memory
> high-water mark or the disk quota is hit — scale out or raise the quota rather than retrying
> blindly.

---

## 6. Deployment

### 6.1 Standalone (Docker Compose)

Use `deployments/docker/standalone/docker-compose.yml` (see the User Guide for the full file).
It launches `milvus-etcd`, `milvus-minio`, and `milvus-standalone` (`milvus run standalone`),
publishing `19530` (API) and `9091` (health/metrics/Web UI). A GPU variant lives at
`deployments/docker/gpu/standalone/`.

```bash
sudo docker compose up -d
curl -f http://localhost:9091/healthz   # -> OK
```

### 6.2 Distributed cluster

For production scale, run roles as independent, separately scaled deployments on Kubernetes
(the official Milvus Helm chart / Milvus Operator; this repository provides reference Ansible
roles under `deployments/docker/cluster-distributed-deployment/`). Typical topology:

- **Multiple Proxies** behind a load balancer (stateless; scale for connection/throughput).
- **MixCoord** with `enableActiveStandby` for HA (one active, standbys ready).
- **QueryNodes** scaled for read/memory (each holds loaded segments).
- **DataNodes** scaled for write/compaction/index throughput.
- **StreamingNodes** for the WAL service.
- External **etcd** (3+ nodes), **object storage**, and **Pulsar/Kafka/Woodpecker**.

> [!TIP]
> Scale QueryNodes for read-heavy or large-resident-data workloads and DataNodes for
> write-heavy/ingest workloads — this independent scaling is the main reason to choose the
> cluster topology over standalone.

### 6.3 High availability

- Coordinators support **active-standby** (`*.enableActiveStandby`): standbys take over via
  etcd session/lease on failure.
- Workers are stateless; losing one triggers QueryCoord auto-balance to redistribute segments.
- The WAL plus object storage make data durable independent of any Milvus process; recovery
  replays from the WAL and reloads from object storage.
- **Replicas** (multiple loaded copies of segments across query nodes) increase read
  throughput and fault tolerance — manage with resource groups and `TransferReplica`.

### 6.4 Resource groups

Resource groups partition QueryNodes into pools so you can isolate workloads/tenants
(`client/milvusclient/resource_group.go`): `CreateResourceGroup`, `UpdateResourceGroup`,
`TransferReplica`, `DescribeReplica`. `quotaAndLimits.limits.maxResourceGroupNumOfQueryNode`
(1024) bounds the count.

---

## 7. Operations

### 7.1 Starting and stopping

```bash
milvus run standalone            # or: run mixcoord / proxy / querynode / datanode / streamingnode
milvus stop standalone           # graceful stop (sends SIGTERM via PID file)
```

Each role honors a `gracefulStopTimeout`; on timeout the process dumps goroutine/heap/block/
mutex pprof profiles to `ProfileCfg.PprofPath` and force-exits (`cmd/components/util.go`).

### 7.2 Health and readiness

- `GET /healthz` on `9091` — liveness.
- Component readiness is tracked internally; the Web UI and `GetComponentStates` expose
  per-role health (`Healthy`/`Abnormal`/`StandBy`).

### 7.3 Monitoring

- **Prometheus** metrics at `:9091/metrics` (Milvus registry) and `/metrics_default`
  (Go runtime). Build info, per-role metrics, meta/msgstream/storage metrics are registered at
  startup (`cmd/roles/roles.go`).
- **Grafana** dashboards are provided under `deployments/monitor/`.
- **OpenTelemetry tracing** via `trace.exporter` (`jaeger`/`otlp`); sampling via
  `trace.sampleFraction`.
- **Access logs** (`proxy.accessLog.*`) record every request with method, status, user,
  latency, and trace ID; optionally uploaded to object storage.
- **Slow queries** are flagged when they exceed `proxy.slowQuerySpanInSeconds` (1 s) and shown
  in the Web UI.

### 7.4 The Web UI

Enabled by `proxy.http.enableWebUI` (default true) on `:9091/webui` — inspect collections,
segment distribution, query-node balance, slow queries, and component health.

### 7.5 Consistency check (`mck`)

`milvus mck run` scans etcd and object storage for orphaned task/segment references and
reports invalid tasks, segments, and missing object-storage paths; `milvus mck cleanTrash`
removes the orphaned entries (with backup to an `mck-trash` prefix). Flags: `-etcdIp`,
`-etcdRootPath`, `-minioAddress`, `-minioUsername`, `-minioPassword`, `-minioUseSSL`,
`-minioBucketName`.

> [!WARNING]
> `mck` mutates metadata. Run it against a quiesced cluster and keep the etcd backup it
> creates. It is a recovery tool, not routine maintenance.

### 7.6 Backup and restore

- **RBAC**: `BackupRBAC` / `RestoreRBAC` (SDK) export/import the full access-control model.
- **Snapshots**: `CreateSnapshot` / `RestoreSnapshot` capture and restore point-in-time
  collection data; `dataCoord.snapshot.*` governs retention and compaction protection.
- **Data**: because data lives in object storage and metadata in etcd, a consistent backup
  pairs an etcd snapshot with the object-storage bucket. The community **Milvus Backup** tool
  automates this.

### 7.7 Upgrades and migration

`cmd/tools/migration` performs metadata migrations between Milvus versions
(`deployments/migrate-meta/`). DataCoord can rebuild segment indexes to a target index-engine
version (`dataCoord.autoUpgradeSegmentIndex`, `targetVecIndexVersion`,
`forceRebuildSegmentIndex`). Always back up etcd before a version upgrade and follow the
official rolling-upgrade procedure for clusters.

### 7.8 Change data capture (CDC) and replication

The `cdc` role and `streaming.replication.*` settings support replicating data between
clusters (`internal/cdc/*`). SDK surface: `GetReplicateInfo`, `CreateReplicateStream`,
`UpdateReplicateConfiguration`. Use for cross-region DR or migration.

---

## 8. Internals (Developer Reference)

### 8.1 Repository layout

| Path | Contents |
|---|---|
| `cmd/` | Entry points: `cmd/milvus` (CLI), `cmd/roles`, `cmd/components`, `cmd/tools/*` |
| `internal/proxy` | The gateway: request validation, RBAC, rate limiting, result reduction |
| `internal/rootcoord` | DDL/DCL coordinator subsystem |
| `internal/datacoord` | Segment/data lifecycle, compaction, GC, index scheduling |
| `internal/querycoordv2` | Query topology, load balancing, checkers/observers |
| `internal/querynodev2` | Search/query execution, segment loading, delegators |
| `internal/datanode` | WAL consumption, flush, compaction, indexing |
| `internal/streamingnode`, `internal/streamingcoord` | WAL service and coordination |
| `internal/distributed/*` | gRPC servers/clients per role |
| `internal/core/src` | C++ Segcore + Knowhere bindings |
| `internal/storage`, `internal/storagev2` | Binlog/segment serialization, object-storage chunk managers |
| `internal/metastore` | etcd/TiKV-backed metadata models and KV catalogs |
| `pkg/` | Shared libraries: `paramtable` (config), `proto` (gRPC defs), `streaming`, `mq`, `log`, `metrics`, `util/*` |
| `client/` | The official Go SDK (`client/v2`) |

### 8.2 Process orchestration

`cmd/roles/roles.go:Run` starts enabled components concurrently via `conc.Future`, waits for
all to be ready, registers them with the health subsystem and Prometheus, wires signal
handling for graceful shutdown, and tears down in order (coordinators → nodes → proxy →
shared etcd client). Config changes propagate through `paramtable` watchers on etcd.

### 8.3 API surface

The wire API is the gRPC `milvuspb.MilvusService` (defined in the external `milvus-proto`
repository) plus the REST v2 layer (`internal/distributed/proxy/httpserver`, paths
`/v2/vectordb/<group>/<action>`). The extraction TSVs only see test handlers; the real API is
the proto service and the SDK methods enumerated in the User Guide.

> [!NOTE]
> Because the protobuf service definitions live in the separate `milvus-io/milvus-proto`
> repository (vendored as `github.com/milvus-io/milvus-proto/go-api/v3`), the gRPC method list
> is not present in this repository's source. The authoritative client-visible surface is the
> Go SDK in `client/` (and the equivalent pymilvus methods).

### 8.4 Key dependencies

From `go.mod`: `go.etcd.io/etcd` (metadata), `minio-go` + cloud SDKs (storage),
`apache/pulsar-client-go` (MQ), `apache/arrow/go` (columnar), `antlr4-go` (search-expression
parser), `gin-gonic/gin` (REST), `casbin` (RBAC), `prometheus/client_golang` (metrics),
`go.opentelemetry.io/otel` (tracing), `grpc`, `tikv/client-go`, `cockroachdb/errors`.

### 8.5 Build and test

`make` builds the Go binaries and C++ core. `make verifiers` runs linters; `make unittest`
runs unit tests. The dev environment is reproduced by the root `docker-compose.yml` (builder
image plus etcd/pulsar/minio/azurite/jaeger). Client linting uses a strict golangci-lint
config (`client/.golangci.yml`) that, for example, forbids the stdlib `errors` package in
favor of `cockroachdb/errors`.

---

## 9. Code Quality and Conventions

- **Error handling** uses a centralized `merr` package (`pkg/util/merr`) producing a rich,
  catalogued error set (2,000+ messages) with codes — this is what surfaces to clients and
  feeds troubleshooting.
- **Config access** is centralized in `paramtable`; code reads typed accessors
  (`.GetAsBool()`, `.GetAsInt()`, `.GetAsDuration()`) rather than parsing YAML directly,
  enabling hot reload and type checking (`common.enableConfigParamTypeCheck`).
- **Concurrency** uses `conc.Future`/pools and structured goroutine management.
- **cgo boundaries** are isolated in `*cgowrapper` packages with explicit lifetime management
  for the C++ core.
- **Linters** are strict and enforced in CI (`make verifiers`, ruleguard custom rules).

---

## 10. Potential Improvements / Operational Watch-list

These are observations for operators and contributors, drawn from configuration and code:

- **Deprecated knobs remain in `milvus.yaml`** (e.g. `queryNode.cache.memoryLimit`,
  `queryNode.segcore.multipleChunkedEnable`, `dataCoord.compaction.maxParallelTaskNum`). Audit
  your config against the current defaults on upgrade rather than copying an old file forward.
- **Storage format is evolving** — `dataNode.storage.format` supports `parquet` (default) and
  `vortex`; `deltalog` supports `json`/`parquet`; Storage V2 (`common.storage.stv2.*`) and
  `useLoonFFI` are newer paths. Pin and test the format before large migrations.
- **WAL choice matters** — RocksMQ does not work in cluster mode; the project recommends
  Woodpecker for new installs. Migrating an existing cluster's WAL backend is non-trivial.
- **Default credentials and disabled auth** are the single biggest production risk; secure
  them before any network exposure.
- **Memory-bound query nodes** — without tiered storage/mmap tuning, large loaded collections
  can exhaust RAM and trigger write throttling cluster-wide. Tune
  `queryNode.segcore.tieredStorage.*` and `queryNode.mmap.*`, and use resource groups for
  isolation.

---

## 11. Quick Reference

### 11.1 Default ports

| Port | Component | Purpose |
|---|---|---|
| `19530` | Proxy | gRPC + REST client API |
| `19529` | Proxy | Internal port |
| `9091` | All | Health (`/healthz`), metrics (`/metrics`), Web UI (`/webui`) |
| `22125` | RootCoord | gRPC |
| `19531` | QueryCoord | gRPC |
| `13333` | DataCoord | gRPC |
| `21123` | QueryNode | gRPC |
| `21124` | DataNode | gRPC |
| `22222` | StreamingNode | gRPC |
| `2379` | etcd | Metadata (external dep) |
| `9000` / `9001` | MinIO | Storage / console (external dep) |

### 11.2 Essential commands

```bash
milvus run standalone                 # start all-in-one
milvus run mixcoord                    # start unified coordinator
milvus run proxy|querynode|datanode|streamingnode
milvus stop <server type>             # graceful stop
milvus mck run -etcdIp <ip> ...       # consistency check
milvus mck cleanTrash                  # clean orphaned data
curl -f http://localhost:9091/healthz  # health probe
```

### 11.3 Critical config keys (change before first start)

| Key | Why |
|---|---|
| `common.security.authorizationEnabled` | Turn on auth |
| `common.security.defaultRootPassword` | Change from `Milvus` |
| `etcd.rootPath` / `msgChannel.chanNamePrefix.cluster` | Per-cluster isolation; immutable after data exists |
| `minio.bucketName` / `rootPath` | Per-cluster storage isolation |
| `mq.type` | Pick the WAL/MQ (Woodpecker recommended) |
| `tls.*` / `internaltls.*` | Enable transport security |

### 11.4 External dependency checklist

- [ ] etcd reachable at `etcd.endpoints` (3+ nodes for HA)
- [ ] Object storage reachable at `minio.address`, bucket creatable/writable
- [ ] WAL/MQ reachable (Pulsar/Kafka) or Woodpecker storage configured
- [ ] Auth enabled and root password changed
- [ ] TLS configured (external + internal)
- [ ] Prometheus scraping `:9091/metrics`; Grafana dashboards loaded
- [ ] Backups configured (etcd snapshot + object storage + RBAC export)

---

_Generated June 2026 from milvus-io/milvus at d9bbbc1. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._