etcd — Administrator & Developer Guide

This guide is for the people who run etcd in production and the developers who extend or embed it: operators, SREs, platform architects, and contributors. It documents etcd's architecture, the complete server configuration surface, deployment and scaling strategies, day-2 operations, performance tuning, security hardening, and the internal module layout.

It complements the etcd User Guide (install + etcdctl/etcdutl usage). Where the User Guide shows you how to use a command, this guide explains how etcd works and how to operate it safely. All facts are drawn from the source at commit 7d330a1 (the v3.7 development line, v3.7.0-beta.0).

Note

etcd is the consistent, distributed key-value store at the heart of many systems — most prominently as the backing store for Kubernetes. It is optimized for storing a relatively small amount of critical metadata with strong consistency and high availability, not for bulk data.

1. Architecture Overview

etcd is a replicated state machine. Clients issue requests to any member over a gRPC API; mutating requests are proposed to the Raft consensus module, which replicates them to a quorum of members and commits them in a total order. Each member then applies committed entries to its local MVCC key-value store, which is persisted in an embedded bbolt database. A write-ahead log (WAL) records every Raft entry to stable storage before it is applied, and periodic snapshots compact the log.

                          ┌──────────────────────────────────────────────┐
   etcdctl / clientv3 ───▶│  gRPC API (api/etcdserverpb)                   │
   (gRPC, port 2379)      │  KV · Watch · Lease · Cluster · Maintenance ·  │
                          │  Auth      [+ optional gRPC-gateway → HTTP]    │
                          └───────────────┬────────────────────────────────┘
                                          │
                                ┌─────────▼──────────┐
                                │   EtcdServer       │  server/etcdserver
                                │   (request router, │
                                │    apply loop)     │
                                └───┬───────────┬────┘
                  propose mutating │           │ apply committed
                                   ▼           ▼
                        ┌──────────────┐   ┌───────────────────────┐
   peer traffic ◀──────▶│  Raft node   │   │  MVCC store           │ server/storage/mvcc
   (port 2380, rafthttp)│ go.etcd.io/  │   │  (revisions, watch,   │
                        │   raft/v3    │   │   lease, auth)        │
                        └──────┬───────┘   └──────────┬────────────┘
                               │ persist             │ persist
                       ┌───────▼────────┐    ┌────────▼───────────┐
                       │  WAL + snapshot │    │  bbolt backend     │ server/storage/backend
                       │ server/storage/ │    │  (data-dir/*.db)   │ (go.etcd.io/bbolt)
                       │  wal, snap      │    └────────────────────┘
                       └─────────────────┘

Component inventory

Technology stack

Note

etcd is split into independently-versioned Go modules — go.etcd.io/etcd/api/v3, client/v3, server/v3, etcdctl/v3, etcdutl/v3, pkg/v3, and cache/v3 — all currently at v3.7.0-beta.0. This lets applications depend on just the client (client/v3) without pulling in the server. The server module even uses gomodguard to forbid importing the umbrella go.etcd.io/etcd module, keeping the dependency graph clean.

Data flow: a write

1. A client sends Put to any member over gRPC (port 2379).
2. EtcdServer checks the request against the backend quota (server/storage/quota.go); an oversized keyspace returns etcdserver: mvcc: database space exceeded and raises a NOSPACE alarm.
3. The request becomes an InternalRaftRequest and is proposed to the Raft node.
4. Raft replicates the entry to peers (port 2380 via rafthttp). Once a quorum persists it to their WALs, the entry is committed.
5. Each member's apply loop applies the committed entry to the MVCC store, creating a new revision. The write is persisted in the bbolt backend in a batched transaction.
6. The response (with the new revision in its header) returns to the client. Linearizable reads go through a ReadIndex confirmation so they observe all prior committed writes.

2. System Requirements

Important

Put etcd's data directory on storage with consistently low write latency. A shared or network disk with variable fsync latency will cause leader elections, etcdserver: request timed out errors, and apply backpressure even when CPU and memory look healthy.

3. Configuration Reference

etcd is configured by command-line flags, by environment variables (ETCD_ prefix, upper-cased, dashes→underscores), or by a single config file (--config-file). Flags and the config file map to the same embed.Config struct.

Important

If --config-file is provided, all other command-line flags and environment variables are ignored. Also note: duration fields in the JSON/YAML config file must be given as integer nanoseconds (e.g. 600000000000 for 10 minutes); human-readable strings like "10m" are only accepted on the command line. This is a Go standard-library limitation around time.Duration unmarshalling.

3.1 Member flags

3.2 Clustering flags

Warning

The legacy v2 --discovery URL bootstrap and --discovery-proxy are deprecated (planned removal in v3.8). Use static --initial-cluster, DNS SRV (--discovery-srv), or the v3 discovery service for new clusters.

3.3 Security (TLS) flags

3.4 Auth flags

3.5 Profiling & monitoring flags

3.6 Logging flags

3.7 Distributed tracing flags

3.8 Feature, maintenance & unsafe flags

Caution

--force-new-cluster and --unsafe-no-fsync break the guarantees of the consensus protocol. --force-new-cluster is a disaster-recovery escape hatch to recover a quorum-loss situation from one surviving member; --unsafe-no-fsync is only for throwaway benchmarking. Never run either in normal production.

3.9 Feature gates

--feature-gates takes Feature=true/false pairs. The gates defined in this version:

Note

LeaseCheckpointPersist requires LeaseCheckpoint to also be enabled — etcd refuses to start otherwise (enabling feature gate LeaseCheckpointPersist requires enabling feature gate LeaseCheckpoint).

3.10 YAML configuration file

The config file uses the JSON keys from embed.Config (e.g. name, data-dir, initial-cluster). TLS is nested under client-transport-security / peer-transport-security:

name: infra1
data-dir: /var/lib/etcd
listen-client-urls: https://10.0.0.10:2379
advertise-client-urls: https://10.0.0.10:2379
listen-peer-urls: https://10.0.0.10:2380
initial-advertise-peer-urls: https://10.0.0.10:2380
initial-cluster: infra1=https://10.0.0.10:2380,infra2=https://10.0.0.11:2380,infra3=https://10.0.0.12:2380
initial-cluster-token: prod-etcd
initial-cluster-state: new
client-transport-security:
  cert-file: /etc/etcd/server.pem
  key-file: /etc/etcd/server-key.pem
  client-cert-auth: true
  trusted-ca-file: /etc/etcd/ca.pem
peer-transport-security:
  cert-file: /etc/etcd/peer.pem
  key-file: /etc/etcd/peer-key.pem
  client-cert-auth: true
  trusted-ca-file: /etc/etcd/ca.pem
auto-compaction-mode: periodic
auto-compaction-retention: "1h"
quota-backend-bytes: 8589934592

Run with: etcd --config-file /etc/etcd/etcd.conf.yaml.

4. Deployment & Scaling

4.1 Static bootstrap (recommended)

Every member is told the full membership up front. Start all members with --initial-cluster-state new:

etcd --name infra1 \
  --data-dir /var/lib/etcd \
  --listen-client-urls https://10.0.0.10:2379 \
  --advertise-client-urls https://10.0.0.10:2379 \
  --listen-peer-urls https://10.0.0.10:2380 \
  --initial-advertise-peer-urls https://10.0.0.10:2380 \
  --initial-cluster infra1=https://10.0.0.10:2380,infra2=https://10.0.0.11:2380,infra3=https://10.0.0.12:2380 \
  --initial-cluster-token prod-etcd \
  --initial-cluster-state new

Deployment checklist:

• [ ] Odd number of members (3 or 5), spread across failure domains
• [ ] Unique --name, --initial-advertise-peer-urls, --advertise-client-urls per member
• [ ] Identical --initial-cluster and --initial-cluster-token on every member
• [ ] Dedicated fast disk for --data-dir
• [ ] TLS configured for both client and peer traffic
• [ ] --quota-backend-bytes and auto-compaction set
• [ ] LimitNOFILE raised in the unit file

4.2 DNS SRV discovery

Instead of a static list, point members at a DNS domain holding _etcd-server-ssl._tcp (peer) and _etcd-client-ssl._tcp (client) SRV records:

etcd --name infra1 --discovery-srv example.com \
  --initial-advertise-peer-urls https://infra1.example.com:2380 \
  --advertise-client-urls https://infra1.example.com:2379 \
  --listen-peer-urls https://0.0.0.0:2380 \
  --listen-client-urls https://0.0.0.0:2379 \
  --initial-cluster-token prod-etcd \
  --initial-cluster-state new

4.3 systemd (multi-node)

A production unit file (adapted from the project's contrib/systemd/etcd3-multinode example):

[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
ExecStart=/usr/local/bin/etcd --name %H \
    --data-dir /var/lib/etcd \
    --listen-client-urls https://${IP}:2379 \
    --advertise-client-urls https://${IP}:2379 \
    --listen-peer-urls https://${IP}:2380 \
    --initial-advertise-peer-urls https://${IP}:2380 \
    --initial-cluster infra1=https://${IP_1}:2380,infra2=https://${IP_2}:2380,infra3=https://${IP_3}:2380 \
    --initial-cluster-token prod-etcd \
    --initial-cluster-state new

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now etcd.service
sudo journalctl -u etcd.service -f

Prepare the data directory first:

sudo mkdir -p /var/lib/etcd
sudo chown -R etcd:etcd /var/lib/etcd

4.4 Scaling: adding/removing members safely

etcd does not scale horizontally for throughput — adding members increases write latency (every write needs quorum). Add members for fault tolerance, not performance. The safe sequence uses learners:

1. etcdctl member add <name> --peer-urls=<peerURL> (or member add --learner where supported) — for a learner, it replicates but doesn't vote.
2. Start the new member with the printed ETCD_* variables and --initial-cluster-state existing.
3. Wait until etcdctl endpoint status shows the new member caught up (matching raft index).
4. Promote it to a voter (etcdctl member promote <id>) only when in sync — otherwise you get can only promote a learner member which is in sync with the leader.
5. To remove: etcdctl member remove <id> after confirming the cluster still has quorum without it.

Warning

--strict-reconfig-check (default true) rejects reconfigurations that would cause quorum loss. Leave it on. Change membership one member at a time and verify health between steps.

5. Operations

5.1 Start / stop / health

sudo systemctl start etcd
sudo systemctl stop etcd
etcdctl endpoint --cluster health
etcdctl -w table endpoint --cluster status

5.2 Backup and disaster recovery

Back up from a live cluster (writes an integrity hash):

etcdctl snapshot save /backups/etcd-$(date +%F).db
etcdutl snapshot status /backups/etcd-$(date +%F).db -w table

Restore a new cluster from one snapshot — run etcdutl snapshot restore once per member, each with its own name/peer URLs but the same --initial-cluster, then start all with --initial-cluster-state new:

etcdutl snapshot restore /backups/etcd.db \
  --name infra1 \
  --initial-cluster infra1=https://10.0.0.10:2380,infra2=https://10.0.0.11:2380,infra3=https://10.0.0.12:2380 \
  --initial-advertise-peer-urls https://10.0.0.10:2380 \
  --data-dir /var/lib/etcd-new

Caution

Recovering a cluster from a single surviving member after permanent quorum loss uses etcd --force-new-cluster (it rewrites membership to a one-member cluster), then you re-add members. This is destructive and discards uncommitted state — take a snapshot first and only do it when quorum is truly unrecoverable.

• [ ] Snapshots scheduled (cron/systemd timer)
• [ ] Snapshots stored off-box
• [ ] Restore procedure tested in a sandbox
• [ ] snapshot status verified after each backup

5.3 Compaction and defragmentation

History (MVCC revisions) grows forever until compacted; the DB file does not shrink until defragmented. Operate both:

# automatic compaction (set on the server; preferred)
etcd ... --auto-compaction-mode=periodic --auto-compaction-retention=1h

# manual compaction to a recent revision
etcdctl compaction "$(etcdctl endpoint status -w json | grep -o '\"revision\":[0-9]*' | head -1 | cut -d: -f2)"

# reclaim disk (one member at a time; blocks that member)
etcdctl defrag --cluster

Note

Compaction and defrag are independent. Compaction removes old revisions inside the database; defrag returns the freed pages to the filesystem. Run compaction frequently (auto), defrag occasionally and never on all members simultaneously.

5.4 Alarms

etcdctl alarm list      # e.g. alarm:NOSPACE  or  alarm:CORRUPT
etcdctl alarm disarm

A NOSPACE alarm means the backend exceeded its quota and etcd is read-only. Recovery: compact → defrag → disarm. A CORRUPT alarm (raised by the corruption checker) means a member's data diverged — investigate before disarming.

5.5 Monitoring (Prometheus)

etcd exposes Prometheus metrics on the client URLs (or --listen-metrics-urls). Key series to alert on:

Set --metrics=extensive to also export per-method gRPC histograms. The project ships Grafana dashboards and Prometheus alert rules in contrib/mixin.

5.6 Upgrades and downgrades

etcd supports rolling upgrades one minor version at a time; members agree on a cluster version (the lowest member minor version), and new members refuse to join a cluster newer than themselves. Downgrade is the reverse and is experimental (v3.6+):

etcdctl downgrade validate 3.6
etcdctl downgrade enable 3.6
# replace each member's binary with the older version, one at a time
# watch the leader log for "the cluster has been downgraded"

etcdutl migrate --data-dir <dir> --target-version X.Y migrates an offline data directory's schema (minimum target 3.5).

Important

Always take a snapshot before an upgrade or downgrade. For downgrade, confirm each member logs The server is ready to downgrade before swapping its binary, and use etcdctl downgrade cancel to abort if anything looks wrong.

6. Performance Tuning

Tip

Use the built-in etcdctl check perf against your real hardware to find the workload tier your cluster sustains, and etcdctl check datascale to estimate memory for a given keyspace before you grow it.

Warning

Raising --quota-backend-bytes above 8 GiB triggers a warning (quota exceeds the maximum value) because large databases increase boot time, snapshot size, defrag duration, and memory use. Prefer aggressive compaction over a huge quota.

7. Security

7.1 Transport security (TLS)

etcd has two independent TLS domains: client↔server and peer↔peer. Configure both for production. Mutual TLS is enabled with --client-cert-auth / --peer-client-cert-auth plus a --trusted-ca-file / --peer-trusted-ca-file.

etcd --name infra1 ... \
  --cert-file=/etc/etcd/server.pem --key-file=/etc/etcd/server-key.pem \
  --client-cert-auth --trusted-ca-file=/etc/etcd/ca.pem \
  --peer-cert-file=/etc/etcd/peer.pem --peer-key-file=/etc/etcd/peer-key.pem \
  --peer-client-cert-auth --peer-trusted-ca-file=/etc/etcd/ca.pem \
  --tls-min-version=TLS1.2

--auto-tls / --peer-auto-tls generate self-signed certificates for quick setups (validity set by --self-signed-cert-validity, default 1 year). Restrict identities with --client-cert-allowed-hostname, --peer-cert-allowed-cn, and --peer-cert-allowed-hostname.

Caution

When TLS is not configured, etcd serves clients over plain HTTP and logs serving client traffic insecurely; this is strongly discouraged!. An insecure server is also exposed to DNS-rebinding attacks (CVE-2018-5702), which is why --host-whitelist defaults to * but should be restricted to known hostnames for any insecure HTTP listener.

7.2 Authentication & RBAC

Once auth enable is run, every request must authenticate. The model is users → roles → per-key/prefix permissions (read, write, readwrite). A root user with the root role is mandatory before enabling. Token type is set by --auth-token:

• simple (default) — in-memory tokens with TTL --auth-token-ttl (300s). Simple, but tokens don't survive restarts and don't work well behind L4 proxies.
• jwt — signed JWT tokens; configure signing key/method via the --auth-token options string. Stateless and proxy-friendly.

Passwords are hashed with bcrypt at cost --bcrypt-cost. See the User Guide §4.10 for the full command sequence.

Warning

Authentication is an authorization layer on top of, not a replacement for, TLS. Without client TLS, credentials and data still cross the wire in cleartext. Always combine RBAC with TLS. There is no root-password recovery — losing it means restoring from a snapshot.

7.3 Data integrity

etcd can detect silent data divergence between members:

• --corrupt-check-time runs periodic cluster corruption checks; a mismatch raises a CORRUPT alarm and logs found data inconsistency with peers.
• The InitialCorruptCheck feature gate verifies integrity before serving any traffic at startup.
• The CompactHashCheck feature gate (with --compact-hash-check-time, default 1m) has the leader compare followers' compaction hashes.
• etcdctl endpoint hashkv and etcdutl hashkv compute the KV history hash for manual comparison.

7.4 Operational security checklist

• [ ] Client and peer TLS enabled, with mTLS (--client-cert-auth, --peer-client-cert-auth)
• [ ] --host-whitelist restricted if any insecure HTTP listener exists
• [ ] Authentication enabled with least-privilege roles scoped by prefix
• [ ] --auth-token=jwt behind proxies
• [ ] Corruption checking enabled (--corrupt-check-time, InitialCorruptCheck)
• [ ] --enable-pprof left off in production (it exposes profiling data)
• [ ] Peer and client ports firewalled to known hosts
• [ ] Dependencies kept patched (the project tracks CVE bumps, e.g. golang.org/x/crypto — issues #21896, #21270)

8. Developer Internals

8.1 Repository structure

8.2 Architecture deep dive

Request path. EtcdServer (server/etcdserver/server.go) is the hub. Read requests with linearizable consistency go through a ReadIndex round trip; serializable reads are served locally. Mutating requests are marshalled into an InternalRaftRequest and proposed to the embedded Raft node. The apply loop consumes committed entries and dispatches them to the appropriate subsystem (server/etcdserver/apply, txn/).

Consensus. Raft (go.etcd.io/raft/v3) handles leader election, log replication, and membership changes (ConfChange entries for add/remove/learner). Pre-Vote (--pre-vote) stops a node that was partitioned away from incrementing terms and disrupting the cluster when it returns. Membership entries are reconstructed from the snapshot + WAL via GetEffectiveNodeIDsFromWALEntries (server/storage/util.go).

Durability. Every Raft entry is written to the WAL (server/storage/wal) and fsynced before being applied — this is the durability guarantee and the dominant latency cost. The Storage interface (server/storage/storage.go) wraps WAL + snapshotter: Save persists entries, SaveSnap writes a snapshot file before the WAL snapshot record (so a snapshot file may be orphaned but a WAL record never lacks its file), and Release frees old WAL locks and .snap.db files.

State machine. Committed entries are applied to the MVCC store (server/storage/mvcc). Every change bumps a global revision; each key tracks create/mod revisions and a version. The watchable store streams events to watchers and supports replay from any uncompacted revision. The store is persisted in the bbolt backend (server/storage/backend), where keys live in buckets (key, lease, auth, meta, ...). A consistentIndex is stored in the backend so recovery knows which entries are already applied (RecoverSnapshotBackend replaces the DB with a leader-sent snapshot when snapshot.Index > consistentIndex).

Quota & alarms. server/storage/quota.go charges each mutating request a cost (kvOverhead=256 per key, leaseOverhead=64 per lease, plus key+value bytes) and rejects writes once backend.Size() would exceed the limit, raising a NOSPACE alarm. The default quota is 2 GiB (DefaultQuotaBytes), and the backend mmap is sized at quota + 10% to leave room to disarm.

Leases. server/lease tracks TTL leases and the keys attached to them, expiring keys when leases lapse. The LeaseCheckpoint feature persists remaining TTL so a leader change doesn't reset long-lived leases; FastLeaseKeepAlive (beta, default on) speeds renewal by skipping the applied-index wait.

Schema & versioning. server/storage/schema records the on-disk storage version and runs migrations; DetectSchemaVersion/Migrate drive etcdutl migrate. The cluster version (lowest member minor version) is negotiated at runtime and gates joins/downgrades.

8.3 The v3 gRPC API surface

Defined in api/etcdserverpb/rpc.proto. Six services:

When --enable-grpc-gateway=true (default), the grpc-gateway exposes these as JSON/HTTP.

8.4 Code patterns and conventions

• Modular Go modules with replace directives during development and gomodguard rules forbidding the umbrella module — keeps client/v3 lightweight for downstream users.
• Protobuf everywhere — the wire API, internal Raft requests, lease/auth/membership persistence, and WAL records are all protobuf.
• gofail injection points — comments like // gofail: var raftBeforeWALSaveSnaphot struct{} mark failpoints the robustness tests use to inject faults.
• Structured logging via zap, with --warning-apply-duration / --warning-unary-request-duration emitting slow-path warnings.
• Feature gates (pkg/featuregate) gate alpha/beta behavior behind --feature-gates.
• Cobra command trees for all CLIs with generated shell completion.

8.5 Testing strategy

etcd carries an unusually deep test suite, reflecting its role as critical infrastructure:

• tests/integration — in-process multi-member clusters exercising the client and server APIs.
• tests/e2e — spawn real etcd/etcdctl binaries (via pkg/expect).
• tests/robustness — linearizability checking with porcupine, driving failpoints (gofail) and validating history against a formal model.
• tests/antithesis — deterministic fault-injection testing.
• tests/framework — a shared abstraction so the same test runs under integration and e2e.

8.6 Bundled tools

8.7 Potential improvements & known issues

From the project's own issue tracker at this commit:

• Memory usage with 3.6 event reuse (#21355) — increased memory reported under event reuse; relevant when sizing nodes.
• Promote member with auth enabled (#20757) — promoting a follower can fail when auth is enabled.
• Dependency hygiene (#21270, #21896, #21885) — archived-dependency cleanup and CVE-driven bumps (golang.org/x/crypto, Go toolchain).
• Exit-code classification (#21032) — an effort to classify all etcdserver exit codes.
• v3.8 performance ideas (#19806) — ongoing performance work.
• Deprecations to track: v2 store/discovery (--v2-deprecation, --discovery), --max-snapshots, and the LeaseCheckpointPersist gate are all on removal paths (v3.7/v3.8).

8.8 Contributing

The project is Go-module-based with build/test scripts under scripts/ and ./scripts/build.sh producing the three binaries. Contributor documentation lives in Documentation/contributor-guide. Protobuf changes are validated by tools/proto-annotations (which enforces etcd_version annotations on API fields). The robustness and e2e suites gate behavioral changes.

9. Integration Points

Note

For application developers, prefer the clientv3 library over shelling out to etcdctl. The library handles endpoint balancing, retries, and watch reconnection; etcdctl is a thin wrapper over the same library intended for humans and scripts.