# LocalAI Admin Guide

This guide is for the people who **operate, deploy, secure, scale, and extend** LocalAI: system administrators, DevOps/platform engineers, and software architects. It documents LocalAI's architecture, its complete configuration surface, security model, distributed/clustered operation, day-2 operations, and internals deep enough that an engineer could reason about — or rebuild — the system.

If you simply want to install LocalAI and call its API, start with the companion **User Guide**; this document assumes that baseline and goes deeper.

> [!NOTE]
> LocalAI is a Go application (`module github.com/mudler/LocalAI`, Go 1.26) licensed MIT. The HTTP server is built on Echo v4; the CLI uses kong; inference backends communicate over gRPC. This guide reflects the v4.3.0-era codebase.

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [Architecture](#2-architecture)
3. [System Requirements](#3-system-requirements)
4. [Configuration Mastery](#4-configuration-mastery)
5. [Authentication & Multi-User](#5-authentication--multi-user)
6. [Security Hardening](#6-security-hardening)
7. [Distributed Mode & Horizontal Scaling](#7-distributed-mode--horizontal-scaling)
8. [P2P & Federated Mode](#8-p2p--federated-mode)
9. [Operations](#9-operations)
10. [Observability](#10-observability)
11. [Repository Structure (for Developers)](#11-repository-structure-for-developers)
12. [Internals Deep Dive](#12-internals-deep-dive)
13. [Code Quality & Conventions](#13-code-quality--conventions)
14. [Potential Improvements](#14-potential-improvements)
15. [Quick Reference](#15-quick-reference)

---

## 1. Executive Summary

LocalAI is a self-hosted inference platform that presents **OpenAI-, Anthropic-, Ollama-, ElevenLabs-, and Jina-compatible APIs** over a fleet of pluggable inference backends. Its defining architectural choice is a **small composable core**: the Go binary contains the API server, model/backend management, routing, auth, and orchestration, while the actual inference engines (llama.cpp, vLLM, transformers, whisper.cpp, diffusers, MLX, kokoro, parakeet, and 30+ more) are **separate processes** packaged as OCI images and pulled on demand.

Key operational properties:

- **One binary, many roles.** The `local-ai` binary runs the server, CLI media tools, distributed workers, agent workers, the P2P explorer, and a stdio MCP control surface.
- **On-demand backends.** Backends install themselves from a gallery (OCI images) the first time a model needs them; integrity can be enforced with cosign keyless verification and SHA256 checks.
- **Scales from laptop to cluster.** A single CPU-only container at one extreme; a PostgreSQL + NATS + S3-backed distributed cluster with VRAM-aware routing at the other.
- **Multi-tenant ready.** API-key and OIDC/GitHub/local auth, per-user roles, quotas, and usage attribution.
- **Privacy-first.** No data leaves the operator's infrastructure; outbound calls are limited to fetching explicitly requested models/backends.

---

## 2. Architecture

### 2.1 High-level component map

```
                              ┌─────────────────────────────────────────────┐
   Clients (OpenAI/Anthropic/ │              local-ai (Go core)              │
   Ollama/ElevenLabs SDKs,    │                                             │
   Web UI, MCP clients)       │  ┌───────────────────────────────────────┐  │
            │                 │  │  HTTP server (Echo v4)                 │  │
            ▼                 │  │  • middleware: auth, CSRF, security    │  │
   ┌──────────────────┐       │  │    headers, request mapping, usage,    │  │
   │  HTTP :8080      │──────▶│  │    admission, PII, baseurl, node hdr   │  │
   │  (+ React UI)    │       │  │  • routes: openai, anthropic, ollama,  │  │
   └──────────────────┘       │  │    elevenlabs, jina, localai, agents,  │  │
                              │  │    auth, nodes, pii, finetuning,       │  │
                              │  │    quantization, ui_api, usage         │  │
                              │  └───────────────┬───────────────────────┘  │
                              │                  ▼                           │
                              │  ┌───────────────────────────────────────┐  │
                              │  │  core/application  +  core/startup     │  │
                              │  │  ModelLoader · ConfigLoader · Gallery  │  │
                              │  │  services: agents, nodes, router,      │  │
                              │  │  galleryop, finetune, quantization,    │  │
                              │  │  messaging(NATS), storage(S3), jobs    │  │
                              │  └───────────────┬───────────────────────┘  │
                              └──────────────────┼───────────────────────────┘
                                                 │ gRPC (localhost:50051+)
                            ┌────────────────────┼────────────────────┐
                            ▼                    ▼                    ▼
                   ┌────────────────┐  ┌────────────────┐  ┌────────────────┐
                   │ Go backends    │  │ Python backends│  │ C++ / Rust     │
                   │ (purego dlopen │  │ (gRPC servers, │  │ backends       │
                   │  of .so libs)  │  │  one per model)│  │ (llama.cpp,    │
                   │ whisper, piper,│  │ vllm, diffusers│  │  ik-llama, ds4,│
                   │ sherpa, opus,  │  │ transformers,  │  │  kokoros/Rust  │
                   │ sd-ggml, sam3, │  │ mlx, sglang,   │  │  via tonic)    │
                   │ rfdetr, voxtral│  │ kokoro, nemo…  │  │                │
                   └────────────────┘  └────────────────┘  └────────────────┘
                            ▲ pulled on demand as OCI images from the backend gallery
```

### 2.2 The composable core

The core is intentionally minimal. It owns:

- **API translation** — every supported dialect (OpenAI, Anthropic, Ollama, ElevenLabs, Jina) is normalized into LocalAI's internal request schema (`core/schema`), then dispatched to a backend.
- **Model lifecycle** — `pkg/model` (the `ModelLoader`) starts, tracks, and stops backend processes; `core/config` loads and validates model YAML and application options.
- **Backend lifecycle** — `core/gallery` + `core/services/galleryop` install/upgrade/remove backends from galleries.
- **Cross-cutting services** — auth, routing, agents, distributed messaging, object storage, jobs, PII filtering, tracing, metrics.

### 2.3 Backends are separate processes

Each backend is a **gRPC server** that implements LocalAI's `backend` proto. The core launches a backend process (allocating an address like `localhost:50051`) and talks to it over gRPC (`pkg/grpc`). This isolation is what makes the core small and lets backends ship independently:

- **Go backends** (`backend/go/*`) use [`purego`](https://github.com/ebitengine/purego) to `dlopen` a native shared library at runtime (e.g. `WHISPER_LIBRARY`, `SD_LIBRARY`, `PARAKEET_LIBRARY`) and register its C ABI symbols — no CGo at build time. The library path is overridable by env var, with a fallback `.so` next to the binary.
- **Python backends** (`backend/python/*`) are gRPC servers (one process per loaded model) started with `--addr`. Each has its own `requirements.txt` and is shipped as its own image (vLLM, diffusers, transformers, MLX, kokoro, nemo, sglang, whisperx, and many more).
- **C++ backends** (`backend/cpp/*`) wrap engines like `llama.cpp`, `ik-llama-cpp`, `ds4`, and `turboquant`.
- **Rust backend** (`backend/rust/kokoros`) uses `tonic` for gRPC and `clap` for its CLI.

> [!NOTE]
> Because backends are external processes, a crashing or hanging backend does not take down the core. Watchdogs and the memory reclaimer (see [§9](#9-operations)) manage misbehaving or idle backends.

### 2.4 Request lifecycle (chat completion)

1. Request hits Echo at e.g. `POST /v1/chat/completions`.
2. **Middleware chain** runs: strip-path-prefix, security headers, logging, recover, metrics, CSRF, auth, request mapping (normalize to internal schema), admission control (per-model concurrency limits), PII filtering, usage attribution.
3. The route handler resolves the model config, ensures the backend is loaded (via `ModelLoader`, installing the backend from the gallery if needed), and applies the model's prompt `template`.
4. The core issues a gRPC `Predict`/`PredictStream` to the backend.
5. Output is post-processed (stopwords, `cutstrings`, `trimspace`, `extract_regex`), optionally filtered for PII, and returned (streamed as SSE if requested).
6. Usage is recorded per API key/user.

### 2.5 Process entry points

| Entry point | Role |
|---|---|
| `cmd/local-ai/main.go` | The `local-ai` binary; loads `.env` files, parses kong CLI, dispatches to a command |
| `core/cli/cli.go` | Top-level CLI command tree |
| `core/http/app.go` | Builds the Echo app, registers middleware and routes, embeds the React UI and static assets |
| `cmd/launcher/main.go` | Desktop Launcher (Fyne GUI + system tray) for macOS/Linux |
| `backend/*/main.go` (+ `kokoros/src/main.rs`) | Per-backend gRPC servers, started internally by the core |
| `pkg/grpc/server.go` | gRPC server scaffolding shared by backends |
| `pkg/mcp/localaitools/server.go` | Admin tool surface exposed as a stdio MCP server |

---

## 3. System Requirements

| Dimension | Minimum | Recommended |
|---|---|---|
| OS | Linux, macOS (Apple Silicon or Intel), Windows | Linux for servers |
| CPU | x86-64 or ARM64 | Many physical cores; set `--threads` to physical core count |
| RAM | Enough to hold the model + KV cache | 16 GB+ for small models, more for large |
| GPU (optional) | — | NVIDIA (CUDA 12/13), AMD (ROCm), Intel (oneAPI/SYCL), Apple (Metal/MLX), Vulkan, Jetson L4T |
| Disk | Room for core image + backend images + model weights | SSD; backends and models can total tens of GB |
| Runtime | Docker/Podman (for containers) | Docker with the matching GPU runtime |

GPU acceleration is selected by the **image tag** (containers) and/or model config (`f16`, `gpu_layers`, `cuda`). LocalAI auto-detects GPU capabilities and pulls the matching backend.

> [!IMPORTANT]
> Backend images are pulled on demand and can be large (multi-GB for GPU engines). Provision disk and network accordingly, and consider pre-installing backends (`local-ai backends install …`) on air-gapped or slow links.

---

## 4. Configuration Mastery

LocalAI is configured at two levels: **server-level** (CLI flags / environment variables on the `run` command) and **model-level** (per-model YAML).

### 4.1 Configuration precedence and sources

1. **`.env` files**, loaded at startup in order: `.env`, `localai.env`, `~/localai.env`, `~/.config/localai.env`, `/etc/localai.env`.
2. **Environment variables** (`LOCALAI_*`, plus legacy unprefixed aliases like `MODELS_PATH`).
3. **CLI flags** on `local-ai run`.
4. **Dynamic config directory** (`--localai-config-dir`, default `./configuration`) — holds `api_keys.json` and `external_backends.json`, hot-reloaded via fsnotify (or polled if `--localai-config-dir-poll-interval` is set).
5. **Model YAML** files in the models directory, and the models config file (`--models-config-file`, legacy alias `--config-file`).

### 4.2 Server flags by group

`local-ai run` flags are organized into groups. Every flag has a `LOCALAI_*` environment equivalent.

#### Storage & paths

| Flag / Env | Default | Purpose |
|---|---|---|
| `--models-path` / `LOCALAI_MODELS_PATH` | `./models` | Model storage |
| `--backends-path` / `LOCALAI_BACKENDS_PATH` | `./backends` | Backend storage |
| `--backends-system-path` / `LOCALAI_BACKENDS_SYSTEM_PATH` | `/var/lib/local-ai/backends` | System backends |
| `--generated-content-path` / `LOCALAI_GENERATED_CONTENT_PATH` | `/tmp/generated/content` | Generated images/audio/video |
| `--upload-path` / `LOCALAI_UPLOAD_PATH` | `/tmp/localai/upload` | Files API uploads |
| `--data-path` / `LOCALAI_DATA_PATH` | `./data` | Persistent data (collection DB, agent state, tasks, jobs) |
| `--localai-config-dir` / `LOCALAI_CONFIG_DIR` | `./configuration` | Hot-reloaded dynamic config |
| `--models-config-file` / `LOCALAI_MODELS_CONFIG_FILE` | — | YAML list of model configs |

#### Models & galleries

| Flag / Env | Default | Purpose |
|---|---|---|
| `--galleries` / `LOCALAI_GALLERIES` | localai | Model galleries (JSON) |
| `--autoload-galleries` / `LOCALAI_AUTOLOAD_GALLERIES` | true | Auto-load model galleries |
| `--preload-models` / `LOCALAI_PRELOAD_MODELS` | — | Models to apply at startup (JSON) |
| `--preload-models-config` / `LOCALAI_PRELOAD_MODELS_CONFIG` | — | YAML path of models to apply at startup |
| `--models` / `LOCALAI_MODELS` | — | Model config URLs to load |
| `--load-to-memory` / `LOCALAI_LOAD_TO_MEMORY` | — | Models to load into memory at startup |

#### Backends

| Flag / Env | Default | Purpose |
|---|---|---|
| `--backend-galleries` / `LOCALAI_BACKEND_GALLERIES` | localai | Backend galleries (JSON) |
| `--autoload-backend-galleries` / `LOCALAI_AUTOLOAD_BACKEND_GALLERIES` | true | Auto-load backend galleries |
| `--external-backends` / `LOCALAI_EXTERNAL_BACKENDS` | — | External backends to load on boot |
| `--external-grpc-backends` / `LOCALAI_EXTERNAL_GRPC_BACKENDS` | — | External gRPC backends |
| `--auto-upgrade-backends` / `LOCALAI_AUTO_UPGRADE_BACKENDS` | false | Auto-upgrade backends |
| `--backend-images-release-tag` / `LOCALAI_BACKEND_IMAGES_RELEASE_TAG` | latest | Fallback release tag |
| `--max-active-backends` / `LOCALAI_MAX_ACTIVE_BACKENDS` | 0 (unlimited) | Max backends loaded at once (LRU eviction) |
| `--single-active-backend` / `LOCALAI_SINGLE_ACTIVE_BACKEND` | — | Deprecated; use `--max-active-backends=1` |
| `--preload-backend-only` / `LOCALAI_PRELOAD_BACKEND_ONLY` | false | Start preloaded models/backends only, no API |

#### Performance

| Flag / Env | Default | Purpose |
|---|---|---|
| `--f16` / `LOCALAI_F16` | false | Enable GPU acceleration (half precision) |
| `--threads` (`-t`) / `LOCALAI_THREADS` | — | Computation threads (use physical core count) |
| `--context-size` / `LOCALAI_CONTEXT_SIZE` | — | Default model context size |

#### API & web

| Flag / Env | Default | Purpose |
|---|---|---|
| `--address` / `LOCALAI_ADDRESS` | `:8080` | Bind address |
| `--cors` / `LOCALAI_CORS` | false | Enable CORS |
| `--cors-allow-origins` / `LOCALAI_CORS_ALLOW_ORIGINS` | — | Allowed CORS origins |
| `--upload-limit` / `LOCALAI_UPLOAD_LIMIT` | 15 | Upload limit (MB) |
| `--disable-webui` / `LOCALAI_DISABLE_WEBUI` | false | API-only mode |
| `--ollama-api-root-endpoint` / `LOCALAI_OLLAMA_API_ROOT_ENDPOINT` | false | Ollama-style health check on `/` |
| `--disable-runtime-settings` | false | Don't load `runtime_settings.json` |
| `--disable-gallery-endpoint` | — | Disable gallery endpoints |
| `--disable-mcp` / `LOCALAI_DISABLE_MCP` | false | Disable MCP support |
| `--disable-metrics-endpoint` | false | Disable `/metrics` |
| `--machine-tag` / `LOCALAI_MACHINE_TAG` | — | `Machine-Tag` response header (P2P tracking) |
| `--enable-tracing` / `LOCALAI_ENABLE_TRACING` | — | Enable API tracing (`--tracing-max-items`, `--tracing-max-body-bytes`) |
| `--agent-job-retention-days` | 30 | Agent job history retention |

#### Backend lifecycle (watchdogs & reclaimer)

| Flag / Env | Default | Purpose |
|---|---|---|
| `--enable-watchdog-idle` / `LOCALAI_WATCHDOG_IDLE` | false | Stop backends idle past the timeout |
| `--watchdog-idle-timeout` | 15m | Idle threshold |
| `--enable-watchdog-busy` / `LOCALAI_WATCHDOG_BUSY` | false | Stop backends busy past the timeout |
| `--watchdog-busy-timeout` | 5m | Busy threshold |
| `--watchdog-interval` | 500ms | Watchdog check interval |
| `--enable-memory-reclaimer` / `LOCALAI_MEMORY_RECLAIMER` | false | Auto-evict backends when memory exceeds threshold |
| `--memory-reclaimer-threshold` | 0.95 | Memory threshold (0.0–1.0) triggering eviction |
| `--force-eviction-when-busy` | false | Evict even with active calls |

### 4.3 Model YAML reference

The model YAML (`core/config.ModelConfig`) is the per-model contract. Core keys:

```yaml
name: my-model              # model name in API requests
backend: llama-cpp          # engine
f16: true                   # half precision
threads: 8
context_size: 4096
gpu_layers: 35              # layers offloaded to GPU
mmap: true
embeddings: false           # serve embeddings endpoint?
cuda: false                 # explicitly enable CUDA for backends that need it
known_usecases: [chat]      # declared capabilities
concurrency_groups: [gpu0]  # mutual-exclusion groups on a node
download_files: []          # files fetched on install

parameters:                 # OpenAI-style request defaults (PredictionOptions)
  temperature: 0.7
  top_p: 0.9
  top_k: 40
  max_tokens: 512
  seed: -1
  repeat_penalty: 1.1

template:                   # Go prompt templates
  chat: "{{.Input}}"
  use_tokenizer_template: false

stopwords: ["</s>"]
```

Specialized blocks: `diffusers`, `tts`, `function`, `reasoning`, `feature_flags`, `mcp`, `agent`, `pii`, `router`, `proxy`, `mitm`, `limits`, `grpc`. The most operationally relevant:

- **`limits:`** — admission control. `max_concurrent` caps simultaneous in-flight requests for the model (0 = unlimited); the admission middleware returns `503` with a `Retry-After` header (`retry_after_seconds`, default 1s) when exceeded. Use this to protect rate-limited cloud upstreams or memory-bound local backends.
- **`proxy:`** — the cloud-proxy backend. `upstream_url`, `mode` (`passthrough` keeps the wire format end-to-end; `translate` converts via internal proto for cross-provider routing), `provider` (`openai`/`anthropic`), `api_key_env` / `api_key_file` (mutually exclusive; the file form integrates with K8s secret mounts / Vault), `upstream_model`, `request_timeout_seconds`.
- **`router:`** — route a request across `candidates` by `policies`, with a `classifier`, optional `classifier_model`, embedding cache, and a `fallback`.
- **`mitm:`** — declare `hosts` this model intercepts when the cloud-proxy MITM listener is enabled; the model's `pii:` settings then filter intercepted traffic. A host claimed by two configs is a configuration error and disables the MITM listener.
- **`mcp:`** — remote + stdio MCP servers the model can call.

LLM tuning keys live inline (the `LLMConfig` struct): `gpu_layers`, `tensor_split`, `main_gpu`, `low_vram`, `mmap`/`mmlock`, `flash_attention`, `cache_type_k`/`cache_type_v`, `prompt_cache_*`, `mirostat*`, `rope_scaling`/`yarn_*`, `lora_*`, `grammar`, `draft_model`/`n_draft` (speculative decoding), and vLLM-specific `gpu_memory_utilization`, `tensor_parallel_size`, `max_model_len`, `dtype`, `quantization`, `trust_remote_code`, `enforce_eager`, `swap_space`, `limit_mm_per_prompt`, and `engine_args` (raw passthrough).

> [!TIP]
> `prompt_cache_all` / llama.cpp prompt caching is on by default in recent releases — repeated system prompts collapse from minutes to seconds. Keep it enabled for chat workloads with stable system prompts.

---

## 5. Authentication & Multi-User

LocalAI supports two complementary auth models: **static API keys** (simplest) and **full user authentication** (multi-tenant).

### 5.1 Static API keys

```bash
local-ai run --api-key "sk-key-one" --api-key "sk-key-two"
```

When any key is set, all requests must present `Authorization: Bearer <key>`. Keys can also be managed dynamically through `api_keys.json` in the config directory (hot-reloaded).

### 5.2 Full user auth

| Flag / Env | Default | Purpose |
|---|---|---|
| `--auth-enabled` / `LOCALAI_AUTH` | false | Enable user authentication/authorization |
| `--auth-database-url` / `LOCALAI_AUTH_DATABASE_URL` | `{DataPath}/database.db` | Auth DB (`postgres://` or SQLite path) |
| `GITHUB_CLIENT_ID` / `GITHUB_CLIENT_SECRET` | — | GitHub OAuth (auto-enables auth when set) |
| `--oidc-issuer` / `--oidc-client-id` / `--oidc-client-secret` | — | OIDC SSO (auto-enables auth) |
| `--auth-base-url` / `LOCALAI_BASE_URL` | — | Base URL for OAuth callbacks |
| `--auth-admin-email` / `LOCALAI_ADMIN_EMAIL` | — | Email auto-promoted to admin |
| `--auth-registration-mode` / `LOCALAI_REGISTRATION_MODE` | open | `open`, `approval`, or `invite` |
| `--disable-local-auth` / `LOCALAI_DISABLE_LOCAL_AUTH` | false | OAuth/OIDC-only (no local email/password) |
| `--auth-api-key-hmac-secret` / `LOCALAI_AUTH_HMAC_SECRET` | auto-gen | HMAC secret for API-key hashing |
| `--default-api-key-expiry` / `LOCALAI_DEFAULT_API_KEY_EXPIRY` | none | Default API-key expiry (e.g. `90d`, `1y`) |

The auth subsystem (`core/http/auth`, routes in `core/http/routes/auth.go`) provides registration/login (local, GitHub, OIDC), session management, per-user API key issuance, roles, per-user model access, per-user **quotas**, and **usage attribution** (per API key and per user). Admin endpoints under `/api/auth/admin/*` manage users, roles, permissions, quotas, and invites.

> [!IMPORTANT]
> Setting `GITHUB_CLIENT_ID`, an OIDC client ID, or an OIDC issuer **auto-enables** authentication even without `--auth-enabled`. Plan callback URLs (`--auth-base-url`) and the admin email before exposing the server.

---

## 6. Security Hardening

LocalAI ships several guardrails. Review each before exposing the server beyond localhost.

### 6.1 Network bind safety

By default, LocalAI **refuses to start** when binding to a public-internet address with no auth configured. Loopback, RFC 1918 LAN, ULA, link-local, and CGNAT/Tailscale ranges are always allowed.

| Flag | Default | Effect |
|---|---|---|
| `--allow-insecure-public-bind` / `LOCALAI_ALLOW_INSECURE_PUBLIC_BIND` | false | Permit public bind with no auth (**not recommended**) |

### 6.2 Request-surface hardening

| Flag | Default | Effect |
|---|---|---|
| `--disable-csrf` / `LOCALAI_DISABLE_CSRF` | false (CSRF on) | Disable CSRF middleware |
| `--subtle-key-comparison` / `LOCALAI_SUBTLE_KEY_COMPARISON` | false | Constant-time API-key comparison (timing-attack resistance) |
| `--opaque-errors` / `LOCALAI_OPAQUE_ERRORS` | false | Replace all error bodies with blank 500s (anti-leak) |
| `--disable-api-key-requirement-for-http-get` | false | Allow unauthenticated GETs to parts of the UI |
| `--http-get-exempted-endpoints` | (UI/static/swagger regexes) | Endpoints exempt when the above is enabled |

Security headers (CSP, `X-Content-Type-Options`, `X-Frame-Options`, `Referrer-Policy`) are applied early to **every** response, including 404s and errors (`core/http/middleware`).

### 6.3 Supply-chain integrity

Backends are OCI images. LocalAI supports **cosign keyless verification** of backend images and SHA256 verification of tarball/HTTP backends. The gallery's `verification:` block declares the expected `issuer`/`identity` (with regex variants).

| Flag | Default | Effect |
|---|---|---|
| `--require-backend-integrity` / `LOCALAI_REQUIRE_BACKEND_INTEGRITY` | false | Reject backend installs lacking a verification policy (OCI) or SHA256 (tarball/HTTP) |
| `--disable-predownload-scan` / `LOCALAI_DISABLE_PREDOWNLOAD_SCAN` | false | Disable the best-effort security scan before downloads |

> [!CAUTION]
> In production with a populated gallery `verification:` block, set `--require-backend-integrity`. Without it, integrity issues only **warn** and install proceeds. Leaving the predownload scan enabled is also recommended.

### 6.4 PII filtering

LocalAI can detect and filter PII (`core/services/routing/pii`, routes under `/api/pii/*`). PII rules can be tested (`POST /api/pii/test`), decided (`POST /api/pii/decide`), and persisted. The streaming PII filter also runs on cloud-proxy passthrough traffic (operating on extracted token text), and on MITM-intercepted bodies for declared hosts.

### 6.5 Hardening checklist

- [ ] Auth configured (`--api-key` or full auth) before any non-loopback bind
- [ ] `--require-backend-integrity` set with a populated gallery `verification:` block
- [ ] Predownload scan left enabled
- [ ] CSRF left enabled; `--subtle-key-comparison` on for public endpoints
- [ ] `/metrics` and `X-LocalAI-Node` header not exposed publicly (topology leak)
- [ ] TLS terminated at a reverse proxy (LocalAI binds plain HTTP)
- [ ] Per-user quotas set in multi-tenant deployments

---

## 7. Distributed Mode & Horizontal Scaling

Distributed mode turns a set of LocalAI instances into a cluster with VRAM-aware smart routing and autoscaling. It requires **PostgreSQL + NATS** (and optionally S3-compatible object storage).

```bash
local-ai run --distributed \
  --nats-url nats://nats:4222 \
  --auth-database-url postgres://user:pass@pg:5432/localai \
  --storage-url http://minio:9000 --storage-bucket localai \
  --storage-access-key KEY --storage-secret-key SECRET
```

### 7.1 Roles

- **Frontend** — the `run --distributed` instance: serves the API/UI, routes requests, mints worker credentials, schedules models, tracks nodes.
- **Worker node** — `local-ai worker` (generic, backend-agnostic): registers with the frontend, runs models, reports heartbeats.
- **Agent worker** — `local-ai agent-worker`: a pure executor that subscribes to the NATS `agent.execute` queue and runs agent chats via cogito; it also handles MCP tool execution, MCP discovery, and MCP CI jobs routed over NATS. Config and skills arrive in the job payload, so it needs no direct DB access.

### 7.2 Key distributed flags

| Flag / Env | Default | Purpose |
|---|---|---|
| `--distributed` / `LOCALAI_DISTRIBUTED` | false | Enable distributed mode |
| `--instance-id` / `LOCALAI_INSTANCE_ID` | auto UUID | Unique instance ID |
| `--nats-url` / `LOCALAI_NATS_URL` | — | NATS server URL |
| `--storage-url` / `--storage-bucket` / `--storage-region` / `--storage-access-key` / `--storage-secret-key` | — | S3-compatible object storage |
| `--registration-token` / `LOCALAI_REGISTRATION_TOKEN` | — | Token nodes must present to register |
| `--auto-approve-nodes` / `LOCALAI_AUTO_APPROVE_NODES` | false | Skip admin approval for new nodes |
| `--distributed-prefix-cache` | true | Prefix-cache-aware routing (else round-robin) |
| `--distributed-prefix-cache-ttl` | 5m | Idle timeout for prefix-cache index entries |
| `--expose-node-header` / `LOCALAI_EXPOSE_NODE_HEADER` | false | Add `X-LocalAI-Node` to inference responses (topology leak — keep off publicly) |

### 7.3 NATS security

NATS auth uses JWT + nkey seeds. The frontend mints per-node worker JWTs at registration (`--nats-account-seed`, `--nats-service-jwt`, `--nats-service-seed`, `--nats-worker-jwt-ttl`). Enforce credentials with `--nats-require-auth`, and use mTLS with `--nats-tls-ca`/`--nats-tls-cert`/`--nats-tls-key` and a `tls://` URL. Workers auto-refresh minted credentials before expiry; if refresh permanently fails, the worker shuts down so it can re-register rather than linger unauthenticated.

### 7.4 Routing and scheduling

The frontend exposes node and scheduling APIs (`core/http/routes/nodes.go`): register/heartbeat/drain/resume/deregister nodes, set per-node labels and `max-replicas-per-model`, and query/define scheduling decisions. The router (`core/services/routing`) supports prefix-cache-aware placement, admission control, billing, and PII adaptation. Decision introspection is available via `/api/router/decisions` and `/api/router/cache/stats`.

> [!NOTE]
> Backend install/upgrade requests are sent to worker nodes over NATS request-reply with generous timeouts (`--nats-backend-install-timeout` / `--nats-backend-upgrade-timeout`, default 15m) — increase for slow links pulling multi-GB images.

---

## 8. P2P & Federated Mode

For peer-to-peer inference without a central control plane, LocalAI integrates libp2p (via `mudler/edgevpn`).

```bash
# Start with P2P; if no token is given, one is generated and printed
local-ai run --p2p

# On a worker node, join with the token
export TOKEN="<token>"
local-ai worker p2p-llama-cpp-rpc
```

P2P-related flags: `--p2p`, `--p2p-token` (`TOKEN`), `--p2p-network-id` (group instances), `--p2p-dht-interval`, `--p2p-otp-interval`. Library-level behavior is tuned with `LOCALAI_P2P_*` env vars (DHT disable, listen multiaddrs, bootstrap peers, log level). The **explorer** (`local-ai explorer`) discovers P2P networks. **Federated mode** (`--federated` / `local-ai federated`) lets a federated instance balance requests across peers.

---

## 9. Operations

### 9.1 Start / stop / restart

```bash
# Containers
docker start -i local-ai          # resume existing
docker stop local-ai
docker logs -f local-ai           # follow logs

# Binary — graceful shutdown on SIGINT/SIGTERM
```

### 9.2 Managing backends and models

```bash
local-ai backends list
local-ai backends install <backend>
local-ai backends upgrade            # all, or name specific ones
local-ai backends uninstall <backend>
```

Backend management is also available over the API (`/backends/*`, `/api/backends/*`) and the UI, including upgrade checks (`/backends/upgrades`, `POST /backends/upgrades/check`).

### 9.3 Backend logs and monitoring

| Endpoint | Purpose |
|---|---|
| `GET /api/backend-logs/:modelId` | Per-backend logs |
| `GET /ws/backend-logs/:modelId` | Live backend logs (WebSocket) |
| `GET /backend/monitor` | Backend status |
| `POST /backend/shutdown` | Stop a backend |
| `GET /api/operations` | In-flight operations (installs, jobs) |

### 9.4 Memory management

- **Idle/busy watchdogs** stop backends that sit idle or run too long.
- **Memory reclaimer** evicts backends when GPU VRAM (or RAM) usage crosses `--memory-reclaimer-threshold` (default 0.95).
- **`--max-active-backends`** caps loaded backends with LRU eviction.
- **`concurrency_groups`** in a model config prevent co-loading models that share a group on a node (see VRAM-management docs).

### 9.5 Backup

Persist and back up these paths:

| Path | Contents |
|---|---|
| `--models-path` | Model weights + YAML configs |
| `--backends-path` | Installed backends |
| `--data-path` | Collection DB, agent state, tasks, jobs |
| Auth DB (`--auth-database-url`) | Users, API keys, quotas, usage (SQLite file or Postgres) |
| `--localai-config-dir` | `api_keys.json`, `external_backends.json` |

### 9.6 Health checks & readiness

```bash
curl http://localhost:8080/healthz   # liveness
curl http://localhost:8080/readyz    # readiness
```

Use `/readyz` for load-balancer/orchestrator readiness gates.

---

## 10. Observability

- **Metrics** — Prometheus metrics at `GET /metrics` (OpenTelemetry-based; disable with `--disable-metrics-endpoint`). The project depends on `go.opentelemetry.io/otel` with a Prometheus exporter.
- **Tracing** — enable in-app API tracing with `--enable-tracing`; inspect via `/api/traces` and `/api/backend-traces`. Bound memory with `--tracing-max-items` and `--tracing-max-body-bytes`.
- **Usage** — per API key and per user usage via `/api/usage` and `/api/auth/usage` (with source breakdowns).
- **Logging** — `xlog`-based structured logging. Set level with `--log-level` / `--debug`; format with `LOCALAI_LOG_FORMAT` (text vs. structured). Quiet paths (UI polling, `/healthz`, `/readyz`, `/api/resources`) log at DEBUG to reduce noise.

> [!TIP]
> In containers where stdout is captured to a file, set `LOCALAI_LOG_FORMAT=text` (backends do this automatically when stdout isn't a TTY) to avoid ANSI color codes in logs.

---

## 11. Repository Structure (for Developers)

```
LocalAI/
├── cmd/
│   ├── local-ai/        # main binary entry point (kong CLI, .env loading)
│   └── launcher/        # Fyne desktop launcher + system tray
├── core/
│   ├── cli/             # CLI command tree (run, models, backends, worker, agent, …)
│   ├── config/          # ApplicationConfig + ModelConfig (YAML schema), gallery, gguf, hooks
│   ├── http/            # Echo app, routes/, middleware/, endpoints/, auth/, react-ui/, views/
│   ├── schema/          # internal request/response schema (OpenAI-compatible types)
│   ├── application/     # app bootstrap and wiring
│   ├── startup/         # startup sequence
│   ├── backend/         # backend invocation glue (predict, embeddings, tts, image, …)
│   ├── gallery/         # model + backend gallery (install/upgrade/verify)
│   ├── services/        # agents, nodes, router, messaging(NATS), storage(S3), jobs,
│   │                    #   galleryop, finetune, quantization, cloudproxy, pii, mcp, skills
│   ├── p2p/             # libp2p/edgevpn integration
│   ├── explorer/        # P2P explorer
│   └── trace/           # tracing buffer
├── backend/
│   ├── go/              # Go backends (purego dlopen of native libs)
│   ├── python/          # Python gRPC backends (vllm, diffusers, transformers, mlx, …)
│   ├── cpp/             # C++ backends (llama-cpp, ik-llama-cpp, ds4, turboquant)
│   └── rust/            # Rust backend (kokoros, tonic gRPC)
├── pkg/
│   ├── model/           # ModelLoader: backend process lifecycle
│   ├── grpc/            # gRPC server/client scaffolding
│   ├── functions/       # function calling, grammars (PEG), JSON-schema → grammar
│   ├── downloader/      # model/file download (HF, OCI, HTTP) with URI schemes
│   ├── gallery / oci    # OCI image handling, cosign verification
│   ├── mcp/             # MCP tool surface (localaitools)
│   ├── reasoning/       # reasoning/thinking extraction
│   ├── vram / xsysinfo  # VRAM + system probing
│   └── …                # audio, sound, store, radixtree, clusterrouting, natsauth, etc.
├── gallery/             # model gallery index + definitions
├── swagger/             # generated OpenAPI spec
├── prompt-templates/    # default prompt templates
├── docs/                # Hugo (Docsy) documentation site
└── tests/               # e2e (aio, backends, ui, distributed), integration, fixtures
```

### Core engine modules at a glance

| Module | Responsibility |
|---|---|
| `pkg/model` | Start/track/stop backend processes; the `ModelLoader` |
| `core/config` | Load/validate application options and model YAML; gallery + GGUF parsing; backend capability hooks |
| `core/http` | Echo app, middleware chain, all route groups, embedded React UI |
| `core/schema` | Internal, OpenAI-compatible request/response types |
| `core/gallery` + `core/services/galleryop` | Backend/model install, upgrade, verification |
| `core/services/agents` | Agent execution, NATS dispatcher, event bridge (cogito) |
| `core/services/messaging` | NATS client wrapper (subjects, queues, JWT/mTLS) |
| `pkg/functions` | Tool/function calling and grammar constraint generation |
| `pkg/grpc` | The backend gRPC contract |

---

## 12. Internals Deep Dive

### 12.1 CLI parsing (kong)

`cmd/local-ai/main.go` builds the kong model from `core/cli.CLI`. Kong `Vars` inject defaults: `basepath` (`.`), the default model and backend galleries (`github:mudler/LocalAI/...@master`), and `version`. The `run` command is `default:"withargs"`, so bare `local-ai <model>` runs the server with that model. Logging level resolves from `--debug`/`--log-level`, with optional dedup (`--log-dedup-logs`).

### 12.2 The backend gRPC contract

Every backend implements the `backend` proto served via `pkg/grpc`. The core launches a backend, connects over gRPC, and calls `LoadModel`, then `Predict`/`PredictStream`, `Embeddings`, TTS/transcription/image RPCs as appropriate. Go backends register native C functions through purego at startup (`RegisterLibFunc`), allowing a single Go process to drive a native engine without CGo. Backends probe optional ABI symbols (`Dlsym`) so a newer Go shim still loads against an older library, degrading gracefully.

### 12.3 Cloud-proxy (`backend/go/cloud-proxy`)

A backend that forwards traffic to an external provider. **Passthrough mode** serves a `Forward` RPC and preserves the client wire format end-to-end. **Translate mode** serves `Predict`/`PredictStream` and converts LocalAI's internal proto to the provider's wire format, enabling cross-provider routing (an OpenAI client hitting an Anthropic upstream). API keys are resolved once at load time from `api_key_env` or `api_key_file`.

### 12.4 Agents and cogito

Agents are built on `mudler/cogito` (the execution engine) and `mudler/LocalAGI` (agent state/types). The agent worker (`core/cli/agent_worker.go`) is a NATS-driven executor: it streams reasoning/content/tool-call events back over NATS as `ProgressEvent`s, buffering tokens and flushing on stream-type changes. It handles three NATS surfaces — agent execution (`agent.execute`), MCP tool/discovery request-reply, and MCP CI jobs — and cleans up MCP sessions on backend-stop events.

### 12.5 MCP

The `mcp-server` command exposes LocalAI's admin tool surface as a **stdio MCP server** (controlling a remote LocalAI over HTTP). In-process, the LocalAI Assistant is an admin MCP server (disable with `--disable-assistant`). MCP sessions are created from per-model config (remote + stdio servers), discovered, and cached; tool calls are dispatched to the right session.

### 12.6 Inference defaults

`core/config/gen_inference_defaults` is a `go generate` tool that fetches unsloth's `inference_defaults.json`, validates and remaps it (e.g. `repetition_penalty` → `repeat_penalty`), filters to allowed sampling fields, and writes `inference_defaults.json`. This seeds sensible per-family sampling defaults (temperature/top_p/top_k/min_p/repeat_penalty/presence_penalty) matched by model-name patterns.

### 12.7 Persistence

LocalAI uses GORM with SQLite (`gorm.io/driver/sqlite`, WAL) by default and PostgreSQL (`gorm.io/driver/postgres`) for distributed/multi-user deployments. The auth DB defaults to `{DataPath}/database.db`. Agent collections, tasks, and jobs persist under the data path; vector knowledge bases default to the `chromem` engine.

---

## 13. Code Quality & Conventions

- **Language/build:** Go 1.26; native engines via purego/`dlopen` (no build-time CGo for Go backends); Rust backend via `tonic`; Python backends as standalone gRPC servers with pinned `grpcio`.
- **HTTP:** Echo v4 with an explicit, ordered middleware chain; SPA fallback serves the React UI for HTML 404s while returning JSON errors otherwise.
- **Config:** struct-tag-driven — kong tags (`env:`, `group:`, `default:`) for CLI, and dual `yaml:`/`json:` tags for model config, so the same structs serialize to YAML files and JSON API bodies.
- **Testing:** Ginkgo/Gomega specs across unit, integration, and e2e suites (`tests/e2e-aio`, `tests/e2e-backends`, `tests/e2e-ui`, `tests/e2e/distributed`), with testcontainers for NATS/Postgres. Face-recognition e2e fixtures download from public NASA images at runtime (or use `_FILE` env overrides for offline runs).
- **Observability-by-default:** structured `xlog`, OpenTelemetry metrics, optional tracing with bounded buffers, quiet-path log demotion.
- **Safety-by-default:** public-bind refusal without auth, CSRF on, security headers on every response, optional constant-time key comparison and opaque errors.

---

## 14. Potential Improvements

These are observations for operators and contributors, derived from the codebase — not defects.

- **Backend integrity defaults to warn-only.** `--require-backend-integrity` is `false` by default; integrity problems only warn. Production deployments should flip it on once the gallery `verification:` block is populated, and the default could move toward enforcement as galleries mature.
- **Topology leakage knobs.** `--expose-node-header` and `/metrics` reveal internal topology; they're correctly off/guardable but warrant explicit review in any public deployment.
- **Large surface area.** Five API dialects plus extensive management APIs (395 routes) is a broad attack/compatibility surface; per-dialect contract tests and clear deprecation of legacy unprefixed env aliases (e.g. `MODELS_PATH`) would reduce drift.
- **Deprecated flags linger.** `--single-active-backend` is superseded by `--max-active-backends=1`, and `--p2ptoken` by `--p2p-token`; a deprecation timeline would tidy the CLI.
- **On-demand backend downloads** make first-use latency unpredictable on slow links. Operators should pre-install backends or preload models; a documented "warm" step in orchestration manifests would help.
- **Known community issues** to track for stability: GLIBCXX mismatches on some CUDA backends (issue [#10056](https://github.com/mudler/LocalAI/issues/10056)), diffusers/SD regressions (issue [#9979](https://github.com/mudler/LocalAI/issues/9979)), embedding-dimension mismatches for some models (issue [#8721](https://github.com/mudler/LocalAI/issues/8721)), and container RAM misreporting (issue [#8059](https://github.com/mudler/LocalAI/issues/8059)).

---

## 15. Quick Reference

### Commands

| Command | Purpose |
|---|---|
| `local-ai run [models…]` | Run the server (default) |
| `local-ai models …` | Manage models |
| `local-ai backends list\|install\|upgrade\|uninstall` | Manage backends |
| `local-ai worker` | Distributed worker (generic) |
| `local-ai p2p-worker` | P2P worker (llama.cpp) |
| `local-ai agent-worker` | Distributed agent worker |
| `local-ai agent run\|list` | Standalone agents |
| `local-ai federated` | Federated mode |
| `local-ai explorer` | P2P explorer |
| `local-ai mcp-server` | stdio MCP control surface |
| `local-ai tts\|transcript\|sound-generation` | One-off media tasks |
| `local-ai completion bash\|zsh\|fish` | Shell completions |

### Operational endpoints

| Endpoint | Purpose |
|---|---|
| `GET /healthz`, `/readyz` | Liveness / readiness |
| `GET /metrics` | Prometheus metrics |
| `GET /version` | Version |
| `GET /system` | System info |
| `GET /api/backend-logs/:modelId` | Backend logs |
| `GET /backend/monitor` | Backend status |
| `GET /api/traces`, `/api/backend-traces` | Tracing |
| `GET /api/usage`, `/api/auth/usage` | Usage attribution |
| `GET /swagger/` | OpenAPI docs |
| `*/nodes/*`, `*/scheduling/*` | Distributed node & scheduling control |

### Critical env vars

| Variable | Purpose |
|---|---|
| `LOCALAI_ADDRESS` | Bind address |
| `LOCALAI_API_KEY` | Static API key(s) |
| `LOCALAI_AUTH` + DB/OIDC/GitHub vars | Full user auth |
| `LOCALAI_MODELS_PATH` / `LOCALAI_BACKENDS_PATH` / `LOCALAI_DATA_PATH` | Storage |
| `LOCALAI_DISTRIBUTED` + `LOCALAI_NATS_URL` + `LOCALAI_STORAGE_*` | Cluster mode |
| `LOCALAI_REQUIRE_BACKEND_INTEGRITY` | Enforce backend signatures |
| `LOCALAI_ALLOW_INSECURE_PUBLIC_BIND` | Override public-bind guard |
| `LOCALAI_MAX_ACTIVE_BACKENDS` | Cap loaded backends |
| `LOCALAI_WATCHDOG_IDLE` / `LOCALAI_MEMORY_RECLAIMER` | Reclaim resources |

### Default ports

| Port | Service |
|---|---|
| `8080` | LocalAI HTTP API + web UI |
| `50051`+ | Backend gRPC servers (internal, one per model) |
| `4222` | NATS (distributed mode, external) |
| `5432` | PostgreSQL (distributed/auth, external) |

---

_Generated June 2026 from mudler/LocalAI at 9f11b09. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._