# exo Admin & Architecture Guide

This guide is the deep reference for operators, architects, and developers running or extending **exo** — a distributed AI inference engine that turns a set of devices into a single cluster capable of serving language and image models too large for any one machine. It covers the system architecture, every component, configuration, deployment, security posture, operational procedures, and the internals needed to reason about (or replicate) the system.

For installation and day-to-day usage, see the companion **exo User Guide**.

> [!NOTE]
> exo is written in **Python 3.13** (orchestration, API, inference glue), **Rust** (networking and native bindings via PyO3), **Swift** (the macOS app), and **TypeScript/Svelte** (the dashboard). It is Apache-2.0 licensed and maintained by [exo labs](https://x.com/exolabs).

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [System Architecture](#2-system-architecture)
3. [Component Inventory](#3-component-inventory)
4. [The Distributed Model: Roles, Election & Event Sourcing](#4-the-distributed-model-roles-election--event-sourcing)
5. [Placement & Sharding Engine](#5-placement--sharding-engine)
6. [The Inference Pipeline (Worker → Runner)](#6-the-inference-pipeline-worker--runner)
7. [Networking Layer](#7-networking-layer)
8. [The API Server](#8-the-api-server)
9. [Model Cards](#9-model-cards)
10. [Configuration Reference](#10-configuration-reference)
11. [Filesystem Layout](#11-filesystem-layout)
12. [Requirements & Capacity Planning](#12-requirements--capacity-planning)
13. [Deployment](#13-deployment)
14. [Operations](#14-operations)
15. [Security Analysis](#15-security-analysis)
16. [Observability & Tracing](#16-observability--tracing)
17. [Performance Tuning](#17-performance-tuning)
18. [Code Quality & Conventions](#18-code-quality--conventions)
19. [Testing Strategy](#19-testing-strategy)
20. [Known Issues & Improvement Areas](#20-known-issues--improvement-areas)
21. [Building from Source & Packaging](#21-building-from-source--packaging)
22. [Quick Reference](#22-quick-reference)

---

## 1. Executive Summary

exo is a **peer-to-peer, masterless-by-default cluster** in which every node runs the same binary and can take on any role. At startup, nodes discover each other automatically over the local network, elect a coordinator ("master"), and continuously share a view of the cluster's topology, memory, and download state. When a user requests a model, the master computes a **placement** — how to shard the model across a connected ring of devices given their memory and link speeds — and the relevant workers spin up inference **runner** subprocesses that execute the model using Apple's MLX framework.

Key architectural properties:

- **No central server, no external dependencies.** No database, no message broker, no cloud. State is replicated via an append-only event log that flows over a zenoh pub/sub mesh.
- **Self-organizing.** Discovery, election, topology mapping, and re-placement on failure are automatic.
- **Topology-aware.** Placement considers per-link types (Thunderbolt/RDMA, Ethernet, Wi-Fi) and per-device memory.
- **Multi-protocol API.** A single FastAPI server exposes OpenAI Chat Completions, OpenAI Responses, Anthropic Claude Messages, and Ollama dialects, plus image generation/editing and cluster-management endpoints.
- **Heterogeneous performance backends.** A socket "ring" backend works anywhere; an RDMA-over-Thunderbolt "JACCL" backend delivers low-latency multi-device inference on Apple Silicon.

> [!IMPORTANT]
> exo is under heavy active development. The macOS (Apple Silicon, GPU) path is Tier 1 and the most mature. Linux runs on CPU. Several features (prefill/decode disaggregation, image models) are gated behind feature flags and off by default.

---

## 2. System Architecture

### 2.1 Single-node view

Every `exo` process constructs one `Node` (`src/exo/main.py`) that wires together a fixed set of long-running async services under a single anyio task group:

```text
                         ┌──────────────────────────────────────────────┐
                         │                  exo Node                     │
                         │                                               │
   network  ◀───────────┤  Router (zenoh)  ──  EventRouter (event bus)   │
   (other nodes)        │     │                      │                   │
                        │     │     ┌────────────────┼───────────────┐   │
                        │     │     │                │               │   │
                        │  Election │   Master   DownloadCoordinator │   │
                        │           │     │            │             │   │
                        │           │   Worker ── runner subprocs    │   │
                        │           │     │                          │   │
   HTTP clients ◀───────┤          API (FastAPI + dashboard) ────────┘   │
   :52415               │                                                │
                         └──────────────────────────────────────────────┘
```

The services created in `Node.create()`:

| Service | Always present? | Role |
|---------|-----------------|------|
| `Router` | Yes | zenoh transport: registers topics, sends/receives messages between nodes |
| `EventRouter` | Yes | Local event bus that fans events to/from the master and subscribers |
| `Election` | Yes | Participates in master election (every node, even non-candidates) |
| `DownloadCoordinator` | Unless `--no-downloads` | Coordinates model shard downloads |
| `Worker` | Unless `--no-worker` | Runs inference via runner subprocesses |
| `Master` | Yes (created on every node) | Active only on the elected master; orchestrates placement & tasks |
| `API` | Unless `--no-api` | FastAPI server + dashboard on `--api-port` |

Each node communicates over a set of named topics registered with the router: `GLOBAL_EVENTS`, `LOCAL_EVENTS`, `COMMANDS`, `ELECTION_MESSAGES`, `CONNECTION_MESSAGES`, `DOWNLOAD_COMMANDS`.

### 2.2 Cluster view

```text
   ┌─────────┐      zenoh mesh (pub/sub) + UDP discovery       ┌─────────┐
   │ Node A  │◀───────────────────────────────────────────────▶│ Node B  │
   │ (master)│                                                  │(worker) │
   └────┬────┘                                                  └────┬────┘
        │   global event log (append-only, replicated)               │
        │                                                            │
        │            ┌─────────┐                                     │
        └───────────▶│ Node C  │◀────────────────────────────────────┘
                     │(worker) │     RDMA/Thunderbolt or socket links
                     └─────────┘     carry model activations between shards
```

The master holds the authoritative event log; other nodes replicate it. Inference activations (the tensors passed between model shards) flow **directly between worker runners** over the chosen backend (ring sockets or RDMA), *not* through the event bus — the event bus carries control plane data (commands, state, tokens/chunks back to the API), while the data plane (model activations) goes peer-to-peer between runners.

### 2.3 Request lifecycle

```text
client → API.chat_completions()
       → adapter converts request → TextGenerationTaskParams
       → API._send(TextGeneration command) ──(COMMANDS topic)──▶ Master
       → Master._command_processor matches TextGeneration:
            picks least-busy instance for the model → emits TaskCreated event
       → event replicated to Workers
       → Worker.plan_step → plan() decides the task is runnable
            → Worker._start_runner_task → runner subprocess executes shard
       → runner emits ChunkGenerated events (tokens) back through the bus
       → API._apply_state routes chunks to the request's stream queue
       → API streams SSE tokens to the client
       → on completion, API emits TaskFinished → Master emits TaskDeleted
```

---

## 3. Component Inventory

### 3.1 Repository structure

```text
exo/
├── src/exo/                  # Python core
│   ├── main.py               # Entry point: Node, Args (CLI), main()
│   ├── api/                  # FastAPI server + protocol adapters
│   │   ├── main.py           #   class API, all routes
│   │   ├── adapters/         #   chat_completions, claude, ollama, responses
│   │   ├── types/            #   request/response pydantic models
│   │   └── keepalive.py      #   SSE keep-alive wrapper
│   ├── master/               # Cluster orchestration
│   │   ├── main.py           #   class Master, command/event processors
│   │   ├── placement.py      #   place_instance(), instance lifecycle
│   │   ├── placement_utils.py#   sharding, layer allocation, IP prioritization
│   │   └── image_store.py    #   ephemeral generated-image storage
│   ├── worker/               # Inference execution
│   │   ├── main.py           #   class Worker, plan_step
│   │   ├── plan.py           #   pure planning state machine
│   │   ├── runner/           #   runner supervisor + inference processes
│   │   ├── engines/          #   mlx (LLM) + image (FLUX/Qwen) engines
│   │   └── disaggregated/    #   prefill/decode disaggregation server
│   ├── routing/              # EventRouter, Router, topics
│   ├── download/             # DownloadCoordinator, shard downloader
│   ├── shared/               # Cross-cutting: constants, models, types, election, topology, apply
│   └── utils/                # channels, async_process, info_gatherer, banner, ports, tracing
├── rust/
│   ├── exo_rs/               # PyO3 bindings: Pidfile, networking handle
│   └── networking/           # zenoh-based discovery + swarm
├── app/EXO/                  # Swift macOS menu-bar app
├── dashboard/                # SvelteKit + Tailwind dashboard SPA
├── bench/                    # exo_bench, exo_eval, prefill_decode_bench, eval_tool_calls
├── resources/                # Built-in model cards (TOML)
│   ├── inference_model_cards/#   122 LLM cards
│   └── image_model_cards/    #   18 image-model cards
├── packaging/                # PyInstaller spec + dmg packaging
├── nix/, flake.nix           # Nix build
├── tools/                    # exo-tools helper package
└── docs/                     # api.md, benchmarks, images
```

### 3.2 Python module map (HIGH-priority)

| Module | Responsibility |
|--------|----------------|
| `src/exo/main.py` | `Node` assembly, `Args` CLI parsing, PID locking, election loop, signal handling |
| `src/exo/api/main.py` | `API` class: ~40 FastAPI routes, SSE streaming, image store, traces |
| `src/exo/master/main.py` | `Master`: command processor (match on command type), event processor, plan loop (kill dead instances, time out nodes), trace merging |
| `src/exo/master/placement.py` | `place_instance()`: cycle selection, memory filtering, sharding validation, backend selection |
| `src/exo/master/placement_utils.py` | Layer allocation (largest-remainder by memory), shard assignment for pipeline/CFG/tensor, IP prioritization, MLX ring/JACCL host matrices |
| `src/exo/worker/main.py` | `Worker`: plan step, runner creation, image-chunk assembly, connection polling, info gathering |
| `src/exo/worker/plan.py` | Pure `plan()` state machine: cancel → kill → create runner → download → connect → load → warmup → run |
| `src/exo/shared/models/model_cards.py` | `ModelCard`, card cache, HF fetch, tensor-support logic |
| `src/exo/utils/channels.py` | In-process and multiprocessing channels (anyio-based) |
| `src/exo/utils/async_process.py` | `AsyncProcess` wrapper around subprocess lifecycle |

### 3.3 Rust crates

| Crate | Purpose |
|-------|---------|
| `rust/networking` | zenoh-based discovery (`Discovery`, `Hello`/`WhatsUp` handshake) and swarm (`Swarm`, `ToSwarm`/`FromSwarm`) |
| `rust/exo_rs` | PyO3 extension module `exo_rs` exposed to Python: `Pidfile`/`PidfileError` (single-instance lock), `PyNetworkingHandle`, stub generation |

The Rust workspace (`Cargo.toml`) uses edition 2024, resolver 3, and a **patched fork of zenoh** (`github.com/evanev7/zenoh` branch `exo`) for all zenoh crates. The networking crate is built as a `cdylib` (for Python import) plus `rlib` (for stub-gen).

---

## 4. The Distributed Model: Roles, Election & Event Sourcing

### 4.1 Every node runs every role

A node is constructed with a `Master`, and (unless disabled) a `Worker`, `DownloadCoordinator`, and `API`. Only one node's `Master` is *active* at a time; the others sit dormant until election promotes them. This means any node can become the coordinator if the current one fails — there is no special "server."

### 4.2 Election

The `Election` service (`src/exo/shared/election.py`) runs on every node and exchanges `ELECTION_MESSAGES`. A node started with `--force-master` is given a huge `seniority` (1,000,000) so it wins elections; ordinary nodes have seniority 0. Election results stream into `Node._elect_loop()`, which reacts:

- **Becomes master:** if not already master, construct and start a `Master`.
- **Loses master:** shut down the local `Master`.
- **New master elected (any):** rebuild the `EventRouter` for the new session, recreate the `DownloadCoordinator` and `Worker`, and `reset()` or `unpause()` the `API`.

Each election produces a `SessionId` (master node id + election clock). Events are tagged with the session; the master discards events not from its own session, which prevents stale messages from a prior master from corrupting state.

> [!NOTE]
> The election loop is explicitly flagged in-code as entangled logic slated for refactoring. Operators should be aware that rapid master churn (e.g. a flapping network link) triggers repeated worker/API resets and is worth avoiding via stable connectivity.

### 4.3 Event sourcing

State is derived by folding an append-only **event log** through a pure `apply(state, event)` reducer (`src/exo/shared/apply.py`). The `Master` is the source of truth: it ingests local events, assigns them monotonic indices, persists them to a `DiskEventLog`, and re-broadcasts them as `GlobalForwarderEvent`s on the `GLOBAL_EVENTS` topic. Each node (API, Worker, Master) maintains its own `State` by applying the global event stream.

`State` includes: `topology`, `node_memory`, `node_network`, `node_backends`, `instances`, `tasks`, `downloads`, `instance_links`, `last_seen`, and more. Commands (intent) flow on the `COMMANDS` topic; events (facts) flow on the event topics. The `MultiSourceBuffer` and `OrderedBuffer` utilities reorder events from multiple origins by index before applying.

The `Master._plan()` loop runs every 10 seconds to reconcile reality against state: it deletes instances whose nodes have disconnected and emits `NodeTimedOut` for nodes not seen in 30 seconds.

> [!TIP]
> Because the filesystem-backed event log is the persistent memory, an operator debugging cluster behavior can inspect `/state` and `/events` on the API to see exactly what the cluster believes is true.

---

## 5. Placement & Sharding Engine

Placement (`src/exo/master/placement.py`, `place_instance()`) is the heart of exo's "automatic" behavior. Given a `PlaceInstance` command (model card + requested sharding + backend + min nodes), it selects which devices run the model and how.

### 5.1 Algorithm

1. **Find cycles.** Ask the topology for device cycles (rings) with at least `min_nodes` members. If `required_nodes` is specified, keep only cycles containing all of them.
2. **Memory filter.** Keep only cycles whose **combined** available RAM ≥ the model's `storage_size`. If none, raise `No cycles found with sufficient memory`.
3. **Sharding constraints:**
   - **Tensor** sharding requires `model_card.supports_tensor`; cycles must divide `hidden_size` evenly (and KV-head divisibility unless the model head-parallelises, e.g. DeepSeek V4). Otherwise raise a descriptive `ValueError`.
   - Specific guards exist: DeepSeek V3.1 (8-bit) can't use pipeline parallelism; Gemma 4 can't use pipeline parallelism (forced to single-node / tensor).
4. **Smallest cycles.** Prefer the cycles with the fewest nodes (less communication overhead).
5. **Backend feasibility.** Intersect the model's `backends` with the requested engine's required backends (`MlxRing` → Metal/CUDA/CPU; `MlxJaccl` → Metal only). Every node in a cycle must support a compatible backend, or raise.
6. **RDMA filter.** For `MlxJaccl`, keep only cycles that are RDMA-connected *and* have `rdma_ctl` enabled on every node; otherwise raise `Requested RDMA (MlxJaccl) but no RDMA-connected cycles available`.
7. **Leaf preference + selection.** Prefer cycles containing leaf nodes, then pick the cycle maximizing (download progress for the model, then total available memory) — i.e. favor devices that already have the model and the most free RAM.
8. **Single-node fallback.** A 1-node cycle is forced to `MlxRing` + `Pipeline`.
9. **Shard assignment + instance construction** (below).

### 5.2 Layer allocation

`allocate_layers_proportionally()` distributes the model's layers across nodes **proportional to each node's available RAM**, using the largest-remainder method, guaranteeing at least one layer per node. Each node's required memory for its layers is validated against availability before the instance is built (`Node N has insufficient memory…`).

### 5.3 Sharding strategies

| Strategy | Function | Description |
|----------|----------|-------------|
| **Pure pipeline** | `_get_shard_assignments_for_pure_pipeline` | Contiguous layer ranges per device; tokens flow around the ring |
| **CFG-parallel pipeline** | `_get_shard_assignments_for_cfg_parallel` | For classifier-free-guidance image models on an even ring ≥2: two mirror pipelines (positive/negative prompt), arranged so both last stages are neighbors |
| **Tensor** | `get_shard_assignments_for_tensor_parallel` | Every device holds all layers but a slice of each; requires `supports_tensor` |

Each assignment maps `node_to_runner` and `runner_to_shard` (with `ShardMetadata` carrying `device_rank`, `world_size`, `start_layer`, `end_layer`).

### 5.4 Instance types & connection wiring

- **`MlxRingInstance`** — built with `hosts_by_node`: for each node, a host list where self is `0.0.0.0:<ephemeral_port>`, left/right neighbors carry real IPs, and non-neighbors get an RFC-5737 placeholder (`198.51.100.1:0`). Neighbor IPs are chosen by `find_ip_prioritised(..., ring=True)`, which prefers **thunderbolt > ethernet > wifi**.
- **`MlxJacclInstance`** — built with a `jaccl_devices` all-to-all RDMA interface matrix and `jaccl_coordinators` (rank-0 coordinator address per node). Coordinator IP selection prefers **ethernet > wifi** (`ring=False`).

> [!IMPORTANT]
> The ring backend prioritizes Thunderbolt for the data path (lowest latency for neighbor exchange), while JACCL prioritizes Ethernet for the *coordinator* control channel. This split is intentional and reflects how each backend uses its links.

---

## 6. The Inference Pipeline (Worker → Runner)

### 6.1 The plan state machine

`Worker.plan_step()` calls the pure function `plan()` (`src/exo/worker/plan.py`) every 100 ms. `plan()` short-circuits through an ordered set of checks and returns the single next `Task` to perform (or `None`):

```text
_cancel_tasks          → cancel any task marked Cancelled
_kill_runner           → Shutdown runners for deleted/failed instances
_create_runner         → CreateRunner for instances assigned to this node
_model_needs_download  → DownloadModel when an idle runner lacks its shard
_init_distributed_backend → ConnectToGroup to form the ring/RDMA group
_load_model            → LoadModel once all shards are downloaded & connected
_ready_to_warmup       → StartWarmup once all ranks are loaded
_pending_tasks         → forward TextGeneration / ImageGeneration / ImageEdits
```

This produces a runner lifecycle: **idle → (download) → connecting → connected → loading → loaded → warming up → ready → running**. The handshake between ranks is carefully ordered (rank `n-1` initiates connection; rank 0 coordinates warmup) so all devices in an instance come up together.

### 6.2 Runner subprocesses

Workers don't run models in-process. `Worker._create_supervisor()` creates a `RunnerSupervisor` (`src/exo/worker/runner/supervisor.py`) which manages a separate **runner process** (started with multiprocessing `spawn`). This isolates the heavy MLX/Metal execution from the async control plane, so a crashed or hung runner doesn't take down the node. Runners are bounded by `EXO_MAX_INSTANCE_RETRIES` (5); an instance that exceeds it is requested for deletion.

### 6.3 Engines

- **MLX LLM engine** (`src/exo/worker/engines/mlx/`) — text generation on Apple Silicon via MLX and a forked `mlx-lm`. Supports the ring and JACCL distributed backends, continuous batching (unless `--no-batch`), and an optional KV prefix cache.
- **Image engines** (`src/exo/worker/engines/image/`) — FLUX and Qwen image models via a forked `mflux`, with model adapters (`ModelAdapter`, `PromptData`) and a generation pipeline. Gated behind `EXO_ENABLE_IMAGE_MODELS`.

### 6.4 Multimodal & chunked transfer

Images sent to the API (vision prompts or img2img) are SHA-256 hashed, deduplicated, and split into ≤512 KB chunks (`EXO_MAX_CHUNK_SIZE`) sent as `InputImageChunk`s; the worker reassembles them by hash into an image cache before running the task. Generated images stream back as `ImageChunk`s and are reassembled by the API.

### 6.5 Prefill/Decode disaggregation (experimental)

Behind `ENABLE_DISAGGREGATION`, exo can split inference into separate prefill and decode instances linked via `InstanceLink`s, with a dedicated disaggregation server (`src/exo/worker/disaggregated/server.py`). The master routes a decode task to a `prefill_endpoint` chosen from linked prefill instances by least in-flight load. This is off by default and the placement preview path for it is still TODO.

---

## 7. Networking Layer

### 7.1 Transport: zenoh

exo's `Router` (`src/exo/routing/router.py`) wraps **zenoh** (a patched fork) for pub/sub messaging between nodes. Topics are registered per concern (events, commands, election, downloads). The default zenoh TCP listen port is **52414** (`--zenoh-port`).

### 7.2 Discovery

A custom UDP discovery service (Rust, `rust/networking/src/discovery.rs`) runs on **52413** (`--discovery-port`). Nodes broadcast `Hello`/`WhatsUp` messages within a **namespace** (default = exo version, override with `--namespace` / `EXO_ZENOH_NAMESPACE`); only same-namespace nodes connect. This is how clusters auto-form and how clusters on a shared network stay isolated.

### 7.3 Topology mapping

Each `Worker._poll_connection_updates()` actively probes reachability (`check_reachable`) to other nodes every 10 s, emitting `TopologyEdgeCreated`/`TopologyEdgeDeleted` events as links appear and disappear. The `info_gatherer` collects each node's memory, network interfaces (with `interface_type`: thunderbolt/ethernet/wifi/unknown), backends, and RDMA control status, broadcast as `NodeGatheredInfo`. Together these build the `Topology` graph that placement consumes.

> [!NOTE]
> Connection types matter for placement. `interface_type` values drive the priority ordering in `find_ip_prioritised`. Mislabeled or undiscovered interfaces degrade placement quality, so accurate interface detection (and physical cabling) directly affects performance.

---

## 8. The API Server

The `API` class (`src/exo/api/main.py`) is a **FastAPI** app served by **hypercorn** bound to `0.0.0.0:<api-port>` (default 52415). The SvelteKit dashboard is mounted as static files at `/`.

### 8.1 Endpoint catalog

**Meta / state**

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/node_id` | Current node id |
| GET | `/state`, `/state/{path}` | Full cluster state (or a JSON subpath) |
| GET | `/events` | Replay the API's event log (debug) |
| GET | `/v1/feature-flags` | Returns `{disaggregation: bool}` |
| GET | `/onboarding`, POST `/onboarding` | Onboarding completion flag |

**Instance management**

| Method | Path | Purpose |
|--------|------|---------|
| POST | `/instance` | Create instance from explicit placement |
| POST | `/place_instance` | Place using server-side placement logic |
| GET | `/instance/placement` | Compute a placement (no create) |
| GET | `/instance/previews` | All valid placements for a model |
| GET | `/instance/await` | SSE wait until instance ready (≤300 s) |
| GET / DELETE | `/instance/{id}` | Inspect / delete instance |

**Instance links (disaggregation, feature-gated)**

| Method | Path |
|--------|------|
| GET/POST | `/v1/instance-links` |
| PUT/DELETE | `/v1/instance-links/{link_id}` |

**Models**

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/models`, `/v1/models` | List models (`?status=downloaded`) |
| POST | `/models/add` | Add custom model from HF |
| DELETE | `/models/custom/{model_id}` | Remove custom model |
| GET | `/models/search` | Search HF (mlx-community first) |

**Inference**

| Method | Path | Dialect |
|--------|------|---------|
| POST | `/v1/chat/completions`, `/ollama/v1/chat/completions` | OpenAI chat |
| POST | `/v1/messages` | Claude Messages |
| POST | `/v1/responses` | OpenAI Responses |
| POST | `/ollama/api/chat` (+ aliases), `/ollama/api/generate` | Ollama |
| GET | `/ollama/api/tags`, `/ollama/api/ps`, POST `/ollama/api/show`, GET `/ollama/api/version` | Ollama meta |
| POST | `/bench/chat/completions` | Chat + perf stats |
| POST | `/v1/cancel/{command_id}` | Cancel a generation |

**Images** (feature-gated by `EXO_ENABLE_IMAGE_MODELS`)

| Method | Path |
|--------|------|
| POST | `/v1/images/generations`, `/bench/images/generations` |
| POST | `/v1/images/edits`, `/bench/images/edits` |
| GET | `/images`, `/images/{image_id}` |

**Downloads & traces**

| Method | Path |
|--------|------|
| POST | `/download/start`, `/download/cancel` |
| DELETE | `/download/{node_id}/{model_id}` |
| GET | `/v1/traces`, `/v1/traces/{task_id}`, `/v1/traces/{task_id}/stats`, `/v1/traces/{task_id}/raw` |
| POST | `/v1/traces/delete` |

### 8.2 Request flow internals

Each inference adapter converts the dialect-specific request into a common `TextGenerationTaskParams` (or image task params), validates that an instance exists for the model (`_validate_model_has_instance`, 404 otherwise), sends a command to the master, and returns a `StreamingResponse`. Streaming text uses SSE (`text/event-stream`) wrapped in `with_sse_keepalive` (a 10 s keep-alive comment prevents idle disconnects); Ollama streaming uses NDJSON. Tokens arrive as `ChunkGenerated` events routed by `command_id` into per-request channels in `_apply_state`.

### 8.3 Concurrency & lifecycle

`EXO_MAX_CONCURRENT_REQUESTS` (default 8) bounds in-flight work. During an election, the API **pauses** (`_pause_on_new_election`): `_send` blocks until unpaused, so requests aren't lost mid-transition. On master change the API is `reset()` (fresh state + event log) or `unpause()`d. The image store is cleaned of expired entries every 5 minutes; generated images expire after ~1 hour.

> [!CAUTION]
> CORS is configured wide open: `allow_origins=["*"]`, `allow_credentials=True`, all methods and headers. Combined with the bind to `0.0.0.0` and **no authentication**, this means anyone who can reach the API port can use (and manage) the cluster. See [Security](#15-security-analysis).

---

## 9. Model Cards

Models are described by **TOML model cards** that define metadata and capabilities. exo loads cards from:

- `resources/inference_model_cards/` — built-in LLM cards (122 shipped)
- `resources/image_model_cards/` — built-in image-model cards (18 shipped)
- `~/.local/share/exo/custom_model_cards/` (`EXO_CUSTOM_MODEL_CARDS_DIR`) — user-added cards, synced across the cluster

### 9.1 Card schema (example)

```toml
model_id = "mlx-community/Llama-3.2-1B-Instruct-4bit"
n_layers = 16
hidden_size = 2048
num_key_value_heads = 8
supports_tensor = true
tasks = ["TextGeneration"]
family = "llama"
quantization = "4bit"
base_model = "Llama 3.2 1B"
capabilities = ["text"]
context_length = 131072
backends = ["MlxMetal", "MlxCuda", "MlxCpu"]

[storage_size]
in_bytes = 729808896

[sampling_defaults]
temperature = 0.6
top_p = 0.9
```

| Field | Meaning |
|-------|---------|
| `model_id` | HuggingFace repo id (also the API model name) |
| `n_layers`, `hidden_size`, `num_key_value_heads` | Architecture; drive layer allocation and tensor-sharding feasibility |
| `supports_tensor` | Whether tensor parallelism is allowed |
| `tasks` | e.g. `TextGeneration`, image tasks |
| `backends` | Which execution backends can run it (`MlxMetal`/`MlxCuda`/`MlxCpu`) |
| `storage_size.in_bytes` | Used for memory-fit checks during placement |
| `sampling_defaults` | Applied when the request omits sampling params |
| `vision`, `capabilities`, `context_length`, `reasoning_dialect` | Optional capabilities surfaced through `/models` and Ollama `show` |

### 9.2 Adding cards

To add a model, drop a TOML card in the appropriate directory, or use `POST /models/add` with a HuggingFace id — exo fetches the config via `ModelCard.fetch_from_hf`, writes a custom card, and syncs it cluster-wide (the worker's `_reconcile_custom_cards` loop keeps the on-disk cache in step with cluster state).

---

## 10. Configuration Reference

exo is configured through **CLI flags** and **environment variables** — there is no required config file (a `config.toml` location exists at `~/.config/exo/config.toml` but the runtime is driven primarily by flags/env).

### 10.1 CLI flags

| Flag | Default | Effect |
|------|---------|--------|
| `-q/--quiet`, `-v/--verbose` | verbosity 0 | Logging level (count) |
| `-m/--force-master` | off | Win elections (seniority 1,000,000) |
| `--no-api` | API on | Skip API/dashboard |
| `--api-port` | 52415 | API/dashboard port |
| `--no-worker` | worker on | Coordinator-only node |
| `--no-downloads` | downloads on | No download coordinator |
| `--offline` | off | No internet; local models only |
| `--no-batch` | batching on | Sequential generation (sets `EXO_NO_BATCH=1`) |
| `--legacy-daemon` | off | Double-fork daemonization |
| `--namespace` | exo version | Discovery namespace |
| `--zenoh-port` | 52414 | zenoh TCP port |
| `--discovery-port` | 52413 | discovery UDP port |
| `--fast-synch` / `--no-fast-synch` | auto | Force MLX `FAST_SYNCH` (sets `EXO_FAST_SYNCH`) |

> [!WARNING]
> `--bootstrap-peers` is **disabled** — it raises `Bootstrap peers has been temporarily removed`. Discovery is automatic.

### 10.2 Environment variables (complete)

| Variable | Default | Purpose |
|----------|---------|---------|
| `EXO_HOME` | unset | Override base of all state dirs (relative to `$HOME`) |
| `XDG_CONFIG_HOME`/`XDG_DATA_HOME`/`XDG_CACHE_HOME` | XDG defaults | Linux base dirs (non-Linux defaults to `~/.exo`) |
| `EXO_DEFAULT_MODELS_DIR` | `<data>/models` | Primary writable model dir |
| `EXO_MODELS_DIRS` | none | Colon-separated extra writable model dirs |
| `EXO_MODELS_READ_ONLY_DIRS` | none | Colon-separated read-only model dirs (NFS, shared) |
| `EXO_RESOURCES_DIR` | auto-detected | Override resources path |
| `EXO_DASHBOARD_DIR` | auto-detected | Override dashboard assets path |
| `EXO_OFFLINE` | `false` | Offline mode |
| `EXO_ENABLE_IMAGE_MODELS` | `false` | Enable image models |
| `EXO_TRACING_ENABLED` | `false` | Enable distributed tracing |
| `EXO_FAST_SYNCH` | auto | MLX `MLX_METAL_FAST_SYNCH` control |
| `EXO_ZENOH_NAMESPACE` | exo version | Cluster namespace (replaces `EXO_LIBP2P_NAMESPACE`) |
| `ENABLE_DISAGGREGATION` | `false` | Prefill/decode disaggregation |
| `EXO_MAX_CONCURRENT_REQUESTS` | `8` | Concurrency cap |
| `EXO_NO_BATCH` | unset | Set by `--no-batch` |

Internal constants (not env-tunable): `EXO_MAX_CHUNK_SIZE` = 512 KB, `EXO_MAX_INSTANCE_RETRIES` = 5.

> [!CAUTION]
> Setting `EXO_LIBP2P_NAMESPACE` (the old name) causes exo to **fail fast** at startup with `EXO_LIBP2P_NAMESPACE has been removed - use EXO_ZENOH_NAMESPACE instead`. Update any automation that still sets it.

---

## 11. Filesystem Layout

exo follows the XDG Base Directory spec on Linux; on macOS everything roots at `~/.exo`. `EXO_HOME` overrides the base.

| Path (Linux default) | Constant | Contents |
|----------------------|----------|----------|
| `~/.local/share/exo/models` | `EXO_DEFAULT_MODELS_DIR` | Downloaded model weights |
| `~/.local/share/exo/custom_model_cards/` | `EXO_CUSTOM_MODEL_CARDS_DIR` | User-added cards |
| `~/.local/share/exo/event_log/` | `EXO_EVENT_LOG_DIR` | Master/API on-disk event logs |
| `~/.config/exo/config.toml` | `EXO_CONFIG_FILE` | Optional config |
| `~/.cache/exo/exo_log/exo.log` | `EXO_LOG` | Main log (rotated) |
| `~/.cache/exo/exo_log/runner_log/` | `EXO_RUNNER_LOG_DIR` | Runner stdout/stderr |
| `~/.cache/exo/exo.pid` | `EXO_PID_FILE` | Single-instance lock |
| `~/.cache/exo/node_zid` | `EXO_NODE_ZID` | Persistent node identity |
| `~/.cache/exo/images/` | `EXO_IMAGE_CACHE_DIR` | Generated images (TTL ~1h) |
| `~/.cache/exo/traces/` | `EXO_TRACING_CACHE_DIR` | Trace JSON files |

On macOS these live under `~/.exo/` (config/data/cache collapse to one root).

---

## 12. Requirements & Capacity Planning

### 12.1 Platform support tiers

| Tier | Platforms |
|------|-----------|
| **1 (tested/maintained)** | Apple Silicon macOS — Mac Studio M3 Ultra, Mac Mini M4 Pro, MacBook Pro M5/M4 Max |
| **Planned** | Linux CUDA (e.g. Nvidia DGX Spark), Linux CPU, Linux Vulkan, eventually Windows |

Today: macOS uses the GPU (Metal); Linux runs on CPU.

### 12.2 Sizing a cluster

The binding constraint is **combined available RAM ≥ model `storage_size`**. Plan as follows:

1. Find the target model's `storage_size.in_bytes` (its card or `/models`).
2. Ensure the sum of `ram_available` across the devices you intend to use exceeds it, with headroom for the KV cache and activations.
3. For best throughput, keep the cycle **small** (exo prefers fewest nodes) and **fast-linked** (Thunderbolt/RDMA).
4. For tensor parallelism, verify the model supports it and `hidden_size` divides evenly across the node count.

> [!TIP]
> The benchmark `2 devices ≈ 1.8×` and `4 devices ≈ 3.2×` (tensor parallel) is a useful planning heuristic: scaling is sublinear, so add devices for *capacity* (bigger models) first and *speed* second.

### 12.3 macOS memory tuning

For large models that fit in RAM, raising the wired (GPU-accessible) memory limit helps:

```bash
sudo sysctl iogpu.wired_limit_mb=N
```

Choose `N` larger than the model size in MB but smaller than total RAM.

---

## 13. Deployment

exo is not a traditional service you deploy to a server — it's a process you run on each device. Three patterns:

### 13.1 Foreground (recommended for servers)

Run under a supervisor that keeps it in the foreground:

=== "systemd (Linux)"

    ```ini
    # /etc/systemd/system/exo.service
    [Unit]
    Description=exo node
    After=network-online.target
    Wants=network-online.target

    [Service]
    Type=simple
    User=exo
    WorkingDirectory=/opt/exo
    ExecStart=/usr/bin/uv run exo
    Restart=on-failure
    Environment=EXO_ENABLE_IMAGE_MODELS=false

    [Install]
    WantedBy=multi-user.target
    ```

    > [!IMPORTANT]
    > Do **not** combine systemd/launchd with `--legacy-daemon`. The daemon flag double-forks; supervisors expect a foreground process. Use foreground mode and let the supervisor manage lifecycle.

=== "launchd (macOS)"

    Use a standard `launchd` plist invoking `uv run exo` in the foreground, or simply install the **macOS app**, which manages background execution and the network profile for you.

=== "Legacy init"

    ```bash
    uv run exo --legacy-daemon
    ```

    Only for SysV-style init scripts.

### 13.2 macOS app

`brew install --cask exo` or the [.dmg](https://assets.exolabs.net/EXO-latest.dmg) installs a menu-bar app (macOS 26.2+). It requests permission to install a **network LaunchDaemon** and configuration profile (for discovery and RDMA). Uninstall cleanly via **menu bar → Advanced → Uninstall**, or run `sudo ./app/EXO/uninstall-exo.sh`. (Remember to remove EXO from **Login Items** manually.)

### 13.3 Multi-node rollout

1. Deploy exo to each device (same version — namespace defaults to the version string, so mismatched versions won't cluster).
2. Cable devices (Thunderbolt 5 for RDMA) and put them on one network.
3. Start exo on each. They self-discover and elect.
4. Optionally designate coordinator-only nodes with `--no-worker`.
5. Pin a master with `--force-master` on a stable, well-connected device if you want deterministic coordination.

> [!WARNING]
> Version skew breaks clustering by default because the discovery **namespace defaults to the exo version**. Roll out the same version everywhere, or set an explicit shared `--namespace`.

---

## 14. Operations

### 14.1 Health & inspection

```bash
# Cluster's belief about itself
curl http://localhost:52415/state | jq

# Current master
curl http://localhost:52415/node_id

# Running instances (Ollama-style)
curl http://localhost:52415/ollama/api/ps

# Feature flags in effect
curl http://localhost:52415/v1/feature-flags
```

### 14.2 Managing model lifecycle

```bash
# Start a download to a specific node
curl -X POST http://localhost:52415/download/start -H 'Content-Type: application/json' -d '{...}'

# Cancel / delete a download
curl -X POST http://localhost:52415/download/cancel -H 'Content-Type: application/json' -d '{...}'
curl -X DELETE "http://localhost:52415/download/NODE_ID/mlx-community/My-Model"

# Tear down an instance to reclaim memory
curl -X DELETE http://localhost:52415/instance/INSTANCE_ID
```

### 14.3 Logs

| What | Where |
|------|-------|
| Main process log (rotated) | `~/.cache/exo/exo_log/exo.log` (macOS: `~/.exo/...`) |
| Runner stdout/stderr | `~/.cache/exo/exo_log/runner_log/` |

Increase verbosity with `-v`/`-vv` at launch. hypercorn access logging is disabled; error logging goes to stderr through a custom `InterceptLogger`.

### 14.4 Backup & recovery

- **Models** are large but reproducible (re-downloadable). Back them up only to save bandwidth, or stage them on a shared `EXO_MODELS_READ_ONLY_DIRS` mount.
- **Event logs** under `event_log/` are operational state; they're rebuilt as the cluster runs and generally don't need backup.
- **Custom model cards** under `custom_model_cards/` *are* worth backing up if you've curated many.

### 14.5 Graceful shutdown

`SIGINT`/`SIGTERM` trigger `Node.shutdown()`, which cancels the task group; a second signal forces `sys.exit(1)`. Runners are shut down with their supervisors. The PID file is always released in a `finally` block.

---

## 15. Security Analysis

> [!CAUTION]
> **exo exposes an unauthenticated, CORS-open control plane on `0.0.0.0`. Treat the API port as fully trusted-network-only.** Anyone who can reach `:52415` can run inference, add/delete models, create/delete instances, and read cluster state. There is no API key, token, or auth middleware.

### 15.1 Threat model & exposure

| Surface | Exposure | Notes |
|---------|----------|-------|
| API `:52415` | `0.0.0.0`, no auth, CORS `*` with credentials | Restrict via host firewall / private network / VPN |
| zenoh `:52414` (TCP) | LAN | Cluster control + data plane |
| Discovery `:52413` (UDP) | LAN broadcast | Namespace-scoped; not authenticated |
| Dashboard | served at `/` | Same exposure as the API |

**Recommendations:**
- Bind exo to a trusted network only; never expose `:52415` to the public internet. Put it behind a firewall, reverse proxy with auth, or VPN/Tailscale if remote access is needed.
- Use a non-default `--namespace` to avoid accidental cluster joins and to compartmentalize.
- Run exo as a non-privileged user (the macOS app needs elevated rights only to install its network profile/RDMA config).

### 15.2 Remote code execution via models

`trust_remote_code` is **disabled by default**. Custom models that require it execute repository code on the worker. Only enable it for fully trusted models — this is the single highest-impact RCE vector in normal use.

### 15.3 Path-traversal hardening

Trace endpoints resolve task ids against `EXO_TRACING_CACHE_DIR` and reject paths that escape it (`Invalid task ID`), preventing arbitrary file reads via `../`. This is the correct pattern; any new file-serving endpoint should follow it.

### 15.4 Secrets hygiene

> [!WARNING]
> Issue **#2116** ("Hardcoded AWS credentials in Xcode scheme file") was reported and closed. Operators building the macOS app from source should verify no credentials are baked into Xcode scheme files or build artifacts, and scrub any local clones accordingly. Never commit credentials to scheme/plist files.

### 15.5 Data locality

A genuine security *benefit*: all inference runs on your own hardware. Prompts and outputs do not leave your network (in `--offline` mode, not even model downloads). This is a core reason to deploy exo over a hosted API for sensitive workloads.

---

## 16. Observability & Tracing

Enable distributed tracing with `EXO_TRACING_ENABLED=true`. When on, runners emit per-rank `TraceEventData`; the master collects traces from all expected ranks for a task, merges them (`_merge_and_save_traces`), and the API persists them to `~/.cache/exo/traces/trace_<task_id>.json`.

Trace API:

| Endpoint | Returns |
|----------|---------|
| `GET /v1/traces` | List of trace files (task id, created, size) |
| `GET /v1/traces/{task_id}` | Raw trace events (name, start_us, duration_us, rank, category) |
| `GET /v1/traces/{task_id}/stats` | Aggregated stats by category and by rank (total/min/max/avg µs) |
| `GET /v1/traces/{task_id}/raw` | The raw JSON file download |
| `POST /v1/traces/delete` | Delete traces by id |

The `/bench/*` endpoints additionally return generation stats (prompt/generation tokens-per-second, peak memory) and power usage sampled via `PowerSampler`, which on Apple Silicon draws on `macmon`.

> [!TIP]
> Traces use a Chrome-trace-like schema (name/start/duration/category/rank), so merged traces can be visualized to find per-rank bottlenecks in a distributed run.

---

## 17. Performance Tuning

| Lever | How | Effect |
|-------|-----|--------|
| **Faster links** | Thunderbolt 5 + RDMA (`MlxJaccl`) | Largest latency win for multi-device |
| **Tensor parallelism** | Model with `supports_tensor` | ~1.8× (2 dev) / ~3.2× (4 dev) |
| **Smaller cycles** | Fewer, beefier nodes | Less inter-shard communication |
| **Continuous batching** | default on (avoid `--no-batch`) | Higher throughput under concurrency |
| **KV prefix cache** | `--use-prefix-cache` in bench / engine | Faster warm prompts |
| **Wired memory limit** | `sudo sysctl iogpu.wired_limit_mb=N` | Speeds large in-RAM models on macOS |
| **`EXO_MAX_CONCURRENT_REQUESTS`** | raise/lower | Tune concurrency vs. memory pressure |
| **`--fast-synch`** | force MLX FAST_SYNCH | JACCL backend tuning |
| **Concurrency** | `--concurrency` in `exo_bench.py` | Measure the throughput knee |

Use `bench/exo_bench.py` to empirically find the best (sharding, backend, node-count) tuple for a model on your hardware before committing it to production placement.

> [!NOTE]
> Known performance pitfalls from the issue tracker: a pipeline-parallel deadlock at 32+ concurrent requests (all_gather blocking), some MacBook Pro pairings running at ~½ expected RDMA speed, and a vision cache key that omits image dimensions. Track these in the upstream issues if they affect you.

---

## 18. Code Quality & Conventions

exo enforces unusually strict standards (see `Cargo.toml` lints, `CONTRIBUTING.md`, `RULES.md`):

- **Language choice:** "prefer Rust unless there's a good reason otherwise." Rust handles networking and performance-critical paths; Python handles orchestration and the API; Swift the app; TypeScript the dashboard.
- **Rust lints:** Clippy `correctness = deny`; `pedantic`, `nursery`, `cargo`, and a large `restriction` set at `warn` (e.g. `unwrap_used`, `panic`, `indexing_slicing`, `arithmetic_side_effects`, `expect_used`, `as_conversions`). `disallowed_script_idents` and `self_named_module_files` are `deny`.
- **Python:** full type hints with `basedpyright`, formatted by `ruff`, Pydantic models everywhere (`FrozenModel` base for immutability).
- **Architecture style:** event sourcing with a pure `apply()` reducer and pure planning functions (`plan()`), favoring testable pure functions. The codebase candidly marks "cracks in the event-sourcing architecture" (e.g. the master/node plan loops) for future refactoring.
- **Comments** explain *why*, not *what*.

Common developer commands (`justfile`):

```bash
just fmt           # treefmt / nix fmt
just lint          # ruff check --fix
just check         # basedpyright type check
just test          # pytest src
just sync          # uv sync --all-packages
just rust-rebuild  # regenerate stubs + rebuild exo_rs
just build-dashboard
just package       # PyInstaller bundle
just build-app     # full macOS app build
```

---

## 19. Testing Strategy

Tests live alongside code (`src/exo/**/tests/`) and in top-level `tests/`. Coverage spans:

- **Placement & sharding** (`src/exo/master/tests`) — layer allocation, cycle selection, shard assignment correctness.
- **Worker plan/runner** (`src/exo/worker/tests/unittests/` — `test_plan`, `test_runner`, `test_download`, `test_image`, `test_mlx`).
- **API** (`src/exo/api/tests`) and **shared apply/types** (`src/exo/shared/tests`).
- **Utilities** (`src/exo/utils/tests`) — channels, info gatherer.
- **Rust** (`rust/exo_rs/tests`, `rust/networking/examples`).
- **App** (`app/EXO/EXOTests`, `EXOUITests`) and **eval/bench** harnesses.

Run with `just test` (`uv run pytest src`) or `uv run pytest tests`. The macOS app and dashboard have their own test targets.

> [!NOTE]
> The `tmp/old_tests/` directory contains legacy tests (e.g. `test_vision_cache.py`) kept for reference; the canonical suites are under each package's `tests/`.

---

## 20. Known Issues & Improvement Areas

Drawn from the codebase's own TODOs and the community issue tracker (as of commit `09f9ea3`):

**Open bugs to be aware of operationally:**

- **#2151** Vision feature cache key omits image dimensions → wrong inference on shape-colliding images.
- **#1973** RDMA ports stop working after idle.
- **#2070** Some MacBook Pro pairings run at ~½ expected RDMA speed.
- **#2136** `DownloadPending` entries for retired node ids are never GC'd → infinite `_delete_download` loop survives restarts.
- **#2120** Runner process can disappear while the instance stays registered → unusable instance.
- **#2141** Prefill step size should be 1024, not 4096.
- **#2108** Pipeline-parallel deadlock at 32+ concurrent requests (all_gather blocks).
- **#2122** Claude Messages API: `stop_sequences` reporting and multi-token stop-sequence leakage.

**Architectural TODOs flagged in-source:**

- The `Node._elect_loop` and master/worker plan loops are explicitly marked as entangled / "cracks in the event-sourcing architecture" awaiting refactor — more behaviors should become commands.
- Shard assignments should carry rank information directly (the JACCL coordinator code works around this).
- Prefill/decode disaggregation placement previews are unimplemented (TODO in `get_placement_previews`).
- Tensor-parallel feasibility check (`hidden_size % n == 0`, KV-head divisibility) is acknowledged as "good enough for now," not fully correct.
- User-download notification on missing models is a logged TODO, not yet a real notification.

**Improvement opportunities:**

- Add authentication / authorization to the API for any non-isolated deployment.
- GC retired-node download entries (#2136).
- Surface a placement-preview path for disaggregation.

---

## 21. Building from Source & Packaging

### 21.1 Toolchain

- Python 3.13, `uv` for dependency management (workspace members: `rust/exo_rs`, `bench`, `tools`).
- Rust **nightly** (edition 2024) for the native crates; built via PyO3 into the importable `exo_rs` module.
- Node for the dashboard (SvelteKit + Vite + Tailwind v4).
- macOS: Xcode (Metal toolchain), Homebrew, pinned `macmon` fork.

### 21.2 Dependency notes

The MLX stack is pinned to **forks** for distributed/RDMA fixes: `mlx==0.32.0`, `mlx-lm` (fork `leo/deepseek-v4`), `mflux` (fork `exo2`), plus platform-specific CUDA/CPU MLX wheels for Linux. Optional extras select the backend: `exo[mlx]` (Metal), `exo[mlx-cpu]`, `exo[mlx-cuda12]`, `exo[mlx-cuda13]`. The zenoh stack is a patched fork across ~25 crates.

### 21.3 Build steps

```bash
just build-dashboard          # npm install + build (required before running)
just rust-rebuild             # stub-gen + rebuild exo_rs
just package                  # PyInstaller bundle (packaging/pyinstaller/exo.spec)
just build-app                # full macOS .app (xcodebuild)
```

Packaging artifacts: a PyInstaller bundle, a DMG (`packaging/dmg/`), and the Xcode-built `.app`. Nix users get a reproducible build via `flake.nix` (`nix run .#exo`).

---

## 22. Quick Reference

### Ports

| Port | Proto | Purpose | Flag |
|------|-------|---------|------|
| 52415 | TCP | API + dashboard | `--api-port` |
| 52414 | TCP | zenoh transport | `--zenoh-port` |
| 52413 | UDP | discovery | `--discovery-port` |

### Backends & sharding

| Concept | Values |
|---------|--------|
| Sharding | `Pipeline`, `Tensor` |
| Instance meta (backend) | `MlxRing` (Metal/CUDA/CPU), `MlxJaccl` (Metal + RDMA) |
| Link priority (ring) | thunderbolt > ethernet > wifi > unknown |
| Link priority (RDMA coordinator) | ethernet > wifi > unknown > thunderbolt |

### Key constants

| Constant | Value |
|----------|-------|
| `EXO_MAX_CHUNK_SIZE` | 512 KB |
| `EXO_MAX_INSTANCE_RETRIES` | 5 |
| `EXO_MAX_CONCURRENT_REQUESTS` | 8 (env-tunable) |
| Image TTL | ~3600 s |
| Node timeout | 30 s unseen |
| Plan reconcile interval | 10 s (master), 0.1 s (worker) |

### Critical operational rules

- Run the same exo version on all nodes (namespace = version by default).
- Never expose `:52415` to untrusted networks — no auth, CORS open.
- Don't pair `--legacy-daemon` with systemd/launchd.
- Build the dashboard before running from source.
- Only enable `trust_remote_code` for trusted models.

### Links

- Source: <https://github.com/exo-explore/exo>
- API reference: `docs/api.md`
- Platform roadmap: `PLATFORMS.md`
- Contributing: `CONTRIBUTING.md` · Discord: <https://discord.gg/TJ4P57arEm>
- License: Apache-2.0

---

_Generated June 2026 from exo-explore/exo at 09f9ea3. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._