# exo User Guide

**exo** connects all of your devices into a single AI cluster so you can run frontier-scale models — large language models *and* image models — across your own hardware. Instead of being limited to what fits on one machine, exo pools the memory and compute of every Mac (and, on CPU, Linux box) you own, automatically discovers them on the network, figures out the best way to split a model across them, and serves the result through APIs you already know: OpenAI, Claude, Ollama, and the OpenAI Responses API.

This guide shows you how to install exo, start a cluster, load models, and run inference — from a single laptop to a multi-node Thunderbolt cluster.

> [!NOTE]
> exo is built and maintained by [exo labs](https://x.com/exolabs). On macOS it runs on the GPU via Apple's MLX framework. On Linux it currently runs on CPU (GPU support is in development). Models run faster as you add more devices, especially with RDMA over Thunderbolt 5.

---

## Table of Contents

1. [Quick Start](#1-quick-start)
2. [Requirements](#2-requirements)
3. [Installation](#3-installation)
4. [Running exo](#4-running-exo)
5. [Using the Dashboard](#5-using-the-dashboard)
6. [Loading and Running a Model (the API Workflow)](#6-loading-and-running-a-model-the-api-workflow)
7. [Everyday Usage: Talking to Your Cluster](#7-everyday-usage-talking-to-your-cluster)
8. [Working with Models](#8-working-with-models)
9. [Image Generation and Editing](#9-image-generation-and-editing)
10. [Building a Multi-Device Cluster](#10-building-a-multi-device-cluster)
11. [Enabling RDMA over Thunderbolt (macOS)](#11-enabling-rdma-over-thunderbolt-macos)
12. [Command-Line Reference](#12-command-line-reference)
13. [Environment Variables](#13-environment-variables)
14. [Benchmarking Your Cluster](#14-benchmarking-your-cluster)
15. [How It Works (Concepts)](#15-how-it-works-concepts)
16. [Troubleshooting](#16-troubleshooting)
17. [FAQ](#17-faq)
18. [Reference](#18-reference)

---

## 1. Quick Start

If you have [Nix](https://nixos.org/) on a Mac, you can launch exo in one command:

```bash
nix run .#exo
```

Otherwise, the source workflow is three steps — clone, build the dashboard, run:

```bash
# Clone exo
git clone https://github.com/exo-explore/exo
cd exo

# Build the dashboard
cd dashboard && npm install && npm run build && cd ..

# Run exo
uv run exo
```

When exo starts, the dashboard and API are available at:

```text
http://localhost:52415/
```

Open that URL in a browser. Every device running exo serves its own copy of the dashboard and API on this port, and devices on the same network discover each other automatically — no manual configuration needed.

> [!TIP]
> Start with a small model such as `mlx-community/Llama-3.2-1B-Instruct-4bit` to confirm everything works before loading anything large.

---

## 2. Requirements

### Supported platforms

| Platform | Acceleration | Status |
|----------|-------------|--------|
| Apple Silicon macOS (M-series) | GPU (Metal via MLX) | Tier 1 — tested and maintained |
| Linux (x86_64 / aarch64) | CPU | Runs; GPU support under development |
| Windows | — | Planned, not yet supported |

Tier 1 hardware that the project actively tests includes the Mac Studio (M3 Ultra), Mac Mini (M4 Pro), and MacBook Pro (M4 Max / M5).

> [!IMPORTANT]
> The macOS background app requires **macOS Tahoe 26.2 or later**. RDMA over Thunderbolt is a macOS 26.2 capability and works on Macs with Thunderbolt 5 (M4 Pro Mac Mini, M4 Max Mac Studio, M4 Max MacBook Pro, M3 Ultra Mac Studio).

### Software prerequisites (run-from-source)

- **Python 3.13** (exo requires `==3.13.*`)
- [**uv**](https://github.com/astral-sh/uv) — Python dependency management
- [**node**](https://github.com/nodejs/node) — builds the dashboard (Linux: v18+)
- [**rust**](https://github.com/rust-lang/rustup) — builds the native bindings (nightly toolchain)
- **macOS only:** [Xcode](https://developer.apple.com/xcode/) (Metal toolchain for MLX), [Homebrew](https://brew.sh), and [macmon](https://github.com/vladkens/macmon) (hardware monitoring on Apple Silicon)

### Memory

exo's whole purpose is to run models that don't fit on one device, so "requirements" scale with the model. The cluster needs enough **combined** RAM across participating nodes to hold the model. exo computes this for you and refuses placements that don't fit (see [Troubleshooting](#16-troubleshooting)).

---

## 3. Installation

exo offers three installation paths. Choose the one that matches your setup.

=== "macOS — Run from Source"

    Install the prerequisites:

    ```bash
    # Homebrew (if you don't have it)
    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

    # uv and node
    brew install uv node

    # Rust (nightly)
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
    rustup toolchain install nightly

    # macmon — install the pinned fork revision (Homebrew's macmon 0.6.1 crashes on M5)
    cargo install --git https://github.com/vladkens/macmon \
      --rev a1cd06b6cc0d5e61db24fd8832e74cd992097a7d \
      macmon --force
    ```

    Then clone, build the dashboard, and run:

    ```bash
    git clone https://github.com/exo-explore/exo
    cd exo/dashboard && npm install && npm run build && cd ..
    uv run exo
    ```

=== "macOS — Nix"

    With [Nix](https://nixos.org/) installed:

    ```bash
    nix run .#exo
    ```

    To accept the Cachix binary cache (and avoid building the Xcode Metal toolchain), add to `/etc/nix/nix.conf`:

    ```text
    trusted-users = root        # or your username
    experimental-features = nix-command flakes
    ```

    Then restart the Nix daemon:

    ```bash
    sudo launchctl kickstart -k system/org.nixos.nix-daemon
    ```

=== "macOS — App (.dmg / Homebrew Cask)"

    Download the prebuilt menu-bar app:

    - **Direct download:** [EXO-latest.dmg](https://assets.exolabs.net/EXO-latest.dmg)
    - **Homebrew:**

      ```bash
      brew install --cask exo
      ```

    The app runs in the background and will ask for permission to modify system settings and install a network profile (needed for device discovery and RDMA). Requires macOS Tahoe 26.2+.

=== "Linux — Run from Source"

    ```bash
    # Node.js + npm (Ubuntu/Debian)
    sudo apt update && sudo apt install nodejs npm

    # uv
    curl -LsSf https://astral.sh/uv/install.sh | sh

    # Rust (nightly)
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
    rustup toolchain install nightly

    # Clone, build dashboard, run
    git clone https://github.com/exo-explore/exo
    cd exo/dashboard && npm install && npm run build && cd ..
    uv run exo
    ```

    `macmon` is not required on Linux.

> [!WARNING]
> On Linux, exo currently runs inference on the **CPU**. It is fully functional but slower than GPU. If you want GPU support for your Linux hardware, search the [issue tracker](https://github.com/exo-explore/exo/issues) and add a 👍 to the relevant request.

> [!NOTE]
> The dashboard must be built (`npm run build`) before running from source. If you skip it, exo can't find the dashboard assets and will tell you so on startup (see [Troubleshooting](#16-troubleshooting)).

---

## 4. Running exo

The simplest invocation is:

```bash
uv run exo
```

```text
http://localhost:52415/
```

That single process becomes a full cluster node: it discovers peers, can act as the cluster coordinator (master), downloads models, runs inference (worker), and serves the API and dashboard.

### Common run modes

=== "Default (everything)"

    ```bash
    uv run exo
    ```

    Discovery + worker + API + downloads, all on.

=== "Coordinator-only node"

    ```bash
    uv run exo --no-worker
    ```

    Runs networking and orchestration but does **not** execute inference. Useful for a machine with good network connectivity but little GPU/RAM.

=== "Offline / air-gapped"

    ```bash
    uv run exo --offline
    ```

    Skips internet checks and uses only models you've already staged locally.

=== "Background daemon (legacy)"

    ```bash
    uv run exo --legacy-daemon
    ```

    Double-fork SysV-style daemonization. Intended for legacy init scripts only — under systemd or launchd, run exo in the foreground (no flag).

When the API node starts, exo prints a startup banner showing the address. Leave the process running; press `Ctrl-C` to shut it down cleanly.

> [!TIP]
> Add `-v` for more logging detail (repeat for more, e.g. `-vv`), or `-q` to quiet it down. See the [Command-Line Reference](#12-command-line-reference).

---

## 5. Using the Dashboard

exo ships a built-in web dashboard for managing your cluster and chatting with models. Once exo is running, open:

```text
http://localhost:52415/
```

From the dashboard you can:

- **See your cluster** — every discovered device, its memory, and the network links between devices (Thunderbolt, Ethernet, Wi-Fi).
- **Browse and download models** — search HuggingFace, start downloads, and watch progress.
- **Load (place) a model** — pick a model and let exo choose a placement across your devices, or preview the options.
- **Chat** — send prompts to a loaded model and stream the response, with support for reasoning/"thinking" models.
- **Generate images** — when image models are enabled.
- **Inspect performance traces** — when tracing is enabled.

The dashboard is the easiest way to get started. The rest of this guide focuses on the API, which gives you the same capabilities programmatically and lets you point existing tools at your cluster.

---

## 6. Loading and Running a Model (the API Workflow)

Loading a model in exo is a deliberate, two-phase process: first you **place an instance** (decide how the model is split across devices and start loading it), then you **send inference requests** to it. This section walks through the full lifecycle with `curl`.

The base URL is your exo node, e.g. `http://localhost:52415`.

### Step 1 — Preview placements

Ask exo for all valid ways to place a model across your current cluster:

```bash
curl "http://localhost:52415/instance/previews?model_id=llama-3.2-1b"
```

```json
{
  "previews": [
    {
      "model_id": "mlx-community/Llama-3.2-1B-Instruct-4bit",
      "sharding": "Pipeline",
      "instance_meta": "MlxRing",
      "instance": { "...": "..." },
      "memory_delta_by_node": { "local": 729808896 },
      "error": null
    }
  ]
}
```

Each preview describes a sharding strategy (`Pipeline` or `Tensor`), a backend (`MlxRing` or `MlxJaccl`), and the per-node memory cost. Placements that won't work include a non-null `error` explaining why.

To grab the first valid placement programmatically, pipe through `jq`:

```bash
curl "http://localhost:52415/instance/previews?model_id=llama-3.2-1b" \
  | jq -c '.previews[] | select(.error == null) | .instance' | head -n1
```

### Step 2 — Create the instance

POST the placement you chose to `/instance`:

```bash
curl -X POST http://localhost:52415/instance \
  -H 'Content-Type: application/json' \
  -d '{
    "instance": { "...": "...paste the instance object from step 1..." }
  }'
```

```json
{
  "message": "Command received.",
  "command_id": "e9d1a8ab-...."
}
```

This is **asynchronous** — the model now begins downloading (if needed) and loading across the assigned devices.

### Step 3 — Wait until the model is ready

Don't send inference yet. Wait for the instance to come online using the streaming `await` endpoint:

```bash
curl -N "http://localhost:52415/instance/await?model_id=mlx-community/Llama-3.2-1B-Instruct-4bit"
```

This returns a Server-Sent Events (SSE) stream. A successful wait emits a message with `"type": "ready"` and the matching instance; a timeout emits `"type": "timeout"`. By default it waits indefinitely; add `&timeout_seconds=120` (max `300`) to bound the wait.

```text
data: {"type": "ready", "instance": {...}}
```

### Step 4 — Run inference

Now send a chat completion (OpenAI format):

```bash
curl -N -X POST http://localhost:52415/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mlx-community/Llama-3.2-1B-Instruct-4bit",
    "messages": [
      {"role": "user", "content": "What is Llama 3.2 1B?"}
    ],
    "stream": true
  }'
```

### Step 5 — Delete the instance when done

Free the memory by deleting the instance (find its ID via `/state` or `/instance`):

```bash
curl -X DELETE http://localhost:52415/instance/YOUR_INSTANCE_ID
```

> [!NOTE]
> If you send a chat request for a model that has no running instance, exo responds with `404 No instance found for model <id>`. Place the instance first (steps 1–3). The dashboard performs this whole flow for you when you click "load".

---

## 7. Everyday Usage: Talking to Your Cluster

Because exo speaks four popular API dialects, you can usually point an existing tool at it by **changing only the base URL** to your exo node. All inference endpoints below assume a model instance is already loaded (see [Section 6](#6-loading-and-running-a-model-the-api-workflow)).

### OpenAI Chat Completions

```bash
curl -N -X POST http://localhost:52415/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mlx-community/Llama-3.2-1B-Instruct-4bit",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello"}
    ],
    "stream": false
  }'
```

Non-streaming responses include `usage` statistics:

```json
{
  "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello! How can I help you?"}, "finish_reason": "stop"}],
  "usage": {"prompt_tokens": 15, "completion_tokens": 8, "total_tokens": 23}
}
```

**Supported request parameters** include `model`, `messages`, `stream`, `max_tokens`, `temperature`, `top_p`, `top_k`, `stop`, `seed`, `tools` (function calling), `logprobs`, `top_logprobs`, and `enable_thinking` (turns on reasoning mode for capable models such as DeepSeek V3.1, Qwen3, and GLM-4.7).

### Claude Messages API

Point an Anthropic-style client at `/v1/messages`:

```bash
curl -N -X POST http://localhost:52415/v1/messages \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mlx-community/Llama-3.2-1B-Instruct-4bit",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 1024,
    "stream": true
  }'
```

Streaming uses Claude's event types: `message_start`, `content_block_start`, `content_block_delta`, `content_block_stop`, `message_delta`, `message_stop`.

### OpenAI Responses API

```bash
curl -N -X POST http://localhost:52415/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mlx-community/Llama-3.2-1B-Instruct-4bit",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'
```

Streaming uses Responses event types: `response.created`, `response.in_progress`, `response.output_item.added`, `response.output_item.done`, `response.done`.

### Ollama API (works with OpenWebUI and friends)

```bash
# Chat
curl -X POST http://localhost:52415/ollama/api/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "mlx-community/Llama-3.2-1B-Instruct-4bit",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": false
  }'

# List models, Ollama-style
curl http://localhost:52415/ollama/api/tags
```

exo also exposes `/ollama/api/generate` (prompt-completion), `/ollama/api/show`, `/ollama/api/ps` (running models), and `/ollama/api/version`.

> [!TIP]
> Configuring a tool that expects "Ollama"? Use `http://<your-exo-node>:52415/ollama` as the Ollama base URL. For OpenAI-compatible tools, use `http://<your-exo-node>:52415/v1`.

### Cancelling a request

You can stop an in-flight generation two ways:

1. **Close the HTTP connection** (for streaming requests) — the server detects the disconnect and stops.
2. **Call the cancel endpoint** with the command's ID:

   ```bash
   curl -X POST http://localhost:52415/v1/cancel/COMMAND_ID
   ```

   Returns `404` if the command is unknown or already finished.

---

## 8. Working with Models

### Listing what's available

```bash
# All known models
curl http://localhost:52415/models

# Only models already downloaded to the cluster
curl "http://localhost:52415/models?status=downloaded"
```

Model descriptors include the model ID, family, quantization, context length, supported tasks, capabilities, whether the model supports tensor parallelism, and an `is_custom` flag.

exo ships with a large built-in catalog of model cards — DeepSeek (V3.1/V3.2/V4), GLM (4.5/4.7/5/5.1), Llama, Qwen, Kimi K2, and many more — predominantly in `mlx-community` quantized form.

### Searching HuggingFace

```bash
curl "http://localhost:52415/models/search?query=llama&limit=10"
```

exo searches `mlx-community` first (the quantized models it runs best), then falls back to all of HuggingFace.

### Adding a custom model

Pull any compatible model from the HuggingFace hub and register it as a custom model card that syncs across your cluster:

```bash
curl -X POST http://localhost:52415/models/add \
  -H 'Content-Type: application/json' \
  -d '{ "model_id": "mlx-community/my-custom-model" }'
```

Remove a custom model card later:

```bash
curl -X DELETE http://localhost:52415/models/custom/mlx-community/my-custom-model
```

> [!CAUTION]
> Models whose configuration requires `trust_remote_code` execute code from the model repository. exo disables this by default. Only enable it for models you fully trust — remote code runs on your machine.

### Where models are stored

Downloaded models live in exo's data directory by default (`~/.exo/models` on macOS, `~/.local/share/exo/models` on Linux). You can redirect downloads to another disk, or point exo at a shared read-only model store, using environment variables — see [Section 13](#13-environment-variables).

---

## 9. Image Generation and Editing

exo can run image models (e.g. FLUX) in addition to LLMs. Image support is **off by default** and must be enabled when you start the node:

```bash
EXO_ENABLE_IMAGE_MODELS=true uv run exo
```

Load an image model the same way you load an LLM (place an instance, wait for ready), then call the OpenAI-compatible image endpoints.

### Generate an image

```bash
curl -X POST http://localhost:52415/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{
    "prompt": "a robot playing chess",
    "model": "exolabs/FLUX.1-dev",
    "n": 1,
    "size": "1024x1024",
    "response_format": "b64_json"
  }'
```

**Key parameters:**

| Parameter | Default | Notes |
|-----------|---------|-------|
| `prompt` | — (required) | Text description of the image |
| `model` | — (required) | Image model ID |
| `n` | `1` | Number of images |
| `size` | `auto` | One of `512x512`, `768x768`, `1024x768`, `768x1024`, `1024x1024`, `1024x1536`, `1536x1024`, `1024x1365`, `1365x1024` |
| `response_format` | `b64_json` | `b64_json` or `url` |
| `quality` | `medium` | `high`, `medium`, or `low` |
| `output_format` | `png` | `png`, `jpeg`, or `webp` |
| `stream` | `false` | Stream partial images |
| `partial_images` | `0` | How many in-progress previews to stream |

Fine-grained control lives in `advanced_params`:

| Advanced param | Range | Description |
|----------------|-------|-------------|
| `seed` | ≥ 0 | Reproducible generation (exo auto-assigns one if omitted, for consistency across devices) |
| `num_inference_steps` | 1–100 | Denoising steps |
| `guidance` | 1.0–20.0 | Classifier-free guidance scale |
| `negative_prompt` | — | What to avoid |

### Edit an image (img2img)

Send the source image as multipart form data to `/v1/images/edits`:

```bash
curl -X POST http://localhost:52415/v1/images/edits \
  -F image=@input.png \
  -F prompt="make it snowy" \
  -F model="exolabs/FLUX.1-Kontext-dev" \
  -F input_fidelity=high \
  -F output_format=png
```

`input_fidelity` (`low`/`high`) controls how closely the output follows the original image.

### Retrieving images

When `response_format` is `url`, exo stores the image and returns a link. Stored images expire after about an hour.

```bash
curl http://localhost:52415/images               # list stored images
curl http://localhost:52415/images/IMAGE_ID -o out.png   # fetch one
```

> [!NOTE]
> Streaming image responses emit SSE events of `type: "partial"` (in-progress previews) and `type: "final"`, terminated by `data: [DONE]`.

---

## 10. Building a Multi-Device Cluster

The payoff of exo is pooling devices. The process is intentionally hands-off:

1. **Install exo on each device** (Section 3).
2. **Connect the devices** to the same network. For maximum speed on Apple Silicon, connect them with Thunderbolt cables (and enable RDMA — Section 11). Ethernet and Wi-Fi also work.
3. **Run `exo` on each device.** They discover each other automatically — no IP addresses or config files to edit.
4. **Open the dashboard** on any node and you'll see all devices and the links between them.
5. **Place a model.** exo's topology-aware placement automatically splits the model across devices based on each device's memory and the speed of the links between them. A model too large for any single device will run across several.

### Keeping clusters separate

If multiple exo clusters share a network (e.g. dev vs. production), isolate them with a namespace. Nodes with different namespaces will not connect:

```bash
EXO_ZENOH_NAMESPACE=my-dev-cluster uv run exo
# or equivalently
uv run exo --namespace my-dev-cluster
```

In the macOS app, set this under **Advanced settings**.

> [!TIP]
> A device with lots of bandwidth but little RAM/GPU makes a good `--no-worker` coordinator: it helps route and orchestrate without being asked to hold model layers.

### How exo decides the split

You don't have to choose, but it helps to know the options exo weighs:

- **Pipeline parallelism** — different layers of the model live on different devices; tokens flow around the ring. Works with the most models.
- **Tensor parallelism** — each layer is split across devices, for up to ~1.8× speedup on 2 devices and ~3.2× on 4. Requires a model that supports it.
- **Ring backend (MlxRing)** — neighbour-to-neighbour communication over sockets/Thunderbolt; the general-purpose default.
- **JACCL backend (MlxJaccl)** — all-to-all RDMA over Thunderbolt 5 for the lowest latency. Requires RDMA to be enabled and every device connected to every other.

exo picks among the valid combinations using available memory and link types, preferring faster links. See [How It Works](#15-how-it-works-concepts).

---

## 11. Enabling RDMA over Thunderbolt (macOS)

RDMA (Remote Direct Memory Access) over Thunderbolt 5 dramatically reduces inter-device latency, making multi-device inference much faster. It's a macOS 26.2 capability available on Macs with Thunderbolt 5.

**Enable RDMA on each Mac:**

1. Shut down the Mac.
2. Hold the power button for ~10 seconds until the boot menu appears.
3. Choose **Options** to enter Recovery mode.
4. In Recovery, open **Terminal** from the **Utilities** menu.
5. Run:

   ```text
   rdma_ctl enable
   ```

6. Reboot.

After that, exo detects and uses RDMA automatically.

> [!IMPORTANT]
> RDMA caveats:
> - Every device in an RDMA cluster must be physically connected to **every other** device.
> - Cables must support **Thunderbolt 5**.
> - On a Mac Studio, do **not** use the Thunderbolt 5 port next to the Ethernet port.
> - If running from source, run `tmp/set_rdma_network_config.sh` to disable Thunderbolt Bridge and set DHCP on each RDMA port.
> - **All devices must run the exact same macOS version** (including beta build numbers), or RDMA ports may fail to discover each other.

Pre-flight checklist:

- [ ] All Macs have Thunderbolt 5 ports
- [ ] All Macs run identical macOS versions (26.2+)
- [ ] Every device cabled to every other device with TB5 cables
- [ ] `rdma_ctl enable` run in Recovery on each Mac
- [ ] (From source) `tmp/set_rdma_network_config.sh` executed

---

## 12. Command-Line Reference

Run `uv run exo [OPTIONS]`. All options are optional; defaults make a sensible single-node setup.

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `-q`, `--quiet` | flag | — | Quiet logging |
| `-v`, `--verbose` | count | — | Increase log verbosity (repeatable: `-vv`) |
| `-m`, `--force-master` | flag | off | Strongly prefer this node as the cluster master |
| `--no-api` | flag | API on | Don't start the API/dashboard server on this node |
| `--api-port` | int | `52415` | Port for the API and dashboard |
| `--no-worker` | flag | worker on | Don't run inference here (coordinator-only node) |
| `--no-downloads` | flag | downloads on | Don't download models on this node |
| `--offline` | flag | off | Air-gapped mode: no internet checks, local models only |
| `--no-batch` | flag | batching on | Disable continuous batching; use sequential generation |
| `--legacy-daemon` | flag | off | Run as a SysV-style background daemon (legacy init only) |
| `--namespace` | str | exo version | Discovery namespace; nodes with different namespaces won't connect |
| `--zenoh-port` | int | `52414` | Fixed TCP port for the zenoh networking layer |
| `--discovery-port` | int | `52413` | Fixed UDP port for the discovery service |
| `--fast-synch` | flag | auto | Force MLX `FAST_SYNCH` on (for the JACCL backend) |
| `--no-fast-synch` | flag | auto | Force MLX `FAST_SYNCH` off |

> [!NOTE]
> `--fast-synch` and `--no-fast-synch` are mutually exclusive; omit both to let exo decide automatically.

> [!WARNING]
> `--bootstrap-peers` (and the `EXO_BOOTSTRAP_PEERS` variable) are currently **not functional** — supplying them raises `Bootstrap peers has been temporarily removed`. Likewise, the old `EXO_LIBP2P_NAMESPACE` variable was renamed; from source use `EXO_ZENOH_NAMESPACE` instead (the macOS app still exposes the namespace as a setting).

---

## 13. Environment Variables

Set these before launching exo (e.g. `VAR=value uv run exo`).

| Variable | Default | Description |
|----------|---------|-------------|
| `EXO_DEFAULT_MODELS_DIR` | `~/.exo/models` (macOS) / `~/.local/share/exo/models` (Linux) | Primary directory for model downloads. Always first in the writable list. |
| `EXO_MODELS_DIRS` | none | Colon-separated extra **writable** model dirs. Used in order; first with enough free space wins. |
| `EXO_MODELS_READ_ONLY_DIRS` | none | Colon-separated **read-only** dirs to search for pre-staged models (e.g. NFS, shared SSD). Never written to or deleted from. |
| `EXO_OFFLINE` | `false` | Run without internet (same as `--offline`). |
| `EXO_ENABLE_IMAGE_MODELS` | `false` | Enable image model support. |
| `EXO_TRACING_ENABLED` | `false` | Record distributed performance traces. |
| `EXO_FAST_SYNCH` | auto | Control MLX `MLX_METAL_FAST_SYNCH` for the JACCL backend. |
| `EXO_ZENOH_NAMESPACE` | none | Cluster isolation namespace (replaces the old `EXO_LIBP2P_NAMESPACE`). |
| `EXO_HOME` | none | Override the base directory for all exo state (relative to your home dir). |

**Examples:**

```bash
# Use pre-downloaded models from an NFS mount (read-only)
EXO_MODELS_READ_ONLY_DIRS=/mnt/nfs/models:/opt/ai-models uv run exo

# Download models to an external SSD (falls back to default dir if full)
EXO_MODELS_DIRS=/Volumes/ExternalSSD/exo-models uv run exo

# Offline mode
EXO_OFFLINE=true uv run exo

# Enable image models
EXO_ENABLE_IMAGE_MODELS=true uv run exo
```

> [!TIP]
> On Linux, exo follows the XDG Base Directory spec: config in `~/.config/exo/`, data in `~/.local/share/exo/`, cache and logs in `~/.cache/exo/`. Override with the standard `XDG_*` variables.

For advanced/internal variables (`EXO_MAX_CONCURRENT_REQUESTS`, `ENABLE_DISAGGREGATION`, `EXO_RESOURCES_DIR`, `EXO_DASHBOARD_DIR`, and more), see the **Admin Guide**.

---

## 14. Benchmarking Your Cluster

exo includes `exo-bench`, a tool to measure prefill and generation speed across placement configurations — useful for finding the fastest way to run a model on your hardware.

**Prerequisite:** nodes must already be running (`uv run exo`); the tool drives the `/bench/chat/completions` endpoint.

```bash
uv run bench/exo_bench.py \
  --model Llama-3.2-1B-Instruct-4bit \
  --pp 128,256,512 \
  --tg 128,256
```

**Frequently used options:**

| Option | Description |
|--------|-------------|
| `--model` | Model to benchmark (short ID or HuggingFace ID) |
| `--pp` | Prompt-size hints (comma-separated integers) |
| `--tg` | Generation lengths (comma-separated integers) |
| `--concurrency` | Concurrency levels, e.g. `1,2,4,8` |
| `--repeat` | Repetitions per configuration (default `1`) |
| `--warmup` | Warmup runs per placement (default `0`) |
| `--use-prefix-cache` | Enable the KV prefix cache (default off, for cold-cache numbers) |
| `--json-out` | Results file (default `bench/results.json`) |
| `--stdout` | Print results to stdout instead of a file |
| `--dry-run` | List the selected placements and exit |

The output reports prompt tokens/sec, generation tokens/sec, and peak memory per configuration.

> [!NOTE]
> exo also bundles `exo_eval.py` (run standard benchmarks/evals against a model), `prefill_decode_bench.py` (measure prefill/decode disaggregation), and `eval_tool_calls.py` (test tool-calling across the API dialects). These are advanced tools covered in the Admin Guide.

---

## 15. How It Works (Concepts)

You don't need this section to *use* exo, but it explains the words you'll see in the dashboard and API.

**Node.** One device running exo. Every node is equal at startup and can take on any role.

**Master (coordinator).** Exactly one node coordinates the cluster at a time. Nodes elect a master automatically; if it disappears, a new one is elected. `--force-master` biases the election toward a specific node.

**Worker.** The part of a node that actually runs model code in a separate runner process. Disable with `--no-worker`.

**Topology.** exo's live map of which devices are connected and how (Thunderbolt, Ethernet, Wi-Fi), including link quality. Placement decisions use this map.

**Instance.** A loaded, ready-to-serve copy of a model, spread across one or more devices according to a placement. You create instances, run inference against them, and delete them to reclaim memory.

**Placement / sharding.** How a model is divided across devices. exo finds device "cycles" (rings) with enough combined memory, then allocates layers proportionally to each device's available RAM. It supports **pipeline** parallelism (layers split across devices) and **tensor** parallelism (each layer split across devices), over either the **ring** (`MlxRing`) or **RDMA** (`MlxJaccl`) backend.

**Continuous batching.** By default, exo batches concurrent requests for throughput. `--no-batch` falls back to sequential generation.

The flow of a single request: your client hits the API → the master turns it into a task and assigns it to the least-busy instance → workers on each device run their shard of the model → tokens (or image chunks) stream back to the API → the API relays them to you as SSE.

---

## 16. Troubleshooting

### Error: `No instance found for model <model_id>`

**Cause:** You sent an inference request for a model that has no running instance (or the model isn't downloaded).
**Solution:** Place an instance first — preview placements, POST to `/instance`, then wait on `/instance/await` (Section 6). Confirm the model appears in `curl http://localhost:52415/models?status=downloaded`.
**Prevention:** Use the dashboard's "load" flow, which handles placement and readiness for you.

### Error: `Insufficient memory to create instance. Required: X GB, Available: Y GB`

**Cause:** The cluster's combined available RAM can't hold the model.
**Solution:** Add more devices, free memory on existing devices, or choose a smaller/more-quantized model (e.g. a 4-bit instead of 8-bit variant).
**Prevention:** Check `/instance/previews` first — `memory_delta_by_node` shows the per-node cost before you commit.

### Error: `No cycles found with sufficient memory`

**Cause:** No connected group of devices has enough total memory for this model.
**Solution:** Connect more devices into the cluster, or pick a smaller model.

### Error: `Requested Tensor sharding but this model does not support tensor parallelism`

**Cause:** You asked for tensor parallelism on a model whose card doesn't support it.
**Solution:** Use pipeline sharding for this model, or pick a model with `supports_tensor: true` (visible in `/models`).

### Error: `Requested RDMA (MlxJaccl) but no RDMA-connected cycles available`

**Cause:** The JACCL backend needs an all-to-all RDMA-connected set of devices, and none is available.
**Solution:** Enable RDMA on every device (Section 11), cable every device to every other with TB5, and ensure identical macOS versions. Otherwise use the ring backend.

### Error: `Unable to locate dashboard assets - you probably forgot to run cd dashboard && npm install && npm run build && cd ..`

**Cause:** Running from source without building the dashboard.
**Solution:** Build it: `cd dashboard && npm install && npm run build && cd ..`, then restart exo.

### Error: `Unable to locate resources. Did you clone the repo properly?`

**Cause:** exo can't find its bundled resource files (model cards, etc.).
**Solution:** Ensure you cloned the full repository (not a partial/shallow copy missing `resources/`), and run from the repo root.

### Error: `EXO_LIBP2P_NAMESPACE has been removed - use EXO_ZENOH_NAMESPACE instead`

**Cause:** You set the old namespace variable when running from source.
**Solution:** Use `EXO_ZENOH_NAMESPACE` (or the `--namespace` flag) instead.

### Error: `Bootstrap peers has been temporarily removed`

**Cause:** You passed `--bootstrap-peers` / `EXO_BOOTSTRAP_PEERS`.
**Solution:** Remove it. Discovery is automatic; manual peer bootstrapping is not currently supported.

### exo won't start: PID file error

**Cause:** Another exo process is already running (it holds a PID lock), or a stale lock remains after a crash.
**Solution:** Stop the other instance. If none is running, the stale PID file lives in exo's cache dir (`~/.exo/` or `~/.cache/exo/`); remove it and retry.

### Devices don't see each other

**Cause:** Different namespaces, blocked discovery ports, or no network path between devices.
**Solution:** Ensure all nodes use the same namespace (or none), that UDP discovery port `52413` and TCP zenoh port `52414` aren't firewalled, and that the devices share a network. On macOS, allow the network profile the app installs.

### Generation seems stuck or very slow

**Cause:** Model still loading, slow links (Wi-Fi instead of Thunderbolt), or a large model on minimal hardware. On Linux, inference runs on CPU.
**Solution:** Wait for `/instance/await` to report ready; prefer Thunderbolt/RDMA links; benchmark placements with `exo-bench` to find the fastest configuration.

### macOS RDMA ports stop working after idle, or run at half speed

**Cause:** Known issues on some hardware/macOS combinations (e.g. RDMA ports going inactive, or certain MacBook Pro pairings running slower over RDMA).
**Solution:** Reconnect/recheck cabling, confirm identical macOS versions, and consult the [issue tracker](https://github.com/exo-explore/exo/issues) for your specific hardware.

---

## 17. FAQ

**Q: Do I have to configure IP addresses or a config file to make a cluster?**
No. Devices on the same network discover each other automatically. You only configure things like ports or namespaces if you have a specific need.

**Q: What's the difference between "place an instance" and just sending a chat request?**
Placing an instance loads the model across your devices and makes it ready to serve. Chat requests then run against that loaded instance. A chat request for a model with no instance returns `404`.

**Q: Which API should I use — OpenAI, Claude, Ollama, or Responses?**
Whichever your existing tool already speaks. exo supports all four; just point the tool's base URL at your exo node. Capabilities are equivalent for text generation.

**Q: Can I run models bigger than any single device's memory?**
Yes — that's the point. exo pools memory across devices and splits the model so the *combined* memory holds it.

**Q: Does exo run on Windows?**
Not yet. macOS (GPU) and Linux (CPU) are supported today; Windows is on the roadmap.

**Q: Is a GPU required?**
On macOS, exo uses the Apple Silicon GPU automatically. On Linux it runs on CPU today.

**Q: How do I make a multi-Mac cluster as fast as possible?**
Connect the Macs with Thunderbolt 5 and enable RDMA (Section 11). Use tensor parallelism on models that support it. Benchmark with `exo-bench`.

**Q: How do I add a model that isn't in the built-in list?**
POST its HuggingFace ID to `/models/add`. It becomes a custom model card synced across the cluster. Only enable `trust_remote_code` for models you trust.

**Q: Where are my downloaded models?**
In `~/.exo/models` (macOS) or `~/.local/share/exo/models` (Linux) by default. Redirect with `EXO_DEFAULT_MODELS_DIR` / `EXO_MODELS_DIRS`, or share a read-only store with `EXO_MODELS_READ_ONLY_DIRS`.

**Q: Can I use exo without internet?**
Yes — run with `--offline` (or `EXO_OFFLINE=true`) once your models are staged locally.

**Q: How do I generate images?**
Start exo with `EXO_ENABLE_IMAGE_MODELS=true`, load an image model, and call `/v1/images/generations` or `/v1/images/edits`.

**Q: How do I stop a runaway generation?**
Close the connection, or `POST /v1/cancel/{command_id}`.

**Q: Can I dedicate a low-power machine to coordination only?**
Yes — run it with `--no-worker`. It helps with networking and orchestration without holding model layers.

**Q: Why is there a `--no-batch` flag?**
exo batches concurrent requests for throughput by default. `--no-batch` switches to sequential generation, which can be useful for debugging or strict reproducibility.

---

## 18. Reference

### Default ports

| Port | Protocol | Purpose |
|------|----------|---------|
| `52415` | TCP | API + dashboard (`--api-port`) |
| `52414` | TCP | zenoh networking (`--zenoh-port`) |
| `52413` | UDP | Device discovery (`--discovery-port`) |

### Core inference endpoints (instance must be loaded)

| Method | Path | Purpose |
|--------|------|---------|
| POST | `/v1/chat/completions` | OpenAI chat completions |
| POST | `/v1/messages` | Claude Messages API |
| POST | `/v1/responses` | OpenAI Responses API |
| POST | `/ollama/api/chat` | Ollama chat |
| POST | `/ollama/api/generate` | Ollama generate |
| POST | `/v1/images/generations` | Image generation |
| POST | `/v1/images/edits` | Image editing (img2img) |
| POST | `/v1/cancel/{command_id}` | Cancel a generation |

### Cluster / model management endpoints

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/instance/previews?model_id=...` | Preview valid placements |
| POST | `/instance` | Create an instance from a placement |
| GET | `/instance/await?model_id=...` | Wait (SSE) until an instance is ready |
| GET | `/instance/{instance_id}` | Inspect an instance |
| DELETE | `/instance/{instance_id}` | Delete an instance |
| GET | `/models` | List models (`?status=downloaded` to filter) |
| GET | `/models/search?query=...` | Search HuggingFace |
| POST | `/models/add` | Add a custom model |
| DELETE | `/models/custom/{model_id}` | Remove a custom model |
| GET | `/state` | Inspect cluster state |
| GET | `/node_id` | Current master node ID |

### Useful links

- Source: <https://github.com/exo-explore/exo>
- API technical reference: `docs/api.md` in the repository
- Community / support: [Discord](https://discord.gg/TJ4P57arEm) · [@exolabs](https://x.com/exolabs)
- License: Apache-2.0

---

_Generated June 2026 from exo-explore/exo at 09f9ea3. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._