# FastChat Admin & Architecture Guide

This guide is for operators, architects, and developers deploying, scaling, securing, or extending FastChat. It covers the distributed serving architecture, the complete configuration surface, deployment strategies (including Docker), security considerations, operational procedures, and the internal module layout. For end-user install and usage, see the **User Guide**.

FastChat (`fschat` `0.2.36`) is an open platform for training, serving, and evaluating LLM-based chatbots. In production it powers Chatbot Arena ([lmarena.ai](https://lmarena.ai)), which has served 10M+ chat requests across 70+ models. The serving layer is a **controller + worker + front-end** system built on FastAPI and Uvicorn.

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [System Architecture](#2-system-architecture)
3. [Component Inventory](#3-component-inventory)
4. [System Requirements](#4-system-requirements)
5. [Configuration Reference](#5-configuration-reference)
6. [Deployment](#6-deployment)
7. [Security](#7-security)
8. [Operations](#8-operations)
9. [Performance & Scaling](#9-performance--scaling)
10. [Repository Structure](#10-repository-structure)
11. [Internal API Reference](#11-internal-api-reference)
12. [Code Patterns & Conventions](#12-code-patterns--conventions)
13. [Testing](#13-testing)
14. [Known Issues & Potential Improvements](#14-known-issues--potential-improvements)
15. [Quick Reference](#15-quick-reference)

---

## 1. Executive Summary

FastChat decomposes LLM serving into three independently deployable tiers that communicate over HTTP:

- A **controller** that maintains a registry of model workers and routes each request to one.
- One or more **model workers**, each hosting one or more models on accelerators, using a pluggable inference backend (Hugging Face Transformers, vLLM, SGLang, LightLLM, MLX, DashInfer, or remote HF APIs).
- **Front ends** — a Gradio web UI and an OpenAI-compatible REST API server — that ask the controller where to send requests.

| Concern | Summary |
|---|---|
| **Language / runtime** | Python ≥ 3.8 |
| **Web framework** | FastAPI + Uvicorn (ASGI) |
| **Serving topology** | Controller / worker / front-end, loosely coupled over HTTP |
| **State** | In-memory worker registry; no database. Logs to `LOGDIR` |
| **Worker discovery** | Workers self-register and send heartbeats; controller expires stale ones |
| **API surface** | OpenAI-compatible (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/models`) |
| **Auth** | Optional static API keys on the API server; optional SSL on every service |
| **Default ports** | Controller `21001`, worker `21002`, API server `8000`, Gradio (auto) |
| **Packaging** | `setuptools`; Docker image based on `nvidia/cuda:12.2.0-runtime-ubuntu20.04` |

> [!IMPORTANT]
> FastChat keeps **no persistent server state** beyond log files. The worker registry is rebuilt from heartbeats, so a controller restart is recovered automatically as workers re-register. There is no database to back up.

---

## 2. System Architecture

```text
                         clients (browser, OpenAI SDK, curl)
                                       │
              ┌────────────────────────┼────────────────────────┐
              ▼                        ▼                         ▼
   ┌────────────────────┐   ┌────────────────────┐   (optional Nginx
   │ gradio_web_server  │   │ openai_api_server  │    gateway in front:
   │  / _multi (Arena)  │   │   (FastAPI :8000)  │    fastchat/serve/
   └─────────┬──────────┘   └─────────┬──────────┘    gateway/nginx.conf)
             │  "address for model X?" │
             └───────────┬─────────────┘
                         ▼
                ┌──────────────────┐   register_worker / receive_heart_beat
                │   controller     │ ◀─────────────────────────────────────┐
                │  (FastAPI :21001)│   get_worker_address (dispatch)        │
                └──────────────────┘                                        │
                         │ forwards generate request                        │
        ┌────────────────┼────────────────────────┐                        │
        ▼                ▼                         ▼                        │
 ┌─────────────┐  ┌─────────────┐          ┌─────────────┐                  │
 │ model_worker│  │ vllm_worker │   ...    │ multi_model │ ─────────────────┘
 │  (:21002)   │  │  (:21002)   │          │   _worker   │  heartbeats
 │  HF Trans-  │  │   vLLM      │          │ many models │
 │  formers    │  │  backend    │          │ one process │
 └─────────────┘  └─────────────┘          └─────────────┘
        │ loads weights from
        ▼
  ~/.cache/huggingface/hub/  (or ModelScope, or local path)
```

### Request lifecycle (OpenAI API path)

1. A client `POST`s to `/v1/chat/completions` on the **API server**.
2. The API server calls the **controller**'s `get_worker_address` for the requested model.
3. The controller selects a worker by its **dispatch method** (`shortest_queue` or `lottery`) and returns the worker URL.
4. The API server streams `/worker_generate_stream` from that **worker**.
5. The worker runs inference on its backend and streams tokens back, which the API server reshapes into the OpenAI response schema.

### Worker registration & health

- On startup, a worker calls the controller's `register_worker` (unless `--no-register`).
- Workers send heartbeats every `FASTCHAT_WORKER_HEART_BEAT_INTERVAL` seconds (default **45**).
- The controller's background `heart_beat_controller` thread removes workers whose last heartbeat is older than `FASTCHAT_CONTROLLER_HEART_BEAT_EXPIRATION` seconds (default **90**).
- `DispatchMethod` (in `controller.py`) is `SHORTEST_QUEUE` (default) or `LOTTERY`; `shortest_queue` balances by reported queue length and worker speed.

> [!NOTE]
> The controller-to-worker timeout is `FASTCHAT_WORKER_API_TIMEOUT` (default **100s**). Slow models on slow hardware should raise this or they will appear to fail.

---

## 3. Component Inventory

| Component | Module | Default port | Role |
|---|---|---|---|
| Controller | `fastchat.serve.controller` | 21001 | Worker registry & dispatch |
| Model worker (Transformers) | `fastchat.serve.model_worker` | 21002 | Default inference backend |
| Base worker class | `fastchat.serve.base_model_worker` | — | Shared worker HTTP API |
| vLLM worker | `fastchat.serve.vllm_worker` | 21002 | High-throughput batched serving |
| SGLang worker | `fastchat.serve.sglang_worker` | 21002 | SGLang backend |
| LightLLM worker | `fastchat.serve.lightllm_worker` | 8000 | LightLLM backend |
| MLX worker | `fastchat.serve.mlx_worker` | 21002 | Apple Silicon (MLX) |
| DashInfer worker | `fastchat.serve.dashinfer_worker` | 21002 | CPU inference (DashInfer) |
| HF API worker | `fastchat.serve.huggingface_api_worker` | 21002 | Proxy to remote HF Inference API |
| Multi-model worker | `fastchat.serve.multi_model_worker` | 21002 | Several models in one process |
| OpenAI API server | `fastchat.serve.openai_api_server` | 8000 | OpenAI-compatible REST API |
| Gradio web server | `fastchat.serve.gradio_web_server` | auto | Single-model chat UI |
| Gradio multi/Arena | `fastchat.serve.gradio_web_server_multi` | auto | Multi-model + side-by-side Arena |
| CLI | `fastchat.serve.cli` | — | Terminal chat client |
| Call monitor | `fastchat.serve.call_monitor` | — | Per-model/user rate-limit accounting |
| Launch helper | `fastchat.serve.launch_all_serve` | — | Spawn controller + workers + API from one command |
| Nginx gateway | `fastchat/serve/gateway/nginx.conf` | 80/443 | Optional reverse proxy / firewall |

---

## 4. System Requirements

| Resource | Requirement |
|---|---|
| Python | ≥ 3.8 |
| OS | Linux (recommended for GPU/`bitsandbytes`), macOS (MPS/MLX), Windows (CPU/limited) |
| GPU | Optional but recommended. CUDA via `nvidia/cuda:12.2.0` in Docker. Also supports MPS, Intel XPU, Ascend NPU |
| GPU memory | ~14 GB (7B), ~28 GB (13B), ~60 GB (33B); halve with `--load-8bit` |
| System RAM (CPU mode) | ~30 GB (7B), ~60 GB (13B) |
| Disk | Model weights cached in `~/.cache/huggingface/hub/` (multiple GB per model) |
| Build tools (macOS) | `rust`, `cmake` (`brew install rust cmake`) |

### Dependency groups

Defined in `pyproject.toml`:

- **Core:** `aiohttp`, `fastapi`, `httpx`, `markdown2[all]`, `nh3`, `numpy`, `prompt_toolkit>=3.0.0`, `pydantic<3,>=2.0.0`, `pydantic-settings`, `psutil`, `requests`, `rich>=10.0.0`, `shortuuid`, `tiktoken`, `uvicorn`.
- **`model_worker`:** `accelerate>=0.21`, `peft`, `sentencepiece`, `torch`, `transformers>=4.31.0`, `protobuf`, `openai`, `anthropic`.
- **`webui`:** `gradio>=4.10`, `plotly`, `scipy`.
- **`train`:** `einops`, `flash-attn>=2.0`, `wandb`.
- **`llm_judge`:** `openai<1`, `anthropic>=0.3`, `ray`.
- **`dev`:** `black==23.3.0`, `pylint==2.8.2`.

> [!WARNING]
> The `llm_judge` extra pins `openai<1` while the `model_worker` extra uses the modern `openai` (≥1). Keep evaluation and serving in **separate virtual environments** to avoid a dependency conflict.

---

## 5. Configuration Reference

FastChat is configured entirely through **command-line flags** and **environment variables** — there is no config file for the servers. (The Arena classifier uses a `config.yaml`, but that is an evaluation tool, not the serving path.)

### Environment variables

All are read in `fastchat/constants.py`, `fastchat/utils.py`, and a few provider modules.

| Variable | Default | Source | Purpose |
|---|---|---|---|
| `FASTCHAT_CONTROLLER_HEART_BEAT_EXPIRATION` | `90` | `constants.py:59` | Seconds before a silent worker is evicted |
| `FASTCHAT_WORKER_HEART_BEAT_INTERVAL` | `45` | `constants.py:61` | Worker heartbeat interval |
| `FASTCHAT_WORKER_API_TIMEOUT` | `100` | `constants.py:62` | Controller→worker request timeout (s) |
| `FASTCHAT_WORKER_API_EMBEDDING_BATCH_SIZE` | `4` | `constants.py:64` | Embedding batch size (lower to avoid OOM) |
| `FASTCHAT_INPUT_CHAR_LEN_LIMIT` | `12000` | `constants.py:43` | Max input characters (web UI) |
| `FASTCHAT_BLIND_MODE_INPUT_CHAR_LEN_LIMIT` | `30000` | `constants.py:45` | Max input chars in Arena blind mode |
| `LOGDIR` | `.` | `constants.py:52` | Directory for log files |
| `CPU_ISA` | *(none)* | `constants.py:54` | CPU instruction set (e.g. `amx`) for accelerated CPU inference |
| `FASTCHAT_USE_MODELSCOPE` | `False` | `model_adapter.py:358` | Download weights from ModelScope |
| `PEFT_SHARE_BASE_WEIGHTS` | `false` | `model_adapter.py:52` | Share base weights across PEFT models |
| `REQUEST_TIMEOUT` | `3` | `utils.py:401` | Generic outbound request timeout (s) |
| `REMOTE_LOGGER_URL` | *(none)* | `remote_logger.py:16` | Endpoint for remote request logging |
| `GCP_PROJECT_ID` / `GCP_LOCATION` | `None` | `api_provider.py` | Vertex AI / Gemini routing |
| `AI2_API_KEY` | *(none)* | `api_provider.py:857` | AI2 (Allen Institute) API provider key |
| `SSL_KEYFILE` / `SSL_CERTFILE` | *(none)* | (read when `--ssl`) | TLS material for any service |
| `WORLD_SIZE` / `LOCAL_RANK` | `1` / *(none)* | `train_lora.py` | Distributed training rank info |

> [!NOTE]
> Other hard-coded constants in `constants.py` are **not** environment-overridable: `CONVERSATION_TURN_LIMIT = 50` and `SESSION_EXPIRATION_TIME = 3600` (seconds). Change these in source if your deployment needs different limits.

### SSL on any service

Every server (controller, workers, API server) accepts `--ssl`, which reads `SSL_KEYFILE` and `SSL_CERTFILE` from the environment:

```bash
export SSL_KEYFILE=/etc/ssl/private/server.key
export SSL_CERTFILE=/etc/ssl/certs/server.crt
python3 -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000 --ssl
```

### Dispatch configuration

```bash
python3 -m fastchat.serve.controller --dispatch-method shortest_queue   # default
python3 -m fastchat.serve.controller --dispatch-method lottery          # random by weight
```

---

## 6. Deployment

### Manual multi-process deployment

The canonical production layout binds services to `0.0.0.0` and places an Nginx gateway in front:

```bash
# Controller
python3 -m fastchat.serve.controller --host 0.0.0.0 --port 21001

# One worker per GPU
CUDA_VISIBLE_DEVICES=0 python3 -m fastchat.serve.model_worker \
  --model-path lmsys/vicuna-7b-v1.5 \
  --controller-address http://localhost:21001 \
  --worker-address http://localhost:31000 \
  --host 0.0.0.0 --port 31000

# OpenAI API front end
python3 -m fastchat.serve.openai_api_server \
  --controller-address http://localhost:21001 \
  --host 0.0.0.0 --port 8000
```

### Docker Compose

The repo ships `docker/docker-compose.yml` and `docker/Dockerfile`. The image is built on `nvidia/cuda:12.2.0-runtime-ubuntu20.04` with Python 3.9 and `pip3 install fschat[model_worker,webui]`. Compose defines three services:

| Service | Entrypoint | Ports |
|---|---|---|
| `fastchat-controller` | `controller --host 0.0.0.0 --port 21001` | `21001:21001` |
| `fastchat-model-worker` | `model_worker` (GPU-reserved, HF cache volume) | internal |
| `fastchat-api-server` | `openai_api_server --port 8000` | `8000:8000` |

Bring it up:

```bash
cd docker
docker compose up --build
```

The model is selected via environment variables consumed by the worker entrypoint:

```bash
export FASTCHAT_WORKER_MODEL_NAMES=vicuna-7b-v1.5
export FASTCHAT_WORKER_MODEL_PATH=lmsys/vicuna-7b-v1.5
docker compose up
```

A named volume `huggingface` is mounted at `/root/.cache/huggingface` so weights persist across container restarts, and the worker service reserves one NVIDIA GPU via the Compose `deploy.resources` block.

> [!CAUTION]
> The Compose worker reserves exactly **one** GPU (`count: 1`). To serve multiple models or scale out, duplicate the `fastchat-model-worker` service with distinct names, ports, and `CUDA_VISIBLE_DEVICES`, each registering to the same controller.

### Nginx gateway

`fastchat/serve/gateway/nginx.conf` provides a reverse proxy that:

1. Protects Gradio servers (acts as a firewall).
2. Supports dynamic mounting/unmounting of Gradio servers.
3. Load-balances across Gradio servers.
4. Enforces a total connection limit.
5. Reduces attack surface to a single public port.

Deploy it by copying `nginx.conf` to `/etc/nginx/nginx.conf`, replacing the upstream port (`server localhost:7860`) with your Gradio port, and reloading:

```bash
sudo nginx -t                 # validate config
sudo systemctl reload nginx
sudo systemctl status nginx
```

> [!IMPORTANT]
> When proxying the streaming API through Nginx, set `proxy_buffering off;` in the relevant `location` block or token streaming will be buffered and appear to hang.

### Cloud training (SkyPilot)

For fine-tuning on cloud spot instances (AWS/GCP/Azure/Lambda), FastChat documents a [SkyPilot](https://github.com/skypilot-org/skypilot) recipe. See `docs/training.md`.

---

## 7. Security

> [!CAUTION]
> FastChat's serving components are designed for **trusted/internal networks or behind a hardened gateway**. They do not provide user authentication, per-user authorization, or input sanitization beyond optional static API keys and content moderation. Treat a directly internet-exposed deployment as a risk.

### Authentication & transport

- **API keys:** the OpenAI API server (and `launch_all_serve`) accept `--api-keys key1,key2`. This is a static shared-secret list, not per-user auth. Without it, the API is open.
- **TLS:** `--ssl` on each service enables HTTPS using `SSL_KEYFILE` / `SSL_CERTFILE`. There is no automatic certificate management.
- **CORS:** the API server exposes `--allowed-origins`, `--allowed-methods`, `--allowed-headers`, and `--allow-credentials`. Defaults are permissive (`*`); restrict them for production.
- **Gradio auth:** the web servers accept `--gradio-auth-path` (a file of `user:password` pairs) and `--password` for basic gating.

### Content moderation

The Gradio servers accept `--moderate` to block unsafe inputs, and `constants.py` defines moderation messages (`MODERATION_MSG`, `TEXT_MODERATION_MSG`, `IMAGE_MODERATION_MSG`). Input length caps (`FASTCHAT_INPUT_CHAR_LEN_LIMIT`, default 12,000) and a 50-turn `CONVERSATION_TURN_LIMIT` bound abuse.

### Rate limiting

`fastchat/serve/call_monitor.py` implements per-model and per-user call accounting with endpoints such as `/get_model_call_limit/{model}`, `/update_model_call_limit/{model}/{limit}`, and `/is_limit_reached`. This is the mechanism the hosted Arena uses to throttle traffic.

### Error codes

`fastchat/constants.py` defines an `ErrorCode` enum aligned to OpenAI error semantics, including `RATE_LIMIT` (42901), `QUOTA_EXCEEDED` (42902), `CONTEXT_OVERFLOW` (40303), `CONTROLLER_NO_WORKER` (50005), and `CUDA_OUT_OF_MEMORY` (50002). Front ends map internal failures to these codes.

### Community-reported security issues (as of the documented commit)

These are open issues on the public tracker. Operators should be aware of them when exposing FastChat:

| Issue | Area | Mitigation |
|---|---|---|
| [#3873](https://github.com/lm-sys/FastChat/issues/3873) | SSRF + arbitrary local file read in `fastchat.utils.load_image` via unvalidated image URL/path | Disable/validate image-URL inputs; run workers with least privilege and no sensitive local files |
| [#3834](https://github.com/lm-sys/FastChat/issues/3834) | Content-moderation bypass in Arena side-by-side views | Keep `--moderate` on; apply upstream fixes |
| [#3833](https://github.com/lm-sys/FastChat/issues/3833) | DoS via blocking event loop in model workers | Front with a gateway connection limit; isolate workers |

> [!WARNING]
> The SSRF/local-file-read report (#3873) is the most serious for self-hosters: if your deployment accepts image URLs (vision arena), validate and allow-list them, and ensure the worker process cannot read sensitive files. Run workers as an unprivileged user in a container.

### Trusting remote code

Several workers expose `--trust_remote_code` (vLLM, SGLang, MLX). Enabling it lets a Hugging Face model execute custom modeling code on download — only enable for models you trust.

---

## 8. Operations

### Starting and verifying the stack

```bash
# Start controller, then worker(s), then a front end (each in its own process)
python3 -m fastchat.serve.controller
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5

# Verify the worker registered and responds
python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5

# Smoke-test throughput
python3 -m fastchat.serve.test_throughput --model-name vicuna-7b-v1.5 --n-thread 8
```

### Graceful shutdown

`fastchat/serve/shutdown_serve.py` can bring down components:

```bash
python3 -m fastchat.serve.shutdown_serve --down all
```

A `tests/killall_python.sh` helper exists to force-kill stray Python serving processes during development.

### Logs

All services log via `build_logger` (in `fastchat/utils.py`) to files under `LOGDIR` (default current directory): `controller.log`, worker logs, etc. Set `LOGDIR` to a persistent path in production. `REMOTE_LOGGER_URL` can forward request logs to an external collector.

### Health & introspection endpoints

The controller exposes operational endpoints for inspecting cluster state:

| Endpoint | Purpose |
|---|---|
| `POST /list_models` | All models currently served |
| `POST /list_language_models` / `POST /list_multimodal_models` | Filter by modality |
| `POST /get_worker_address` | Which worker serves a model |
| `POST /refresh_all_workers` | Force re-poll of all workers |
| `POST /receive_heart_beat` | (internal) worker liveness |

Each worker exposes `POST /worker_get_status`, `GET /test_connection` (controller side), and `POST /model_details`.

### Common operational adjustments

- **Slow model marked dead:** raise `FASTCHAT_WORKER_API_TIMEOUT` and lower the heartbeat expiration sensitivity.
- **Embedding OOM:** set `FASTCHAT_WORKER_API_EMBEDDING_BATCH_SIZE=1`.
- **Concurrency tuning:** `--limit-worker-concurrency` (default 5 for HF worker, 1024 for vLLM/SGLang/LightLLM) caps simultaneous requests per worker.

---

## 9. Performance & Scaling

### Choosing a backend

| Backend | Worker | When to use |
|---|---|---|
| HF Transformers | `model_worker` | Maximum model compatibility; lower throughput |
| vLLM | `vllm_worker` | High-throughput batched serving; paged attention |
| SGLang | `sglang_worker` | High throughput with structured generation |
| LightLLM | `lightllm_worker` | Long-context, fine-grained kernel control |
| MLX | `mlx_worker` | Apple Silicon |
| DashInfer | `dashinfer_worker` | CPU serving |

> [!TIP]
> The default HF worker has "great compatibility but can be slow." For production traffic, switch to `vllm_worker` or `sglang_worker` and tune `--gpu_memory_utilization` / `--mem-fraction-static` (both default `0.9`).

### Scaling out

- **Throughput for one model:** launch several workers for the same model on different GPUs/ports; the controller balances by `shortest_queue`.
- **Many models, one endpoint:** register heterogeneous workers to one controller, or use `multi_model_worker` to co-locate models that share a base (PEFT/LoRA via `PeftModelAdapter`, gated by `PEFT_SHARE_BASE_WEIGHTS`).
- **Memory pressure:** model parallelism (`--num-gpus`), `--max-gpu-memory`, `--load-8bit`, and `--cpu-offloading` trade speed for capacity (see User Guide §4).

### Quantization paths

GPTQ (`--gptq-*`), AWQ (`--awq-*`), ExLlama V2 (`--enable-exllama` + `--exllama-*`), and xFasterTransformer (`--enable-xft`, CPU). Each reduces VRAM at some quality/throughput cost; backend-specific docs live under `docs/`.

---

## 10. Repository Structure

```text
FastChat/
├── fastchat/
│   ├── constants.py            Global constants & env-var defaults; ErrorCode enum
│   ├── conversation.py         Prompt templates / conversation formatting (SepStyle)
│   ├── utils.py                Logging, dtype helpers, image loading, request helpers
│   ├── model/                  Model adapters, quantization, weight ops (19 files)
│   │   ├── model_adapter.py    Adapter registry; add_model_args() (all model/HW flags)
│   │   ├── model_registry.py   Display metadata for known models
│   │   ├── compression.py      8-bit compression
│   │   ├── apply_delta.py / make_delta.py / apply_lora.py / upload_hub.py
│   │   └── model_*.py          Per-architecture adapters (chatglm, falcon, yuan2, ...)
│   ├── modules/                Quantization configs: awq, gptq, exllama, xfastertransformer
│   ├── protocol/               Pydantic schemas
│   │   ├── openai_api_protocol.py   OpenAI request/response models
│   │   └── api_protocol.py          Worker protocol models
│   ├── serve/                  The serving system (30 files) — see Component Inventory
│   │   ├── controller.py           Worker registry & dispatch
│   │   ├── base_model_worker.py    Shared worker HTTP endpoints
│   │   ├── model_worker.py / vllm_worker.py / sglang_worker.py / ...
│   │   ├── openai_api_server.py     OpenAI-compatible REST API
│   │   ├── gradio_web_server.py / _multi.py   Web UIs / Arena
│   │   ├── api_provider.py          External API model providers (OpenAI/Anthropic/Gemini)
│   │   ├── call_monitor.py          Rate-limit accounting
│   │   ├── inference.py             Core generate/stream loop for HF backend
│   │   ├── gateway/nginx.conf       Reverse-proxy config
│   │   └── monitor/                 Arena analytics: Elo, leaderboard, clustering
│   ├── llm_judge/              MT-Bench evaluation pipeline (9 files)
│   ├── train/                  Fine-tuning scripts (12 files: train_mem, train_lora, ...)
│   └── data/                   Data-prep pipeline (clean/split/sample/merge, 16 files)
├── docker/                     Dockerfile + docker-compose.yml
├── docs/                       Integration & how-to docs (vllm, awq, gptq, openai_api, ...)
├── playground/                 API examples, embedding tests, DeepSpeed configs
├── tests/                      Unit / integration tests
├── pyproject.toml              Package metadata + dependency extras
└── README.md
```

### Key modules explained

- **`model/model_adapter.py`** — the heart of model support. `add_model_args()` defines every shared model/hardware CLI flag, and the adapter registry maps model paths to loading logic and conversation templates. Adding a new model means adding an adapter here.
- **`serve/inference.py`** — the streaming generation loop (`chat_loop`, `generate_stream`) used by the CLI and the default worker.
- **`serve/api_provider.py`** — routes Arena requests to external hosted models (OpenAI, Anthropic via `anthropic_message`, Gemini, AI2, Vertex), reading provider keys from the environment.
- **`protocol/openai_api_protocol.py`** — Pydantic models that define the OpenAI-compatible request/response contract.

---

## 11. Internal API Reference

These are the inter-service ("worker protocol") endpoints. They are not the public OpenAI API (see User Guide §6) but are useful for monitoring, debugging, and building custom front ends.

### Worker endpoints (every worker implements)

| Method | Path | Purpose |
|---|---|---|
| `POST` | `/worker_generate_stream` | Streaming token generation |
| `POST` | `/worker_generate` | Non-streaming generation |
| `POST` | `/worker_get_embeddings` | Embeddings |
| `POST` | `/worker_get_status` | Queue length, speed, model names |
| `POST` | `/count_token` | Token count for input |
| `POST` | `/worker_get_conv_template` | Conversation template in use |
| `POST` | `/model_details` | Context length and model metadata |

### Controller endpoints

| Method | Path | Purpose |
|---|---|---|
| `POST` | `/register_worker` | Register a worker |
| `POST` | `/refresh_all_workers` | Re-poll all workers |
| `POST` | `/list_models` | List served models |
| `POST` | `/list_language_models` | Text-only models |
| `POST` | `/list_multimodal_models` | Multimodal models |
| `POST` | `/get_worker_address` | Resolve a model to a worker |
| `POST` | `/receive_heart_beat` | Worker heartbeat |
| `POST` | `/worker_generate_stream` | Proxy generation to a worker |
| `POST` | `/worker_get_status` | Aggregate worker status |
| `GET` | `/test_connection` | Liveness probe |

### Call monitor endpoints

| Method | Path | Purpose |
|---|---|---|
| `GET` | `/get_model_call_limit/{model}` | Current per-model limit |
| `GET` | `/update_model_call_limit/{model}/{limit}` | Set per-model limit |
| `GET` | `/is_limit_reached` | Whether a limit is hit |
| `GET` | `/get_num_users_hr` / `/get_num_users_day` | Active user counts |
| `GET` | `/get_user_call_stats` / `/get_model_call_stats` | Usage statistics |

### Public OpenAI-compatible endpoints (API server)

| Method | Path | Function |
|---|---|---|
| `GET` | `/v1/models` | `show_available_models` |
| `POST` | `/v1/chat/completions` | `create_chat_completion` |
| `POST` | `/v1/completions` | `create_completion` |
| `POST` | `/v1/embeddings` | `create_embeddings` |
| `POST` | `/v1/engines/{model_name}/embeddings` | `create_embeddings` |
| `POST` | `/api/v1/token_check` | `count_tokens` |

---

## 12. Code Patterns & Conventions

- **FastAPI + Uvicorn everywhere.** Each server is a FastAPI app launched under Uvicorn; every worker variant re-implements the same seven `/worker_*` endpoints, inheriting from `base_model_worker.py`. This uniformity is what makes backends swappable.
- **Adapter pattern for models.** `model_adapter.py` resolves a model path to an adapter that knows how to load weights and which conversation template to apply. Unrecognized paths raise `ValueError: No valid model adapter for <path>`.
- **Pydantic v2 protocols.** Request/response shapes live in `protocol/` as Pydantic models, decoupling the wire format from server logic.
- **Configuration via env + argparse.** No server config files; defaults live in `constants.py` and `add_model_args()`. This keeps deployment scriptable but means changes require flags or env vars, not a config file.
- **In-memory, heartbeat-driven state.** The controller holds a `Dict[str, WorkerInfo]` rebuilt from registrations and heartbeats; a background thread expires stale workers. No external state store.
- **`ValueError` for user-facing validation.** The codebase raises `ValueError` with descriptive f-strings for invalid input (43 such sites), e.g. invalid device, style, dispatch method, or stop-field type.

### Notable conventions to follow when extending

- Add a new inference backend by copying an existing `*_worker.py`, implementing the seven worker endpoints, and reusing `base_model_worker.py`.
- Add a new model by registering an adapter in `model_adapter.py` and (if needed) a conversation template in `conversation.py`.
- Add a new external provider by extending `api_provider.py` and reading its key from the environment.

---

## 13. Testing

Tests live in `tests/` (see `tests/README.md`). They are run directly as scripts rather than through a single test runner.

| Test | Command | Covers |
|---|---|---|
| CLI inference | `python3 test_cli.py` | Terminal chat loop |
| OpenAI API | `python3 launch_openai_api_test_server.py` then `python3 test_openai_api.py` | REST API end-to-end |
| OpenAI + LangChain | `python3 test_openai_langchain.py` | LangChain integration |
| Vision API | `python3 test_openai_vision_api.py` | Multimodal requests |
| Image utils | `python3 test_image_utils.py` | Image handling helpers |
| Load test | `python3 load_test.py` | Throughput under load |

GUI and PEFT serving have manual test procedures documented in `tests/README.md` (launch a controller + workers, then a multi web server).

> [!NOTE]
> There is no CI-style single-command suite and limited automated coverage. Treat the tests as integration smoke tests; stand up a controller + worker before running the serving tests.

---

## 14. Known Issues & Potential Improvements

### Open issues (public tracker, as of the documented commit)

| Issue | Type | Notes |
|---|---|---|
| [#1048](https://github.com/lm-sys/FastChat/issues/1048) | Bug | `stop` parameter ignored in OpenAI API since v0.2.5 |
| [#3876](https://github.com/lm-sys/FastChat/issues/3876) | Bug | WebSocket deadlock — stuck generation loop |
| [#3861](https://github.com/lm-sys/FastChat/issues/3861) / [#3866](https://github.com/lm-sys/FastChat/issues/3866) | Bug | Session hang / infinite loading on long conversations |
| [#3833](https://github.com/lm-sys/FastChat/issues/3833) | Security | DoS via blocking event loop in workers |
| [#3834](https://github.com/lm-sys/FastChat/issues/3834) | Security | Content-moderation bypass in Arena |
| [#3873](https://github.com/lm-sys/FastChat/issues/3873) | Security | SSRF + arbitrary local file read in `load_image` |

### Improvement opportunities

- **OpenAI API completeness.** `docs/openai_api.md` lists unimplemented parameters: `logprobs`, `logit_bias`, `user`, `presence_penalty`, `frequency_penalty`, model permission/owner metadata, an Edits API, and built-in rate-limit settings.
- **Async worker I/O.** The DoS report (#3833) points at blocking calls in the worker event loop; moving inference dispatch fully off the event loop would harden against slow-client DoS.
- **Input validation for `load_image`.** Allow-list schemes/hosts and forbid local paths to close the SSRF/file-read vector (#3873).
- **Dependency conflict.** The `llm_judge` extra (`openai<1`) conflicts with `model_worker` (`openai>=1`); consolidating on the modern OpenAI SDK would let serving and evaluation share one environment.
- **Configuration management.** Server config is flags/env only; a structured config file would simplify reproducible deployments.
- **Automated test coverage.** The script-based tests could be wrapped in a pytest suite with fixtures that spin up an ephemeral controller + worker.

> [!IMPORTANT]
> The security issues above are community reports against the documented commit and may be partially fixed upstream. Before exposing FastChat publicly, check the current tracker, run behind the Nginx gateway with a connection limit, require `--api-keys`, enable `--ssl`, and run workers as an unprivileged containerized user.

---

## 15. Quick Reference

### Default ports

| Service | Port |
|---|---|
| Controller | 21001 |
| Model worker | 21002 |
| OpenAI API server | 8000 |
| LightLLM worker | 8000 |
| Gradio web server | auto-assigned |

### Bring up a minimal stack

```bash
python3 -m fastchat.serve.controller --host 0.0.0.0 --port 21001
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5
python3 -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000 --api-keys SECRET
```

### Hardening checklist

- [ ] Bind internal services to `localhost`/private network; expose only the gateway
- [ ] Front with `fastchat/serve/gateway/nginx.conf` (connection limit, single public port)
- [ ] Require `--api-keys` on the OpenAI API server
- [ ] Enable `--ssl` (set `SSL_KEYFILE` / `SSL_CERTFILE`) on public-facing services
- [ ] Restrict CORS (`--allowed-origins`, `--allowed-methods`) away from `*`
- [ ] Run workers as an unprivileged user in a container (mitigate #3873)
- [ ] Keep `--moderate` on for any public Gradio/Arena server
- [ ] Set `LOGDIR` to a persistent, monitored path
- [ ] Avoid `--trust_remote_code` except for trusted models

### Operational env-var cheat sheet

```bash
export FASTCHAT_WORKER_API_TIMEOUT=600                 # slow models
export FASTCHAT_WORKER_API_EMBEDDING_BATCH_SIZE=1       # embedding OOM
export FASTCHAT_CONTROLLER_HEART_BEAT_EXPIRATION=120    # tolerate slow heartbeats
export LOGDIR=/var/log/fastchat                         # persistent logs
export FASTCHAT_USE_MODELSCOPE=True                     # ModelScope weights
```

---

_Generated June 2026 from lm-sys/FastChat at 587d5cf. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._