FastChat Admin & Architecture Guide

This guide is for operators, architects, and developers deploying, scaling, securing, or extending FastChat. It covers the distributed serving architecture, the complete configuration surface, deployment strategies (including Docker), security considerations, operational procedures, and the internal module layout. For end-user install and usage, see the User Guide.

FastChat (fschat 0.2.36) is an open platform for training, serving, and evaluating LLM-based chatbots. In production it powers Chatbot Arena (lmarena.ai), which has served 10M+ chat requests across 70+ models. The serving layer is a controller + worker + front-end system built on FastAPI and Uvicorn.

Table of Contents

1. Executive Summary
2. System Architecture
3. Component Inventory
4. System Requirements
5. Configuration Reference
6. Deployment
7. Security
8. Operations
9. Performance & Scaling
10. Repository Structure
11. Internal API Reference
12. Code Patterns & Conventions
13. Testing
14. Known Issues & Potential Improvements
15. Quick Reference

1. Executive Summary

FastChat decomposes LLM serving into three independently deployable tiers that communicate over HTTP:

• A controller that maintains a registry of model workers and routes each request to one.
• One or more model workers, each hosting one or more models on accelerators, using a pluggable inference backend (Hugging Face Transformers, vLLM, SGLang, LightLLM, MLX, DashInfer, or remote HF APIs).
• Front ends — a Gradio web UI and an OpenAI-compatible REST API server — that ask the controller where to send requests.

Important

FastChat keeps no persistent server state beyond log files. The worker registry is rebuilt from heartbeats, so a controller restart is recovered automatically as workers re-register. There is no database to back up.

2. System Architecture

                         clients (browser, OpenAI SDK, curl)
                                       │
              ┌────────────────────────┼────────────────────────┐
              ▼                        ▼                         ▼
   ┌────────────────────┐   ┌────────────────────┐   (optional Nginx
   │ gradio_web_server  │   │ openai_api_server  │    gateway in front:
   │  / _multi (Arena)  │   │   (FastAPI :8000)  │    fastchat/serve/
   └─────────┬──────────┘   └─────────┬──────────┘    gateway/nginx.conf)
             │  "address for model X?" │
             └───────────┬─────────────┘
                         ▼
                ┌──────────────────┐   register_worker / receive_heart_beat
                │   controller     │ ◀─────────────────────────────────────┐
                │  (FastAPI :21001)│   get_worker_address (dispatch)        │
                └──────────────────┘                                        │
                         │ forwards generate request                        │
        ┌────────────────┼────────────────────────┐                        │
        ▼                ▼                         ▼                        │
 ┌─────────────┐  ┌─────────────┐          ┌─────────────┐                  │
 │ model_worker│  │ vllm_worker │   ...    │ multi_model │ ─────────────────┘
 │  (:21002)   │  │  (:21002)   │          │   _worker   │  heartbeats
 │  HF Trans-  │  │   vLLM      │          │ many models │
 │  formers    │  │  backend    │          │ one process │
 └─────────────┘  └─────────────┘          └─────────────┘
        │ loads weights from
        ▼
  ~/.cache/huggingface/hub/  (or ModelScope, or local path)

Request lifecycle (OpenAI API path)

1. A client POSTs to /v1/chat/completions on the API server.
2. The API server calls the controller's get_worker_address for the requested model.
3. The controller selects a worker by its dispatch method (shortest_queue or lottery) and returns the worker URL.
4. The API server streams /worker_generate_stream from that worker.
5. The worker runs inference on its backend and streams tokens back, which the API server reshapes into the OpenAI response schema.

Worker registration & health

• On startup, a worker calls the controller's register_worker (unless --no-register).
• Workers send heartbeats every FASTCHAT_WORKER_HEART_BEAT_INTERVAL seconds (default 45).
• The controller's background heart_beat_controller thread removes workers whose last heartbeat is older than FASTCHAT_CONTROLLER_HEART_BEAT_EXPIRATION seconds (default 90).
• DispatchMethod (in controller.py) is SHORTEST_QUEUE (default) or LOTTERY; shortest_queue balances by reported queue length and worker speed.

Note

The controller-to-worker timeout is FASTCHAT_WORKER_API_TIMEOUT (default 100s). Slow models on slow hardware should raise this or they will appear to fail.

3. Component Inventory

4. System Requirements

Dependency groups

Defined in pyproject.toml:

• Core: aiohttp, fastapi, httpx, markdown2[all], nh3, numpy, prompt_toolkit>=3.0.0, pydantic<3,>=2.0.0, pydantic-settings, psutil, requests, rich>=10.0.0, shortuuid, tiktoken, uvicorn.
• model_worker: accelerate>=0.21, peft, sentencepiece, torch, transformers>=4.31.0, protobuf, openai, anthropic.
• webui: gradio>=4.10, plotly, scipy.
• train: einops, flash-attn>=2.0, wandb.
• llm_judge: openai<1, anthropic>=0.3, ray.
• dev: black==23.3.0, pylint==2.8.2.

Warning

The llm_judge extra pins openai<1 while the model_worker extra uses the modern openai (≥1). Keep evaluation and serving in separate virtual environments to avoid a dependency conflict.

5. Configuration Reference

FastChat is configured entirely through command-line flags and environment variables — there is no config file for the servers. (The Arena classifier uses a config.yaml, but that is an evaluation tool, not the serving path.)

Environment variables

All are read in fastchat/constants.py, fastchat/utils.py, and a few provider modules.

Note

Other hard-coded constants in constants.py are not environment-overridable: CONVERSATION_TURN_LIMIT = 50 and SESSION_EXPIRATION_TIME = 3600 (seconds). Change these in source if your deployment needs different limits.

SSL on any service

Every server (controller, workers, API server) accepts --ssl, which reads SSL_KEYFILE and SSL_CERTFILE from the environment:

export SSL_KEYFILE=/etc/ssl/private/server.key
export SSL_CERTFILE=/etc/ssl/certs/server.crt
python3 -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000 --ssl

Dispatch configuration

python3 -m fastchat.serve.controller --dispatch-method shortest_queue   # default
python3 -m fastchat.serve.controller --dispatch-method lottery          # random by weight

6. Deployment

Manual multi-process deployment

The canonical production layout binds services to 0.0.0.0 and places an Nginx gateway in front:

# Controller
python3 -m fastchat.serve.controller --host 0.0.0.0 --port 21001

# One worker per GPU
CUDA_VISIBLE_DEVICES=0 python3 -m fastchat.serve.model_worker \
  --model-path lmsys/vicuna-7b-v1.5 \
  --controller-address http://localhost:21001 \
  --worker-address http://localhost:31000 \
  --host 0.0.0.0 --port 31000

# OpenAI API front end
python3 -m fastchat.serve.openai_api_server \
  --controller-address http://localhost:21001 \
  --host 0.0.0.0 --port 8000

Docker Compose

The repo ships docker/docker-compose.yml and docker/Dockerfile. The image is built on nvidia/cuda:12.2.0-runtime-ubuntu20.04 with Python 3.9 and pip3 install fschat[model_worker,webui]. Compose defines three services:

Bring it up:

cd docker
docker compose up --build

The model is selected via environment variables consumed by the worker entrypoint:

export FASTCHAT_WORKER_MODEL_NAMES=vicuna-7b-v1.5
export FASTCHAT_WORKER_MODEL_PATH=lmsys/vicuna-7b-v1.5
docker compose up

A named volume huggingface is mounted at /root/.cache/huggingface so weights persist across container restarts, and the worker service reserves one NVIDIA GPU via the Compose deploy.resources block.

Caution

The Compose worker reserves exactly one GPU (count: 1). To serve multiple models or scale out, duplicate the fastchat-model-worker service with distinct names, ports, and CUDA_VISIBLE_DEVICES, each registering to the same controller.

Nginx gateway

fastchat/serve/gateway/nginx.conf provides a reverse proxy that:

1. Protects Gradio servers (acts as a firewall).
2. Supports dynamic mounting/unmounting of Gradio servers.
3. Load-balances across Gradio servers.
4. Enforces a total connection limit.
5. Reduces attack surface to a single public port.

Deploy it by copying nginx.conf to /etc/nginx/nginx.conf, replacing the upstream port (server localhost:7860) with your Gradio port, and reloading:

sudo nginx -t                 # validate config
sudo systemctl reload nginx
sudo systemctl status nginx

Important

When proxying the streaming API through Nginx, set proxy_buffering off; in the relevant location block or token streaming will be buffered and appear to hang.

Cloud training (SkyPilot)

For fine-tuning on cloud spot instances (AWS/GCP/Azure/Lambda), FastChat documents a SkyPilot recipe. See docs/training.md.

7. Security

Caution

FastChat's serving components are designed for trusted/internal networks or behind a hardened gateway. They do not provide user authentication, per-user authorization, or input sanitization beyond optional static API keys and content moderation. Treat a directly internet-exposed deployment as a risk.

Authentication & transport

• API keys: the OpenAI API server (and launch_all_serve) accept --api-keys key1,key2. This is a static shared-secret list, not per-user auth. Without it, the API is open.
• TLS: --ssl on each service enables HTTPS using SSL_KEYFILE / SSL_CERTFILE. There is no automatic certificate management.
• CORS: the API server exposes --allowed-origins, --allowed-methods, --allowed-headers, and --allow-credentials. Defaults are permissive (*); restrict them for production.
• Gradio auth: the web servers accept --gradio-auth-path (a file of user:password pairs) and --password for basic gating.

Content moderation

The Gradio servers accept --moderate to block unsafe inputs, and constants.py defines moderation messages (MODERATION_MSG, TEXT_MODERATION_MSG, IMAGE_MODERATION_MSG). Input length caps (FASTCHAT_INPUT_CHAR_LEN_LIMIT, default 12,000) and a 50-turn CONVERSATION_TURN_LIMIT bound abuse.

Rate limiting

fastchat/serve/call_monitor.py implements per-model and per-user call accounting with endpoints such as /get_model_call_limit/{model}, /update_model_call_limit/{model}/{limit}, and /is_limit_reached. This is the mechanism the hosted Arena uses to throttle traffic.

Error codes

fastchat/constants.py defines an ErrorCode enum aligned to OpenAI error semantics, including RATE_LIMIT (42901), QUOTA_EXCEEDED (42902), CONTEXT_OVERFLOW (40303), CONTROLLER_NO_WORKER (50005), and CUDA_OUT_OF_MEMORY (50002). Front ends map internal failures to these codes.

Community-reported security issues (as of the documented commit)

These are open issues on the public tracker. Operators should be aware of them when exposing FastChat:

Warning

The SSRF/local-file-read report (#3873) is the most serious for self-hosters: if your deployment accepts image URLs (vision arena), validate and allow-list them, and ensure the worker process cannot read sensitive files. Run workers as an unprivileged user in a container.

Trusting remote code

Several workers expose --trust_remote_code (vLLM, SGLang, MLX). Enabling it lets a Hugging Face model execute custom modeling code on download — only enable for models you trust.

8. Operations

Starting and verifying the stack

# Start controller, then worker(s), then a front end (each in its own process)
python3 -m fastchat.serve.controller
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5

# Verify the worker registered and responds
python3 -m fastchat.serve.test_message --model-name vicuna-7b-v1.5

# Smoke-test throughput
python3 -m fastchat.serve.test_throughput --model-name vicuna-7b-v1.5 --n-thread 8

Graceful shutdown

fastchat/serve/shutdown_serve.py can bring down components:

python3 -m fastchat.serve.shutdown_serve --down all

A tests/killall_python.sh helper exists to force-kill stray Python serving processes during development.

Logs

All services log via build_logger (in fastchat/utils.py) to files under LOGDIR (default current directory): controller.log, worker logs, etc. Set LOGDIR to a persistent path in production. REMOTE_LOGGER_URL can forward request logs to an external collector.

Health & introspection endpoints

The controller exposes operational endpoints for inspecting cluster state:

Each worker exposes POST /worker_get_status, GET /test_connection (controller side), and POST /model_details.

Common operational adjustments

• Slow model marked dead: raise FASTCHAT_WORKER_API_TIMEOUT and lower the heartbeat expiration sensitivity.
• Embedding OOM: set FASTCHAT_WORKER_API_EMBEDDING_BATCH_SIZE=1.
• Concurrency tuning: --limit-worker-concurrency (default 5 for HF worker, 1024 for vLLM/SGLang/LightLLM) caps simultaneous requests per worker.

9. Performance & Scaling

Choosing a backend

Tip

The default HF worker has "great compatibility but can be slow." For production traffic, switch to vllm_worker or sglang_worker and tune --gpu_memory_utilization / --mem-fraction-static (both default 0.9).

Scaling out

• Throughput for one model: launch several workers for the same model on different GPUs/ports; the controller balances by shortest_queue.
• Many models, one endpoint: register heterogeneous workers to one controller, or use multi_model_worker to co-locate models that share a base (PEFT/LoRA via PeftModelAdapter, gated by PEFT_SHARE_BASE_WEIGHTS).
• Memory pressure: model parallelism (--num-gpus), --max-gpu-memory, --load-8bit, and --cpu-offloading trade speed for capacity (see User Guide §4).

Quantization paths

GPTQ (--gptq-*), AWQ (--awq-*), ExLlama V2 (--enable-exllama + --exllama-*), and xFasterTransformer (--enable-xft, CPU). Each reduces VRAM at some quality/throughput cost; backend-specific docs live under docs/.

10. Repository Structure

FastChat/
├── fastchat/
│   ├── constants.py            Global constants & env-var defaults; ErrorCode enum
│   ├── conversation.py         Prompt templates / conversation formatting (SepStyle)
│   ├── utils.py                Logging, dtype helpers, image loading, request helpers
│   ├── model/                  Model adapters, quantization, weight ops (19 files)
│   │   ├── model_adapter.py    Adapter registry; add_model_args() (all model/HW flags)
│   │   ├── model_registry.py   Display metadata for known models
│   │   ├── compression.py      8-bit compression
│   │   ├── apply_delta.py / make_delta.py / apply_lora.py / upload_hub.py
│   │   └── model_*.py          Per-architecture adapters (chatglm, falcon, yuan2, ...)
│   ├── modules/                Quantization configs: awq, gptq, exllama, xfastertransformer
│   ├── protocol/               Pydantic schemas
│   │   ├── openai_api_protocol.py   OpenAI request/response models
│   │   └── api_protocol.py          Worker protocol models
│   ├── serve/                  The serving system (30 files) — see Component Inventory
│   │   ├── controller.py           Worker registry & dispatch
│   │   ├── base_model_worker.py    Shared worker HTTP endpoints
│   │   ├── model_worker.py / vllm_worker.py / sglang_worker.py / ...
│   │   ├── openai_api_server.py     OpenAI-compatible REST API
│   │   ├── gradio_web_server.py / _multi.py   Web UIs / Arena
│   │   ├── api_provider.py          External API model providers (OpenAI/Anthropic/Gemini)
│   │   ├── call_monitor.py          Rate-limit accounting
│   │   ├── inference.py             Core generate/stream loop for HF backend
│   │   ├── gateway/nginx.conf       Reverse-proxy config
│   │   └── monitor/                 Arena analytics: Elo, leaderboard, clustering
│   ├── llm_judge/              MT-Bench evaluation pipeline (9 files)
│   ├── train/                  Fine-tuning scripts (12 files: train_mem, train_lora, ...)
│   └── data/                   Data-prep pipeline (clean/split/sample/merge, 16 files)
├── docker/                     Dockerfile + docker-compose.yml
├── docs/                       Integration & how-to docs (vllm, awq, gptq, openai_api, ...)
├── playground/                 API examples, embedding tests, DeepSpeed configs
├── tests/                      Unit / integration tests
├── pyproject.toml              Package metadata + dependency extras
└── README.md

Key modules explained

• model/model_adapter.py — the heart of model support. add_model_args() defines every shared model/hardware CLI flag, and the adapter registry maps model paths to loading logic and conversation templates. Adding a new model means adding an adapter here.
• serve/inference.py — the streaming generation loop (chat_loop, generate_stream) used by the CLI and the default worker.
• serve/api_provider.py — routes Arena requests to external hosted models (OpenAI, Anthropic via anthropic_message, Gemini, AI2, Vertex), reading provider keys from the environment.
• protocol/openai_api_protocol.py — Pydantic models that define the OpenAI-compatible request/response contract.

11. Internal API Reference

These are the inter-service ("worker protocol") endpoints. They are not the public OpenAI API (see User Guide §6) but are useful for monitoring, debugging, and building custom front ends.

Worker endpoints (every worker implements)

Controller endpoints

Call monitor endpoints

Public OpenAI-compatible endpoints (API server)

12. Code Patterns & Conventions

• FastAPI + Uvicorn everywhere. Each server is a FastAPI app launched under Uvicorn; every worker variant re-implements the same seven /worker_* endpoints, inheriting from base_model_worker.py. This uniformity is what makes backends swappable.
• Adapter pattern for models. model_adapter.py resolves a model path to an adapter that knows how to load weights and which conversation template to apply. Unrecognized paths raise ValueError: No valid model adapter for <path>.
• Pydantic v2 protocols. Request/response shapes live in protocol/ as Pydantic models, decoupling the wire format from server logic.
• Configuration via env + argparse. No server config files; defaults live in constants.py and add_model_args(). This keeps deployment scriptable but means changes require flags or env vars, not a config file.
• In-memory, heartbeat-driven state. The controller holds a Dict[str, WorkerInfo] rebuilt from registrations and heartbeats; a background thread expires stale workers. No external state store.
• ValueError for user-facing validation. The codebase raises ValueError with descriptive f-strings for invalid input (43 such sites), e.g. invalid device, style, dispatch method, or stop-field type.

Notable conventions to follow when extending

• Add a new inference backend by copying an existing *_worker.py, implementing the seven worker endpoints, and reusing base_model_worker.py.
• Add a new model by registering an adapter in model_adapter.py and (if needed) a conversation template in conversation.py.
• Add a new external provider by extending api_provider.py and reading its key from the environment.

13. Testing

Tests live in tests/ (see tests/README.md). They are run directly as scripts rather than through a single test runner.

GUI and PEFT serving have manual test procedures documented in tests/README.md (launch a controller + workers, then a multi web server).

Note

There is no CI-style single-command suite and limited automated coverage. Treat the tests as integration smoke tests; stand up a controller + worker before running the serving tests.

14. Known Issues & Potential Improvements

Open issues (public tracker, as of the documented commit)

Improvement opportunities

• OpenAI API completeness. docs/openai_api.md lists unimplemented parameters: logprobs, logit_bias, user, presence_penalty, frequency_penalty, model permission/owner metadata, an Edits API, and built-in rate-limit settings.
• Async worker I/O. The DoS report (#3833) points at blocking calls in the worker event loop; moving inference dispatch fully off the event loop would harden against slow-client DoS.
• Input validation for load_image. Allow-list schemes/hosts and forbid local paths to close the SSRF/file-read vector (#3873).
• Dependency conflict. The llm_judge extra (openai<1) conflicts with model_worker (openai>=1); consolidating on the modern OpenAI SDK would let serving and evaluation share one environment.
• Configuration management. Server config is flags/env only; a structured config file would simplify reproducible deployments.
• Automated test coverage. The script-based tests could be wrapped in a pytest suite with fixtures that spin up an ephemeral controller + worker.

Important

The security issues above are community reports against the documented commit and may be partially fixed upstream. Before exposing FastChat publicly, check the current tracker, run behind the Nginx gateway with a connection limit, require --api-keys, enable --ssl, and run workers as an unprivileged containerized user.

15. Quick Reference

Default ports

Bring up a minimal stack

python3 -m fastchat.serve.controller --host 0.0.0.0 --port 21001
python3 -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5
python3 -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000 --api-keys SECRET

Hardening checklist

• [ ] Bind internal services to localhost/private network; expose only the gateway
• [ ] Front with fastchat/serve/gateway/nginx.conf (connection limit, single public port)
• [ ] Require --api-keys on the OpenAI API server
• [ ] Enable --ssl (set SSL_KEYFILE / SSL_CERTFILE) on public-facing services
• [ ] Restrict CORS (--allowed-origins, --allowed-methods) away from *
• [ ] Run workers as an unprivileged user in a container (mitigate #3873)
• [ ] Keep --moderate on for any public Gradio/Arena server
• [ ] Set LOGDIR to a persistent, monitored path
• [ ] Avoid --trust_remote_code except for trusted models

Operational env-var cheat sheet

export FASTCHAT_WORKER_API_TIMEOUT=600                 # slow models
export FASTCHAT_WORKER_API_EMBEDDING_BATCH_SIZE=1       # embedding OOM
export FASTCHAT_CONTROLLER_HEART_BEAT_EXPIRATION=120    # tolerate slow heartbeats
export LOGDIR=/var/log/fastchat                         # persistent logs
export FASTCHAT_USE_MODELSCOPE=True                     # ModelScope weights