# ChatTTS Admin & Developer Guide

This guide is the deep reference for engineers deploying, operating, integrating, or extending **ChatTTS** — the generative dialogue-oriented text-to-speech model from the `2noise` organization. It covers the model architecture, the inference pipeline internals, deployment of the bundled API/UI servers, configuration, security considerations, code structure, and improvement opportunities.

For installation and day-to-day usage, see the companion **ChatTTS User Guide**. This document assumes you are comfortable with PyTorch, the Hugging Face Transformers stack, and serving Python models.

> [!IMPORTANT]
> **Licensing has operational consequences.** The code is `AGPLv3+` (network-use copyleft) and the pretrained model is `CC BY-NC 4.0` (**non-commercial only**). If you deploy ChatTTS as a network service, AGPL obligations attach to your modifications, and the model license forbids commercial use. Treat this repo as an **academic/research reference implementation**, not a production product.

---

## 1. Executive Summary

ChatTTS is a **two-stage autoregressive TTS system** wrapped in a single Python class, `ChatTTS.Chat`. The repository is intentionally scoped as "algorithm infrastructure and some simple examples" — it is a model + reference pipeline, not a turnkey hosted product.

| Aspect | Summary |
|---|---|
| **Purpose** | Conversational, expressive TTS (EN/ZH) with prosody control. |
| **Language** | Python (one Go helper for checksums). |
| **Core dependency** | PyTorch ≥ 2.1 + Hugging Face Transformers (Llama backbone) + Vocos vocoder. |
| **Model size** | 40,000-hour pretrained base (open), no SFT. Main internal model trained on 100,000+ hours. |
| **Public surfaces** | Python API, CLI, Gradio web UI, native FastAPI server, OpenAI-compatible FastAPI server. |
| **Sample rate** | 24,000 Hz, mono. |
| **State** | Stateless inference; model loaded once into memory, reused per request. |
| **Persistence** | None beyond model asset files on disk; no database. |
| **Scaling model** | Single-process, single-model-in-memory; concurrency guarded by a lock in the OpenAI server. |

The system's defining design choice is that it generates **discrete audio tokens** (4 vector-quantization codebooks) with a Llama transformer, then decodes those tokens to a mel-spectrogram via a DVAE, and finally to a waveform via the Vocos neural vocoder.

---

## 2. Architecture

### 2.1 End-to-end pipeline

```
                ┌─────────────────────────────────────────────────────────────┐
   text  ─────▶ │  Normalizer (norm.py)                                        │
                │   • language detect (zh/en)                                   │
                │   • text normalization (NeMo / WeTextProcessing, optional)   │
                │   • homophone replacement (numba _fast_replace)              │
                └───────────────────────────┬─────────────────────────────────┘
                                            │ cleaned text
                ┌───────────────────────────▼─────────────────────────────────┐
   STAGE 1      │  Refine text (GPT.generate, infer_text=True)                 │
   (optional)   │   • Tokenizer (BertTokenizerFast) → input_ids               │
                │   • Llama backbone predicts prosody-marked token stream      │
                │   • controlled by RefineTextParams                           │
                └───────────────────────────┬─────────────────────────────────┘
                                            │ refined text
                ┌───────────────────────────▼─────────────────────────────────┐
   STAGE 2      │  Infer code (GPT.generate, infer_text=False)                 │
                │   • Speaker prompt injection ([Stts][spk_emb]…[Ptts])        │
                │   • Embed (audio+text token embeddings) → Llama backbone     │
                │   • emits 4 VQ codebook token streams (num_vq=4)             │
                │   • controlled by InferCodeParams                            │
                └───────────────────────────┬─────────────────────────────────┘
                                            │ audio tokens (hiddens or ids)
                ┌───────────────────────────▼─────────────────────────────────┐
   DECODE       │  DVAE / Decoder (dvae.py)                                    │
                │   • GFSQ de-quantize → ConvNeXt decoder → mel-spectrogram    │
                └───────────────────────────┬─────────────────────────────────┘
                                            │ mel spectrogram
                ┌───────────────────────────▼─────────────────────────────────┐
   VOCODER      │  Vocos (external lib)                                        │
                │   • MelSpectrogram features → ISTFT head → waveform          │
                └───────────────────────────┬─────────────────────────────────┘
                                            ▼
                                   24 kHz float32 PCM  ─▶ tools/audio → mp3/wav/ogg
```

### 2.2 Component inventory

| Component | File | Role |
|---|---|---|
| `Chat` | `ChatTTS/core.py` | Orchestrator. Owns config, normalizer, all sub-models; exposes `load`/`infer`/`unload`/`interrupt`. |
| `Config` (+ nested dataclasses) | `ChatTTS/config/config.py` | Static model hyperparameters and asset paths. |
| `Normalizer` | `ChatTTS/norm.py` | Text cleaning, language detection, homophone replacement (numba-jitted). |
| `Tokenizer` | `ChatTTS/model/tokenizer.py` | Wraps `BertTokenizerFast`; encodes text + speaker prompt into `(input_ids, attention_mask, text_mask)`. |
| `Embed` | `ChatTTS/model/embed.py` | Joint text + 4×audio-codebook embedding tables and output heads. |
| `GPT` | `ChatTTS/model/gpt.py` | Llama backbone + autoregressive `generate()` loop (text and code modes). |
| `Speaker` | `ChatTTS/model/speaker.py` | Speaker-embedding sampling, encode/decode, prompt decoration. |
| `DVAE` | `ChatTTS/model/dvae.py` | Encoder/decoder + GFSQ vector quantizer; mel-spectrogram ↔ tokens. |
| `gen_logits` / processors | `ChatTTS/model/processors.py` | Top-P/Top-K warpers + custom repetition penalty. |
| Utilities | `ChatTTS/utils/` | Asset download + SHA-256 verification, device selection, safetensors I/O, logging. |
| Audio I/O | `tools/audio/` | PyAV-based load/encode, PCM↔WAV/MP3/OGG conversion, float→int16. |
| Normalizer adapters | `tools/normalizer/` | Thin factories for NeMo (en) and WeTextProcessing (zh). |
| Seeder | `tools/seeder/ctx.py` | `TorchSeedContext` for reproducible RNG scopes. |
| Logger | `tools/logger/log.py` | Colorized, timestamped logging formatter. |
| Checksum tool | `tools/checksum/*.go` | Go program that regenerates `sha256_map.json` for asset integrity. |

### 2.3 Model hyperparameters (from `config/config.py`)

These are baked-in dataclass defaults — they describe the *released model*, not user-tunable knobs.

| Group | Key | Value |
|---|---|---|
| **GPT** (Llama backbone) | `hidden_size` | 768 |
| | `intermediate_size` | 3072 |
| | `num_attention_heads` | 12 |
| | `num_hidden_layers` | 20 |
| | `max_position_embeddings` | 4096 |
| | `num_audio_tokens` | 626 |
| | `num_text_tokens` | 21178 |
| | `num_vq` (codebooks) | 4 |
| | `spk_emb_dim` | 192 |
| **DVAE / VQ** | `vq.levels` | (5, 5, 5, 5) |
| | `vq.G` / `vq.R` (groups/residuals) | 2 / 2 |
| | decoder `n_layer` | 12 |
| **Vocos feature extractor** | `sample_rate` | 24000 |
| | `n_fft` / `hop_length` | 1024 / 256 |
| | `n_mels` | 100 |

### 2.4 Model assets

The model ships as a set of files under an `asset/` directory, verified by SHA-256 against the checked-in `ChatTTS/res/sha256_map.json`:

| Asset | Purpose |
|---|---|
| `asset/Vocos.safetensors` | Vocos vocoder weights. |
| `asset/DVAE.safetensors` | DVAE encoder/decoder (used for cloning + fallback decode). |
| `asset/Decoder.safetensors` | High-quality decoder (the default decode path). |
| `asset/Embed.safetensors` | Token embedding tables and heads. |
| `asset/gpt/{config.json, model.safetensors}` | Llama backbone. |
| `asset/tokenizer/{...}` | BERT fast tokenizer files. |

Assets are obtained either via Hugging Face `snapshot_download(repo_id="2Noise/ChatTTS")` (`source="huggingface"`) or via an external Go downloader `rvcmd` (the RVC-Models-Downloader release, used by `source="local"`).

---

## 3. The Inference Engine in Detail

### 3.1 `Chat.infer()` control flow

`infer()` (in `core.py`) is the public entry; `_infer()` is the generator that does the work:

1. **Sentence splitting** — if `split_text=True` and the input is a string, it is split on newlines, or on `。` / `. ` boundaries, into a list.
2. **Normalization** — each sentence passes through `Normalizer.__call__` (language detect → normalize → homophone replace → invalid-char rejection).
3. **Refine stage** (`_refine_text`) — unless `skip_refine_text=True`, the Llama backbone rewrites the text with prosody tokens. With `refine_text_only=True`, it returns here.
4. **Reference-speaker bootstrap** — when splitting multiple sentences and no `spk_smp` is given, the first sentence is synthesized and re-encoded as a speaker sample so the remaining sentences share a consistent voice.
5. **Code stage** (`_infer_code`) — text + speaker prompt is tokenized and the backbone emits 4 VQ token streams.
6. **Decode** (`_decode_to_wavs`) — tokens → mel via `Decoder`/`DVAE`, mel → waveform via `_vocos_decode`.
7. **Silence trimming** — for non-streaming output, samples below `1e-5` magnitude are stripped and (when split) concatenated into one array.

### 3.2 The autoregressive loop (`GPT.generate`)

`gpt.py`'s `generate()` is the heart of the system. Key behaviors operators should understand:

- **Pre-allocation.** It pre-allocates `inputs_ids_buf` and `attention_mask_cache` sized to `prompt_len + max_new_token`, then fills them token by token. This bounds memory but means `max_new_token` directly sets the worst-case allocation.
- **Two modes.** `infer_text=True` predicts text tokens via `head_text`; `infer_text=False` predicts audio tokens across all 4 codebooks via `head_code`.
- **Sampling.** Logits are divided by per-codebook `temperature`, passed through Top-P/Top-K warpers and the repetition penalty, softmaxed, then `torch.multinomial` samples the next token. A `manual_seed` makes this deterministic.
- **Empty-output guard.** If the model emits EOS at step 0 and `ensure_non_empty=True`, it **recursively restarts** generation (logged as "regenerate in order to ensure non-empty"). With a fixed `manual_seed`, it does *not* retry (to stay deterministic).
- **Streaming.** When `stream=True`, partial `GenerationOutputs` are yielded every `stream_batch` tokens.
- **Termination.** Stops when all sequences hit EOS, when `context.get()` is set (a user `interrupt()`), or when `max_new_token` is reached (logged as "incomplete result. hit max_new_token").

> [!NOTE]
> The `Context` object (a tiny interrupt flag in `gpt.py`) is how `Chat.interrupt()` cooperatively stops a running stream. It is checked once per generated token.

### 3.3 Speaker embeddings

`Speaker` (`speaker.py`) represents a voice as a 192-dim vector sampled from a Gaussian whose `mean`/`std` come from the `spk_stat` statistics string baked into `config.py`. Embeddings are serialized for transport by **LZMA-compressing** the float16 bytes and **base16384-encoding** the result — that is why speaker strings are long and opaque, and why a corrupted string raises `lzma.LZMAError` (see Troubleshooting). Voice cloning (`sample_audio_speaker`) instead encodes DVAE tokens of a reference clip via `encode_prompt`.

### 3.4 Prompt grammar (internal control tokens)

The model is conditioned with bracketed control tokens. Two categories matter:

- **Structural tokens** injected by the code, not the user: `[Stts]`, `[Ptts]`, `[spk_emb]`, `[empty_spk]`, `[Sbreak]`, `[Pbreak]`, `[Ebreak]` (EOS for refine). See `Speaker.decorate_code_prompts` / `decorate_text_prompts`.
- **User prosody tokens** documented in the User Guide: `[oral_N]`, `[laugh_N]`, `[break_N]` (refine prompt), `[speed_N]` (infer prompt), and inline `[laugh]`/`[uv_break]`/`[lbreak]`.

> [!TIP]
> The tokenizer special IDs are resolved at load: `spk_emb_ids = "[spk_emb]"`, `break_0_ids = "[break_0]"`, `eos_token = "[Ebreak]"`. If you swap tokenizers, these must exist or generation breaks.

---

## 4. Deployment

ChatTTS itself is a library; the *deployable* artifacts are the example servers. None of them ship with a Dockerfile, process manager, or TLS — that is your responsibility.

### 4.1 Choosing a server

| Server | File | Output | Concurrency | Best for |
|---|---|---|---|---|
| Native API | `examples/api/main.py` | ZIP of MP3s | None built in | Batch synthesis of sentence lists. |
| OpenAI-compatible API | `examples/api/openai_api.py` | mp3/wav/ogg stream | `asyncio.Lock` around inference | Drop-in for OpenAI TTS clients. |
| Web UI | `examples/web/webui.py` | In-browser audio | Single user | Interactive demos and tuning. |

### 4.2 Running the OpenAI-compatible server (recommended for integration)

```bash
# From the project root, with the model reachable
fastapi run examples/api/openai_api.py --host 0.0.0.0 --port 8000
```

On startup it: loads the model from Hugging Face, registers EN/ZH normalizers, creates an `asyncio.Lock`, and preloads the voice files in `VOICE_MAP` (`1528.pt`, `1384.pt`, `2443.pt`) into `app.state.spk_emb_map`. Missing voice files log a warning and fall back to the default speaker.

> [!IMPORTANT]
> The server holds **exactly one model in memory** and serializes inference with an `asyncio.Lock`. Throughput is one synthesis at a time per process. To scale, run multiple processes behind a load balancer, each with its own GPU/VRAM budget (≥4 GB each). There is no built-in queue, batching across requests, or autoscaling.

### 4.3 Production checklist

- [ ] Pin Python to 3.10/3.11 in a clean environment.
- [ ] Pre-download model assets at build time (`chat.load()`), not on first request, to avoid cold-start downloads.
- [ ] Verify asset integrity passes (SHA-256 check in `utils/dl.py`).
- [ ] Run behind a reverse proxy that terminates TLS; the example servers serve plain HTTP.
- [ ] Add authentication/rate limiting at the proxy — **the example servers have none** (see [Security](#6-security-analysis)).
- [ ] Set `--server_name`/`--root_path` appropriately if behind a path prefix (web UI).
- [ ] Run one process per available GPU; cap concurrent requests per process at 1.
- [ ] Set `input` length limits at the edge (the OpenAI server caps at 2048 chars; the native server does not).
- [ ] Monitor VRAM; long `max_new_token` values raise peak memory.
- [ ] Provide the community speaker `.pt` files if you rely on named voices.

### 4.4 Environment variables

| Variable | Read in | Default | Purpose |
|---|---|---|---|
| `CHTTS_VER` | `setup.py` | `v0.0.0` | Package version stamp at build time. |
| `HF_HOME` | `core.py` | `~/.cache/huggingface` | Hugging Face cache root for snapshot lookups. |
| `CHATTTS_SERVICE_HOST` | `client.py` / `postScript.py` | `localhost` / `127.0.0.1` | Host the HTTP clients target. |
| `CHATTTS_SERVICE_PORT` | `client.py` / `postScript.py` | `8000` / `9900` | Port the HTTP clients target. |
| `PYTORCH_ENABLE_MPS_FALLBACK` | set automatically on macOS | `1` | Lets unsupported MPS ops fall back to CPU. |
| `TOKENIZERS_PARALLELISM` | set in `tokenizer.py` | `false` | Silences the HF tokenizer fork warning. |

> [!NOTE]
> There is **no `.env` file, YAML config, or settings module**. "Configuration" is either model hyperparameters (dataclasses in `config.py`, not meant to change) or per-call parameters (`RefineTextParams`/`InferCodeParams`). The handful of environment variables above are the only external config surface.

---

## 5. Operations

### 5.1 Logging

All components use `tools/logger/log.py`'s `get_logger(name)`, which installs a colorized formatter producing lines like:

```text
[+0800 20260411 12:00:03] [INFO] ChatTTS | core | gpt loaded.
```

Color is disabled on Windows. Noisy third-party loggers (`numba`, `httpx`, `wetext-zh_normalizer`, `NeMo-text-processing`) are pinned to `WARNING`. Pass a custom `logging.Logger` to `ChatTTS.Chat(logger=...)` to route logs into your own stack.

Key log signals to watch:

| Log line | Meaning |
|---|---|
| `... loaded.` (vocos/dvae/gpt/speaker/decoder/tokenizer) | Successful staged model load. |
| `Switching to CPU.` | GPU/NPU had < `min_memory` (2047 MB) free. |
| `regenerate in order to ensure non-empty` | Model emitted immediate EOS; auto-retrying. |
| `incomplete result. hit max_new_token` | Output truncated by the token cap. |
| `generation is interrupted` | `interrupt()` was called mid-stream. |
| `replace homophones: X->Y` | Homophone substitution applied. |
| `found invalid characters: {...}` | Characters stripped by the reject pattern. |

### 5.2 Memory management

ChatTTS is aggressive about freeing tensors mid-loop (`del`, `del_all` in `utils/io.py`) because generation pre-allocates large buffers. `Chat.unload()` deletes all sub-models and re-initializes the object so it can be reloaded. Call `unload()` before reloading with new options (the web UI's Reload button does exactly this).

### 5.3 Asset integrity & regeneration

`utils/dl.py` verifies each asset's SHA-256 against `ChatTTS/res/sha256_map.json`; mismatches are renamed to `.bak` and re-downloaded when `update=True`. If you fine-tune or replace assets, regenerate the map with the Go tool:

```bash
cd tools/checksum && go run .   # writes ChatTTS/res/sha256_map.json
```

### 5.4 Device selection

`utils/gpu.py:select_device()` picks, in order: the CUDA/NPU device with the most free memory (if ≥ ~2 GB free), else MPS (only with `experimental=True`), else DirectML (only with `experimental=True`), else CPU. Vocos and the MelSpectrogram run on CPU when the device is MPS/NPU because complex-dtype ops are unsupported there.

---

## 6. Security Analysis

> [!CAUTION]
> The example servers are **reference demos, not hardened services.** Do not expose them directly to untrusted networks. The findings below are inherent to the shipped code.

| Area | Finding | Mitigation |
|---|---|---|
| **Authentication** | None on any endpoint (`/generate_voice`, `/v1/audio/speech`, `/health`, web UI). | Put an authenticating reverse proxy / API gateway in front. |
| **Rate limiting / DoS** | None. `max_new_token` up to 2048 per request; the native API has no input length cap. A flood of long inputs exhausts GPU/CPU. | Enforce request size, length, and rate limits at the edge; cap `max_new_token`. |
| **Untrusted deserialization** | The OpenAI server `torch.load()`s speaker `.pt` files at startup. `torch.load` can execute arbitrary code from a malicious pickle. | Only load speaker files from trusted sources; prefer `safetensors`-format speaker files if you control them. |
| **Speaker/coef strings** | User-supplied `spk_emb`/`coef` are LZMA+base16384 decoded. Malformed input raises (handled), but very large inputs could be a memory vector. | Validate/limit the size of user-supplied embedding strings. |
| **Network bind default** | Web UI binds `0.0.0.0:8080` by default. | Bind to `127.0.0.1` behind a proxy, or restrict with firewall rules. |
| **Asset download** | Models fetched over HTTPS from HF / GitHub releases, integrity-checked by SHA-256. This is a reasonable supply-chain posture. | Keep `sha256_map.json` under review; pin asset versions. |
| **CORS** | Not configured on the FastAPI apps. | Add explicit CORS rules if browser clients call the API cross-origin. |
| **Model misuse** | The model is anti-cloning-hardened (added noise, MP3 compression) but voice synthesis is inherently dual-use. | Enforce acceptable-use policy; respect the non-commercial license. |

> [!WARNING]
> The startup `torch.load(spk_path, map_location="cpu")` in `openai_api.py` is the single highest-severity issue for a hosted deployment. Treat every `.pt` speaker file as executable code and load only files you produced or fully trust.

---

## 7. Code Quality & Patterns

**Strengths**

- **Clean separation of concerns:** orchestration (`core.py`), model components (`model/`), I/O and infra (`utils/`, `tools/`). Each model piece is an `nn.Module` with `load_pretrained`.
- **Config as typed dataclasses:** hyperparameters are explicit and discoverable in one file.
- **Performance-conscious:** numba-jitted hot paths (homophone replacement, float→int16), pre-allocated generation buffers, eager tensor freeing, optional `torch.compile`/vLLM.
- **Pluggable normalization:** `Normalizer.register()` lets you add language normalizers without touching core code.
- **Multi-surface, single engine:** four front-ends (CLI/UI/two APIs) all funnel into one `Chat.infer`, avoiding logic duplication of the model path.

**Weaknesses / risks**

- **Broad `except` clauses:** `core.py`'s HF download path and the web UI's normalizer registration swallow exceptions broadly (`except:` / `except BaseException`), which can hide real failures.
- **Deprecated FastAPI patterns:** `@app.on_event("startup")` is deprecated in favor of lifespan handlers; both example servers still use it.
- **Vendored Transformers code:** `examples/onnx/modeling_llama.py` is a large copy of HF's Llama implementation for ONNX export — a maintenance/version-drift liability.
- **Mixed-language comments:** core streaming logic (`examples/cmd/stream.py`) is commented in Chinese only, raising the bar for non-Chinese contributors.
- **Global mutable state in the web UI:** `funcs.py` uses module-level globals (`chat`, `enable_cache`, `is_in_generate`) — fine for a single-user demo, unsafe for multi-user serving.
- **No automated test suite of substance:** `tests/` holds issue-reproduction scripts (`#511.py`, `#588.py`, `#655.py`) and a shell runner, not unit tests.

---

## 8. Repository Structure

```
ChatTTS/
├── ChatTTS/                 # the installable package
│   ├── core.py             # Chat orchestrator (load / infer / unload)
│   ├── norm.py             # Normalizer (lang detect, homophones)
│   ├── config/config.py    # model hyperparameter dataclasses
│   ├── model/
│   │   ├── gpt.py          # Llama backbone + generate() loop
│   │   ├── dvae.py         # DVAE encoder/decoder + GFSQ quantizer
│   │   ├── embed.py        # token embedding tables + heads
│   │   ├── speaker.py      # speaker sampling / encode / cloning
│   │   ├── tokenizer.py    # BertTokenizerFast wrapper
│   │   ├── processors.py   # logits warpers + repetition penalty
│   │   ├── cuda/           # experimental TransformerEngine path
│   │   └── velocity/       # vLLM-based accelerated GPT backend
│   ├── res/                # homophones_map.json, sha256_map.json
│   └── utils/              # dl (download+verify), gpu, io, log
├── tools/
│   ├── audio/              # PyAV load + PCM↔mp3/wav/ogg
│   ├── normalizer/         # NeMo (en) + WeTextProcessing (zh) adapters
│   ├── seeder/             # TorchSeedContext
│   ├── logger/             # colorized logging formatter
│   ├── llm/                # ChatOpenAI helper (LLM→TTS text prep)
│   └── checksum/           # Go tool to regenerate sha256_map.json
├── examples/
│   ├── cmd/                # run.py (CLI), stream.py (ChatStreamer)
│   ├── web/                # Gradio webui.py + funcs.py + ex.py
│   ├── api/                # main.py, openai_api.py, client.py, postScript.py
│   ├── onnx/               # ONNX exporter + vendored Llama
│   └── ipynb/              # Colab + example notebooks
├── docs/                   # translated READMEs (cn/jp/ru/es/fr/kr)
├── tests/                  # issue-reproduction scripts
├── requirements.txt
└── setup.py
```

> [!NOTE]
> `ChatTTS/model/velocity/` is a self-contained vLLM-style engine (block manager, scheduler, sampler, worker) used only when `use_vllm=True` on Linux. `ChatTTS/model/cuda/` holds the experimental TransformerEngine Llama, used only when `experimental=True` on Linux+CUDA. Both are inert in a default install.

---

## 9. Internals Reference

### 9.1 Key classes and methods

| Class.method | File:line | Notes |
|---|---|---|
| `Chat.load` | `core.py:142` | Downloads + loads all sub-models; returns `bool`. |
| `Chat.infer` | `core.py:215` | Public synthesis entry; wraps the `_infer` generator. |
| `Chat._infer` | `core.py:395` | Normalize → refine → reference-speaker → code → decode. |
| `Chat._infer_code` | `core.py:551` | Builds prompts, runs GPT code generation (vLLM-aware). |
| `Chat._refine_text` | `core.py:674` | Text-refinement GPT pass. |
| `Chat.sample_random_speaker` | `core.py:183` | Gaussian speaker sample → encoded string. |
| `Chat.sample_audio_speaker` | `core.py:186` | Encode a reference clip into a speaker sample (cloning). |
| `GPT.generate` | `gpt.py:722` | The autoregressive sampling loop. |
| `DVAE.forward` | `dvae.py:271` | Encode (mel→tokens) / decode (tokens→mel). |
| `Speaker.encode_prompt`/`decode_prompt` | `speaker.py` | LZMA+base16384 (de)serialization of prompts. |
| `Tokenizer.encode` | `tokenizer.py:1321` | Left-padded `input_ids` + masks, optional speaker prompt. |
| `Normalizer.__call__` | `norm.py:926` | The full text-cleaning pipeline. |
| `gen_logits` | `processors.py:1108` | Builds Top-P/Top-K warpers + repetition penalty. |

### 9.2 HTTP API reference

| Server | Method | Path | Request | Response |
|---|---|---|---|---|
| native | `POST` | `/generate_voice` | `ChatTTSParams` JSON (`text[]`, flags, `params_refine_text`, `params_infer_code`) | `application/zip` of `{idx}.mp3` |
| openai | `POST` | `/v1/audio/speech` | OpenAI TTS JSON (`model`, `input`, `voice`, `response_format`, `speed`, `stream`) | streamed `audio/mpeg` or `audio/wav` |
| openai | `GET` | `/health` | — | `{"status":"healthy","model_loaded":true}` |

The native server's `ChatTTSParams` (`main.py`) mirrors `infer()` arguments; note it embeds the typed `ChatTTS.Chat.RefineTextParams`/`InferCodeParams` directly as Pydantic-compatible models. The OpenAI server whitelists request params (`ALLOWED_PARAMS`) and forces `model="tts-1"`.

### 9.3 Dependency footprint

| Dependency | Role |
|---|---|
| `torch>=2.1.0`, `torchaudio` | Tensor compute, mel-spectrogram. |
| `transformers>=4.41.1` | Llama backbone + BERT tokenizer. |
| `vocos` | Neural vocoder (mel → waveform). |
| `vector_quantize_pytorch` | `GroupedResidualFSQ` for the DVAE. |
| `numba` | JIT for homophone replace + float→int16. |
| `numpy<3.0.0` | Array ops throughout. |
| `pybase16384` | Base16384 codec for embeddings. |
| `av`, `pydub` | Audio decode/encode (PyAV + FFmpeg detection). |
| `gradio` | Web UI. |
| `fastapi` | API example servers. |
| `IPython`, `tqdm`, `requests` | Notebooks, progress bars, downloads. |
| `pynini`, `WeTextProcessing`, `nemo_text_processing` | Linux-only text normalization. |

---

## 10. Potential Improvements

Derived from the code and community issues — candidates for contributors or downstream forks.

| Area | Improvement | Rationale |
|---|---|---|
| **Serving** | Add request batching + a proper queue in the API servers. | Current lock serializes all inference; GPU is underutilized under load. |
| **Security** | Replace startup `torch.load` of `.pt` voices with `safetensors`. | Removes arbitrary-code-execution risk from speaker files. |
| **FastAPI** | Migrate `@app.on_event("startup")` to lifespan handlers. | Deprecated API; future FastAPI removal. |
| **Robustness** | Narrow broad `except:`/`except BaseException` blocks. | Hidden download/registration failures are hard to diagnose. |
| **Stability** | Guard against empty/too-short input before generation. | Eliminates the recurring `narrow(): length must be non-negative` crash (#955/#978). |
| **Packaging** | Ship a Dockerfile + pinned environment. | Removes Python/CUDA/Transformers version drift (#876, #987, #930). |
| **Testing** | Convert `tests/#NNN.py` reproductions into a real pytest suite. | No regression safety net today. |
| **Docs/i18n** | Translate Chinese-only inline comments in streaming code. | Lowers contribution barrier. |
| **vLLM** | Investigate timbre bias on the vLLM path (#640, #840, #847). | Accelerated path degrades voice quality. |
| **Feature** | Surface the roadmap's multi-emotion control once models ship. | Currently only `[laugh]/[uv_break]/[lbreak]` work at token level. |

---

## 11. Quick Reference

### Minimal programmatic load + infer

```python
import ChatTTS, torch, torchaudio
chat = ChatTTS.Chat()
assert chat.load(source="huggingface")
wavs = chat.infer(["hello world"], skip_refine_text=True)
torchaudio.save("o.wav", torch.from_numpy(wavs[0]).unsqueeze(0), 24000)
```

### Serve the OpenAI-compatible API

```bash
fastapi run examples/api/openai_api.py --host 127.0.0.1 --port 8000
curl -s localhost:8000/health
```

### Regenerate asset checksums

```bash
cd tools/checksum && go run .
```

### Reproducible voice

```python
import torch
torch.manual_seed(1111)
spk = chat.sample_random_speaker()   # reuse spk as InferCodeParams.spk_emb
```

### Key invariants

- Sample rate is always **24000 Hz** mono.
- `num_vq = 4` codebooks; backbone is a **20-layer Llama**, hidden 768.
- One model per process; OpenAI server serializes inference via `asyncio.Lock`.
- Speaker strings are **LZMA + base16384** — store them verbatim.
- Model is **non-commercial (CC BY-NC 4.0)**; code is **AGPLv3+**.

---

_Generated June 2026 from 2noise/ChatTTS at 77b89ee. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._