ChatTTS Admin & Developer Guide

This guide is the deep reference for engineers deploying, operating, integrating, or extending ChatTTS — the generative dialogue-oriented text-to-speech model from the 2noise organization. It covers the model architecture, the inference pipeline internals, deployment of the bundled API/UI servers, configuration, security considerations, code structure, and improvement opportunities.

For installation and day-to-day usage, see the companion ChatTTS User Guide. This document assumes you are comfortable with PyTorch, the Hugging Face Transformers stack, and serving Python models.

Important

Licensing has operational consequences. The code is AGPLv3+ (network-use copyleft) and the pretrained model is CC BY-NC 4.0 (non-commercial only). If you deploy ChatTTS as a network service, AGPL obligations attach to your modifications, and the model license forbids commercial use. Treat this repo as an academic/research reference implementation, not a production product.

1. Executive Summary

ChatTTS is a two-stage autoregressive TTS system wrapped in a single Python class, ChatTTS.Chat. The repository is intentionally scoped as "algorithm infrastructure and some simple examples" — it is a model + reference pipeline, not a turnkey hosted product.

The system's defining design choice is that it generates discrete audio tokens (4 vector-quantization codebooks) with a Llama transformer, then decodes those tokens to a mel-spectrogram via a DVAE, and finally to a waveform via the Vocos neural vocoder.

2. Architecture

2.1 End-to-end pipeline

                ┌─────────────────────────────────────────────────────────────┐
   text  ─────▶ │  Normalizer (norm.py)                                        │
                │   • language detect (zh/en)                                   │
                │   • text normalization (NeMo / WeTextProcessing, optional)   │
                │   • homophone replacement (numba _fast_replace)              │
                └───────────────────────────┬─────────────────────────────────┘
                                            │ cleaned text
                ┌───────────────────────────▼─────────────────────────────────┐
   STAGE 1      │  Refine text (GPT.generate, infer_text=True)                 │
   (optional)   │   • Tokenizer (BertTokenizerFast) → input_ids               │
                │   • Llama backbone predicts prosody-marked token stream      │
                │   • controlled by RefineTextParams                           │
                └───────────────────────────┬─────────────────────────────────┘
                                            │ refined text
                ┌───────────────────────────▼─────────────────────────────────┐
   STAGE 2      │  Infer code (GPT.generate, infer_text=False)                 │
                │   • Speaker prompt injection ([Stts][spk_emb]…[Ptts])        │
                │   • Embed (audio+text token embeddings) → Llama backbone     │
                │   • emits 4 VQ codebook token streams (num_vq=4)             │
                │   • controlled by InferCodeParams                            │
                └───────────────────────────┬─────────────────────────────────┘
                                            │ audio tokens (hiddens or ids)
                ┌───────────────────────────▼─────────────────────────────────┐
   DECODE       │  DVAE / Decoder (dvae.py)                                    │
                │   • GFSQ de-quantize → ConvNeXt decoder → mel-spectrogram    │
                └───────────────────────────┬─────────────────────────────────┘
                                            │ mel spectrogram
                ┌───────────────────────────▼─────────────────────────────────┐
   VOCODER      │  Vocos (external lib)                                        │
                │   • MelSpectrogram features → ISTFT head → waveform          │
                └───────────────────────────┬─────────────────────────────────┘
                                            ▼
                                   24 kHz float32 PCM  ─▶ tools/audio → mp3/wav/ogg

2.2 Component inventory

2.3 Model hyperparameters (from config/config.py)

These are baked-in dataclass defaults — they describe the released model, not user-tunable knobs.

2.4 Model assets

The model ships as a set of files under an asset/ directory, verified by SHA-256 against the checked-in ChatTTS/res/sha256_map.json:

Assets are obtained either via Hugging Face snapshot_download(repo_id="2Noise/ChatTTS") (source="huggingface") or via an external Go downloader rvcmd (the RVC-Models-Downloader release, used by source="local").

3. The Inference Engine in Detail

3.1 Chat.infer() control flow

infer() (in core.py) is the public entry; _infer() is the generator that does the work:

1. Sentence splitting — if split_text=True and the input is a string, it is split on newlines, or on 。 / . boundaries, into a list.
2. Normalization — each sentence passes through Normalizer.__call__ (language detect → normalize → homophone replace → invalid-char rejection).
3. Refine stage (_refine_text) — unless skip_refine_text=True, the Llama backbone rewrites the text with prosody tokens. With refine_text_only=True, it returns here.
4. Reference-speaker bootstrap — when splitting multiple sentences and no spk_smp is given, the first sentence is synthesized and re-encoded as a speaker sample so the remaining sentences share a consistent voice.
5. Code stage (_infer_code) — text + speaker prompt is tokenized and the backbone emits 4 VQ token streams.
6. Decode (_decode_to_wavs) — tokens → mel via Decoder/DVAE, mel → waveform via _vocos_decode.
7. Silence trimming — for non-streaming output, samples below 1e-5 magnitude are stripped and (when split) concatenated into one array.

3.2 The autoregressive loop (GPT.generate)

gpt.py's generate() is the heart of the system. Key behaviors operators should understand:

• Pre-allocation. It pre-allocates inputs_ids_buf and attention_mask_cache sized to prompt_len + max_new_token, then fills them token by token. This bounds memory but means max_new_token directly sets the worst-case allocation.
• Two modes. infer_text=True predicts text tokens via head_text; infer_text=False predicts audio tokens across all 4 codebooks via head_code.
• Sampling. Logits are divided by per-codebook temperature, passed through Top-P/Top-K warpers and the repetition penalty, softmaxed, then torch.multinomial samples the next token. A manual_seed makes this deterministic.
• Empty-output guard. If the model emits EOS at step 0 and ensure_non_empty=True, it recursively restarts generation (logged as "regenerate in order to ensure non-empty"). With a fixed manual_seed, it does not retry (to stay deterministic).
• Streaming. When stream=True, partial GenerationOutputs are yielded every stream_batch tokens.
• Termination. Stops when all sequences hit EOS, when context.get() is set (a user interrupt()), or when max_new_token is reached (logged as "incomplete result. hit max_new_token").

Note

The Context object (a tiny interrupt flag in gpt.py) is how Chat.interrupt() cooperatively stops a running stream. It is checked once per generated token.

3.3 Speaker embeddings

Speaker (speaker.py) represents a voice as a 192-dim vector sampled from a Gaussian whose mean/std come from the spk_stat statistics string baked into config.py. Embeddings are serialized for transport by LZMA-compressing the float16 bytes and base16384-encoding the result — that is why speaker strings are long and opaque, and why a corrupted string raises lzma.LZMAError (see Troubleshooting). Voice cloning (sample_audio_speaker) instead encodes DVAE tokens of a reference clip via encode_prompt.

3.4 Prompt grammar (internal control tokens)

The model is conditioned with bracketed control tokens. Two categories matter:

• Structural tokens injected by the code, not the user: [Stts], [Ptts], [spk_emb], [empty_spk], [Sbreak], [Pbreak], [Ebreak] (EOS for refine). See Speaker.decorate_code_prompts / decorate_text_prompts.
• User prosody tokens documented in the User Guide: [oral_N], [laugh_N], [break_N] (refine prompt), [speed_N] (infer prompt), and inline [laugh]/[uv_break]/[lbreak].

Tip

The tokenizer special IDs are resolved at load: spk_emb_ids = "[spk_emb]", break_0_ids = "[break_0]", eos_token = "[Ebreak]". If you swap tokenizers, these must exist or generation breaks.

4. Deployment

ChatTTS itself is a library; the deployable artifacts are the example servers. None of them ship with a Dockerfile, process manager, or TLS — that is your responsibility.

4.1 Choosing a server

4.2 Running the OpenAI-compatible server (recommended for integration)

# From the project root, with the model reachable
fastapi run examples/api/openai_api.py --host 0.0.0.0 --port 8000

On startup it: loads the model from Hugging Face, registers EN/ZH normalizers, creates an asyncio.Lock, and preloads the voice files in VOICE_MAP (1528.pt, 1384.pt, 2443.pt) into app.state.spk_emb_map. Missing voice files log a warning and fall back to the default speaker.

Important

The server holds exactly one model in memory and serializes inference with an asyncio.Lock. Throughput is one synthesis at a time per process. To scale, run multiple processes behind a load balancer, each with its own GPU/VRAM budget (≥4 GB each). There is no built-in queue, batching across requests, or autoscaling.

4.3 Production checklist

• [ ] Pin Python to 3.10/3.11 in a clean environment.
• [ ] Pre-download model assets at build time (chat.load()), not on first request, to avoid cold-start downloads.
• [ ] Verify asset integrity passes (SHA-256 check in utils/dl.py).
• [ ] Run behind a reverse proxy that terminates TLS; the example servers serve plain HTTP.
• [ ] Add authentication/rate limiting at the proxy — the example servers have none (see Security).
• [ ] Set --server_name/--root_path appropriately if behind a path prefix (web UI).
• [ ] Run one process per available GPU; cap concurrent requests per process at 1.
• [ ] Set input length limits at the edge (the OpenAI server caps at 2048 chars; the native server does not).
• [ ] Monitor VRAM; long max_new_token values raise peak memory.
• [ ] Provide the community speaker .pt files if you rely on named voices.

4.4 Environment variables

Note

There is no .env file, YAML config, or settings module. "Configuration" is either model hyperparameters (dataclasses in config.py, not meant to change) or per-call parameters (RefineTextParams/InferCodeParams). The handful of environment variables above are the only external config surface.

5. Operations

5.1 Logging

All components use tools/logger/log.py's get_logger(name), which installs a colorized formatter producing lines like:

[+0800 20260411 12:00:03] [INFO] ChatTTS | core | gpt loaded.

Color is disabled on Windows. Noisy third-party loggers (numba, httpx, wetext-zh_normalizer, NeMo-text-processing) are pinned to WARNING. Pass a custom logging.Logger to ChatTTS.Chat(logger=...) to route logs into your own stack.

Key log signals to watch:

5.2 Memory management

ChatTTS is aggressive about freeing tensors mid-loop (del, del_all in utils/io.py) because generation pre-allocates large buffers. Chat.unload() deletes all sub-models and re-initializes the object so it can be reloaded. Call unload() before reloading with new options (the web UI's Reload button does exactly this).

5.3 Asset integrity & regeneration

utils/dl.py verifies each asset's SHA-256 against ChatTTS/res/sha256_map.json; mismatches are renamed to .bak and re-downloaded when update=True. If you fine-tune or replace assets, regenerate the map with the Go tool:

cd tools/checksum && go run .   # writes ChatTTS/res/sha256_map.json

5.4 Device selection

utils/gpu.py:select_device() picks, in order: the CUDA/NPU device with the most free memory (if ≥ ~2 GB free), else MPS (only with experimental=True), else DirectML (only with experimental=True), else CPU. Vocos and the MelSpectrogram run on CPU when the device is MPS/NPU because complex-dtype ops are unsupported there.

6. Security Analysis

Caution

The example servers are reference demos, not hardened services. Do not expose them directly to untrusted networks. The findings below are inherent to the shipped code.

Warning

The startup torch.load(spk_path, map_location="cpu") in openai_api.py is the single highest-severity issue for a hosted deployment. Treat every .pt speaker file as executable code and load only files you produced or fully trust.

7. Code Quality & Patterns

Strengths

• Clean separation of concerns: orchestration (core.py), model components (model/), I/O and infra (utils/, tools/). Each model piece is an nn.Module with load_pretrained.
• Config as typed dataclasses: hyperparameters are explicit and discoverable in one file.
• Performance-conscious: numba-jitted hot paths (homophone replacement, float→int16), pre-allocated generation buffers, eager tensor freeing, optional torch.compile/vLLM.
• Pluggable normalization: Normalizer.register() lets you add language normalizers without touching core code.
• Multi-surface, single engine: four front-ends (CLI/UI/two APIs) all funnel into one Chat.infer, avoiding logic duplication of the model path.

Weaknesses / risks

• Broad except clauses: core.py's HF download path and the web UI's normalizer registration swallow exceptions broadly (except: / except BaseException), which can hide real failures.
• Deprecated FastAPI patterns: @app.on_event("startup") is deprecated in favor of lifespan handlers; both example servers still use it.
• Vendored Transformers code: examples/onnx/modeling_llama.py is a large copy of HF's Llama implementation for ONNX export — a maintenance/version-drift liability.
• Mixed-language comments: core streaming logic (examples/cmd/stream.py) is commented in Chinese only, raising the bar for non-Chinese contributors.
• Global mutable state in the web UI: funcs.py uses module-level globals (chat, enable_cache, is_in_generate) — fine for a single-user demo, unsafe for multi-user serving.
• No automated test suite of substance: tests/ holds issue-reproduction scripts (#511.py, #588.py, #655.py) and a shell runner, not unit tests.

8. Repository Structure

ChatTTS/
├── ChatTTS/                 # the installable package
│   ├── core.py             # Chat orchestrator (load / infer / unload)
│   ├── norm.py             # Normalizer (lang detect, homophones)
│   ├── config/config.py    # model hyperparameter dataclasses
│   ├── model/
│   │   ├── gpt.py          # Llama backbone + generate() loop
│   │   ├── dvae.py         # DVAE encoder/decoder + GFSQ quantizer
│   │   ├── embed.py        # token embedding tables + heads
│   │   ├── speaker.py      # speaker sampling / encode / cloning
│   │   ├── tokenizer.py    # BertTokenizerFast wrapper
│   │   ├── processors.py   # logits warpers + repetition penalty
│   │   ├── cuda/           # experimental TransformerEngine path
│   │   └── velocity/       # vLLM-based accelerated GPT backend
│   ├── res/                # homophones_map.json, sha256_map.json
│   └── utils/              # dl (download+verify), gpu, io, log
├── tools/
│   ├── audio/              # PyAV load + PCM↔mp3/wav/ogg
│   ├── normalizer/         # NeMo (en) + WeTextProcessing (zh) adapters
│   ├── seeder/             # TorchSeedContext
│   ├── logger/             # colorized logging formatter
│   ├── llm/                # ChatOpenAI helper (LLM→TTS text prep)
│   └── checksum/           # Go tool to regenerate sha256_map.json
├── examples/
│   ├── cmd/                # run.py (CLI), stream.py (ChatStreamer)
│   ├── web/                # Gradio webui.py + funcs.py + ex.py
│   ├── api/                # main.py, openai_api.py, client.py, postScript.py
│   ├── onnx/               # ONNX exporter + vendored Llama
│   └── ipynb/              # Colab + example notebooks
├── docs/                   # translated READMEs (cn/jp/ru/es/fr/kr)
├── tests/                  # issue-reproduction scripts
├── requirements.txt
└── setup.py

Note

ChatTTS/model/velocity/ is a self-contained vLLM-style engine (block manager, scheduler, sampler, worker) used only when use_vllm=True on Linux. ChatTTS/model/cuda/ holds the experimental TransformerEngine Llama, used only when experimental=True on Linux+CUDA. Both are inert in a default install.

9. Internals Reference

9.1 Key classes and methods

9.2 HTTP API reference

The native server's ChatTTSParams (main.py) mirrors infer() arguments; note it embeds the typed ChatTTS.Chat.RefineTextParams/InferCodeParams directly as Pydantic-compatible models. The OpenAI server whitelists request params (ALLOWED_PARAMS) and forces model="tts-1".

9.3 Dependency footprint

10. Potential Improvements

Derived from the code and community issues — candidates for contributors or downstream forks.

11. Quick Reference

Minimal programmatic load + infer

import ChatTTS, torch, torchaudio
chat = ChatTTS.Chat()
assert chat.load(source="huggingface")
wavs = chat.infer(["hello world"], skip_refine_text=True)
torchaudio.save("o.wav", torch.from_numpy(wavs[0]).unsqueeze(0), 24000)

Serve the OpenAI-compatible API

fastapi run examples/api/openai_api.py --host 127.0.0.1 --port 8000
curl -s localhost:8000/health

Regenerate asset checksums

cd tools/checksum && go run .

Reproducible voice

import torch
torch.manual_seed(1111)
spk = chat.sample_random_speaker()   # reuse spk as InferCodeParams.spk_emb

Key invariants

• Sample rate is always 24000 Hz mono.
• num_vq = 4 codebooks; backbone is a 20-layer Llama, hidden 768.
• One model per process; OpenAI server serializes inference via asyncio.Lock.
• Speaker strings are LZMA + base16384 — store them verbatim.
• Model is non-commercial (CC BY-NC 4.0); code is AGPLv3+.