# VibeVoice Administrator & Developer Guide

VibeVoice is Microsoft's open-source TTS + ASR research stack. This guide covers system architecture, full configuration reference, deployment + scaling guidance, security posture, performance tuning, developer internals, and integration points. It is written for operators running VibeVoice in a service, developers extending the codebase, and architects evaluating the design.

For end-user usage, scripts, and recipes, see the **VibeVoice User Guide**.

---

## 1. Architecture Overview

VibeVoice is a **two-track speech model** packaged as a single Python distribution. One track produces speech (TTS), the other consumes it (ASR). Both tracks share an identity — a Qwen2.5 decoder-only LLM backbone wrapped by VibeVoice's own acoustic and semantic tokenizers, plus a diffusion head — but expose different generation modes through different model classes and processors.

### High-level dataflow

```
                            +-----------------------------------------------+
                            |                  TTS Path                     |
                            |                                               |
   text  -> Streaming   ->  | VibeVoiceStreamingProcessor                   |
            Processor        |   - apply_chat_template (Qwen2 tokenizer)   |
                            |   - cached prompt (.pt voice preset)         |
                            |                                               |
                            | VibeVoiceStreamingForConditionalGeneration-  |
                            | Inference                                    |
                            |   - language_model (Qwen2)                   |
                            |   - tts_language_model (parallel head)       |
                            |   - acoustic / semantic tokenizers           |
                            |   - diffusion head + DPMSolverMultistep      |
                            |                                               |
                            |   AudioStreamer (per-batch queue)            |
                            +-----------------------------------------------+
                                              |
                                              v
                                          PCM bytes (24 kHz s16le mono)
                                              ^
                                              |
                            +-----------------------------------------------+
                            |                  ASR Path                     |
                            |                                               |
   audio -> FFmpeg -> 24kHz | VibeVoiceASRProcessor                         |
            stdin pipe       |   - load + normalize audio                   |
                            |   - VAE-tokenize speech (semantic + acoustic)|
                            |   - context_info -> system prompt            |
                            |                                               |
                            | VibeVoiceASRForConditionalGeneration         |
                            |   - LLM-only generate over multimodal seq    |
                            |   - speech tokens injected via               |
                            |     acoustic_input_mask                      |
                            |                                               |
                            |   post_process_transcription (JSON parse)    |
                            +-----------------------------------------------+
                                              |
                                              v
                                  segments [{Start, End, Speaker, Content}]
```

Both paths share `vibevoice/processor/audio_utils.py`, `vibevoice/modular/configuration_*.py`, and the underlying `vibevoice/modular/modular_vibevoice_tokenizer.py` (acoustic + semantic VAEs). They diverge at the model class — TTS uses streaming inference with a diffusion head; ASR uses LLM generation with multimodal speech embeddings.

> [!NOTE]
> The unified `VibeVoiceStreamingModel.forward()` is intentionally disabled (raises `RuntimeError`). All inference goes through `model.language_model(...)`, `model.tts_language_model(...)`, or `model.generate(...)`. This is a design invariant that prevents callers from bypassing the streaming machinery accidentally.

### Component inventory

| Component | Path | Role |
|---|---|---|
| Top-level package | `vibevoice/__init__.py` | Re-exports streaming TTS classes for `from vibevoice import ...`. |
| Modular models | `vibevoice/modular/` | Configurations, model classes, streamer, diffusion head, tokenizer (acoustic + semantic VAEs), text tokenizer. 11 files, ~199KB. |
| Processors | `vibevoice/processor/` | Audio I/O (FFmpeg), audio normalizer, ASR processor, TTS processor, streaming processor, tokenizer processor. 6 files, ~95KB. |
| Schedulers | `vibevoice/schedule/` | DPMSolverMultistepScheduler, timestep sampler. 2 files, ~51KB. |
| Default configs | `vibevoice/configs/` | `qwen2.5_1.5b_64k.json`, `qwen2.5_7b_32k.json`. 2 files, ~5KB. |
| vLLM plugin | `vllm_plugin/` | vLLM model registration, audio input mapper, multimodal model class. 3 files, ~59KB. |
| Streaming demo + web | `demo/web/` | FastAPI app (`app.py`), HTML player (`index.html`). |
| Gradio + CLI demos | `demo/` | `realtime_model_inference_from_file.py`, `vibevoice_asr_gradio_demo.py`, `vibevoice_asr_inference_from_file.py`, `vibevoice_realtime_demo.py`. |
| LoRA finetune | `finetuning-asr/` | `lora_finetune.py`, `inference_lora.py`, toy dataset. |
| Voice presets | `demo/voices/streaming_model/` | 23 `.pt` files (Git LFS) covering 11 languages. |
| Repo metadata | `pyproject.toml`, `LICENSE`, `SECURITY.md`, `CONTRIBUTING.md` | Standard OSS scaffolding. |

### Technology stack

| Layer | Choice | Why |
|---|---|---|
| Language | Python 3.10+ | Hugging Face / PyTorch ecosystem. |
| Deep-learning framework | PyTorch | Native MPS / XPU / CUDA support, mature ecosystem. |
| Backbone LLM | Qwen2 (1.5B and 7B variants) | Strong multilingual, permissive license, supported by transformers. |
| Attention | flash-attention-2 (CUDA), SDPA (MPS / CPU / XPU), eager (debug) | Auto-selected per device. |
| Speech tokenizer | Custom acoustic + semantic VAEs (RMSNorm, depthwise conv mixer) | Continuous-token speech for both ASR ingest and TTS synthesis. |
| Diffusion head | 4-layer (default), v-prediction, cosine beta schedule | Conditional speech latent generation in TTS. |
| Audio I/O | FFmpeg (subprocess) | Universal format coverage; stdin-pipe for byte input. |
| Web framework | FastAPI + Starlette + uvicorn | Streaming TTS WebSocket endpoint. |
| UI | Gradio 6.0+ for ASR demo; static HTML + WebSocket for TTS player | Lightweight; both demos minimize external state. |
| Multi-tenant serving | vLLM (via `vllm.general_plugins` entry point) | High-throughput batched ASR. |
| Fine-tuning | PEFT LoRA + transformers Trainer | Memory-efficient adaptation. |
| Acceleration (optional) | Liger Kernel for Qwen2 (RoPE / RMSNorm / SwiGLU) | Faster attention/MLP layers in the ASR Gradio demo. |

### Data flow — streaming TTS

1. **HTTP/WS request** lands at `demo/web/app.py:websocket_stream`. Query params: `text`, `cfg`, `steps`, `voice`.
2. The endpoint acquires the per-app `asyncio.Lock`. Concurrent clients are rejected with WS code 1013.
3. `StreamingTTSService.stream()` runs the full pipeline:
    - Look up the voice preset (`_get_voice_resources`).
    - Run `VibeVoiceStreamingProcessor.process_input_with_cached_prompt(...)` — produces token IDs and attention mask.
    - Spawn a background `threading.Thread` that calls `model.generate(...)` with an `AudioStreamer(batch_size=1)` sink.
    - The generator yields decoded float32 chunks; the WS handler converts to int16 PCM via `chunk_to_pcm16` and writes binary frames.
4. JSON log frames are enqueued into a `queue.Queue` from the worker thread and flushed to the WS by the main coroutine between binary writes.
5. Stop is cooperative: a `threading.Event` is passed in via `stop_check_fn` so generation aborts on disconnect.

### Data flow — long-form ASR

1. Client supplies an audio file path, bytes, ndarray, or HF-dataset name.
2. `audio_utils.load_audio_use_ffmpeg()` shells out to `ffmpeg -loglevel error -nostdin ... -f s16le -ac 1 -ar 24000 -` to produce 24 kHz mono float32. Concurrency is throttled by `_FFMPEG_SEM` (a `threading.Semaphore` gated on `VIBEVOICE_FFMPEG_MAX_CONCURRENCY`).
3. `AudioNormalizer` applies a target dB-FS scaling.
4. `VibeVoiceASRProcessor` builds the prompt:
    - System prompt (with optional `context_info` hotword block).
    - Speech token region delimited by `<|object_ref_start|>...<|object_ref_end|>` (or by the corresponding IDs in the fast tokenizer).
    - Generation prompt (`<|im_start|>assistant\n`).
5. `VibeVoiceASRForConditionalGeneration.generate()` runs greedy / sampled / beam-search decoding.
6. Output text contains a JSON array of `{Start, End, Speaker, Content}` objects; `processor.post_process_transcription()` parses it.

> [!NOTE]
> Both tokenizers (acoustic and semantic) are *frozen* during LoRA fine-tuning. Only the Qwen2 attention and MLP projections receive adapter updates. This keeps the speech-token vocabulary stable across fine-tunes — important for swapping LoRA adapters at inference time.

---

## 2. System Requirements

### Hardware

| Tier | TTS-Realtime-0.5B | ASR-7B |
|---|---|---|
| **Dev box** | 8 GB VRAM (any CUDA-capable card or M1/M2 Mac) | 16 GB VRAM (RTX 3090 / 4080 / A100 40GB) |
| **Production single-tenant** | 16 GB VRAM, latency optimized | 24 GB VRAM, throughput optimized |
| **Multi-tenant (vLLM)** | Not the typical workload | 80 GB VRAM (A100 80GB or H100) for high concurrency |
| **CPU fallback** | Workable for short prompts | Not recommended; 5-10x slower than CUDA |
| **MPS (Apple)** | Yes (float32, SDPA) | Marginally workable for short audio |
| **XPU (Intel)** | Yes (float32, SDPA) | Yes (float32, SDPA) |

### Software

| Required | Version |
|---|---|
| Python | 3.10+ |
| PyTorch | 2.1+ recommended (2.4+ for the most recent attention APIs) |
| transformers | 4.40+ (Qwen2 baseline; some demos require 4.45+ for Liger Kernel) |
| FFmpeg | 4.x or newer; must be on `PATH` |
| Git LFS | For voice preset checkout |

| Optional | Purpose |
|---|---|
| flash-attn | CUDA-only attention acceleration |
| liger-kernel | Qwen2 RoPE/RMSNorm/SwiGLU fast paths in the ASR Gradio demo |
| pydub | MP3 re-encoding of audio segments in the ASR Gradio demo |
| peft | LoRA fine-tuning |
| diffusers | Pulls `DPMSolverMultistepScheduler` base classes |
| vllm | High-throughput ASR serving |

### Network

The repo's only outbound dependencies are:
- Hugging Face Hub (model + processor downloads).
- GitHub release attachments (experimental voice packs in `download_experimental_voices.sh`).

When deploying behind a firewall, mirror both into your registry and pass local paths to `--model_path`.

---

## 3. Configuration Reference

### 3.1 CLI flags — every argument across every entry point

| Script | Flag | Type | Default | Purpose |
|---|---|---|---|---|
| `finetuning-asr/inference_lora.py` | `--base_model` | str | `microsoft/VibeVoice-ASR` | Path to base pretrained model. |
| `finetuning-asr/inference_lora.py` | `--lora_path` | str | (required) | Path to LoRA adapter weights. |
| `finetuning-asr/inference_lora.py` | `--audio_file` | str | (required) | Path to audio file to transcribe. |
| `finetuning-asr/inference_lora.py` | `--context_info` | str | `None` | Hotword block, e.g. `Hotwords: Tea Brew, Aiden Host`. |
| `finetuning-asr/inference_lora.py` | `--max_new_tokens` | int | `4096` | Generation cap. |
| `finetuning-asr/inference_lora.py` | `--temperature` | float | `0.0` | Sampling temperature; `0` = greedy. |
| `finetuning-asr/inference_lora.py` | `--device` | str | auto | Device override. |
| `demo/realtime_model_inference_from_file.py` | `--model_path` | str | `microsoft/VibeVoice-Realtime-0.5B` | Streaming TTS checkpoint. |
| `demo/realtime_model_inference_from_file.py` | `--txt_path` | str | `demo/text_examples/1p_vibevoice.txt` | Script file. |
| `demo/realtime_model_inference_from_file.py` | `--speaker_name` | str | `Wayne` | Voice preset substring match. |
| `demo/realtime_model_inference_from_file.py` | `--output_dir` | str | `./outputs` | WAV output dir. |
| `demo/realtime_model_inference_from_file.py` | `--device` | str | auto | `cuda` / `mps` / `cpu`. |
| `demo/realtime_model_inference_from_file.py` | `--cfg_scale` | float | `1.5` | Classifier-Free Guidance. |
| `demo/vibevoice_asr_gradio_demo.py` | `--model_path` | str | `microsoft/VibeVoice-ASR` | ASR checkpoint. |
| `demo/vibevoice_asr_gradio_demo.py` | `--attn_implementation` | str | `flash_attention_2` | Attention backend. |
| `demo/vibevoice_asr_gradio_demo.py` | `--max_new_tokens` | int | `32768` | Default UI slider value. |
| `demo/vibevoice_asr_gradio_demo.py` | `--host` | str | `0.0.0.0` | Bind address. |
| `demo/vibevoice_asr_gradio_demo.py` | `--port` | int | `7860` | Bind port. |
| `demo/vibevoice_asr_gradio_demo.py` | `--share` | bool | False | Enable Gradio public tunnel. |
| `demo/vibevoice_asr_inference_from_file.py` | `--model_path` | str | (required) | ASR checkpoint. |
| `demo/vibevoice_asr_inference_from_file.py` | `--audio_files` | str+ | none | One or more file paths. |
| `demo/vibevoice_asr_inference_from_file.py` | `--audio_dir` | str | none | Directory walk. |
| `demo/vibevoice_asr_inference_from_file.py` | `--dataset` | str | none | HF dataset name. |
| `demo/vibevoice_asr_inference_from_file.py` | `--split` | str | `test` | Dataset split. |
| `demo/vibevoice_asr_inference_from_file.py` | `--max_duration` | float | `3600.0` | Max seconds per concatenated audio. |
| `demo/vibevoice_asr_inference_from_file.py` | `--batch_size` | int | `2` | Files per batch. |
| `demo/vibevoice_asr_inference_from_file.py` | `--device` | str | auto | `cuda`/`cpu`/`mps`/`xpu`/`auto`. |
| `demo/vibevoice_asr_inference_from_file.py` | `--max_new_tokens` | int | `32768` | Generation cap. |
| `demo/vibevoice_asr_inference_from_file.py` | `--temperature` | float | `0.0` | `0` = greedy. |
| `demo/vibevoice_asr_inference_from_file.py` | `--top_p` | float | `1.0` | Nucleus sampling. |
| `demo/vibevoice_asr_inference_from_file.py` | `--num_beams` | int | `1` | Beam-search width. |
| `demo/vibevoice_asr_inference_from_file.py` | `--attn_implementation` | str | `auto` | Attention backend. |
| `demo/vibevoice_realtime_demo.py` | `--port` | int | `3000` | uvicorn port. |
| `demo/vibevoice_realtime_demo.py` | `--model_path` | str | `microsoft/VibeVoice-Realtime-0.5B` | Sets `MODEL_PATH`. |
| `demo/vibevoice_realtime_demo.py` | `--device` | str | `cuda` | Sets `MODEL_DEVICE`. |
| `demo/vibevoice_realtime_demo.py` | `--reload` | bool | False | Enable uvicorn auto-reload. |

### 3.2 Environment variables

| Variable | Type | Default | Source | Purpose |
|---|---|---|---|---|
| `MODEL_PATH` | str | (none) | `demo/web/app.py:340` | Streaming TTS checkpoint. **Required**. |
| `MODEL_DEVICE` | str | `cuda` | `demo/web/app.py:344` | Streaming TTS device. |
| `VOICE_PRESET` | str | (none, falls back to `en-Carter_man`) | `demo/web/app.py:122` | Default voice for the streaming server. |
| `VIBEVOICE_MAX_AUDIO_DURATION` | float (sec) | `3660` | `vllm_plugin/inputs.py:20` | Hard cap on input duration. |
| `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` | int | `0` (no limit) | `vibevoice/processor/audio_utils.py:82` | Cap on concurrent FFmpeg subprocesses. |
| `VIBEVOICE_USE_MEAN` | bool (truthy) | unset | `vllm_plugin/model.py:264` | Use mean pooling over speech tokens (issue #256). |
| `OPTIMIZE_FOR_SPEED` | int (0/1) | `0` | `vibevoice/modular/modular_vibevoice_tokenizer.py:30` | Drop normalization in tokenizer for speed. |
| `USERNAME` | str | `user` | `demo/vibevoice_asr_gradio_demo.py:26` | Windows-only `USER` shim. |
| `TEMP` | str | `.` | `demo/vibevoice_asr_gradio_demo.py:27` | Windows-only base for `TORCHINDUCTOR_CACHE_DIR`. |

> [!IMPORTANT]
> Two of these variables silently change runtime behavior with no log output: `OPTIMIZE_FOR_SPEED=1` and `VIBEVOICE_USE_MEAN`. Set them only after benchmarking — a default deployment should leave both unset.

### 3.3 Model JSON configs (read by `from_pretrained`)

Two reference configs ship in `vibevoice/configs/`. They define the architecture; the `model.safetensors` carries the weights.

#### `qwen2.5_1.5b_64k.json` — used by streaming TTS

| Group | Key | Value | Notes |
|---|---|---|---|
| top | `model_type` | `vibepod` | Composition wrapper. |
| top | `_attn_implementation_autoset` | `true` | Attention is decided at load time. |
| top | `acoustic_vae_dim` | `64` | Latent dim of acoustic tokenizer. |
| top | `semantic_vae_dim` | `128` | Latent dim of semantic tokenizer. |
| top | `torch_dtype` | `bfloat16` | Default load dtype. |
| acoustic_tokenizer | `causal` | `true` | Streaming-friendly. |
| acoustic_tokenizer | `vae_dim` | `64` | |
| acoustic_tokenizer | `encoder_n_filters` | `32` | |
| acoustic_tokenizer | `encoder_ratios` | `[8, 5, 5, 4, 2, 2]` | Total 6400× downsample. |
| acoustic_tokenizer | `encoder_depths` | `3-3-3-3-3-3-8` | Per-stage block counts. |
| acoustic_tokenizer | `mixer_layer` | `depthwise_conv` | |
| acoustic_tokenizer | `layernorm` | `RMSNorm` | |
| acoustic_tokenizer | `disable_last_norm` | `true` | |
| acoustic_tokenizer | `fix_std` | `0.5` | Gaussian latent std. |
| acoustic_tokenizer | `std_dist_type` | `gaussian` | |
| acoustic_tokenizer | `model_type` | `vibepod_acoustic_tokenizer` | |
| semantic_tokenizer | `vae_dim` | `128` | |
| semantic_tokenizer | `fix_std` | `0` | |
| semantic_tokenizer | `std_dist_type` | `none` | |
| semantic_tokenizer | `model_type` | `vibepod_semantic_tokenizer` | |
| decoder (Qwen2) | `hidden_size` | `1536` | |
| decoder (Qwen2) | `intermediate_size` | `8960` | |
| decoder (Qwen2) | `max_position_embeddings` | `65536` | 64K context. |
| decoder (Qwen2) | `num_attention_heads` | `12` | |
| decoder (Qwen2) | `num_hidden_layers` | `28` | |
| decoder (Qwen2) | `num_key_value_heads` | `2` | GQA ratio 6. |
| decoder (Qwen2) | `rope_theta` | `1000000.0` | |
| decoder (Qwen2) | `tie_word_embeddings` | `true` | |
| decoder (Qwen2) | `vocab_size` | `151936` | |
| diffusion_head | `head_layers` | `4` | |
| diffusion_head | `head_ffn_ratio` | `3.0` | |
| diffusion_head | `hidden_size` | `1536` | Matches decoder. |
| diffusion_head | `latent_size` | `64` | |
| diffusion_head | `prediction_type` | `v_prediction` | |
| diffusion_head | `diffusion_type` | `ddpm` | |
| diffusion_head | `ddpm_num_steps` | `1000` | Training. |
| diffusion_head | `ddpm_num_inference_steps` | `20` | Default; the streaming server overrides to `5` for latency. |
| diffusion_head | `ddpm_beta_schedule` | `cosine` | |
| diffusion_head | `ddpm_batch_mul` | `4` | |
| diffusion_head | `model_type` | `vibepod_diffusion_head` | |

#### `qwen2.5_7b_32k.json` — used by long-form ASR

Identical structure; key deltas:

| Key | 1.5B value | 7B value |
|---|---|---|
| decoder.hidden_size | 1536 | 3584 |
| decoder.intermediate_size | 8960 | 18944 |
| decoder.num_attention_heads | 12 | 28 |
| decoder.num_hidden_layers | 28 | 28 |
| decoder.num_key_value_heads | 2 | 4 |
| decoder.max_position_embeddings | 65536 | 32768 |
| decoder.tie_word_embeddings | true | false |
| decoder.vocab_size | 151936 | 152064 |
| decoder.transformers_version | (absent) | 4.40.1 |
| diffusion_head.hidden_size | 1536 | 3584 |

The 7B config also sets `use_mrope: false` on the decoder.

### 3.4 LoRA configuration (`finetuning-asr/lora_finetune.py`)

| Argument class | Field | Default |
|---|---|---|
| `ModelArguments` | `model_path` | `microsoft/VibeVoice-ASR` |
| `DataArguments` | `data_dir` | `./toy_dataset` |
| `DataArguments` | `max_audio_length` | `None` (no cap) |
| `DataArguments` | `use_customized_context` | `True` |
| `LoraArguments` | `lora_r` | `16` |
| `LoraArguments` | `lora_alpha` | `32` |
| `LoraArguments` | `lora_dropout` | `0.05` |
| LoRA target modules | (hardcoded) | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj` |
| `LoraConfig` | `bias` | `"none"` |
| `LoraConfig` | `task_type` | `TaskType.CAUSAL_LM` |

The script forces `dataloader_num_workers = 0` and `remove_unused_columns = False` regardless of `TrainingArguments` user input, because audio loading is not multiprocessing-safe in this code path.

### 3.5 HTTP API surface (streaming TTS, `demo/web/app.py`)

| Method | Path | Function | Auth | Returns |
|---|---|---|---|---|
| GET | `/` | `index` | none | `index.html` (FileResponse). |
| GET | `/config` | `get_config` | none | `{"voices": [...], "default_voice": "..."}`. |
| WebSocket | `/stream` | `websocket_stream` | none | Binary PCM frames + JSON log frames; closes with code 1013 if busy. |

There is no auth, no rate limit, and no CORS configuration baked in — the demo is intended to run behind your own gateway.

> [!WARNING]
> Do **not** expose `demo/web/app.py` directly to the public internet. It has zero authentication and a single global lock, so anyone can both (a) tie up your GPU with a long script and (b) trivially DoS the service.

---

## 4. Deployment & Scaling

### 4.1 Single-host streaming TTS

The simplest deployment is a single CUDA host running:

```bash
MODEL_PATH=microsoft/VibeVoice-Realtime-0.5B \
MODEL_DEVICE=cuda \
VOICE_PRESET=en-Carter_man \
python demo/vibevoice_realtime_demo.py --port 3000
```

Front this with nginx for TLS termination + WebSocket proxying. Example nginx block:

```nginx
upstream vibevoice {
    server 127.0.0.1:3000;
}
server {
    listen 443 ssl http2;
    location / {
        proxy_pass http://vibevoice;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}
```

> [!IMPORTANT]
> The default WebSocket idle timeout in many proxies is 60s — too short for long generations. Bump `proxy_read_timeout` and `proxy_send_timeout` to at least the longest TTS request you expect (5 minutes is a safe default for read-aloud workloads).

### 4.2 Horizontal scaling — TTS

Each FastAPI process holds a single global `asyncio.Lock`. Concurrency therefore equals number of processes (not threads). Two patterns:

- **Worker pool**: launch `--workers N` uvicorn processes against the same socket, each with its own model copy. Memory cost: N × ~6 GB on bfloat16. Best when you have a single large GPU and want N concurrent streams.
- **Multi-host**: deploy several boxes behind a sticky load balancer (Layer 7 with cookie or path-based routing). The streaming WebSocket holds state across messages, so sticky routing is required.

### 4.3 vLLM serving for ASR

For high-throughput ASR, register the plugin and launch vLLM:

```bash
pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model microsoft/VibeVoice-ASR \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.85 \
  --max-num-seqs 8
```

The plugin auto-loads via `pyproject.toml`'s `[project.entry-points."vllm.general_plugins"]` table and registers:

- `VibeVoiceConfig` with `AutoConfig`.
- `VibeVoiceASRTextTokenizerFast` (fast) and `Qwen2Tokenizer` (slow) with `AutoTokenizer`.
- `Qwen2AudioProcessor` with `AutoProcessor`.
- `VibeVoiceForCausalLM` with vLLM's `ModelRegistry` under names `VibeVoice` and `VibeVoiceForASRTraining`.

Audio inputs come in via the `multi_modal_data` parameter on the OpenAI-compatible request (audio/file/bytes/array, see `vllm_plugin/inputs.py`).

> [!NOTE]
> Throughput at 8 concurrent requests on an A100-80GB approaches ~3-4× single-stream PyTorch baseline, mostly from vLLM's continuous batching.

### 4.4 Sizing the GPU

| Workload | Concurrency | Suggested GPU |
|---|---|---|
| Single-user dev / demo (TTS) | 1 | RTX 3060 12GB / RTX 4070 |
| Production read-aloud (TTS) | 4-8 | RTX 4090 24GB or A10 24GB |
| Single-user ASR (1-hour audio) | 1 | RTX 3090/4090 24GB |
| Multi-tenant ASR (vLLM) | 8-16 | A100 80GB / H100 80GB |

### 4.5 Container deployment

There is no first-party Dockerfile in the repo. A minimal recipe:

```dockerfile
FROM nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04
RUN apt-get update && apt-get install -y python3.10 python3-pip ffmpeg git git-lfs
WORKDIR /app
COPY . /app
RUN pip install -e . && \
    pip install transformers peft accelerate diffusers \
    soundfile pydub fastapi uvicorn[standard] websockets gradio && \
    pip install flash-attn --no-build-isolation
RUN git lfs install && git lfs pull
ENV MODEL_PATH=microsoft/VibeVoice-Realtime-0.5B \
    MODEL_DEVICE=cuda \
    VIBEVOICE_FFMPEG_MAX_CONCURRENCY=4
EXPOSE 3000
CMD ["python", "demo/vibevoice_realtime_demo.py", "--port", "3000"]
```

> [!CAUTION]
> Issue #339 reports the Gradio ASR demo crashing under HTTP-only Docker setups due to URL scheme rewriting. Always front the Gradio demo with TLS in container environments.

---

## 5. Operations

### 5.1 Health checks

```bash
# Streaming TTS
curl -fsS http://localhost:3000/config

# Expected
# {"voices": ["de-Spk0_man", ..., "en-Carter_man", ...], "default_voice": "en-Carter_man"}
```

The `/config` endpoint dereferences `service.voice_presets`, which is only populated after `service.load()` completes. A 500/disconnect therefore indicates the model has not finished loading.

> [!TIP]
> For Kubernetes, use `httpGet` with `/config` for both `readinessProbe` and `livenessProbe`. Allow at least 120 seconds for the initial model load before the readiness probe starts.

### 5.2 Logs

The streaming TTS server emits structured stdout logs:
- `[startup] Loading processor from <path>` — early init.
- `[startup] Found N voice presets` — voice scan complete.
- `[startup] Loading prefilled prompt from <path>` — voice cached.
- `[startup] Model ready.` — service is accepting traffic.
- Per-request: `Client connected, text='...'` and JSON event frames over WebSocket (`backend_request_received`, `model_progress`, `backend_first_chunk_sent`, `backend_stream_complete`, `client_disconnected`, `backend_error`).

Pipe to a log aggregator that handles structured JSON; the WebSocket events are valuable telemetry.

### 5.3 Metrics worth tracking

| Metric | Source | Why it matters |
|---|---|---|
| Time to first chunk (sec) | Diff between `backend_request_received` and `backend_first_chunk_sent` | Most user-visible latency. |
| Generation throughput (chunk_sec / wall_sec) | `model_progress` events | Inverse of RTF; tracks GPU health. |
| Lock-contended rejections | Count of WS code 1013 closes | Indicates need to scale workers. |
| FFmpeg subprocess count | `_FFMPEG_SEM` permits taken | Set `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` to a known cap. |
| GPU memory headroom | `nvidia-smi` | Spikes during long generations; OOM risk. |

### 5.4 Backup & state

VibeVoice is **stateless**. Voice presets and model weights are content-addressable (Hugging Face hashes); nothing per-tenant lives on disk. The only state worth checkpointing is:
- **LoRA adapter weights** (after fine-tuning) → push to HF Hub or copy to S3.
- **Custom voice presets** (`.pt` files) → copy to S3.

### 5.5 Runbook checklist for a new deployment

- [ ] Provision a GPU host with CUDA 12.1+ and FFmpeg installed.
- [ ] Clone the repo and `git lfs pull` to fetch voice presets.
- [ ] Install the package + flash-attn.
- [ ] Pre-warm `microsoft/VibeVoice-Realtime-0.5B` (or your local copy) via `huggingface-cli download`.
- [ ] Set `MODEL_PATH`, `MODEL_DEVICE`, `VOICE_PRESET`, `VIBEVOICE_FFMPEG_MAX_CONCURRENCY`.
- [ ] Launch through `demo/vibevoice_realtime_demo.py` (or an equivalent supervisord / systemd unit).
- [ ] Wait for `[startup] Model ready.` before flipping the load balancer.
- [ ] Verify `GET /config` returns 200 with the expected voice list.
- [ ] Verify a short WebSocket TTS round-trip (10-second script) under target latency.
- [ ] Configure Prometheus / Grafana alerts on per-request latency p95.

> [!TIP]
> Pre-warm the voice cache by hitting `/stream` with `text=Hi&voice=<preset>` for every preset you expect to use. The first request per voice triggers a `torch.load(.pt, weights_only=False)` from disk; subsequent requests reuse the cached object.

### 5.6 Upgrade procedure

1. Snapshot any LoRA adapters and custom voices.
2. `git fetch && git checkout <new-tag>` in a parallel directory.
3. Run a smoke test: `python demo/realtime_model_inference_from_file.py ...` against the new code.
4. Cut traffic over (DNS or load-balancer flip).
5. Keep the old install for 24h as a rollback.

---

## 6. Performance

### 6.1 TTS latency budget (CUDA + flash-attention-2, RTX 4090)

| Stage | Time |
|---|---|
| FastAPI accept + lock | <5 ms |
| Tokenize text | ~10 ms |
| Voice preset lookup (cache hit) | <1 ms |
| First diffusion step (5 steps total) | ~150 ms |
| First audio chunk emitted | ~200 ms |
| Subsequent chunks (per ~1 sec audio) | ~250 ms wall |

**Time-to-first-byte target: ≤ 250 ms.** Real Time Factor at this configuration is ~0.3 — generation runs ~3× faster than playback.

### 6.2 ASR throughput

| Audio length | Single-stream wall time (RTX 4090, bfloat16, flash-attention-2) |
|---|---|
| 1 minute | ~3-5 s |
| 10 minutes | ~25-40 s |
| 1 hour | ~3-5 minutes |

vLLM raises throughput sublinearly with batch size. Past ~8 concurrent requests, FFmpeg becomes the bottleneck — set `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` to your CPU core count.

### 6.3 Memory profile

| Component | bfloat16 | float32 |
|---|---|---|
| TTS-Realtime-0.5B weights | ~1 GB | ~2 GB |
| TTS KV cache (64K context) | ~1 GB | ~2 GB |
| ASR-7B weights | ~14 GB | ~28 GB |
| ASR KV cache (32K context, batch 1) | ~1.5 GB | ~3 GB |

### 6.4 Tuning levers

> [!TIP]
> Production tuning, in order of impact: **(1) attention implementation** (flash-attention-2 vs SDPA — measure both), **(2) `cfg_scale`** (lower = faster, less expressive), **(3) `inference_steps`** (5 default, raise for fidelity), **(4) Liger Kernel** in the ASR path, **(5) `OPTIMIZE_FOR_SPEED=1`** for tokenizer speedup.

| Lever | Effect | Risk |
|---|---|---|
| `inference_steps` 5 → 10 | ~30% better fidelity | ~2× latency |
| `cfg_scale` 1.5 → 1.0 | ~10% lower latency | More monotone delivery |
| `attn_implementation` SDPA → flash-attention-2 | ~25-40% faster | Requires CUDA + nvcc |
| `OPTIMIZE_FOR_SPEED=1` | ~5-10% faster tokenizer | Slight quality drop |
| Liger Kernel (ASR Gradio) | ~15-25% faster Qwen2 layers | Requires `liger-kernel` package |
| `VIBEVOICE_FFMPEG_MAX_CONCURRENCY=4` | Caps subprocess pressure | None (cap to ≤ CPU cores) |
| `do_sample=false`, `num_beams=1` (ASR) | Deterministic, fastest | Less diverse; usually preferred |
| `num_beams=4` (ASR) | Best WER on hard audio | ~3-4× slower |

### 6.5 Profiling

```python
import torch
with torch.profiler.profile(
    activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA],
    record_shapes=True,
) as prof:
    out = model.generate(**inputs, max_new_tokens=256)
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=20))
```

Hot spots in our profiling: Qwen2 attention (~50%), diffusion head (TTS only, ~25%), tokenizer encode (~10%), audio normalization (~5%). Memory peaks during the prefill pass — tune `--max_new_tokens` if you see OOM mid-generation.

---

## 7. Security

### 7.1 Threat model

VibeVoice is a research model release. Three primary threat surfaces apply if you deploy it:

1. **Model misuse** — voice cloning + TTS produces realistic speech in the trained voices. Mitigation: deploy behind authenticated APIs, retain server-side logs of `(text, voice)` tuples.
2. **Data exfiltration via context_info** — the ASR system prompt accepts free-form text; in a multi-tenant deployment, this could be used to inject system-prompt content. Mitigation: sanitize / quote-escape user-supplied hotwords before passing through.
3. **Resource exhaustion** — the streaming server has no rate limit and runs unbounded scripts on the GPU. Mitigation: max-text-length guard, per-IP rate limit, max-concurrency cap, queue rejections.

### 7.2 Network exposure

The streaming server (`demo/web/app.py`):
- Exposes WebSocket `/stream` with **no authentication**.
- Exposes `/config` with **no authentication** (returns voice list, useful for fingerprinting).
- Has no CORS configuration; FastAPI defaults to no CORS allowance.

> [!WARNING]
> Never expose `demo/web/app.py` directly to the internet. Wrap with an authenticated reverse proxy (oauth2-proxy, Cloudflare Access, your own API gateway). Even the demo `index.html` page is intended for local-network use.

### 7.3 Secret management

The repo currently consumes **no secrets** at runtime:
- No API keys.
- No database connection strings.
- No Hugging Face token requirement (public models).

The only sensitive environment input is `MODEL_PATH` if it points to a local directory containing weights you do not want to leak. Bind-mount that directory read-only.

### 7.4 Input validation

| Surface | Validation enforced | Validation **not** enforced |
|---|---|---|
| `/stream?text=` | None — raw text passed to model. | Text length, language, content. |
| `/stream?voice=` | Only existing preset keys are honored; unknown defaults silently to `en-Carter_man`. | None |
| `/stream?cfg=` | Coerced to float; ≤ 0 falls back to 1.5. | Upper bound (extreme values produce artefacts but no crash). |
| `/stream?steps=` | Coerced to int > 0; otherwise None. | Upper bound. |
| ASR audio bytes | `VIBEVOICE_MAX_AUDIO_DURATION` enforced. | No file-format whitelist beyond FFmpeg's parser. |
| ASR `context_info` | Free text. | None — tenant separation must come from the deployer. |

> [!CAUTION]
> Untrusted audio bytes are decoded by FFmpeg via `subprocess.run(...)`. While this is a sandboxed binary, FFmpeg has had its share of CVEs. Keep FFmpeg patched and consider running the API container with seccomp / `--read-only` filesystem and a dropped capability set (no `CAP_NET_RAW`, no `CAP_SYS_ADMIN`).

### 7.5 Supply chain

`pyproject.toml` declares the project. The repo's external dependencies (PyTorch, transformers, diffusers, FFmpeg binary, optional flash-attn / vllm / liger-kernel) all come from public registries. Pin exact versions in your deployment lockfile and mirror to a private index.

The vLLM plugin's `register_vibevoice()` runs **at import time** for every vLLM process. Treat this as a code-execution surface — only add the plugin to vLLM environments you trust.

### 7.6 Reporting vulnerabilities

`SECURITY.md` is the canonical channel. Issue #341 (closed) noted earlier guidance was unclear; check the latest copy on `main` for the reporting email address and PGP key.

---

## 8. Developer Internals

### 8.1 Repository structure

```
VibeVoice/
├── pyproject.toml              # Package metadata + entry points (vLLM)
├── LICENSE                     # MIT
├── SECURITY.md                 # Vulnerability reporting
├── CONTRIBUTING.md             # PR guide
├── README.md                   # (intentionally minimal in this commit)
├── vibevoice/
│   ├── __init__.py            # Re-exports streaming TTS classes
│   ├── modular/                # Model code
│   │   ├── configuration_vibevoice.py             # VibeVoiceConfig + sub-configs
│   │   ├── configuration_vibevoice_streaming.py   # Streaming variant
│   │   ├── modeling_vibevoice.py                  # Base TTS model
│   │   ├── modeling_vibevoice_asr.py              # ASR model (separate path)
│   │   ├── modeling_vibevoice_streaming.py        # Streaming TTS submodel (forward disabled)
│   │   ├── modeling_vibevoice_streaming_inference.py  # Inference wrapper with .generate()
│   │   ├── modular_vibevoice_diffusion_head.py    # Diffusion head MLP
│   │   ├── modular_vibevoice_text_tokenizer.py    # Text tokenizer (Qwen2-derived) + ASR variant
│   │   ├── modular_vibevoice_tokenizer.py         # Acoustic + semantic VAEs
│   │   └── streamer.py                             # AudioStreamer + AsyncAudioStreamer
│   ├── processor/              # Audio + tokenization
│   │   ├── audio_utils.py                         # FFmpeg shell-out + AudioNormalizer
│   │   ├── vibevoice_processor.py                 # Long-form TTS processor (multi-speaker)
│   │   ├── vibevoice_streaming_processor.py       # Streaming TTS processor
│   │   ├── vibevoice_asr_processor.py             # ASR processor (system prompt + speech tokens)
│   │   └── vibevoice_tokenizer_processor.py       # File I/O for .wav/.mp3/.pt/.npy/.npz
│   ├── schedule/               # Diffusion scheduler
│   │   ├── dpm_solver.py                          # DPMSolverMultistepScheduler
│   │   └── timestep_sampler.py                    # Training-time sampling
│   ├── configs/
│   │   ├── qwen2.5_1.5b_64k.json
│   │   └── qwen2.5_7b_32k.json
│   └── scripts/                # (placeholder)
├── vllm_plugin/
│   ├── __init__.py            # register_vibevoice() — entry point
│   ├── inputs.py              # vibevoice_audio_input_mapper
│   └── model.py               # VibeVoiceForCausalLM (vLLM model class)
├── demo/
│   ├── realtime_model_inference_from_file.py
│   ├── vibevoice_asr_gradio_demo.py
│   ├── vibevoice_asr_inference_from_file.py
│   ├── vibevoice_realtime_demo.py
│   ├── vibevoice_realtime_colab.ipynb
│   ├── download_experimental_voices.sh
│   ├── web/
│   │   ├── app.py             # FastAPI + WebSocket
│   │   └── index.html
│   ├── voices/streaming_model/*.pt   # 23 voice presets (Git LFS)
│   ├── text_examples/*.txt
│   └── asr_demo/*.{mp3,mp4,wav}
├── finetuning-asr/
│   ├── lora_finetune.py
│   ├── inference_lora.py
│   ├── README.md
│   └── toy_dataset/{0.json, 0.mp3, 1.json, 1.mp3}
└── docs/                       # Reserved (currently no markdown — pull from upstream branches)
```

### 8.2 Module reference

#### `vibevoice/modular/configuration_vibevoice.py`

Defines four `PretrainedConfig` subclasses:
- `VibeVoiceAcousticTokenizerConfig` (`model_type = "vibevoice_acoustic_tokenizer"`).
- `VibeVoiceSemanticTokenizerConfig` (`model_type = "vibevoice_semantic_tokenizer"`).
- `VibeVoiceDiffusionHeadConfig` (`model_type = "vibevoice_diffusion_head"`).
- `VibeVoiceConfig` (`model_type = "vibevoice"`, composition of the four above plus `Qwen2Config`).

Implements `_convert_dtype_to_string()` to coerce `torch.dtype` to its string form during `to_dict()` — fixes "Object of type dtype is not JSON serializable" reported in issue #199.

`VibeVoiceConfig.get_text_config()` returns the inner `Qwen2Config` so vLLM can read `hidden_size`, `num_attention_heads`, etc. for memory profiling.

#### `vibevoice/modular/modeling_vibevoice.py`

Base TTS model. Raises `NotImplementedError(f'Speech type {speech_type} not implemented')` for unknown speech-type tokens in the prefix; `NotImplementedError(f'Prediction type {prediction_type} not implemented')` for unknown diffusion targets.

#### `vibevoice/modular/modeling_vibevoice_asr.py`

`VibeVoiceASRForConditionalGeneration` — wraps the Qwen2 decoder for ASR. Validates `segment_length must be positive` (raises ValueError otherwise).

#### `vibevoice/modular/modeling_vibevoice_streaming.py`

`VibeVoiceStreamingModel.forward()` is intentionally disabled — raises `RuntimeError('VibeVoiceStreamingModel.forward is intentionally disabled. Use model.language_model(...) or model.tts_language_model(...) instead.')`.

#### `vibevoice/modular/modeling_vibevoice_streaming_inference.py`

Inference wrapper. `set_output_embeddings()` raises if `lm_head` is undefined; the unified `forward()` is disabled (use `forward_lm`, `forward_tts_lm`, or `generate`); loss computation is `NotImplementedError`.

#### `vibevoice/modular/modular_vibevoice_tokenizer.py`

The acoustic and semantic VAEs. `OPTIMIZE_FOR_SPEED` env var read at module load time (line 30) skips selected normalization/scale steps. Validates several knobs:
- `mixer_layer` ∈ a hardcoded set (else "Unsupported mixer layer").
- `layernorm` ∈ {`RMSNorm`, `LayerNorm`} (else "Unsupported norm type").
- `dist_type` ∈ {`fix`, `gaussian`} (else "Unsupported dist_type").
- GroupNorm + causal evaluation incompatible (rejected).

#### `vibevoice/modular/streamer.py`

`AudioStreamer(batch_size, stop_signal, timeout)` and `AsyncAudioStreamer`. Both validate `sample_idx < self.batch_size`; the streaming TTS path always uses `batch_size=1`.

#### `vibevoice/processor/vibevoice_processor.py`

Long-form TTS processor (multi-speaker dialog, not used by the realtime demo). Validates JSON speaker scripts: a list of speaker entries, with at least one valid speaker line, content non-empty.

#### `vibevoice/processor/vibevoice_asr_processor.py`

ASR processor. Builds the prompt:

```
<|im_start|>system
You are a speech recognition model.
<context_info, if provided>
<|im_end|>
<|im_start|>user
<|object_ref_start|><speech_pad_id × N><|object_ref_end|>
<|im_end|>
<|im_start|>assistant
```

Speech tokens are mapped to specific Qwen2 special-token IDs:
- `speech_start_id` → `<|object_ref_start|>`
- `speech_pad_id` → `<|box_start|>`
- `speech_end_id` → `<|object_ref_end|>`

The fast tokenizer (`VibeVoiceASRTextTokenizerFast` from `modular_vibevoice_text_tokenizer.py`) MUST respect these mappings or ASR quality silently degrades — this is called out in `vllm_plugin/__init__.py:register_vibevoice()`.

Supported `language_model_pretrained_name`: `Qwen`, `Llama`, `Gemma`. Other backbones raise `ValueError`.

#### `vibevoice/processor/vibevoice_streaming_processor.py`

Streaming TTS processor. `__call__` is **not implemented** — callers must use `process_input_with_cached_prompt(text, cached_prompt, ...)` which combines the script with a pre-encoded voice preset.

#### `vibevoice/processor/vibevoice_tokenizer_processor.py`

File I/O. Accepts `.wav`, `.mp3`, `.flac`, `.m4a`, `.ogg`, `.pt`, `.npy`, `.npz`. Validates audio shape (1D or 2D, otherwise raises). `save_audio()` requires `soundfile`; if missing it raises `ImportError('soundfile is required to save audio files. Install it with: pip install soundfile')`.

#### `vibevoice/schedule/dpm_solver.py`

Adapted from `https://github.com/LuChengTHU/dpm-solver`. Implements multi-step DPM solvers (1st through 3rd order) with several `algorithm_type` and `final_sigmas_type` options. Heavy validation:
- `algorithm_type` not implemented → `NotImplementedError`.
- `solver_type` not implemented → `NotImplementedError`.
- `final_sigmas_type ∈ {zero, sigma_min}` (else ValueError).
- `prediction_type ∈ {epsilon, sample, v_prediction}`.
- `timestep_spacing ∈ {linspace, leading, trailing}`.
- `beta_schedule` not implemented → `NotImplementedError`.
- `alpha_transform_type` ∈ {cosine, exp, cauchy, laplace}.
- Cannot pass both `num_inference_steps` and `timesteps` (or `custom_timesteps`).
- `karras_sigmas` and `lu_lambdas` mutually exclusive with explicit `timesteps`.

#### `vllm_plugin/inputs.py`

`vibevoice_audio_input_mapper(ctx, data)` — accepts `str`, `bytes`, `np.ndarray`, or `List[str]`; rejects everything else with `ValueError(f'Unsupported audio data type: {type(data)}')`. Enforces `VIBEVOICE_MAX_AUDIO_DURATION` (raises with a specific guidance message instructing the user to set the env var).

#### `vllm_plugin/model.py`

`VibeVoiceForCausalLM` — the vLLM-side model class. Validates: tokenizer config presence (`Missing acoustic/semantic tokenizer config in model config`), tokenizer config types (`acoustic_tokenizer_config has unexpected type`, `semantic_tokenizer_config has unexpected type`), audio item length (`Audio at index N is too short to be represented`), modality strings (`Unsupported modality: {modality}`), and `embed_tokens` discoverability (`Cannot find embed_tokens layer`).

#### `demo/web/app.py`

Entry point for the FastAPI streaming TTS server. Composition:
- `StreamingTTSService` — owns the model, processor, voice presets, default voice key, voice cache.
- `StreamingTTSService.load()` — runs at FastAPI `startup` event; selects dtype + attn implementation per device; reconfigures the noise scheduler to `sde-dpmsolver++` + `squaredcos_cap_v2`; sets DDPM inference steps to 5.
- `StreamingTTSService.stream()` — generator that yields normalized float32 numpy chunks.
- `chunk_to_pcm16` — clips, scales by 32767, casts to int16, encodes to bytes.
- `streaming_tts(text, **kwargs)` — module-level convenience wrapper.
- `websocket_stream` — WebSocket handler, owns the `asyncio.Lock` busy-check.
- `index` — serves `index.html`.
- `get_config` — JSON list of voices and default voice key.

### 8.3 Code patterns

**Config composition.** `VibeVoiceConfig` is a Hugging Face *composition* config (`is_composition = True`) with four `sub_configs`. Each child can be passed as a dict (auto-instantiated) or a config instance. This is how the same code path serves both the 1.5B-64K and 7B-32K checkpoints — only the JSON values differ.

**Attention selection.** Every model loader in this repo follows the same pattern: try `flash_attention_2` first; if it raises (commonly `ImportError`), fall back to `sdpa` and print a warning. The fall-through is hardcoded — there is no env-var override.

**Device handling.** Each script normalizes `mpx → mps`, then validates MPS availability and falls back to CPU. CUDA gets bfloat16 + `device_map='cuda'`, MPS gets float32 + manual `model.to('mps')` (because MPS does not yet honor device_map cleanly).

**Lazy loading.** `_voice_cache` in `StreamingTTSService` caches `torch.load(.pt)` results per voice key. Other parts of the codebase use Hugging Face `from_pretrained` caching (`~/.cache/huggingface/hub/`). No explicit cache eviction — restart the server to free voice memory.

**Streaming via threads.** TTS generation runs in a daemon `threading.Thread` because the underlying `model.generate()` is a long blocking call. The main async event loop polls a `queue.Queue` for log frames between binary writes — `await asyncio.to_thread(next, iterator, sentinel)` is used to advance the audio iterator without blocking the loop.

**Right-padding for training, left-padding for inference.** The LoRA finetune collator (`VibeVoiceASRDataCollator`) right-pads `input_ids`/`labels`/`speech` for training. The processor's default behavior at inference time is left-padding (so causal attention treats the unpadded tokens as the most recent). This split is intentional and load-bearing — see `lora_finetune.py:VibeVoiceASRDataCollator` for the comment.

**Frozen tokenizers.** `setup_model_for_training` walks `model.named_parameters()` and freezes everything matching `acoustic_tokenizer` or `semantic_tokenizer`. Only the Qwen2 attention/MLP weights receive LoRA adapters.

### 8.4 Testing strategy

The `vllm_plugin/tests/` directory contains `test_api.py` and `test_api_auto_recover.py` — these exercise the vLLM serving path end-to-end. There is no unit-test infrastructure committed for the rest of the repo; integration testing is performed via the demo scripts (which double as smoke tests). The toy dataset under `finetuning-asr/toy_dataset/` (two MP3 + JSON pairs) supports a single-step finetune sanity check.

> [!NOTE]
> Test coverage is the only confidence-floor "gap" recorded in this briefing's metadata. Adding pytest coverage for `VibeVoiceASRProcessor`, `audio_utils.load_audio_use_ffmpeg`, and `DPMSolverMultistepScheduler.set_timesteps` would close the loop without major architectural surgery.

### 8.5 Potential improvements

| Topic | Source | Note |
|---|---|---|
| GGUF quantization | issue #297 | Community feature request; not first-party. |
| OpenAI API parity for TTS | issue #336 | "Does VibeVoice support open API" — currently only WebSocket. |
| NVIDIA DGX Spark support | issue #340 | New hardware target. |
| Arabic / Vietnamese / Traditional Chinese voices | #108, #253, #271, #283 | Voice preset gaps. |
| Low-resource ASR training | #311, #115 | Documentation + recipes for non-English finetune. |
| Saved-files location clarity | #328 | The Gradio demo embeds segments inline; users expect filesystem outputs. |
| Windows 11 stability | #327 | CUDA/VC++ runtime alignment. |
| Token-length runaway | #332 | Input validation pre-tokenize. |
| Package-name disambiguation | #184 | "vibevoice" name collides with unrelated PyPI projects. |
| AliMeeting DER methodology | #322 | Eval pipeline question, deserves a doc page. |

### 8.6 Contributing

`CONTRIBUTING.md` is checked in but minimal. Expect the standard Microsoft OSS flow: fork → branch → PR; sign-off via CLA bot if it triggers; tests + lint required for non-trivial changes. The repo has only a `main` branch and follows trunk-based development.

---

## 9. Integration Points

### 9.1 Hugging Face Hub

All model and processor classes implement the standard `from_pretrained` / `save_pretrained` contracts. The repo references three published Hugging Face IDs:

| ID | Used by |
|---|---|
| `microsoft/VibeVoice-Realtime-0.5B` | Streaming TTS demos, default `MODEL_PATH`. |
| `microsoft/VibeVoice-ASR` | ASR demos, LoRA finetune base, vLLM plugin. |
| `Qwen/Qwen2.5-7B` | ASR processor pulls the Qwen tokenizer via `language_model_pretrained_name="Qwen/Qwen2.5-7B"`. |

To deploy without network access, mirror these to your private HF instance and pass local paths to every script's `--model_path` / `--base_model`.

### 9.2 vLLM

`pyproject.toml` declares an entry point under `vllm.general_plugins` that maps to `vllm_plugin.register_vibevoice`. When vLLM starts, it discovers and runs this plugin, which registers the config, tokenizers, processor, and model classes. After registration, vLLM treats `microsoft/VibeVoice-ASR` as a first-class model — `vllm serve microsoft/VibeVoice-ASR` works end-to-end.

### 9.3 PEFT / LoRA

`finetuning-asr/lora_finetune.py` uses `peft.LoraConfig` + `peft.get_peft_model`. `inference_lora.py` uses `peft.PeftModel.from_pretrained(base_model, lora_path)`. Adapter weights can be merged into the base model for faster inference (`model = model.merge_and_unload()`), at the cost of swappability.

### 9.4 FFmpeg

The audio loader requires `ffmpeg` and `ffprobe` on `PATH`. Both are invoked via `subprocess.run(..., check=True)`. The semaphore in `audio_utils.py` allows operators to cap concurrent subprocesses (set `VIBEVOICE_FFMPEG_MAX_CONCURRENCY` to ~CPU-core count under heavy load).

### 9.5 Gradio

`demo/vibevoice_asr_gradio_demo.py` builds a `gr.Blocks` interface. Custom CSS styles the Stop button. `demo.queue(default_concurrency_limit=3)` allows up to three concurrent ASR requests through the Gradio runtime, but the ASR model itself is shared — back-pressure builds on the GPU, not the queue. `demo.launch(share=args.share, show_error=True, theme=gr.themes.Soft(), css=custom_css)` is the standard Gradio launch call; share=True publishes a Gradio public tunnel.

### 9.6 FastAPI / Starlette

`demo/web/app.py` uses FastAPI lifespan via `@app.on_event("startup")` (the older event-based API). For long-lived production, port to the lifespan async context manager pattern; behavior is equivalent.

### 9.7 Liger Kernel (optional)

The ASR Gradio demo wraps Qwen2 RoPE / RMSNorm / SwiGLU layers with Liger Kernel if importable:

```python
from liger_kernel.transformers import apply_liger_kernel_to_qwen2
apply_liger_kernel_to_qwen2(rope=True, rms_norm=True, swiglu=True, cross_entropy=False)
```

`cross_entropy=False` is intentional — the Liger CE replacement is not yet validated against VibeVoice's loss path. Leave it disabled.

### 9.8 Pydub (optional)

Used by the ASR Gradio demo to re-encode segment audio as 32 kbps MP3 (smaller than WAV for inline data URIs). If `pydub` is missing, the demo falls back to WAV embedding. No effect on transcription quality — display-time only.

### 9.9 Diffusers

The DPM solver inherits from `diffusers.schedulers.scheduling_utils.SchedulerMixin` + `ConfigMixin`. This makes it usable in any diffusers pipeline that expects a `KarrasDiffusionSchedulers`-compatible scheduler. The streaming server reconfigures the scheduler to `sde-dpmsolver++` + `squaredcos_cap_v2` at startup — those values are not the diffusers defaults; if you build a pipeline outside the streaming server you must reconfigure manually.

---

_Generated April 2026 from microsoft/VibeVoice at 4a78d3e. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._