VibeVoice User Guide

VibeVoice is Microsoft's open-source family of speech models for streaming text-to-speech (TTS) and long-form automatic speech recognition (ASR) with speaker diarization. The codebase ships two production-ready model families, several inference scripts, two Gradio demos, a FastAPI WebSocket streaming server, a vLLM serving plugin, and a LoRA fine-tuning recipe — all built on Qwen2.5 backbones.

This guide is for end users running inference, transcribing audio, generating voice-over from scripts, or wiring VibeVoice into an application. For architecture, deployment, and developer internals, see the VibeVoice Administrator & Developer Guide.

1. Quick Start

What is VibeVoice?

VibeVoice is a research-grade speech stack from Microsoft that delivers two capabilities from a single repository:

• VibeVoice-Realtime-0.5B — A 0.5B-parameter streaming TTS model that converts text into 24kHz speech with a chosen voice preset. Output streams chunk-by-chunk as it is synthesized, making it suitable for real-time read-aloud, podcast narration, and web playback.
• VibeVoice-ASR — A larger Qwen2.5-7B-based ASR model that transcribes long-form audio (up to ~61 minutes per request) to time-stamped, speaker-diarized text. It accepts MP3, WAV, FLAC, OGG, and 20+ other formats via FFmpeg.

Key features

• Streaming TTS with WebSocket + browser playback at 24kHz, sub-second time-to-first-byte on CUDA + flash-attention-2.
• Multi-language voice presets shipped in the repository: English, German, French, Italian, Japanese, Korean, Polish, Portuguese, Spanish, Dutch, plus Indian English (in-Samuel_man).
• Long-form ASR with timestamped JSON output ({"Start": ..., "End": ..., "Speaker": ..., "Content": ...}), speaker diarization, and optional context hotwords for proper-noun accuracy.
• Hugging Face Transformers integration: every model loads with from_pretrained() and runs on CUDA, MPS (Apple Silicon), Intel XPU, or CPU with automatic attention-implementation selection.
• A vLLM general plugin (vibevoice.vllm_plugin) for high-throughput ASR serving.
• LoRA fine-tuning recipe for adapting the ASR model to new domains, hotword sets, or low-resource languages.

Prerequisites

Python 3.10+, FFmpeg in PATH, PyTorch with the appropriate backend (CUDA 12.x, MPS, XPU, or CPU), and 16+ GB GPU memory recommended for the 7B ASR model.

Hello-world: TTS from a text file

python demo/realtime_model_inference_from_file.py \
  --model_path microsoft/VibeVoice-Realtime-0.5B \
  --txt_path demo/text_examples/1p_vibevoice.txt \
  --speaker_name Carter \
  --output_dir ./outputs

[startup] Found 23 voice presets
Using device: cuda, torch_dtype: torch.bfloat16, attn_implementation: flash_attention_2
Using voice preset for Carter: .../voices/streaming_model/en-Carter_man.pt
Generation time: 12.37 seconds
Generated audio duration: 38.42 seconds
RTF (Real Time Factor): 0.32x
Saved output to ./outputs/1p_vibevoice_generated.wav

Tip

Real Time Factor (RTF) below 1.0 means generation is faster than playback. On a CUDA GPU with flash-attention-2 you should consistently see RTF in the 0.2 – 0.5 range, leaving headroom for streaming use.

Hello-world: ASR from an audio file

python demo/vibevoice_asr_inference_from_file.py \
  --model_path microsoft/VibeVoice-ASR \
  --audio_files demo/asr_demo/demo1-chat.mp3 \
  --batch_size 1

The script prints per-segment results in the form:

[0.00 - 5.34] Speaker 0: Hey everyone, welcome back to the show...
[5.34 - 9.81] Speaker 1: Thanks for having me, great to be here.

What to try next

• Launch the Gradio ASR demo (demo/vibevoice_asr_gradio_demo.py) for a browser UI with audio upload, microphone recording, and per-segment playback.
• Launch the streaming TTS web app (demo/vibevoice_realtime_demo.py) for live WebSocket TTS playback in any browser.
• Add hotwords with --context_info "Hotwords: Acme Corp, Dr. Tan" to improve proper-noun transcription accuracy.
• Fine-tune the ASR model on your own data with finetuning-asr/lora_finetune.py.

2. Installation

System requirements

Important

FFmpeg must be on PATH before launching any inference script. The audio loader shells out to ffmpeg/ffprobe for every input file. On Windows install via winget install Gyan.FFmpeg or place ffmpeg.exe next to your Python interpreter.

Install from source

The repository is a regular Python package with pyproject.toml. Clone, create a virtual environment, and install editable.

git clone https://github.com/microsoft/VibeVoice.git
cd VibeVoice

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

pip install --upgrade pip
pip install -e .

Add Hugging Face Transformers, PyTorch, PEFT (for LoRA), and audio dependencies. The exact PyTorch wheel depends on your platform — pick one of the rows below.

Then add the rest of the runtime stack:

pip install transformers peft accelerate diffusers
pip install soundfile pydub numpy
pip install fastapi uvicorn[standard] websockets
pip install gradio
pip install vllm                          # optional, for the vLLM plugin
pip install flash-attn --no-build-isolation   # optional, recommended on CUDA
pip install liger-kernel                  # optional, accelerates Qwen2 layers in the ASR Gradio demo

Tip

flash-attn requires nvcc and a matching PyTorch version. If installation fails, omit it — the demos automatically fall back to PyTorch SDPA. SDPA quality is "lower than flash-attention-2" per the demo banners but is functionally complete.

Verify the install

python -c "from vibevoice import VibeVoiceStreamingForConditionalGenerationInference; print('ok')"

ok

For ASR:

python -c "from vibevoice.modular.modeling_vibevoice_asr import VibeVoiceASRForConditionalGeneration; print('ok')"

ok

Download voice presets

The 23 voice presets that ship in the repo (demo/voices/streaming_model/*.pt) are committed as Git LFS objects:

# After cloning, pull the .pt files
git lfs install
git lfs pull

For experimental voice packs (~9 language packs distributed via GitHub release attachments), run the downloader:

bash demo/download_experimental_voices.sh

[INFO] Starting download of experimental voices...
[INFO] Target directory: .../demo/voices/streaming_model/experimental_voices
[INFO] Downloading experimental_voices_de.tar.gz ...
...
[SUCCESS] All experimental speakers installed successfully!

Install troubleshooting

Warning

Several install paths fail silently and only surface as cryptic runtime errors. The entries below are pulled from real community reports.

3. Configuration

VibeVoice is configured almost entirely through CLI flags and environment variables. Model architecture lives in JSON files distributed inside each Hugging Face checkpoint; users never edit those.

Environment variables

Note

MODEL_PATH and MODEL_DEVICE are read by demo/web/app.py at FastAPI startup. They are also set programmatically by demo/vibevoice_realtime_demo.py, so for normal use you launch through that wrapper and pass --model_path / --device instead of exporting variables manually.

Common option combinations

Tip

CFG scale (Classifier-Free Guidance) is the single biggest dial for TTS expressiveness. The default 1.5 is balanced; 1.0 is more literal/calm; 2.0+ exaggerates emphasis but can introduce artefacts. Tune one notch at a time.

For full configuration mastery — every CLI flag, every JSON config knob, every internal env var — see the Admin Guide.

4. Core Usage

VibeVoice ships eight primary user-facing entry points. Use the index below to jump to the one you need.

4.1 Generate speech from a script file

python demo/realtime_model_inference_from_file.py \
  --model_path microsoft/VibeVoice-Realtime-0.5B \
  --txt_path demo/text_examples/1p_vibevoice.txt \
  --speaker_name Carter \
  --output_dir ./outputs \
  --device cuda \
  --cfg_scale 1.5

The script writes one .wav per input file: <txt_filename>_generated.wav at 24 kHz mono. It also prints token statistics, audio duration, and Real Time Factor on completion.

Note

The voice mapper does substring matching: passing --speaker_name Carter matches en-Carter_man. If two presets share a substring (e.g. passing Spk0), the script raises with the message "Multiple voice presets match the speaker name" — pass a more specific name.

4.2 Stream TTS to a browser

demo/vibevoice_realtime_demo.py boots a FastAPI app (demo/web/app.py) that serves three endpoints — an HTML player at /, a JSON config endpoint at /config, and a WebSocket streamer at /stream.

python demo/vibevoice_realtime_demo.py \
  --port 3000 \
  --model_path microsoft/VibeVoice-Realtime-0.5B \
  --device cuda

Then visit http://localhost:3000 in any modern browser.

Direct WebSocket protocol (for embedding in your own app):

ws://host:3000/stream?text=<URL-encoded>&cfg=<float>&steps=<int>&voice=<preset_key>

The server responds with binary frames of 16-bit signed little-endian PCM at 24 kHz mono, interleaved with JSON log frames carrying events like backend_request_received, model_progress (with generated_sec, chunk_sec), backend_first_chunk_sent, and backend_stream_complete. A second client connecting while a generation is in progress receives a backend_busy JSON frame and the WebSocket is closed with code 1013 (Service busy).

Important

Only one TTS request runs concurrently per server instance — the FastAPI app holds an asyncio.Lock for the lifetime of each generation. Run multiple uvicorn workers (and load-balance) if you need parallelism.

4.3 Transcribe audio files (CLI)

python demo/vibevoice_asr_inference_from_file.py \
  --model_path microsoft/VibeVoice-ASR \
  --audio_files podcast.mp3 interview.wav \
  --max_new_tokens 32768 \
  --batch_size 2

Output is printed per file — both raw text and JSON-parsed segments with timestamps and speaker IDs.

4.4 Run the Gradio ASR browser app

python demo/vibevoice_asr_gradio_demo.py \
  --model_path microsoft/VibeVoice-ASR \
  --host 0.0.0.0 \
  --port 7860

The UI provides:

• Audio upload + microphone recording.
• Optional Context Info textbox for hotwords ("John Smith", "Dr. Tan", "Acme Corp") that bias transcription.
• Sampling toggle with Temperature, Top-p, and Repetition Penalty sliders.
• Per-segment playback chips (each segment is clipped, optionally re-encoded as 32 kbps MP3 for compactness, and embedded inline as a data URI).
• A red Stop button that interrupts generation mid-stream via a custom StoppingCriteria.

Tip

The Gradio demo applies Liger Kernel to Qwen2 (RoPE + RMSNorm + SwiGLU) when available — install liger-kernel for a meaningful speedup. If absent, the demo prints a warning and continues.

4.5 Use the Python API directly

Streaming TTS (synchronous chunks)

import torch, copy
from vibevoice.modular.modeling_vibevoice_streaming_inference import (
    VibeVoiceStreamingForConditionalGenerationInference,
)
from vibevoice.processor.vibevoice_streaming_processor import VibeVoiceStreamingProcessor

processor = VibeVoiceStreamingProcessor.from_pretrained("microsoft/VibeVoice-Realtime-0.5B")
model = VibeVoiceStreamingForConditionalGenerationInference.from_pretrained(
    "microsoft/VibeVoice-Realtime-0.5B",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    attn_implementation="flash_attention_2",
).eval()
model.set_ddpm_inference_steps(num_steps=5)

prefilled = torch.load("demo/voices/streaming_model/en-Carter_man.pt",
                      map_location="cuda", weights_only=False)
inputs = processor.process_input_with_cached_prompt(
    text="Welcome to VibeVoice, a streaming TTS engine.",
    cached_prompt=prefilled,
    padding=True, return_tensors="pt", return_attention_mask=True,
)
inputs = {k: v.to("cuda") if torch.is_tensor(v) else v for k, v in inputs.items()}

outputs = model.generate(
    **inputs,
    max_new_tokens=None,
    cfg_scale=1.5,
    tokenizer=processor.tokenizer,
    generation_config={"do_sample": False},
    verbose=True,
    all_prefilled_outputs=copy.deepcopy(prefilled),
)
processor.save_audio(outputs.speech_outputs[0], output_path="hello.wav")

One-shot ASR

import torch
from vibevoice.modular.modeling_vibevoice_asr import VibeVoiceASRForConditionalGeneration
from vibevoice.processor.vibevoice_asr_processor import VibeVoiceASRProcessor

processor = VibeVoiceASRProcessor.from_pretrained(
    "microsoft/VibeVoice-ASR",
    language_model_pretrained_name="Qwen/Qwen2.5-7B",
)
model = VibeVoiceASRForConditionalGeneration.from_pretrained(
    "microsoft/VibeVoice-ASR",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    trust_remote_code=True,
).to("cuda").eval()

inputs = processor(audio="podcast.mp3", return_tensors="pt", add_generation_prompt=True)
inputs = {k: v.to("cuda") if torch.is_tensor(v) else v for k, v in inputs.items()}

with torch.no_grad():
    out = model.generate(
        **inputs, max_new_tokens=32768, do_sample=False, num_beams=1,
        pad_token_id=processor.pad_id,
        eos_token_id=processor.tokenizer.eos_token_id,
    )

text = processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
segments = processor.post_process_transcription(text)
for seg in segments:
    print(f"[{seg['Start']:.2f} - {seg['End']:.2f}] Speaker {seg.get('Speaker', '?')}: {seg['Content']}")

4.6 LoRA fine-tune the ASR model

python finetuning-asr/lora_finetune.py \
  --model_path microsoft/VibeVoice-ASR \
  --data_dir ./my_dataset \
  --output_dir ./output \
  --num_train_epochs 3 \
  --per_device_train_batch_size 1 \
  --gradient_accumulation_steps 8 \
  --learning_rate 1e-4 \
  --bf16 true \
  --logging_steps 10 \
  --save_steps 100 \
  --lora_r 16 --lora_alpha 32 --lora_dropout 0.05

The script uses Hugging Face Trainer with right-padded batches via VibeVoiceASRDataCollator. Audio + JSON labels live side-by-side in data_dir:

my_dataset/
  0.mp3
  0.json
  1.flac
  1.json
  ...

JSON schema:

{
  "audio_path": "0.mp3",
  "audio_duration": 351.73,
  "segments": [
    {"speaker": 0, "text": "Hey everyone...", "start": 0.0, "end": 38.68},
    {"speaker": 1, "text": "Thanks for having me.", "start": 38.68, "end": 42.10}
  ],
  "customized_context": ["Tea Brew", "Aiden Host"]
}

LoRA adapters target the Qwen2 attention/MLP projections (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj); the acoustic and semantic tokenizers are frozen. After training, run inference via inference_lora.py:

python finetuning-asr/inference_lora.py \
  --base_model microsoft/VibeVoice-ASR \
  --lora_path ./output \
  --audio_file finetuning-asr/toy_dataset/0.mp3 \
  --context_info "Hotwords: Tea Brew, Aiden Host"

Warning

The default lora_finetune.py configuration is tuned for the toy dataset (finetuning-asr/toy_dataset/). For production runs you must override learning_rate, num_train_epochs, per_device_train_batch_size, gradient_accumulation_steps, warmup_ratio, and weight_decay via Hugging Face TrainingArguments. The script accepts every standard TrainingArguments flag.

5. Common Tasks

Task 1 — Generate a single-speaker narration WAV

Goal: Turn a script of fewer than 64K tokens into a 24kHz WAV in a chosen voice.

Steps:

1. Place the script as a UTF-8 .txt file.
2. Pick a voice from demo/voices/streaming_model/ — for English, Carter, Davis, Emma, Frank, Grace, Mike.
3. Run python demo/realtime_model_inference_from_file.py --txt_path script.txt --speaker_name Emma --output_dir ./out.
4. Listen to ./out/script_generated.wav.

If it fails: check VRAM (the script halves it for bfloat16 weights + KV cache) and re-run with a shorter script.

Task 2 — Build a podcast transcript with speaker labels

python demo/vibevoice_asr_inference_from_file.py \
  --model_path microsoft/VibeVoice-ASR \
  --audio_files podcast.mp3 \
  --max_new_tokens 32768 \
  --num_beams 1 \
  --temperature 0.0

Pipe the JSON segments to a downstream tool (subtitle generator, summarizer):

import json, subprocess
text = subprocess.check_output(["python", "demo/vibevoice_asr_inference_from_file.py",
                                "--model_path", "microsoft/VibeVoice-ASR",
                                "--audio_files", "podcast.mp3"], text=True)
# Pull the JSON block out of stdout (the script logs it on completion).

For programmatic use prefer the Python API in section 4.5 — it returns parsed segments directly.

Task 3 — Improve transcription accuracy with hotwords

Industry jargon and proper nouns are the most common ASR failure mode. Pass a --context_info block with one term per line:

python demo/vibevoice_asr_gradio_demo.py --model_path microsoft/VibeVoice-ASR
# Then in the UI's "Context Info" textbox:
# Acme Corp
# Dr. Tan
# Continuous integration
# kubectl

The CLI demos accept the same idea via the model API but the Gradio UI is the easiest place to iterate.

Task 4 — Use a non-default voice without renaming files

--speaker_name is a substring match on the preset's stem. To target kr-Spk1_man either pass --speaker_name kr-Spk1_man or any unambiguous substring like kr-Spk1. To force a default in the streaming server, set VOICE_PRESET=de-Spk0_man before launching demo/vibevoice_realtime_demo.py.

Task 5 — Run on Apple Silicon

python demo/realtime_model_inference_from_file.py \
  --model_path microsoft/VibeVoice-Realtime-0.5B \
  --txt_path demo/text_examples/1p_vibevoice.txt \
  --speaker_name Emma \
  --device mps

The script automatically forces torch.float32 on MPS (MPS does not support bfloat16 reliably) and uses SDPA attention. Performance is significantly slower than CUDA + flash-attention-2; expect RTF > 1.0.

Warning

If you pass --device mpx the scripts treat it as mps. This is a deliberate compatibility shim for documentation typos — there is no mpx device. If MPS is unavailable, the scripts auto-fall-back to CPU and print a warning.

Task 6 — Process a HuggingFace dataset

python demo/vibevoice_asr_inference_from_file.py \
  --model_path microsoft/VibeVoice-ASR \
  --dataset openslr/librispeech_asr \
  --split test.clean \
  --max_duration 600 \
  --batch_size 2

The script streams the dataset (no full download), concatenates samples up to max_duration per inference call, and yields one transcription block per concatenated audio.

Task 7 — Embed streaming TTS in a Node.js or Python client

Open a WebSocket to /stream on the running TTS server and feed PCM bytes to a Web Audio context (browser) or a sounddevice.RawOutputStream(samplerate=24000, dtype="int16", channels=1) (Python). JSON log frames arrive on the same socket — distinguish them by checking whether the WebSocket message is text or binary.

import asyncio, websockets, sounddevice as sd

async def play(text):
    uri = f"ws://localhost:3000/stream?text={text}&cfg=1.5&voice=en-Carter_man"
    async with websockets.connect(uri) as ws:
        with sd.RawOutputStream(samplerate=24000, dtype="int16", channels=1) as out:
            async for msg in ws:
                if isinstance(msg, bytes):
                    out.write(msg)
                # else: msg is a JSON log frame, ignore or parse
asyncio.run(play("Hello from VibeVoice"))

Task 8 — Increase concurrent ASR throughput with vLLM

The vllm_plugin/ package registers VibeVoice with vLLM through a general_plugins entry point. Launch a vLLM server pointing at microsoft/VibeVoice-ASR and your existing client code (curl, OpenAI SDK, etc.) gets multimodal audio input automatically. The plugin honors VIBEVOICE_MAX_AUDIO_DURATION and VIBEVOICE_FFMPEG_MAX_CONCURRENCY to keep memory and FFmpeg fork pressure under control. See the Admin Guide → Operations section for cluster sizing.

Task 9 — Quickly test the toy dataset

python finetuning-asr/inference_lora.py \
  --base_model microsoft/VibeVoice-ASR \
  --lora_path microsoft/VibeVoice-ASR \
  --audio_file finetuning-asr/toy_dataset/0.mp3

This requires no real LoRA weights — passing the base model as --lora_path falls through PEFT's loader for a no-op adapter.

Task 10 — Inspect or replace the noise scheduler

The streaming TTS server reconfigures its scheduler at startup to use sde-dpmsolver++ with squaredcos_cap_v2. To reproduce in your own code:

model.model.noise_scheduler = model.model.noise_scheduler.from_config(
    model.model.noise_scheduler.config,
    algorithm_type="sde-dpmsolver++",
    beta_schedule="squaredcos_cap_v2",
)
model.set_ddpm_inference_steps(num_steps=5)   # 5 is the default; raise to 10–20 for higher fidelity at the cost of latency

Task 11 — Stop a long generation gracefully

The Gradio ASR demo wires a StopOnFlag() StoppingCriteria to a global flag toggled by the red Stop button. When you embed VibeVoice in your own app, install the same pattern:

import threading
from transformers import StoppingCriteria, StoppingCriteriaList

stop = threading.Event()
class StopOnFlag(StoppingCriteria):
    def __call__(self, input_ids, scores, **kwargs): return stop.is_set()

model.generate(..., stopping_criteria=StoppingCriteriaList([StopOnFlag()]))
# From another thread: stop.set()

The streaming TTS path accepts a similar pattern via stop_check_fn=stop_event.is_set on model.generate(...).

Task 12 — Process audio segments larger than a single batch

Use transcribe_with_batching from VibeVoiceASRBatchInference (in demo/vibevoice_asr_inference_from_file.py) — it splits a list of files into batches of size --batch_size, generates per-batch, and concatenates results. For audio individually longer than the model's context window, pre-segment with FFmpeg first (e.g. ffmpeg -i in.mp3 -f segment -segment_time 1800 out_%03d.mp3).

6. Troubleshooting

Decision tree

1. Does python -c "from vibevoice import *" import cleanly? — No → install issue (section 2).
2. Does ffmpeg -version print? — No → install FFmpeg.
3. Does nvidia-smi show a GPU and free VRAM ≥ 16GB for ASR / 8GB for TTS? — No → switch --device cpu (slow) or --device mps.
4. Did the script print "Found N voice presets"? — No → run git lfs pull from repo root.
5. Did the model load with flash_attention_2? — No → either pip install flash-attn or accept the SDPA fallback.
6. Did the WebSocket close with code 1013? — server is busy with another generation; wait or run more workers.

Error: RuntimeError: MODEL_PATH not set in environment

Cause: demo/web/app.py reads MODEL_PATH at FastAPI startup, but the user launched uvicorn web.app:app directly without exporting the variable.

Solution:

1. Use the wrapper: python demo/vibevoice_realtime_demo.py --model_path microsoft/VibeVoice-Realtime-0.5B.
2. Or export the variables manually: MODEL_PATH=microsoft/VibeVoice-Realtime-0.5B MODEL_DEVICE=cuda uvicorn web.app:app.

Prevention: Always launch via vibevoice_realtime_demo.py; it sets both env vars before importing the app.

Error: RuntimeError: Voices directory not found

Cause: demo/web/app.py resolves voices from BASE.parent / "voices" / "streaming_model", where BASE = demo/web/. If the repo was cloned without Git LFS the .pt files are missing.

Solution:

1. cd VibeVoice && git lfs install && git lfs pull.
2. Verify with ls demo/voices/streaming_model/*.pt.

Prevention: Add git lfs pull to your bootstrap script.

Error: RuntimeError: No voice preset (.pt) files found in <dir>

Cause: The voices directory exists but is empty (LFS pointer files only).

Solution: Same as above — git lfs pull.

Error: RuntimeError: Voice preset 'foo' not found

Cause: _ensure_voice_cached was passed a voice query parameter (or --speaker_name) that does not match any preset stem.

Solution: Hit GET /config to list available voice keys, or ls demo/voices/streaming_model/. Use the exact stem (e.g. en-Carter_man).

Error: RuntimeError: StreamingTTSService not initialized

Cause: _prepare_inputs was called before service.load() finished. Usually means the FastAPI app received a request during startup.

Solution: Wait for the [startup] Model ready. log line before sending traffic.

Error: ValueError: Multiple voice presets match the speaker name 'X'

Cause: The VoiceMapper.get_voice_path() substring match found two presets containing X.

Solution: Use a more specific name. For example, Spk0 matches every language pack; use de-Spk0_man instead.

Error: ValueError: Audio duration (Ns) exceeds the configured limit (3660s)

Cause: The vLLM plugin enforces a hard duration cap to prevent OOM (issue #210).

Solution: Either pre-split the audio (ffmpeg -i input.mp3 -f segment -segment_time 1800 part_%03d.mp3) or raise the limit: export VIBEVOICE_MAX_AUDIO_DURATION=7200.

Prevention: Profile peak memory at your target duration before raising the limit.

Error: Token indices sequence length is longer than the specified maximum sequence length for this model

Cause: The TTS script you supplied tokenizes to more than the decoder context (issue #332). The 1.5B-64K config supports 65,536 tokens; the 7B-32K supports 32,768.

Solution: Split the script into chunks and call the model per chunk, then concatenate WAVs. The streaming demo already streams; for the file-based demo, slice the input.

Error: ImportError: soundfile is required to save audio files

Cause: processor.save_audio() in vibevoice_tokenizer_processor.py falls back to soundfile and the package is not installed.

Solution: pip install soundfile.

Error: Error loading the model. Trying to use SDPA. However, note that only flash_attention_2 has been fully tested...

Cause: flash-attn failed at import time. Common on Windows or older PyTorch wheels.

Solution: This is a warning, not a hard failure. The model still loads with SDPA. Audio quality is "lower" per the project authors but typically still acceptable. To regain flash-attn install it from source: pip install flash-attn --no-build-isolation.

Error: Connection closed with code 1013 (Service busy)

Cause: Another client already holds the server's TTS lock (asyncio.Lock).

Solution: Wait for the in-flight stream to complete, or run a uvicorn worker pool fronted by a load balancer.

Error: Gradio ASR demo crashes when you connect over plain HTTP through Docker (#339)

Cause: Gradio rewrites internal /file= URLs based on the request scheme. Behind some HTTP-only proxies the rewrite fails.

Solution: Front the demo with HTTPS, or expose the Gradio port directly without rewriting.

Error: Constant crash on Windows 11 (#327)

Cause: Mixed CUDA driver / VC++ runtime / PyTorch wheel ABIs.

Solution: Re-install the latest NVIDIA driver, latest VC++ Redistributable, and a PyTorch wheel matching your CUDA version (pip install torch --index-url https://download.pytorch.org/whl/cu121).

Error: ValueError: Unsupported audio data type: <class '...'>

Cause: The vLLM plugin's audio mapper accepts str (path), bytes, np.ndarray, or List[str]. Anything else raises.

Solution: Convert your input to one of the four supported types before calling.

Error: ValueError: Unsupported file format: .X. Supported formats: .wav, .mp3, .flac, .m4a, .ogg, .pt, .npy, .npz

Cause: vibevoice_tokenizer_processor.py rejects unknown extensions when loading audio.

Solution: Convert with FFmpeg: ffmpeg -i input.amr -ar 24000 input.wav.

7. FAQ

Q: What languages does VibeVoice support?

The shipped voice presets cover German, English, Spanish, French, Italian, Indian English, Japanese, Korean, Dutch, Polish, and Portuguese. ASR can transcribe any language Qwen2.5-7B understands, but accuracy varies by language. Arabic (#108, #253), Vietnamese (#283), and Traditional Chinese (#271) are open community feature requests at the time of this writing.

Q: Is VibeVoice open source?

The code is MIT-licensed (see LICENSE). Pretrained weights are distributed via Hugging Face under their own terms. See LICENSE and the project's SECURITY.md for the full picture.

Q: Can I run VibeVoice without a GPU?

Yes — pass --device cpu. Expect Real Time Factor far above 1.0 (i.e. generation is slower than playback). The 0.5B streaming model is more usable on CPU than the 7B ASR model.

Q: How do I get speaker diarization in ASR output?

It is automatic. The ASR model emits JSON segments with Start, End, Speaker (integer ID), and Content. Call processor.post_process_transcription(text) to get the parsed list.

Q: How do I bias transcription toward a known vocabulary?

Pass context_info (CLI: --context_info) with one hotword per line. The processor injects it into the system prompt; the LLM treats it as a hint. The Gradio demo exposes a textbox; the LoRA finetune script reads customized_context from each JSON label.

Q: What sample rate does VibeVoice use?

24 kHz mono throughout. All inputs are resampled to 24 kHz at load time and outputs are produced at 24 kHz. The streaming server emits 16-bit PCM at 24 kHz.

Q: How do I add a new voice preset?

Run processor.encode_voice_to_pt(audio_path, output_path) (or follow the encoding scripts under vllm_plugin/tools/). Drop the resulting .pt file into demo/voices/streaming_model/ and it will be discovered at server startup.

Q: Why does streaming TTS sometimes hang for several seconds before playing?

The first chunk requires the model's prefill pass to complete. Subsequent chunks stream every few hundred milliseconds. Check the WebSocket log frames: backend_request_received to model_progress (first instance) is your time-to-first-byte.

Q: Why does --cfg_scale 1.0 sound flat?

CFG below 1.5 reduces guidance strength, biasing toward the unconditional voice prior. 1.5 is the project default; raise to 2.0 for more emphatic delivery, lower for a calmer read.

Q: Can I use VibeVoice commercially?

The code is MIT. The pretrained weight licenses live on the Hugging Face model pages — read them carefully before commercial deployment. As of this writing the project explicitly publishes microsoft/VibeVoice-Realtime-0.5B and microsoft/VibeVoice-ASR; check the model cards for use restrictions.

Q: Is there a JavaScript or REST API?

Yes — the streaming server speaks WebSocket (/stream) and HTTP GET (/, /config). For ASR, the vLLM plugin adds OpenAI-compatible REST endpoints (/v1/chat/completions with multimodal audio input). Issue #336 tracks deeper OpenAI API parity. There is no first-party REST endpoint for ASR shipped in this repo — use the vLLM plugin.

Q: How much GPU memory do I need?

The 0.5B streaming TTS model needs ~4-6 GB at bfloat16 + cache. The 7B ASR model needs ~16 GB at bfloat16 for a single-batch 1-hour audio run. Multi-batch inference scales linearly.

Q: Where do generated files end up?

The file-based TTS demo writes to --output_dir (default ./outputs). The Gradio ASR demo embeds segment audio inline as data URIs (no disk writes per segment). Issue #328 tracks user confusion about file locations.

Q: Does VibeVoice support voice cloning?

The streaming TTS path uses voice presets that are pre-encoded .pt files. The repo ships 23 such presets, plus optional experimental packs. Encoding new voices requires running the audio through the tokenizer pipeline; see the Admin Guide → Developer Internals for the workflow.

Q: Can I quantize the model?

Issue #297 requests GGUF Q-quantization. As shipped the codebase exercises bfloat16 (CUDA), float32 (MPS/CPU), and the underlying Qwen2 backbone supports the standard bitsandbytes 4-bit/8-bit modes via transformers.BitsAndBytesConfig. Quantization is community-driven, not first-party.

Q: Why does the Gradio demo hide the sampling sliders by default?

do_sample=False (greedy) is the recommended default for ASR. Sampling sliders only appear when Enable Sampling is checked, to avoid configuration drift in production.

Q: How do I report a security issue?

SECURITY.md lays out the disclosure path. Issue #341 noted the original guidance was unclear; check the latest copy on main.