# Whisper Administrator & Developer Guide This guide is the source-of-truth reference for operators, ML platform engineers, and contributors. It covers Whisper's architecture, every configurable parameter, deployment patterns, performance characteristics, security/privacy implications, and the internals of the encoder/decoder, the language ID head, the temperature-fallback decoder, and the word-alignment pass. For end-user CLI recipes and the Python API tour, see the **Whisper User Guide**. This document complements that one — they should be read together. ## 1. Architecture Overview Whisper is a **single end-to-end Transformer** trained for multilingual, multitask audio-to-text transduction. The same model handles speech recognition (X→X), speech translation (X→English), language identification, and timestamp prediction — all controlled by special tokens at the start of the decoder sequence. The Python package is intentionally thin: it owns the model weights, the tokenizer, an FFmpeg-based audio loader, a sliding-window decoder loop with temperature fallback, and writers for five output formats. There is no server, no daemon, no queue, no GPU pool — the package is meant to be embedded in your application or invoked as a CLI. ### 1.1 Data flow ``` audio file (mp3 / wav / flac / mp4 / …) │ ┌────────────┴───────────────┐ │ ffmpeg subprocess │ load_audio() │ → 16 kHz mono float32 │ whisper/audio.py └────────────┬───────────────┘ │ ┌────────────┴───────────────┐ │ STFT + Mel filterbank │ log_mel_spectrogram() │ → (n_mels, n_frames) │ 80 or 128 mels │ CHUNK_LENGTH = 30 s │ N_FRAMES = 3000 └────────────┬───────────────┘ │ ┌────────────┴───────────────┐ │ AudioEncoder │ whisper/model.py │ Conv1D ×2 → Transformer │ sinusoidal pos. embed │ → (n_audio_ctx, n_state) │ └────────────┬───────────────┘ │ ┌────────────┴───────────────┐ ┌───────────────────┐ │ language detection (opt.) │ ←─ │ detect_language() │ │ argmax over lang tokens │ │ whisper/decoding │ └────────────┬───────────────┘ └───────────────────┘ │ ┌────────────┴───────────────┐ │ TextDecoder (autoregress) │ whisper/model.py │ cross-attn over audio │ kv-cache enabled │ + LogitFilter rules │ timestamp / suppress │ + Greedy/BeamSearch │ temperature fallback │ → tokens + logprobs │ └────────────┬───────────────┘ │ ┌────────────┴───────────────┐ │ optional: word DTW pass │ add_word_timestamps() │ cross-attn alignment │ whisper/timing.py │ → per-word start/end │ triton_ops on CUDA └────────────┬───────────────┘ │ ┌────────────┴───────────────┐ │ ResultWriter (per format) │ whisper/utils.py │ txt / srt / vtt / tsv / │ │ json │ └────────────┬───────────────┘ │ ▼ transcript files ``` > [!NOTE] > Whisper processes audio in **30-second non-overlapping windows**. Anything > longer is sliced internally; anything shorter is zero-padded. The > `condition_on_previous_text` flag controls whether each window's prompt > includes the previous window's output, which is what gives long-form > transcription continuity at the cost of occasional looped degeneration. ### 1.2 Component inventory | Component | Path | Role | |-----------|------|------| | Package init / model registry | `whisper/__init__.py` | URL table, SHA-256 verification, `load_model()`, `available_models()`. | | CLI entrypoint | `whisper/__main__.py` | One-liner: `from .transcribe import cli; cli()`. | | Audio I/O | `whisper/audio.py` | `load_audio` (FFmpeg shell-out), `pad_or_trim`, `log_mel_spectrogram`. | | Decoder & inference | `whisper/decoding.py` | `DecodingOptions`, `DecodingResult`, `DecodingTask`, beam/greedy decoders, logit filters. | | Transformer model | `whisper/model.py` | `Whisper`, `AudioEncoder`, `TextDecoder`, attention, `ModelDimensions`. | | Word timestamps | `whisper/timing.py` | DTW alignment, median filter, `find_alignment`, `add_word_timestamps`. | | Tokenizer | `whisper/tokenizer.py` | tiktoken wrapper, special tokens, language list, word-token splitter. | | Long-form pipeline | `whisper/transcribe.py` | `transcribe()` (sliding window + fallback), `cli()` (argparse). | | Triton GPU kernels | `whisper/triton_ops.py` | DTW kernel + median filter for CUDA acceleration. | | Output writers | `whisper/utils.py` | `WriteTXT/WriteSRT/WriteVTT/WriteTSV/WriteJSON`, `format_timestamp`, helpers. | | Text normalizers | `whisper/normalizers/` | English number/spelling normalizer + basic Unicode normalizer (used for WER scoring). | | Version stamp | `whisper/version.py` | `__version__ = "20250625"`. | ### 1.3 Technology stack | Layer | Tech | |-------|------| | Modeling framework | PyTorch (≥ 1.13 recommended for `weights_only=True` torch.load) | | Tokenizer | `tiktoken` (BPE) | | Numerics | NumPy | | JIT compilation | Numba (CPU DTW), Triton (CUDA DTW + median filter) | | Progress bars | `tqdm` | | Audio decoding | `ffmpeg` CLI (subprocess shell-out) | | Iteration helpers | `more-itertools` | The dependency list is small by design (`requirements.txt`): ```text numba numpy torch tqdm more-itertools tiktoken triton>=2.0.0;platform_machine=="x86_64" and sys_platform=="linux" or sys_platform=="linux2" ``` ### 1.4 Model dimensions The `ModelDimensions` dataclass in `whisper/model.py` parameterises the Transformer. Each checkpoint stores its own dims, so `load_model` can build the correct architecture from the file: ```python @dataclass class ModelDimensions: n_mels: int n_audio_ctx: int n_audio_state: int n_audio_head: int n_audio_layer: int n_vocab: int n_text_ctx: int n_text_state: int n_text_head: int n_text_layer: int ``` The audio encoder consumes `(batch, n_mels, 3000)` log-mel frames and emits `(batch, n_audio_ctx, n_audio_state)` features (e.g. `1500 × 1280` for `large-v3`). The text decoder runs autoregressively over `n_text_ctx = 448` tokens with cross-attention into the encoder output. > [!NOTE] > `is_multilingual` is detected by `n_vocab >= 51865`, and `num_languages` is > derived from `n_vocab - 51765 - int(is_multilingual)`. This is how the > same code paths support `large-v3` (99 langs), `large-v2` (100 langs), and > the English-only `.en` variants in one library. ## 2. System Requirements ### 2.1 Software | Dependency | Minimum | Notes | |------------|---------|-------| | Python | 3.8 | Older versions not supported. | | PyTorch | ≥ 1.13 recommended | The loader sets `weights_only=True` if the installed `torch.__version__ >= "1.13"`. Earlier still works but without the safer load. | | FFmpeg | any modern | Required at runtime for audio decoding. | | OS | Linux, macOS, Windows | Triton acceleration is Linux/x86_64 only. | Python dependencies are installed transitively by `pip`: `numba`, `numpy`, `torch`, `tqdm`, `more-itertools`, `tiktoken`, `triton` (Linux/x86_64). ### 2.2 Hardware | Workload | CPU | GPU | RAM | |----------|-----|-----|-----| | `tiny` / `tiny.en` | any modern x86/ARM | optional | 2 GB | | `base` / `base.en` | any modern x86/ARM | optional | 2 GB | | `small` / `small.en` | 4+ cores recommended | helpful | 4 GB | | `medium` / `medium.en` | 8+ cores | strongly recommended | 8 GB | | `large-v1/v2/v3`, `large` | 16+ cores | required for usable speed | 16 GB | | `turbo` (`large-v3-turbo`) | 8+ cores | recommended | 12 GB | Per-model VRAM (peak, FP16, batch size 1): `tiny` ~1 GB, `base` ~1 GB, `small` ~2 GB, `medium` ~5 GB, `large-v*` ~10 GB, `turbo` ~6 GB. Add ~10–20% headroom for the operating system and the cudnn workspace. > [!IMPORTANT] > For production batch transcription with `large-v3`, plan for an A100-40GB > or H100 if you want headroom for batched decoding. A consumer 24 GB GPU > (RTX 3090/4090/5090) handles `large-v3` for single-stream FP16 jobs but > may OOM on long word-timestamp runs because the Triton DTW kernel needs > additional contiguous workspace. ### 2.3 Storage Model weights live in `~/.cache/whisper/` by default (override with `XDG_CACHE_HOME` or the `download_root` argument to `load_model`). Sizes: | Model | Approx size on disk | |-------|---------------------| | `tiny[.en]` | 39 MB / 75 MB | | `base[.en]` | 142 MB | | `small[.en]` | 461 MB | | `medium[.en]` | 1.42 GB | | `large-v1`, `large-v2`, `large-v3`, `large` | 2.87 GB each | | `large-v3-turbo`, `turbo` | 1.51 GB | The `assets/` directory ships ~6 MB of mel filter banks and tiktoken vocab files; these are bundled with the wheel and require no separate download. ## 3. Configuration Reference Whisper is configured exclusively through CLI arguments and Python kwargs. There are no config files, no XML/YAML, and only a single environment variable. ### 3.1 Environment variables | Variable | Read by | Default | Purpose | |----------|---------|---------|---------| | `XDG_CACHE_HOME` | `whisper/__init__.py:134` | `~/.cache` | Sets the parent of `whisper/` cache dir. The full default is `~/.cache/whisper/`. | There are no other environment variables. Whisper does not read `WHISPER_*`, `TRANSCRIBE_*`, or any custom env namespace. ### 3.2 CLI flags — full reference Every flag accepted by `whisper` (in source order, from `cli()` in `whisper/transcribe.py`): | Flag | Type | Default | Description | |------|------|---------|-------------| | `audio` | str (positional, `nargs="+"`) | required | One or more audio files to transcribe. | | `--model` | `valid_model_name` | `turbo` | Model name from `available_models()` or path to a `.pt` checkpoint. | | `--model_dir` | str | `None` | Directory to download/cache models in. Defaults to `~/.cache/whisper`. | | `--device` | str | `cuda` if available else `cpu` | PyTorch device selector. | | `--output_dir` / `-o` | str | `.` | Output directory for transcript files. | | `--output_format` / `-f` | str | `all` | One of `txt`, `vtt`, `srt`, `tsv`, `json`, `all`. | | `--verbose` | `str2bool` | `True` | Print decoded segments as they complete. | | `--task` | str | `transcribe` | `transcribe` or `translate` (X→English). | | `--language` | str | `None` | Language code or name; `None` triggers auto-detect. | | `--temperature` | float | `0` | Sampling temperature (greedy at 0). | | `--best_of` | `optional_int` | `5` | Sample candidates when `temperature > 0`. | | `--beam_size` | `optional_int` | `5` | Beam search width when `temperature == 0`. | | `--patience` | float | `None` | Beam-search patience (`arxiv:2204.05424`). | | `--length_penalty` | float | `None` | Length penalty alpha (must be in `[0, 1]`). | | `--suppress_tokens` | str | `-1` | Comma-separated token ids to suppress; `-1` uses `non_speech_tokens`. | | `--initial_prompt` | str | `None` | First-window decoder prompt (custom vocabulary, context). | | `--carry_initial_prompt` | `str2bool` | `False` | Re-prepend `initial_prompt` to every internal `decode()` call. | | `--condition_on_previous_text` | `str2bool` | `True` | Use prior window output as next window's prompt. | | `--fp16` | `str2bool` | `True` | Run inference in FP16 (auto-disabled on CPU). | | `--temperature_increment_on_fallback` | `optional_float` | `0.2` | Step added to `temperature` on each fallback. | | `--compression_ratio_threshold` | `optional_float` | `2.4` | Treat segment as failure when gzip ratio is above this. | | `--logprob_threshold` | `optional_float` | `-1.0` | Treat segment as failure when avg logprob is below this. | | `--no_speech_threshold` | `optional_float` | `0.6` | Treat as silence when no_speech prob > threshold AND avg logprob < `logprob_threshold`. | | `--word_timestamps` | `str2bool` | `False` | Run cross-attention DTW alignment, populate per-word timings. | | `--prepend_punctuations` | str | `"'“¿([{-` | Punctuation merged with the next word in word timing. | | `--append_punctuations` | str | `"'.。,，!！?？:：”)]}、` | Punctuation merged with the previous word in word timing. | | `--highlight_words` | `str2bool` | `False` | Underline currently-spoken word in SRT/VTT (requires `--word_timestamps True`). | | `--max_line_width` | `optional_int` | `None` | Max chars per subtitle line (requires `--word_timestamps True`). | | `--max_line_count` | `optional_int` | `None` | Max lines per subtitle block (requires `--word_timestamps True`). | | `--max_words_per_line` | `optional_int` | `None` | Max words per subtitle line (requires `--word_timestamps True`; no effect with `--max_line_width`). | | `--threads` | `optional_int` | `0` | `torch.set_num_threads()`. Supersedes `MKL_NUM_THREADS` / `OMP_NUM_THREADS`. | | `--clip_timestamps` | str | `0` | `start,end,start,end,...` seconds; only transcribe these clips. | | `--hallucination_silence_threshold` | `optional_float` | `None` | Skip silent periods longer than this (seconds) when a hallucination is suspected. Requires `--word_timestamps True`. | ### 3.3 `transcribe()` keyword arguments Programmatic equivalents to the CLI flags. Defaults differ for `temperature` and `logprob_threshold` between CLI and API: | Argument | Type | Default (API) | Mapping | |----------|------|---------------|---------| | `verbose` | `Optional[bool]` | `None` | CLI `--verbose` (default `True`). `None` suppresses all output. | | `temperature` | `Union[float, Tuple[float, ...]]` | `(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)` | API uses tuple for fallback; CLI builds the tuple from `--temperature` and `--temperature_increment_on_fallback`. | | `compression_ratio_threshold` | `Optional[float]` | `2.4` | `--compression_ratio_threshold`. | | `logprob_threshold` | `Optional[float]` | `-1.0` | `--logprob_threshold`. | | `no_speech_threshold` | `Optional[float]` | `0.6` | `--no_speech_threshold`. | | `condition_on_previous_text` | `bool` | `True` | `--condition_on_previous_text`. | | `initial_prompt` | `Optional[str]` | `None` | `--initial_prompt`. | | `carry_initial_prompt` | `bool` | `False` | `--carry_initial_prompt`. | | `word_timestamps` | `bool` | `False` | `--word_timestamps`. | | `prepend_punctuations` | `str` | `"'“¿([{-` | `--prepend_punctuations`. | | `append_punctuations` | `str` | `"'.。,，!！?？:：”)]}、` | `--append_punctuations`. | | `clip_timestamps` | `Union[str, List[float]]` | `"0"` | `--clip_timestamps`. | | `hallucination_silence_threshold` | `Optional[float]` | `None` | `--hallucination_silence_threshold`. | | `**decode_options` | dict | `{}` | Splat into `DecodingOptions` (e.g. `language`, `task`, `beam_size`, `best_of`, `patience`, `length_penalty`, `suppress_tokens`, `prompt`, `prefix`, `suppress_blank`, `without_timestamps`, `max_initial_timestamp`, `fp16`). | > [!IMPORTANT] > `temperature` defaults differ between CLI and API. The CLI passes a single > float through `--temperature` (default `0`) and lets > `--temperature_increment_on_fallback` build the fallback tuple. The Python > API expects a tuple directly (default > `(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)`). If you call `transcribe()` with > `temperature=0`, you disable the fallback ladder entirely — usually not > what you want for long-form audio. ### 3.4 `DecodingOptions` (single 30-s window) Every field of the dataclass in `whisper/decoding.py`: | Field | Type | Default | Description | |-------|------|---------|-------------| | `task` | str | `transcribe` | `transcribe` or `translate`. | | `language` | `Optional[str]` | `None` | Language code; `None` triggers detection. | | `temperature` | float | `0.0` | Single temperature for this decode call. | | `sample_len` | `Optional[int]` | `None` | Max tokens to sample (defaults to `n_text_ctx // 2`). | | `best_of` | `Optional[int]` | `None` | Independent samples when `temperature > 0`. | | `beam_size` | `Optional[int]` | `None` | Beams when `temperature == 0`. | | `patience` | `Optional[float]` | `None` | Beam patience. | | `length_penalty` | `Optional[float]` | `None` | Google-NMT length penalty alpha (`[0, 1]`). | | `prompt` | `Optional[Union[str, List[int]]]` | `None` | Text or token ids prepended to the decoder input. | | `prefix` | `Optional[Union[str, List[int]]]` | `None` | Forces the start of the output. | | `suppress_tokens` | `Optional[Union[str, Iterable[int]]]` | `"-1"` | `-1` uses `tokenizer.non_speech_tokens`. | | `suppress_blank` | `bool` | `True` | Suppress blank-only outputs. | | `without_timestamps` | `bool` | `False` | If `True`, sample text-only tokens (no `<\|0.00\|>` etc.). | | `max_initial_timestamp` | `Optional[float]` | `1.0` | Cap on the first timestamp the model can emit. | | `fp16` | `bool` | `True` | FP16 inference. | ### 3.5 Audio constants Hard-coded in `whisper/audio.py` (do **not** change without retraining): | Constant | Value | Meaning | |----------|-------|---------| | `SAMPLE_RATE` | `16000` | Hz. All inputs are resampled to this. | | `N_FFT` | `400` | FFT window. | | `HOP_LENGTH` | `160` | Frame hop. | | `CHUNK_LENGTH` | `30` | Seconds per encoder window. | | `N_SAMPLES` | `480000` | Samples per 30-second chunk. | | `N_FRAMES` | `3000` | Mel frames per chunk. | | `N_SAMPLES_PER_TOKEN` | `320` | Stride-2 conv ⇒ 20 ms per output token. | | `FRAMES_PER_SECOND` | `100` | 10 ms per audio frame. | | `TOKENS_PER_SECOND` | `50` | Reciprocal of token stride. | ## 4. Deployment & Scaling ### 4.1 Embedded library The simplest deployment: ship Whisper as a Python dependency of your application. Hold one model in memory per process and serve transcription requests via your own queue/router. ```python # server-startup.py import whisper MODEL = whisper.load_model("turbo", device="cuda") def handle_request(audio_path, language=None): return MODEL.transcribe(audio_path, language=language) ``` This is appropriate for moderate-traffic services where you can size one GPU per process. Whisper does **not** support shared-GPU multi-tenancy out of the box — you cannot load one model and have multiple Python processes hit it concurrently. ### 4.2 CLI in a Docker container A minimal Linux container: ```dockerfile FROM python:3.11-slim RUN apt-get update && apt-get install -y --no-install-recommends \ ffmpeg \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir -U openai-whisper WORKDIR /work ENTRYPOINT ["whisper"] ``` Build and use: ```bash docker build -t whisper:turbo . docker run --rm -v $(pwd):/work whisper:turbo audio.mp3 --model small.en ``` For GPU use, base on `nvidia/cuda:12.1.1-runtime-ubuntu22.04`, install `python3` + `pip`, install a CUDA-matched PyTorch wheel, then `pip install -U openai-whisper`. Run with `--gpus all`. ### 4.3 Pre-baked model cache Bake the model into the image to avoid the first-run download: ```dockerfile RUN python -c "import whisper; whisper.load_model('turbo')" # Layer above caches /root/.cache/whisper/large-v3-turbo.pt (1.5 GB) ``` For air-gapped deployments, build the image somewhere with internet, then ship the layer. > [!TIP] > In Kubernetes, mount `~/.cache/whisper` from a `PersistentVolume` shared > across pods (read-only) so each new pod doesn't re-download the > checkpoint. Whisper SHA-256-verifies on load — concurrent reads are > safe. ### 4.4 Scaling out Whisper is embarrassingly parallel at the file level. The recommended pattern: ``` input queue (S3, Redis, RabbitMQ, …) │ ▼ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │ Worker pod (GPU) │ │ Worker pod (GPU) │ │ Worker pod (GPU) │ │ load_model("…") │ │ load_model("…") │ │ load_model("…") │ │ for path in queue │ │ for path in queue │ │ for path in queue │ │ transcribe(...) │ │ transcribe(...) │ │ transcribe(...) │ └───────────────────┘ └───────────────────┘ └───────────────────┘ │ │ │ └───────────────────────┴───────────────────────┘ │ ▼ output store (S3, DB) ``` Each worker holds one model resident, pulls jobs from a queue, and writes results back. With `large-v3-turbo` you can typically run one worker per GPU at saturated utilisation; with `tiny`/`base` you may be able to run two or three workers per GPU because the model is small enough that multiple CUDA contexts fit. > [!NOTE] > There is no built-in **batched** inference path. If you have many short > clips, you'll get more throughput by collecting them into 30-second > super-clips with a sentinel token between them — but that requires > custom code on top of `decode()`. The stock `transcribe()` is > single-stream. ### 4.5 Health checks A useful liveness check is to transcribe `tests/jfk.flac` (a 12-second sample bundled with the source repo) and assert the result contains the word "country": ```python import whisper model = whisper.load_model("tiny.en") out = model.transcribe("tests/jfk.flac") assert "country" in out["text"].lower() ``` ## 5. Operations ### 5.1 Capacity planning Throughput is a function of model size, GPU FP16 throughput, and audio duration. Real-world numbers, single A100-40GB, FP16, batch=1, no word timestamps: | Model | Real-time factor (RTF) | |-------|------------------------| | `tiny` | 30–40x | | `base` | 20–30x | | `small` | 8–15x | | `medium` | 4–8x | | `large-v3` | 1.5–3x | | `turbo` (`large-v3-turbo`) | 6–12x | "RTF" here means seconds of audio processed per second of wall time. Word timestamps add ~10–25% overhead on GPU (the Triton DTW kernel) and ~50–100% on CPU. > [!TIP] > A useful heuristic for batch jobs: assume `RTF/2` to account for > first-window language detection, occasional fallback re-decodes at > higher temperature, and disk I/O. ### 5.2 Logging and monitoring The package emits very little — `tqdm` progress bars and the per-segment prints when `verbose=True`. To capture structured logs, wrap the call: ```python import time, json, logging log = logging.getLogger("whisper.adapter") def transcribe_logged(model, path, **kwargs): t0 = time.monotonic() try: result = model.transcribe(path, verbose=False, **kwargs) log.info(json.dumps({ "event": "transcribe.ok", "path": path, "language": result["language"], "n_segments": len(result["segments"]), "wall_seconds": round(time.monotonic() - t0, 3), })) return result except Exception as e: log.exception("transcribe.fail path=%s", path) raise ``` Useful per-segment metrics inside `result["segments"]`: - `avg_logprob` (lower than `logprob_threshold` ⇒ low confidence). - `compression_ratio` (above `compression_ratio_threshold` ⇒ likely loop). - `no_speech_prob` (above `no_speech_threshold` ⇒ silent / non-speech). - `temperature` (the temperature at which this segment finally succeeded; > 0 means the first attempt was rejected). ### 5.3 Cache management The on-disk cache layout: ``` ~/.cache/whisper/ tiny.pt base.en.pt small.pt … ``` Each `.pt` file is verified against the SHA-256 in its download URL on every load. If the hash mismatches, Whisper warns and re-downloads. Manual cleanup: just `rm` the unwanted `.pt` files; Whisper recreates the cache directory on next run. > [!NOTE] > The cache directory is shared across all users on the host who use the > same `XDG_CACHE_HOME`. In multi-tenant environments, set `XDG_CACHE_HOME` > per user or run with `download_root=...` to isolate. ### 5.4 Backup and restore There is no state to back up. Whisper is stateless — model weights are re-downloadable, and transcription outputs are whatever your application writes them to. Snapshot the cache dir if you want to skip download latency on restore. ### 5.5 Upgrades ```bash pip install -U openai-whisper ``` The version is in `whisper/version.py` (`__version__ = "20250625"` at commit `04f449b`). Major upgrades typically bump the model URL table — existing checkpoints continue to load, but the default model may change (`turbo` became the default in 2024). Pin `--model` explicitly in production scripts to avoid surprises on upgrade. > [!WARNING] > Whisper does not have a plugin or extension API. Anything you build on > top is on top of the Python imports — be aware that internal APIs in > `whisper.decoding`, `whisper.timing`, and `whisper.transcribe` are not > stability-guaranteed across releases. ## 6. Performance ### 6.1 The model size table (operations view) | Model | Params | Layers (audio/text) | n_state | Heads | n_mels | Vocab | Languages | Disk | VRAM (FP16) | |-------|--------|---------------------|---------|-------|--------|-------|-----------|------|-------------| | `tiny` | 39 M | 4 / 4 | 384 | 6 | 80 | 51865 | 99 | 39 MB | ~1 GB | | `base` | 74 M | 6 / 6 | 512 | 8 | 80 | 51865 | 99 | 142 MB | ~1 GB | | `small` | 244 M | 12 / 12 | 768 | 12 | 80 | 51865 | 99 | 461 MB | ~2 GB | | `medium` | 769 M | 24 / 24 | 1024 | 16 | 80 | 51865 | 99 | 1.42 GB | ~5 GB | | `large-v1`, `large-v2` | 1550 M | 32 / 32 | 1280 | 20 | 80 | 51865 | 99–100 | 2.87 GB | ~10 GB | | `large-v3`, `large` | 1550 M | 32 / 32 | 1280 | 20 | 128 | 51866 | 99 | 2.87 GB | ~10 GB | | `turbo`, `large-v3-turbo` | 809 M | 32 / 4 | 1280 | 20 | 128 | 51866 | 99 | 1.51 GB | ~6 GB | `turbo` is "large-v3 with a much smaller decoder" — encoder is the same 1.55B-class network, but the text decoder shrinks from 32 layers to 4, which is where the speedup comes from. ### 6.2 Latency vs. throughput For interactive use (transcribe one file as fast as possible), pick `turbo` and enable FP16 on a recent GPU. For batch processing of many files, the bottleneck shifts: even on `tiny`, throughput is GPU-bound once your storage is fast enough. A rough breakdown of where time goes during a typical `large-v3` call on a 1-hour audio file: - Audio decode (FFmpeg): 1–3% (often bound by disk). - Mel spectrogram: <1%. - Encoder forward pass: ~30%. - Decoder + temperature fallback: ~60–65%. - Word-timestamp DTW (if enabled): ~5–10%. - Result writers: <1%. > [!TIP] > Word timestamps are NOT free. If your downstream system doesn't actually > consume them, leave `word_timestamps=False` (the default) — you'll > save 5–25% wall time on GPU and 50–100% on CPU. ### 6.3 FP16 vs. FP32 | Setting | Where | Effect | |---------|-------|--------| | `fp16=True` | GPU | Default. Roughly 1.5–2x faster than FP32 with no measurable WER impact. | | `fp16=True` | CPU | Auto-disabled with a warning (`FP16 is not supported on CPU; using FP32 instead`). | | `fp16=False` | GPU | Forces FP32 — slower, no benefit unless debugging FP16 instability. | The fallback ladder (temperature `0 → 0.2 → 0.4 → ... → 1.0`) re-decodes the same 30-second window up to 6 times if the output keeps failing the compression-ratio or logprob thresholds. A pathological window can blow out the per-window budget. ### 6.4 KV cache `whisper/model.py` installs forward hooks on every key/value projection in the decoder via `Whisper.install_kv_cache_hooks()`. During autoregressive sampling, only the last token's logits are computed and the cached K/V tensors are reused. Without this, decoding a single 30-second window would require quadratic compute in the token count; with it, it is linear. The KV cache is per-`DecodingTask`, lives in a `dict` mapping `nn.Module → Tensor`, and is destroyed at the end of `_main_loop()` via `PyTorchInference.cleanup_caching()`. There is **no cross-window cache reuse** — every new 30-second window starts with an empty KV cache. ### 6.5 Threading CPU inference uses PyTorch's intra-op thread pool. Set it via `--threads N` (CLI) or `torch.set_num_threads(N)` (Python). The CLI explicitly notes this **supersedes `MKL_NUM_THREADS` and `OMP_NUM_THREADS`**. For multiple parallel transcriptions on CPU, prefer multiple processes over multiple threads — Whisper itself is not thread-safe at the model instance level, and the Python GIL limits intra-process parallelism. ### 6.6 Profiling tips - Set `verbose=False` and wrap with `time.monotonic()` for end-to-end measurement. - Inspect `result["segments"][i]["temperature"]` — non-zero values signal that fallback re-decodes happened. - Watch `result["segments"][i]["compression_ratio"]` — values close to the `2.4` threshold mean you are one bad window away from a fallback cascade. - For GPU profiling, run with `CUDA_LAUNCH_BLOCKING=1` and use Nsight Systems or `torch.profiler`. ## 7. Security Whisper is an inference library that runs entirely on the local machine. This makes its attack surface small but not zero. ### 7.1 Privacy properties The single most important property: **audio is processed locally and is never sent over the network**. Whisper's only network call is the one-time download of model weights from `https://openaipublic.azureedge.net/main/whisper/models/...`. Once those weights are cached, the package is fully offline-capable. Implications: - Recordings of customer conversations, medical dictation, legal interviews, etc. stay on the host you control. Data residency is whatever the host's residency is. - Whisper itself does not log audio, transcripts, or metadata. Anything your wrapper code logs is on you. Be careful logging `result["segments"][i]["text"]` — it contains the actual transcript. - There is no telemetry. No usage pings, no error reporting, no anonymous metrics. > [!CAUTION] > If your transcripts contain personally identifying information (PII), > regulated health data (PHI), or attorney-client material, treat them as > the same sensitivity class as the original audio. Don't log them > verbatim, don't store them in shared queues without encryption, and > consider segment-level redaction before persisting. ### 7.2 Model weight provenance Whisper downloads weights over **HTTPS** and verifies the file's SHA-256 against the hash baked into its URL. The hash table for every model is `_MODELS` in `whisper/__init__.py`. Example: ```python "large-v3": "https://openaipublic.azureedge.net/main/whisper/models/e5b1a55b89c1367dacf97e3e19bfd829a01529dbfdeefa8caeb59b3f1b81dadb/large-v3.pt", ``` The 64-hex segment between `models/` and the filename **is** the SHA-256 the loader checks against. A man-in-the-middle who modifies the bytes will change the hash; Whisper raises: ```text RuntimeError: Model has been downloaded but the SHA256 checksum does not not match. Please retry loading the model. ``` > [!WARNING] > The integrity check protects against in-flight corruption and naive > tampering. It does **not** protect against an attacker who compromises > `openaipublic.azureedge.net` and serves a pre-image-collision-resistant > backdoored checkpoint with the same SHA-256 — currently believed > infeasible, but worth noting. For ultra-high-trust environments, > distribute weights out-of-band. ### 7.3 Untrusted input `whisper.load_model(name)` accepts a string that is either a model name **or a filesystem path**. If you call it with attacker-controlled input, you give them path-controlled `torch.load()` access. The package mitigates this by setting `weights_only=True` (when `torch.__version__ >= "1.13"`), which restricts which Python types `pickle` will reconstruct — but `weights_only` is not a hard sandbox. For audio inputs: `whisper.load_audio()` shells out to `ffmpeg` with the file path as `-i`. Pass only paths your application controls. FFmpeg has its own attack surface against malformed media — keep it patched. ### 7.4 What NOT to log - The full transcript (`result["text"]`). - `result["segments"][i]["text"]`. - The user's `initial_prompt` if it contains business secrets (e.g. customer names, internal codenames you fed in for vocab biasing). - The audio file's path if it is constructed from PII (`recordings/.mp3`). - Token IDs in `result["segments"][i]["tokens"]` — they aren't directly human-readable, but a tiktoken decode reproduces the transcript. Safe to log: language, segment count, temperatures, average log-probs, durations, model name, error class names. ### 7.5 Network policies Whisper's only outbound traffic targets `openaipublic.azureedge.net` over HTTPS (port 443). For locked-down environments: 1. Pre-populate `~/.cache/whisper/` with the models you need. 2. Block egress to `openaipublic.azureedge.net`. 3. Set `XDG_CACHE_HOME` to a read-only volume in production. ## 8. Developer Internals ### 8.1 Repository layout ``` whisper/ ├── __init__.py model registry, load_model, available_models ├── __main__.py CLI entry: from .transcribe import cli; cli() ├── audio.py load_audio (ffmpeg), pad_or_trim, log_mel_spectrogram ├── decoding.py DecodingOptions, DecodingResult, DecodingTask, decoders, filters ├── model.py Whisper, AudioEncoder, TextDecoder, attention, kv-cache hooks ├── timing.py DTW alignment, find_alignment, add_word_timestamps ├── tokenizer.py tiktoken wrapper, LANGUAGES, get_tokenizer ├── transcribe.py transcribe(), cli() — long-form pipeline + argparse ├── triton_ops.py DTW + median filter Triton kernels (Linux/x86_64 GPU) ├── utils.py ResultWriter (txt/srt/vtt/tsv/json), helpers ├── version.py __version__ = "20250625" ├── assets/ gpt2.tiktoken, multilingual.tiktoken, mel_filters.npz └── normalizers/ ├── basic.py BasicTextNormalizer (Unicode-symbol stripping) ├── english.py EnglishTextNormalizer (number/spelling normalization) └── english.json Spelling correction table ``` ### 8.2 The encoder/decoder model `whisper.model.Whisper` is a standard Transformer with a few audio-specific twists: - **AudioEncoder** (`model.py:174`): 1. Two `Conv1d` layers (kernel 3) — the second has stride 2, so 30s → 1500 frames at the encoder output. 2. Add a **fixed sinusoidal** positional embedding (`sinusoids()`). 3. `n_audio_layer` `ResidualAttentionBlock`s with self-attention only. 4. Final `LayerNorm`. - **TextDecoder** (`model.py:207`): 1. Token embedding (shared with the output projection — no separate `lm_head`, the logits are computed as `x @ token_embedding.T`). 2. **Learned** positional embedding (length `n_text_ctx = 448`). 3. `n_text_layer` `ResidualAttentionBlock`s with self-attention plus **cross-attention** into the encoder output. 4. Causal mask (`triu(1)`). - **MultiHeadAttention** (`model.py:81`): - Uses `torch.nn.functional.scaled_dot_product_attention` (SDPA) when available — falls back to a manual softmax+matmul otherwise. - The `disable_sdpa()` context manager is used during word-timestamp DTW so the raw attention weights can be captured by hooks. ### 8.3 Special tokens and the language ID head The tokenizer (`whisper/tokenizer.py`) wraps `tiktoken.Encoding` and adds a fixed set of specials. The decoder sequence template is: ``` <|startoftranscript|> <||> <|transcribe|> <|endoftext|> <|startoftranscript|> <||> <|translate|> <|endoftext|> <|startoftranscript|> <||> <|transcribe|> <|notimestamps|> ... <|endoftext|> ``` Plus 1501 timestamp tokens `<|0.00|>` through `<|30.00|>` in 0.02-second increments — these are why Whisper can emit timestamps natively without a separate alignment model. `detect_language()` (`decoding.py:19`) is the language ID head: 1. Run the encoder to get audio features. 2. Forward the decoder for **one step** with just `<|startoftranscript|>`. 3. Mask out non-language-token logits. 4. `argmax` → most probable language token. The "head" is not a separate module — it's the same decoder, queried at exactly position 1 of the sequence with a logit mask. Elegant and zero extra parameters. ### 8.4 The decoder loop and logit filters `DecodingTask` (`decoding.py:508`) is the per-30s-window controller. Its `run()` method: 1. Builds the initial token sequence (sot + lang + task + optional prompt/prefix). 2. Picks a decoder strategy: `BeamSearchDecoder` if `beam_size` is set, else `GreedyDecoder` (which becomes sampling at `temperature > 0`). 3. Picks logit filters: always `SuppressTokens` for non-speech symbols, optionally `SuppressBlank`, and (unless `without_timestamps`) `ApplyTimestampRules`. 4. Runs `_main_loop`: for each step, get logits, apply filters, ask the decoder to pick the next token, repeat until EOT or context full. 5. Use `MaximumLikelihoodRanker` to pick the best of beam-search / best-of-N candidates. `ApplyTimestampRules` (`decoding.py:441`) is the most interesting filter. It enforces: - Timestamps appear in **pairs** (each segment has a start and end). - Each segment is non-zero length. - The **first** sampled token must be a timestamp (so segments don't start with text). - If the total probability mass over timestamp tokens exceeds the max text-token probability at any step, force the next token to be a timestamp (this is what gives Whisper its "predict timestamps natively" behaviour). ### 8.5 Long-form transcription (sliding window + temperature fallback) `transcribe()` in `whisper/transcribe.py` implements long-form transcription via a sliding-window state machine: ``` prev_seek = 0 while seek < total_frames: take 30-second mel window starting at `seek` decode with temperature fallback ladder: for t in temperatures (default (0, 0.2, 0.4, 0.6, 0.8, 1.0)): kwargs = decode_options excluding incompatible (beam_size vs best_of) result = model.decode(window, DecodingOptions(**kwargs, temperature=t)) failed = (compression_ratio > 2.4 OR avg_logprob < -1.0) AND not silence if not failed: break if no_speech_prob > 0.6 and avg_logprob < -1.0: skip this window (silence fast-forward) else: carve segments out of timestamp-token boundaries if word_timestamps: run DTW alignment seek += min(N_FRAMES, last-segment-end-in-frames) if condition_on_previous_text: prepend last window's tokens to next prompt ``` The result dict aggregates segments across windows and re-decodes the final concatenated tokens to produce `result["text"]`. ### 8.6 Word-level timestamps (cross-attention DTW) `add_word_timestamps()` in `whisper/timing.py`: 1. Re-run the model **with hooks** that capture cross-attention weights from the alignment heads. The alignment heads are pre-determined per model — base85-encoded boolean masks in `_ALIGNMENT_HEADS` in `whisper/__init__.py:38`. 2. Median-filter the attention matrix along the time axis (Triton CUDA kernel if available, else NumPy/Numba). 3. Run dynamic time warping (DTW) between predicted tokens and audio frames to find the best monotonic alignment. The CUDA path uses the Triton kernel `dtw_kernel` in `whisper/triton_ops.py`; the CPU path uses `dtw_cpu` decorated with `@numba.jit(parallel=True)`. 4. Convert frame indices to seconds via `TOKENS_PER_SECOND = 50`. 5. Merge punctuation tokens with adjacent words per `prepend_punctuations`/`append_punctuations`. 6. Smooth out word durations at sentence boundaries to suppress pathological long words. The output is per-word `{word, start, end, probability}` injected into the segment dicts as `"words"`. > [!NOTE] > Word timestamps are an **inference-time** capability, not a training > objective. Whisper was not trained directly on word-level alignments — > the alignment is derived post hoc from cross-attention. This is why > `--task translate` warns: the cross-attention pattern between English > output tokens and non-English audio is less interpretable. ### 8.7 Logit filters reference | Class (decoding.py) | Purpose | |---------------------|---------| | `SuppressBlank` | At sample position 0, suppress `' '` and `<\|endoftext\|>` so output cannot be empty. | | `SuppressTokens` | Mask a fixed set of token IDs (`-1` expands to `non_speech_tokens` plus task/sot tokens). | | `ApplyTimestampRules` | Enforces timestamp pairing, monotonicity, max initial timestamp, and "prefer timestamps" behaviour. | ### 8.8 Decoder strategies | Class (decoding.py) | Used when | |---------------------|-----------| | `GreedyDecoder` | `temperature > 0` (sampling) or as a fallback when `beam_size is None` and `temperature == 0`. | | `BeamSearchDecoder` | `beam_size is not None` and `temperature == 0`. Optional `patience` extends beam exploration. | | `MaximumLikelihoodRanker` | Final pick across beams or best-of-N samples, with optional length penalty. | ### 8.9 Output writers `whisper.utils.get_writer()` returns a callable for the requested format. The class hierarchy: ``` ResultWriter ├── WriteTXT .txt one segment per line ├── SubtitlesWriter │ ├── WriteVTT .vtt WEBVTT, dot-decimal millis │ └── WriteSRT .srt SRT, comma-decimal millis, hours always shown ├── WriteTSV .tsv start_ms\tend_ms\ttext (no header decoration) └── WriteJSON .json full result dict via json.dump ``` When `output_format` is `"all"`, `get_writer` returns a `write_all` callable that fans out to all five. ### 8.10 Code patterns - **`@torch.no_grad()`** is applied at every entry point that runs the model (`detect_language`, `decode`, `DecodingTask.run`). - **`@dataclass(frozen=True)`** for `DecodingOptions` and `DecodingResult` — option objects are immutable; `replace()` returns a copy. - **Forward hooks** for KV-cache and attention capture. Hooks are always paired with a `cleanup` step to avoid leaks. - **`@lru_cache(maxsize=None)`** on tokenizer construction (`get_tokenizer`, `get_encoding`) and the mel filter loader — they are pure functions of their arguments. - **Numba JIT** for hot loops on CPU (`dtw_cpu`, `backtrace`). - **Triton JIT** for the same loops on CUDA, with a graceful fallback via try/except around `RuntimeError` and `subprocess.CalledProcessError`. ### 8.11 Testing strategy The repository ships a `tests/` directory with pytest-style tests: - `tests/test_audio.py` — `load_audio`, `log_mel_spectrogram`, sample rate / frame count invariants. - `tests/test_normalizer.py` — `BasicTextNormalizer`, `EnglishTextNormalizer`. - `tests/test_timing.py` — DTW alignment correctness on synthetic input. - `tests/test_tokenizer.py` — special-token IDs, language token lookup. - `tests/test_transcribe.py` — end-to-end transcription on `tests/jfk.flac`. Run them with `pytest tests/`. Tests are CPU-friendly; `test_transcribe.py` uses `tiny.en` to keep wall time manageable. ### 8.12 Code quality tooling `.pre-commit-config.yaml` (in the root) configures: | Hook | Version | Purpose | |------|---------|---------| | `pre-commit-hooks` | v5.0.0 | `check-json`, `end-of-file-fixer`, `trailing-whitespace`, `mixed-line-ending`, `check-added-large-files` (4 MB cap). | | `psf/black` | 25.1.0 | Code formatter. | | `pycqa/isort` | 6.0.0 | Import sorting (`--profile black`, line length 88). | | `pycqa/flake8` | 7.1.1 | Linter (`max-line-length 88`, ignore `E203,E501,W503,W504`). | Match these before submitting a PR upstream. ### 8.13 Potential improvements The codebase is mature, but the source has a few visible TODOs and non-obvious behaviours worth flagging: - **`transcribe.py`** has a comment `"NOTE: This loop is obscurely flattened to make the diff readable. A later commit should turn this into a simpler nested loop."` near `while clip_idx < len(seek_clips)`. Refactor candidate. - **No batched inference** at the `transcribe()` level — each file serially decodes through 30s windows. A batched-window decoder would improve throughput on short-clip workloads. - **Triton fallback warnings** are silent by default (`warnings.warn(...)`). If you depend on the GPU DTW path, add a `simplefilter("error", UserWarning)` guard in development. - **No ONNX/TensorRT export** included in the package. Community ports exist (whisper.cpp, faster-whisper, distil-whisper) but are not part of this repo. ### 8.14 Contributing The repo accepts PRs at . Practical guidance: 1. Run `pre-commit run --all-files` before pushing — black + isort + flake8 will be enforced. 2. Add a test under `tests/` for any user-visible behaviour change. 3. Avoid breaking `DecodingOptions` — it's the public Python API. 4. For new CLI flags, the pattern is: add an `argparse.add_argument()` in `cli()`, plumb the arg through `transcribe()`, and update the `transcribe()` docstring. ## 9. Integration Points ### 9.1 Public Python API The package's top-level imports in `whisper/__init__.py`: ```python from .audio import load_audio, log_mel_spectrogram, pad_or_trim from .decoding import DecodingOptions, DecodingResult, decode, detect_language from .model import ModelDimensions, Whisper from .transcribe import transcribe from .version import __version__ available_models() -> List[str] load_model(name, device=None, download_root=None, in_memory=False) -> Whisper ``` Anything else (`DecodingTask`, `BeamSearchDecoder`, `find_alignment`, `get_writer`, `Tokenizer`, `LANGUAGES`, `TO_LANGUAGE_CODE`, etc.) is accessible by full module path but not part of the surface guaranteed across releases. ### 9.2 Bundled `Whisper` instance methods Methods attached to a loaded `model` (assigned at the bottom of `whisper/model.py`): | Method | Source | Description | |--------|--------|-------------| | `model.transcribe(audio, **opts)` | `transcribe.py:38` | Long-form transcription pipeline. | | `model.decode(mel, options, **kwargs)` | `decoding.py:793` | Single 30-second decode. | | `model.detect_language(mel, tokenizer=None)` | `decoding.py:19` | Returns `(language_tokens, language_probs)`. | | `model.embed_audio(mel)` | `model.py:287` | Run encoder only. | | `model.logits(tokens, audio_features)` | `model.py:290` | Decoder forward pass given pre-encoded features. | | `model.set_alignment_heads(dump)` | `model.py:278` | Override the heads used for word timing. | | `model.install_kv_cache_hooks(cache=None)` | `model.py:310` | Returns `(cache_dict, hooks)`. | | `model.is_multilingual` (property) | `model.py:303` | `True` if vocab includes language tokens. | | `model.num_languages` (property) | `model.py:307` | Number of supported languages for this checkpoint. | | `model.device` (property) | `model.py:299` | The device the parameters live on. | ### 9.3 Audio input formats `load_audio()` accepts any path FFmpeg can decode. There is no constraint on container or codec. The function: 1. Spawns `ffmpeg -nostdin -threads 0 -i -f s16le -ac 1 -acodec pcm_s16le -ar 16000 -` as a subprocess. 2. Reads `int16` samples from stdout. 3. Normalises to `float32` in `[-1, 1]` (divide by 32768). So Whisper supports MP3, WAV, FLAC, OGG, OPUS, M4A, AAC, WMA, MP4 video (audio extracted), MOV, MKV, WebM, AIFF, AU, CAF, AMR, GSM, and anything else FFmpeg builds with. `log_mel_spectrogram()` also accepts a NumPy array or a PyTorch tensor — useful when audio is already in memory (e.g. from a microphone capture library). ### 9.4 Output integrations The five built-in formats cover the common cases: | Format | Typical consumer | |--------|------------------| | `txt` | Simple text indexing, search. | | `srt` | Video player subtitles (HandBrake, MPV, Plex). | | `vtt` | HTML5 ``, web video. | | `tsv` | Spreadsheet, pandas, custom analytics — `start_ms\tend_ms\ttext`. | | `json` | Programmatic consumption with full `result` shape. | For non-standard formats, write a small `ResultWriter` subclass: ```python from whisper.utils import ResultWriter class WriteMyFormat(ResultWriter): extension: str = "myfmt" def write_result(self, result, file, options=None, **kwargs): for seg in result["segments"]: file.write(f"{seg['start']:.2f},{seg['end']:.2f}: {seg['text']}\n") ``` Or just consume `result` as a dict and write whatever you want. ### 9.5 No HTTP / RPC interface Whisper does not ship a REST API, gRPC server, or any networked interface. If you need one, build it on top of the Python package — for example with FastAPI + a model held resident in process. There is no plugin system to extend Whisper itself; integration is done at the caller layer. --- _Generated April 2026 from openai/whisper at 04f449b. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._