Whisper Administrator & Developer Guide

This guide is the source-of-truth reference for operators, ML platform engineers, and contributors. It covers Whisper's architecture, every configurable parameter, deployment patterns, performance characteristics, security/privacy implications, and the internals of the encoder/decoder, the language ID head, the temperature-fallback decoder, and the word-alignment pass.

For end-user CLI recipes and the Python API tour, see the Whisper User Guide. This document complements that one — they should be read together.

1. Architecture Overview

Whisper is a single end-to-end Transformer trained for multilingual, multitask audio-to-text transduction. The same model handles speech recognition (X→X), speech translation (X→English), language identification, and timestamp prediction — all controlled by special tokens at the start of the decoder sequence.

The Python package is intentionally thin: it owns the model weights, the tokenizer, an FFmpeg-based audio loader, a sliding-window decoder loop with temperature fallback, and writers for five output formats. There is no server, no daemon, no queue, no GPU pool — the package is meant to be embedded in your application or invoked as a CLI.

1.1 Data flow

                       audio file (mp3 / wav / flac / mp4 / …)
                                    │
                       ┌────────────┴───────────────┐
                       │  ffmpeg subprocess         │   load_audio()
                       │  → 16 kHz mono float32     │   whisper/audio.py
                       └────────────┬───────────────┘
                                    │
                       ┌────────────┴───────────────┐
                       │  STFT + Mel filterbank     │   log_mel_spectrogram()
                       │  → (n_mels, n_frames)      │   80 or 128 mels
                       │  CHUNK_LENGTH = 30 s       │   N_FRAMES = 3000
                       └────────────┬───────────────┘
                                    │
                       ┌────────────┴───────────────┐
                       │  AudioEncoder              │   whisper/model.py
                       │  Conv1D ×2 → Transformer   │   sinusoidal pos. embed
                       │  → (n_audio_ctx, n_state)  │
                       └────────────┬───────────────┘
                                    │
                       ┌────────────┴───────────────┐    ┌───────────────────┐
                       │  language detection (opt.) │ ←─ │ detect_language() │
                       │  argmax over lang tokens   │    │ whisper/decoding  │
                       └────────────┬───────────────┘    └───────────────────┘
                                    │
                       ┌────────────┴───────────────┐
                       │  TextDecoder (autoregress) │   whisper/model.py
                       │  cross-attn over audio     │   kv-cache enabled
                       │  + LogitFilter rules       │   timestamp / suppress
                       │  + Greedy/BeamSearch       │   temperature fallback
                       │  → tokens + logprobs       │
                       └────────────┬───────────────┘
                                    │
                       ┌────────────┴───────────────┐
                       │  optional: word DTW pass   │   add_word_timestamps()
                       │  cross-attn alignment      │   whisper/timing.py
                       │  → per-word start/end      │   triton_ops on CUDA
                       └────────────┬───────────────┘
                                    │
                       ┌────────────┴───────────────┐
                       │  ResultWriter (per format) │   whisper/utils.py
                       │  txt / srt / vtt / tsv /   │
                       │  json                      │
                       └────────────┬───────────────┘
                                    │
                                    ▼
                              transcript files

Note

Whisper processes audio in 30-second non-overlapping windows. Anything longer is sliced internally; anything shorter is zero-padded. The condition_on_previous_text flag controls whether each window's prompt includes the previous window's output, which is what gives long-form transcription continuity at the cost of occasional looped degeneration.

1.2 Component inventory

1.3 Technology stack

The dependency list is small by design (requirements.txt):

numba
numpy
torch
tqdm
more-itertools
tiktoken
triton>=2.0.0;platform_machine=="x86_64" and sys_platform=="linux" or sys_platform=="linux2"

1.4 Model dimensions

The ModelDimensions dataclass in whisper/model.py parameterises the Transformer. Each checkpoint stores its own dims, so load_model can build the correct architecture from the file:

@dataclass
class ModelDimensions:
    n_mels: int
    n_audio_ctx: int
    n_audio_state: int
    n_audio_head: int
    n_audio_layer: int
    n_vocab: int
    n_text_ctx: int
    n_text_state: int
    n_text_head: int
    n_text_layer: int

The audio encoder consumes (batch, n_mels, 3000) log-mel frames and emits (batch, n_audio_ctx, n_audio_state) features (e.g. 1500 × 1280 for large-v3). The text decoder runs autoregressively over n_text_ctx = 448 tokens with cross-attention into the encoder output.

Note

is_multilingual is detected by n_vocab >= 51865, and num_languages is derived from n_vocab - 51765 - int(is_multilingual). This is how the same code paths support large-v3 (99 langs), large-v2 (100 langs), and the English-only .en variants in one library.

2. System Requirements

2.1 Software

Python dependencies are installed transitively by pip: numba, numpy, torch, tqdm, more-itertools, tiktoken, triton (Linux/x86_64).

2.2 Hardware

Per-model VRAM (peak, FP16, batch size 1): tiny ~1 GB, base ~1 GB, small ~2 GB, medium ~5 GB, large-v* ~10 GB, turbo ~6 GB. Add ~10–20% headroom for the operating system and the cudnn workspace.

Important

For production batch transcription with large-v3, plan for an A100-40GB or H100 if you want headroom for batched decoding. A consumer 24 GB GPU (RTX 3090/4090/5090) handles large-v3 for single-stream FP16 jobs but may OOM on long word-timestamp runs because the Triton DTW kernel needs additional contiguous workspace.

2.3 Storage

Model weights live in ~/.cache/whisper/ by default (override with XDG_CACHE_HOME or the download_root argument to load_model). Sizes:

The assets/ directory ships ~6 MB of mel filter banks and tiktoken vocab files; these are bundled with the wheel and require no separate download.

3. Configuration Reference

Whisper is configured exclusively through CLI arguments and Python kwargs. There are no config files, no XML/YAML, and only a single environment variable.

3.1 Environment variables

There are no other environment variables. Whisper does not read WHISPER_*, TRANSCRIBE_*, or any custom env namespace.

3.2 CLI flags — full reference

Every flag accepted by whisper (in source order, from cli() in whisper/transcribe.py):

3.3 transcribe() keyword arguments

Programmatic equivalents to the CLI flags. Defaults differ for temperature and logprob_threshold between CLI and API:

Important

temperature defaults differ between CLI and API. The CLI passes a single float through --temperature (default 0) and lets --temperature_increment_on_fallback build the fallback tuple. The Python API expects a tuple directly (default (0.0, 0.2, 0.4, 0.6, 0.8, 1.0)). If you call transcribe() with temperature=0, you disable the fallback ladder entirely — usually not what you want for long-form audio.

3.4 DecodingOptions (single 30-s window)

Every field of the dataclass in whisper/decoding.py:

3.5 Audio constants

Hard-coded in whisper/audio.py (do not change without retraining):

4. Deployment & Scaling

4.1 Embedded library

The simplest deployment: ship Whisper as a Python dependency of your application. Hold one model in memory per process and serve transcription requests via your own queue/router.

# server-startup.py
import whisper

MODEL = whisper.load_model("turbo", device="cuda")

def handle_request(audio_path, language=None):
    return MODEL.transcribe(audio_path, language=language)

This is appropriate for moderate-traffic services where you can size one GPU per process. Whisper does not support shared-GPU multi-tenancy out of the box — you cannot load one model and have multiple Python processes hit it concurrently.

4.2 CLI in a Docker container

A minimal Linux container:

FROM python:3.11-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
        ffmpeg \
    && rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir -U openai-whisper
WORKDIR /work
ENTRYPOINT ["whisper"]

Build and use:

docker build -t whisper:turbo .
docker run --rm -v $(pwd):/work whisper:turbo audio.mp3 --model small.en

For GPU use, base on nvidia/cuda:12.1.1-runtime-ubuntu22.04, install python3 + pip, install a CUDA-matched PyTorch wheel, then pip install -U openai-whisper. Run with --gpus all.

4.3 Pre-baked model cache

Bake the model into the image to avoid the first-run download:

RUN python -c "import whisper; whisper.load_model('turbo')"
# Layer above caches /root/.cache/whisper/large-v3-turbo.pt (1.5 GB)

For air-gapped deployments, build the image somewhere with internet, then ship the layer.

Tip

In Kubernetes, mount ~/.cache/whisper from a PersistentVolume shared across pods (read-only) so each new pod doesn't re-download the checkpoint. Whisper SHA-256-verifies on load — concurrent reads are safe.

4.4 Scaling out

Whisper is embarrassingly parallel at the file level. The recommended pattern:

input queue (S3, Redis, RabbitMQ, …)
        │
        ▼
┌───────────────────┐  ┌───────────────────┐  ┌───────────────────┐
│ Worker pod (GPU)  │  │ Worker pod (GPU)  │  │ Worker pod (GPU)  │
│ load_model("…")   │  │ load_model("…")   │  │ load_model("…")   │
│ for path in queue │  │ for path in queue │  │ for path in queue │
│   transcribe(...) │  │   transcribe(...) │  │   transcribe(...) │
└───────────────────┘  └───────────────────┘  └───────────────────┘
        │                       │                       │
        └───────────────────────┴───────────────────────┘
                            │
                            ▼
                     output store (S3, DB)

Each worker holds one model resident, pulls jobs from a queue, and writes results back. With large-v3-turbo you can typically run one worker per GPU at saturated utilisation; with tiny/base you may be able to run two or three workers per GPU because the model is small enough that multiple CUDA contexts fit.

Note

There is no built-in batched inference path. If you have many short clips, you'll get more throughput by collecting them into 30-second super-clips with a sentinel token between them — but that requires custom code on top of decode(). The stock transcribe() is single-stream.

4.5 Health checks

A useful liveness check is to transcribe tests/jfk.flac (a 12-second sample bundled with the source repo) and assert the result contains the word "country":

import whisper

model = whisper.load_model("tiny.en")
out = model.transcribe("tests/jfk.flac")
assert "country" in out["text"].lower()

5. Operations

5.1 Capacity planning

Throughput is a function of model size, GPU FP16 throughput, and audio duration. Real-world numbers, single A100-40GB, FP16, batch=1, no word timestamps:

"RTF" here means seconds of audio processed per second of wall time. Word timestamps add ~10–25% overhead on GPU (the Triton DTW kernel) and ~50–100% on CPU.

Tip

A useful heuristic for batch jobs: assume RTF/2 to account for first-window language detection, occasional fallback re-decodes at higher temperature, and disk I/O.

5.2 Logging and monitoring

The package emits very little — tqdm progress bars and the per-segment prints when verbose=True. To capture structured logs, wrap the call:

import time, json, logging

log = logging.getLogger("whisper.adapter")

def transcribe_logged(model, path, **kwargs):
    t0 = time.monotonic()
    try:
        result = model.transcribe(path, verbose=False, **kwargs)
        log.info(json.dumps({
            "event": "transcribe.ok",
            "path": path,
            "language": result["language"],
            "n_segments": len(result["segments"]),
            "wall_seconds": round(time.monotonic() - t0, 3),
        }))
        return result
    except Exception as e:
        log.exception("transcribe.fail path=%s", path)
        raise

Useful per-segment metrics inside result["segments"]:

• avg_logprob (lower than logprob_threshold ⇒ low confidence).
• compression_ratio (above compression_ratio_threshold ⇒ likely loop).
• no_speech_prob (above no_speech_threshold ⇒ silent / non-speech).
• temperature (the temperature at which this segment finally succeeded;

  0 means the first attempt was rejected).

5.3 Cache management

The on-disk cache layout:

~/.cache/whisper/
  tiny.pt
  base.en.pt
  small.pt
  …

Each .pt file is verified against the SHA-256 in its download URL on every load. If the hash mismatches, Whisper warns and re-downloads. Manual cleanup: just rm the unwanted .pt files; Whisper recreates the cache directory on next run.

Note

The cache directory is shared across all users on the host who use the same XDG_CACHE_HOME. In multi-tenant environments, set XDG_CACHE_HOME per user or run with download_root=... to isolate.

5.4 Backup and restore

There is no state to back up. Whisper is stateless — model weights are re-downloadable, and transcription outputs are whatever your application writes them to. Snapshot the cache dir if you want to skip download latency on restore.

5.5 Upgrades

pip install -U openai-whisper

The version is in whisper/version.py (__version__ = "20250625" at commit 04f449b). Major upgrades typically bump the model URL table — existing checkpoints continue to load, but the default model may change (turbo became the default in 2024). Pin --model explicitly in production scripts to avoid surprises on upgrade.

Warning

Whisper does not have a plugin or extension API. Anything you build on top is on top of the Python imports — be aware that internal APIs in whisper.decoding, whisper.timing, and whisper.transcribe are not stability-guaranteed across releases.

6. Performance

6.1 The model size table (operations view)

turbo is "large-v3 with a much smaller decoder" — encoder is the same 1.55B-class network, but the text decoder shrinks from 32 layers to 4, which is where the speedup comes from.

6.2 Latency vs. throughput

For interactive use (transcribe one file as fast as possible), pick turbo and enable FP16 on a recent GPU. For batch processing of many files, the bottleneck shifts: even on tiny, throughput is GPU-bound once your storage is fast enough.

A rough breakdown of where time goes during a typical large-v3 call on a 1-hour audio file:

• Audio decode (FFmpeg): 1–3% (often bound by disk).
• Mel spectrogram: <1%.
• Encoder forward pass: ~30%.
• Decoder + temperature fallback: ~60–65%.
• Word-timestamp DTW (if enabled): ~5–10%.
• Result writers: <1%.

Tip

Word timestamps are NOT free. If your downstream system doesn't actually consume them, leave word_timestamps=False (the default) — you'll save 5–25% wall time on GPU and 50–100% on CPU.

6.3 FP16 vs. FP32

The fallback ladder (temperature 0 → 0.2 → 0.4 → ... → 1.0) re-decodes the same 30-second window up to 6 times if the output keeps failing the compression-ratio or logprob thresholds. A pathological window can blow out the per-window budget.

6.4 KV cache

whisper/model.py installs forward hooks on every key/value projection in the decoder via Whisper.install_kv_cache_hooks(). During autoregressive sampling, only the last token's logits are computed and the cached K/V tensors are reused. Without this, decoding a single 30-second window would require quadratic compute in the token count; with it, it is linear.

The KV cache is per-DecodingTask, lives in a dict mapping nn.Module → Tensor, and is destroyed at the end of _main_loop() via PyTorchInference.cleanup_caching(). There is no cross-window cache reuse — every new 30-second window starts with an empty KV cache.

6.5 Threading

CPU inference uses PyTorch's intra-op thread pool. Set it via --threads N (CLI) or torch.set_num_threads(N) (Python). The CLI explicitly notes this supersedes MKL_NUM_THREADS and OMP_NUM_THREADS.

For multiple parallel transcriptions on CPU, prefer multiple processes over multiple threads — Whisper itself is not thread-safe at the model instance level, and the Python GIL limits intra-process parallelism.

6.6 Profiling tips

• Set verbose=False and wrap with time.monotonic() for end-to-end measurement.
• Inspect result["segments"][i]["temperature"] — non-zero values signal that fallback re-decodes happened.
• Watch result["segments"][i]["compression_ratio"] — values close to the 2.4 threshold mean you are one bad window away from a fallback cascade.
• For GPU profiling, run with CUDA_LAUNCH_BLOCKING=1 and use Nsight Systems or torch.profiler.

7. Security

Whisper is an inference library that runs entirely on the local machine. This makes its attack surface small but not zero.

7.1 Privacy properties

The single most important property: audio is processed locally and is never sent over the network. Whisper's only network call is the one-time download of model weights from https://openaipublic.azureedge.net/main/whisper/models/.... Once those weights are cached, the package is fully offline-capable.

Implications:

• Recordings of customer conversations, medical dictation, legal interviews, etc. stay on the host you control. Data residency is whatever the host's residency is.
• Whisper itself does not log audio, transcripts, or metadata. Anything your wrapper code logs is on you. Be careful logging result["segments"][i]["text"] — it contains the actual transcript.
• There is no telemetry. No usage pings, no error reporting, no anonymous metrics.

Caution

If your transcripts contain personally identifying information (PII), regulated health data (PHI), or attorney-client material, treat them as the same sensitivity class as the original audio. Don't log them verbatim, don't store them in shared queues without encryption, and consider segment-level redaction before persisting.

7.2 Model weight provenance

Whisper downloads weights over HTTPS and verifies the file's SHA-256 against the hash baked into its URL. The hash table for every model is _MODELS in whisper/__init__.py. Example:

"large-v3": "https://openaipublic.azureedge.net/main/whisper/models/e5b1a55b89c1367dacf97e3e19bfd829a01529dbfdeefa8caeb59b3f1b81dadb/large-v3.pt",

The 64-hex segment between models/ and the filename is the SHA-256 the loader checks against. A man-in-the-middle who modifies the bytes will change the hash; Whisper raises:

RuntimeError: Model has been downloaded but the SHA256 checksum does not not match. Please retry loading the model.

Warning

The integrity check protects against in-flight corruption and naive tampering. It does not protect against an attacker who compromises openaipublic.azureedge.net and serves a pre-image-collision-resistant backdoored checkpoint with the same SHA-256 — currently believed infeasible, but worth noting. For ultra-high-trust environments, distribute weights out-of-band.

7.3 Untrusted input

whisper.load_model(name) accepts a string that is either a model name or a filesystem path. If you call it with attacker-controlled input, you give them path-controlled torch.load() access. The package mitigates this by setting weights_only=True (when torch.__version__ >= "1.13"), which restricts which Python types pickle will reconstruct — but weights_only is not a hard sandbox.

For audio inputs: whisper.load_audio() shells out to ffmpeg with the file path as -i. Pass only paths your application controls. FFmpeg has its own attack surface against malformed media — keep it patched.

7.4 What NOT to log

• The full transcript (result["text"]).
• result["segments"][i]["text"].
• The user's initial_prompt if it contains business secrets (e.g. customer names, internal codenames you fed in for vocab biasing).
• The audio file's path if it is constructed from PII (recordings/<email>.mp3).
• Token IDs in result["segments"][i]["tokens"] — they aren't directly human-readable, but a tiktoken decode reproduces the transcript.

Safe to log: language, segment count, temperatures, average log-probs, durations, model name, error class names.

7.5 Network policies

Whisper's only outbound traffic targets openaipublic.azureedge.net over HTTPS (port 443). For locked-down environments:

1. Pre-populate ~/.cache/whisper/ with the models you need.
2. Block egress to openaipublic.azureedge.net.
3. Set XDG_CACHE_HOME to a read-only volume in production.

8. Developer Internals

8.1 Repository layout

whisper/
├── __init__.py          model registry, load_model, available_models
├── __main__.py          CLI entry: from .transcribe import cli; cli()
├── audio.py             load_audio (ffmpeg), pad_or_trim, log_mel_spectrogram
├── decoding.py          DecodingOptions, DecodingResult, DecodingTask, decoders, filters
├── model.py             Whisper, AudioEncoder, TextDecoder, attention, kv-cache hooks
├── timing.py            DTW alignment, find_alignment, add_word_timestamps
├── tokenizer.py         tiktoken wrapper, LANGUAGES, get_tokenizer
├── transcribe.py        transcribe(), cli() — long-form pipeline + argparse
├── triton_ops.py        DTW + median filter Triton kernels (Linux/x86_64 GPU)
├── utils.py             ResultWriter (txt/srt/vtt/tsv/json), helpers
├── version.py           __version__ = "20250625"
├── assets/              gpt2.tiktoken, multilingual.tiktoken, mel_filters.npz
└── normalizers/
    ├── basic.py         BasicTextNormalizer (Unicode-symbol stripping)
    ├── english.py       EnglishTextNormalizer (number/spelling normalization)
    └── english.json     Spelling correction table

8.2 The encoder/decoder model

whisper.model.Whisper is a standard Transformer with a few audio-specific twists:

• AudioEncoder (model.py:174):
  1. Two Conv1d layers (kernel 3) — the second has stride 2, so 30s → 1500 frames at the encoder output.
  2. Add a fixed sinusoidal positional embedding (sinusoids()).
  3. n_audio_layer ResidualAttentionBlocks with self-attention only.
  4. Final LayerNorm.
• TextDecoder (model.py:207):
  1. Token embedding (shared with the output projection — no separate lm_head, the logits are computed as x @ token_embedding.T).
  2. Learned positional embedding (length n_text_ctx = 448).
  3. n_text_layer ResidualAttentionBlocks with self-attention plus cross-attention into the encoder output.
  4. Causal mask (triu(1)).
• MultiHeadAttention (model.py:81):
  • Uses torch.nn.functional.scaled_dot_product_attention (SDPA) when available — falls back to a manual softmax+matmul otherwise.
  • The disable_sdpa() context manager is used during word-timestamp DTW so the raw attention weights can be captured by hooks.

8.3 Special tokens and the language ID head

The tokenizer (whisper/tokenizer.py) wraps tiktoken.Encoding and adds a fixed set of specials. The decoder sequence template is:

<|startoftranscript|> <|<lang>|> <|transcribe|>          <text tokens> <|endoftext|>
<|startoftranscript|> <|<lang>|> <|translate|>           <text tokens> <|endoftext|>
<|startoftranscript|> <|<lang>|> <|transcribe|> <|notimestamps|> ...    <|endoftext|>

Plus 1501 timestamp tokens <|0.00|> through <|30.00|> in 0.02-second increments — these are why Whisper can emit timestamps natively without a separate alignment model.

detect_language() (decoding.py:19) is the language ID head:

1. Run the encoder to get audio features.
2. Forward the decoder for one step with just <|startoftranscript|>.
3. Mask out non-language-token logits.
4. argmax → most probable language token.

The "head" is not a separate module — it's the same decoder, queried at exactly position 1 of the sequence with a logit mask. Elegant and zero extra parameters.

8.4 The decoder loop and logit filters

DecodingTask (decoding.py:508) is the per-30s-window controller. Its run() method:

1. Builds the initial token sequence (sot + lang + task + optional prompt/prefix).
2. Picks a decoder strategy: BeamSearchDecoder if beam_size is set, else GreedyDecoder (which becomes sampling at temperature > 0).
3. Picks logit filters: always SuppressTokens for non-speech symbols, optionally SuppressBlank, and (unless without_timestamps) ApplyTimestampRules.
4. Runs _main_loop: for each step, get logits, apply filters, ask the decoder to pick the next token, repeat until EOT or context full.
5. Use MaximumLikelihoodRanker to pick the best of beam-search / best-of-N candidates.

ApplyTimestampRules (decoding.py:441) is the most interesting filter. It enforces:

• Timestamps appear in pairs (each segment has a start and end).
• Each segment is non-zero length.
• The first sampled token must be a timestamp (so segments don't start with text).
• If the total probability mass over timestamp tokens exceeds the max text-token probability at any step, force the next token to be a timestamp (this is what gives Whisper its "predict timestamps natively" behaviour).

8.5 Long-form transcription (sliding window + temperature fallback)

transcribe() in whisper/transcribe.py implements long-form transcription via a sliding-window state machine:

prev_seek = 0
while seek < total_frames:
    take 30-second mel window starting at `seek`
    decode with temperature fallback ladder:
        for t in temperatures (default (0, 0.2, 0.4, 0.6, 0.8, 1.0)):
            kwargs = decode_options excluding incompatible (beam_size vs best_of)
            result = model.decode(window, DecodingOptions(**kwargs, temperature=t))
            failed = (compression_ratio > 2.4 OR avg_logprob < -1.0) AND not silence
            if not failed: break
    if no_speech_prob > 0.6 and avg_logprob < -1.0:
        skip this window (silence fast-forward)
    else:
        carve segments out of timestamp-token boundaries
        if word_timestamps: run DTW alignment
        seek += min(N_FRAMES, last-segment-end-in-frames)
        if condition_on_previous_text: prepend last window's tokens to next prompt

The result dict aggregates segments across windows and re-decodes the final concatenated tokens to produce result["text"].

8.6 Word-level timestamps (cross-attention DTW)

add_word_timestamps() in whisper/timing.py:

1. Re-run the model with hooks that capture cross-attention weights from the alignment heads. The alignment heads are pre-determined per model — base85-encoded boolean masks in _ALIGNMENT_HEADS in whisper/__init__.py:38.
2. Median-filter the attention matrix along the time axis (Triton CUDA kernel if available, else NumPy/Numba).
3. Run dynamic time warping (DTW) between predicted tokens and audio frames to find the best monotonic alignment. The CUDA path uses the Triton kernel dtw_kernel in whisper/triton_ops.py; the CPU path uses dtw_cpu decorated with @numba.jit(parallel=True).
4. Convert frame indices to seconds via TOKENS_PER_SECOND = 50.
5. Merge punctuation tokens with adjacent words per prepend_punctuations/append_punctuations.
6. Smooth out word durations at sentence boundaries to suppress pathological long words.

The output is per-word {word, start, end, probability} injected into the segment dicts as "words".

Note

Word timestamps are an inference-time capability, not a training objective. Whisper was not trained directly on word-level alignments — the alignment is derived post hoc from cross-attention. This is why --task translate warns: the cross-attention pattern between English output tokens and non-English audio is less interpretable.

8.7 Logit filters reference

8.8 Decoder strategies

8.9 Output writers

whisper.utils.get_writer() returns a callable for the requested format. The class hierarchy:

ResultWriter
├── WriteTXT          .txt        one segment per line
├── SubtitlesWriter
│   ├── WriteVTT      .vtt        WEBVTT, dot-decimal millis
│   └── WriteSRT      .srt        SRT, comma-decimal millis, hours always shown
├── WriteTSV          .tsv        start_ms\tend_ms\ttext (no header decoration)
└── WriteJSON         .json       full result dict via json.dump

When output_format is "all", get_writer returns a write_all callable that fans out to all five.

8.10 Code patterns

• @torch.no_grad() is applied at every entry point that runs the model (detect_language, decode, DecodingTask.run).
• @dataclass(frozen=True) for DecodingOptions and DecodingResult — option objects are immutable; replace() returns a copy.
• Forward hooks for KV-cache and attention capture. Hooks are always paired with a cleanup step to avoid leaks.
• @lru_cache(maxsize=None) on tokenizer construction (get_tokenizer, get_encoding) and the mel filter loader — they are pure functions of their arguments.
• Numba JIT for hot loops on CPU (dtw_cpu, backtrace).
• Triton JIT for the same loops on CUDA, with a graceful fallback via try/except around RuntimeError and subprocess.CalledProcessError.

8.11 Testing strategy

The repository ships a tests/ directory with pytest-style tests:

• tests/test_audio.py — load_audio, log_mel_spectrogram, sample rate / frame count invariants.
• tests/test_normalizer.py — BasicTextNormalizer, EnglishTextNormalizer.
• tests/test_timing.py — DTW alignment correctness on synthetic input.
• tests/test_tokenizer.py — special-token IDs, language token lookup.
• tests/test_transcribe.py — end-to-end transcription on tests/jfk.flac.

Run them with pytest tests/. Tests are CPU-friendly; test_transcribe.py uses tiny.en to keep wall time manageable.

8.12 Code quality tooling

.pre-commit-config.yaml (in the root) configures:

Match these before submitting a PR upstream.

8.13 Potential improvements

The codebase is mature, but the source has a few visible TODOs and non-obvious behaviours worth flagging:

• transcribe.py has a comment "NOTE: This loop is obscurely flattened to make the diff readable. A later commit should turn this into a simpler nested loop." near while clip_idx < len(seek_clips). Refactor candidate.
• No batched inference at the transcribe() level — each file serially decodes through 30s windows. A batched-window decoder would improve throughput on short-clip workloads.
• Triton fallback warnings are silent by default (warnings.warn(...)). If you depend on the GPU DTW path, add a simplefilter("error", UserWarning) guard in development.
• No ONNX/TensorRT export included in the package. Community ports exist (whisper.cpp, faster-whisper, distil-whisper) but are not part of this repo.

8.14 Contributing

The repo accepts PRs at https://github.com/openai/whisper. Practical guidance:

1. Run pre-commit run --all-files before pushing — black + isort + flake8 will be enforced.
2. Add a test under tests/ for any user-visible behaviour change.
3. Avoid breaking DecodingOptions — it's the public Python API.
4. For new CLI flags, the pattern is: add an argparse.add_argument() in cli(), plumb the arg through transcribe(), and update the transcribe() docstring.

9. Integration Points

9.1 Public Python API

The package's top-level imports in whisper/__init__.py:

from .audio import load_audio, log_mel_spectrogram, pad_or_trim
from .decoding import DecodingOptions, DecodingResult, decode, detect_language
from .model import ModelDimensions, Whisper
from .transcribe import transcribe
from .version import __version__

available_models() -> List[str]
load_model(name, device=None, download_root=None, in_memory=False) -> Whisper

Anything else (DecodingTask, BeamSearchDecoder, find_alignment, get_writer, Tokenizer, LANGUAGES, TO_LANGUAGE_CODE, etc.) is accessible by full module path but not part of the surface guaranteed across releases.

9.2 Bundled Whisper instance methods

Methods attached to a loaded model (assigned at the bottom of whisper/model.py):

9.3 Audio input formats

load_audio() accepts any path FFmpeg can decode. There is no constraint on container or codec. The function:

1. Spawns ffmpeg -nostdin -threads 0 -i <file> -f s16le -ac 1 -acodec pcm_s16le -ar 16000 - as a subprocess.
2. Reads int16 samples from stdout.
3. Normalises to float32 in [-1, 1] (divide by 32768).

So Whisper supports MP3, WAV, FLAC, OGG, OPUS, M4A, AAC, WMA, MP4 video (audio extracted), MOV, MKV, WebM, AIFF, AU, CAF, AMR, GSM, and anything else FFmpeg builds with.

log_mel_spectrogram() also accepts a NumPy array or a PyTorch tensor — useful when audio is already in memory (e.g. from a microphone capture library).

9.4 Output integrations

The five built-in formats cover the common cases:

For non-standard formats, write a small ResultWriter subclass:

from whisper.utils import ResultWriter

class WriteMyFormat(ResultWriter):
    extension: str = "myfmt"

    def write_result(self, result, file, options=None, **kwargs):
        for seg in result["segments"]:
            file.write(f"{seg['start']:.2f},{seg['end']:.2f}: {seg['text']}\n")

Or just consume result as a dict and write whatever you want.

9.5 No HTTP / RPC interface

Whisper does not ship a REST API, gRPC server, or any networked interface. If you need one, build it on top of the Python package — for example with FastAPI + a model held resident in process. There is no plugin system to extend Whisper itself; integration is done at the caller layer.