Hugging Face Transformers — Administrator & Developer Guide

1. Architecture Overview

Hugging Face Transformers v5.6.0.dev0 is the model-definition framework for state-of-the-art machine learning across text, vision, audio, and multimodal tasks, supporting both inference and training. The library is built around a small number of foundational architectural patterns that govern how the 1,000+ supported model families are structured, imported, and executed.

Note

Transformers follows a lazy import architecture. No backend-specific code (PyTorch, TensorFlow, JAX, vision, audio) is loaded until the symbol is actually accessed. This means import transformers is cheap regardless of which optional dependencies are installed.

1.1 Lazy Import System

The central design pattern is deferred loading via three cooperating components:

1. _import_structure dict — Defined in src/transformers/__init__.py, this maps every public symbol to the module path that contains it. The dict is populated at module load time but no modules are actually imported.

2. _LazyModule — A custom module subclass that intercepts __getattr__ calls. When code accesses transformers.BertModel for the first time, _LazyModule resolves the path from _import_structure and imports only that specific module.

3. define_import_structure() — Helper that populates the _import_structure dict from structured declarations, keeping the init file maintainable at scale.

# Conceptual illustration of how lazy loading works
import transformers
# At this point, torch has NOT been imported

model_class = transformers.BertModel
# Now, and only now, src/transformers/models/bert/modeling_bert.py is imported

1.2 Optional Dependency Pattern

Backend-specific code is guarded by two mechanisms:

• OptionalDependencyNotAvailable — A sentinel exception raised when a required optional dependency (torch, vision libraries, audio libraries) is not installed.
• requires_backends() — Decorator/guard that attaches a meaningful error message to any class or function that requires a specific backend, rather than surfacing a raw ImportError.

This means a user without PyTorch installed can still import transformers, use tokenizers, and access configuration objects — only model classes raise errors at instantiation time.

1.3 Auto Classes and Model Registry

The Auto class system (AutoModel, AutoTokenizer, AutoConfig, AutoModelForCausalLM, and ~40 more task-specific variants) provides a unified interface for loading any supported model by name.

The registry is maintained in src/transformers/models/auto/auto_mappings.py via two families of mapping dicts:

When AutoModel.from_pretrained("bert-base-uncased") is called, the library:

1. Downloads or loads the config.json from the model hub
2. Reads the "model_type" field
3. Looks up the matching class in CONFIG_MAPPING_NAMES / MODEL_FOR_*_MAPPING_NAMES
4. Instantiates and returns the concrete class

1.4 Model Directory Convention

Each model family lives at src/transformers/models/{model_name}/ and contains a standardized set of files:

1.5 Modular Model System

New models may be defined using modular_{model}.py files that subclass existing models and override only what changes. These files are not shipped as-is — the repo maintenance tool utils/modular_model_converter.py converts them to full standalone modeling files. This workflow:

• Reduces copy-paste across similar architectures
• Keeps diffs small during model development
• Produces self-contained output files that the lazy import system can load normally

1.6 Pipeline System

The pipeline() factory function in src/transformers/pipelines/__init__.py is the primary high-level inference interface. It:

1. Accepts a task string (e.g., "text-generation", "image-classification") or a model identifier
2. Auto-detects the task from the loaded model config when not specified
3. Loads the appropriate pipeline class from the 20+ implementations in src/transformers/pipelines/
4. Returns a callable that handles tokenization, inference, and post-processing

1.7 Generation Subsystem

The src/transformers/generation/ directory implements text generation for all autoregressive models:

• GenerationMixin — Mixed into PreTrainedModel, provides generate(), greedy_search(), beam_search(), sample(), and related methods
• GenerationConfig — Dataclass capturing all generation hyperparameters (temperature, top-k, top-p, repetition penalty, etc.)
• Logits processors — Composable transforms applied to the logit distribution at each decoding step (see §11.1 for the complete list)
• Stopping criteria — Pluggable conditions that terminate generation
• Streaming — TextStreamer, TextIteratorStreamer, AsyncTextIteratorStreamer for token-by-token output
• Continuous batching — ContinuousBatchingManager and ContinuousMixin in the continuous_batching subdirectory

1.8 Training Stack

The Trainer class in src/transformers/trainer.py wraps a full PyTorch training loop with:

• Callback system for extensibility
• Logging integrations (TensorBoard, W&B, etc.) via src/transformers/integrations/
• Distributed training support (DDP, FSDP, DeepSpeed) via Accelerate
• TrainingArguments dataclass in src/transformers/training_args.py for all hyperparameters and infrastructure options

1.9 Quantization Architecture

22+ quantization backends are supported via a plugin-style system in src/transformers/quantizers/. Each backend implements a common interface and is activated by passing the corresponding config object to from_pretrained().

1.10 Serving Architecture

src/transformers/cli/serving/server.py exposes a build_server() function that returns a FastAPI application with OpenAI-compatible endpoints. A ModelManager handles model lifecycle (load, reset). Request ID tracking is provided via X-Request-ID middleware. CORS middleware is optionally enabled.

┌─────────────────────────────────────────────────────────┐
│  transformers serve (CLI)                               │
│     └─ build_server() → FastAPI app                    │
│           ├── POST /v1/chat/completions                 │
│           ├── POST /v1/responses                        │
│           ├── POST /v1/audio/transcriptions             │
│           ├── POST /load_model                          │
│           ├── POST /reset                               │
│           ├── GET  /v1/models                           │
│           └── GET  /health                              │
└─────────────────────────────────────────────────────────┘

2. System Requirements

2.1 Python

2.2 Required Dependencies

These packages are installed automatically with pip install transformers:

Important

The tokenizers version is upper-bounded at 0.23.0. Installing a newer tokenizers release without first checking compatibility may cause silent failures in fast tokenizer behavior. Pin this dependency explicitly in production environments.

2.3 Optional Backend Dependencies

None of these are required for the library to import, but are needed for the capabilities shown:

2.4 Hardware

3. Configuration Reference

3.1 PreTrainedConfig Base Class

All model configs inherit from PreTrainedConfig (src/transformers/configuration_utils.py). Common attributes shared across all models:

3.2 TrainingArguments Reference

TrainingArguments (src/transformers/training_args.py) is a @dataclass that controls all aspects of the Trainer. Key fields:

3.3 GenerationConfig Reference

GenerationConfig controls all autoregressive decoding behavior. Key fields:

3.4 Environment Variables

All 69 recognized environment variables are listed below, organized by domain.

Library behavior:

XLA / TPU:

Attention:

Distributed training:

Debugging:

SageMaker:

Testing:

Important

LOCAL_RANK defaults to -1, which signals single-process mode. Multi-GPU training launched via torchrun or accelerate launch will set this automatically. Do not manually set LOCAL_RANK in production unless you are managing process launch yourself.

4. Deployment & Scaling

4.1 Installation

Standard installation (PyTorch, CPU):

pip install transformers

With PyTorch GPU support:

pip install transformers torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

With all extras for training:

pip install transformers[torch,accelerate,deepspeed,peft]

From source (development):

git clone https://github.com/huggingface/transformers
cd transformers
pip install -e ".[dev]"

4.2 Serving via CLI

The transformers serve command starts an OpenAI-compatible HTTP server backed by build_server() in src/transformers/cli/serving/server.py.

# Basic serve
transformers serve --model-name-or-path meta-llama/Llama-3.1-8B-Instruct

# With custom host/port
transformers serve \
  --model-name-or-path Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 8080

Available endpoints on the running server:

All requests include an X-Request-ID header for tracing.

4.3 Distributed Training

Using torchrun (PyTorch native DDP):

torchrun --nproc_per_node=4 examples/pytorch/text-generation/run_clm.py \
  --model_name_or_path gpt2 \
  --dataset_name wikitext \
  --dataset_config_name wikitext-2-raw-v1 \
  --output_dir /tmp/clm-output \
  --per_device_train_batch_size 4 \
  --num_train_epochs 3

Using Accelerate:

accelerate config  # One-time setup
accelerate launch examples/pytorch/token-classification/run_ner.py \
  --model_name_or_path bert-base-cased \
  --output_dir /tmp/ner-output

Using DeepSpeed:

# Set ACCELERATE_USE_DEEPSPEED=true or pass --deepspeed config.json
deepspeed --num_gpus=8 examples/pytorch/text-generation/run_clm.py \
  --deepspeed ds_config_zero3.json \
  --model_name_or_path meta-llama/Llama-3.1-8B \
  --output_dir /tmp/llama-finetuned

4.4 Quantization for Deployment

Pass a quantization config to from_pretrained() to load a model in reduced precision:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config,
    device_map="auto",
)

All 22+ quantization backends follow the same pattern — swap the config class. See §11.3 for the complete backend list.

4.5 KV-Cache Strategies

Set via GenerationConfig.cache_implementation or pass directly to generate().

4.6 Compile for Production Throughput

import torch
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B")
model = torch.compile(model, mode="reduce-overhead")

# Use StaticCache to avoid recompilation on shape changes
from transformers import StaticCache
model.generation_config.cache_implementation = "static"

Use CompileConfig for fine-grained control over compilation behavior via GenerationConfig.

5. Operations

5.1 CLI Reference

The transformers command is the single entry point, backed by a Typer app at src/transformers/cli/transformers.py.

transformers --help

5.2 Interactive Chat

transformers chat --model-name-or-path Qwen/Qwen2.5-7B-Instruct

The chat command uses RichInterface from src/transformers/cli/chat.py to render a terminal TUI. Requires rich to be installed.

5.3 Environment Diagnostics

transformers env

Prints installed versions of Python, PyTorch, TensorFlow, JAX, Transformers, Tokenizers, and relevant system info. Always run this first when filing a bug report.

5.4 Hub Operations

# Download a model
transformers download meta-llama/Llama-3.1-8B --cache-dir /data/hf-cache

# Enable Git LFS tracking for .bin and .safetensors
transformers lfs-enable-largefiles .

5.5 Model Conversion

# Convert PyTorch checkpoint to TensorFlow
transformers pt-to-tf \
  --model-name-or-path bert-base-uncased \
  --local-dir ./tf-bert

5.6 Continuous Batching

For high-throughput serving, ContinuousBatchingManager (in src/transformers/generation/continuous_batching/) implements iteration-level batching — new requests are inserted into an in-flight batch at each decoding step, avoiding the padding waste of static batching.

from transformers.generation import ContinuousBatchingConfig

gen_config = GenerationConfig(
    continuous_batching=ContinuousBatchingConfig(max_batch_size=32)
)

Note

Continuous batching is most effective when combined with a static KV-cache (cache_implementation="static") and torch.compile. It is not compatible with beam search (num_beams > 1).

5.7 Monitoring and Logging

Set the verbosity level via environment variable or programmatically:

export TRANSFORMERS_VERBOSITY=debug

import transformers
transformers.logging.set_verbosity_debug()
transformers.logging.enable_progress_bar()

Available levels (lowest to highest verbosity): critical, error, warning, info, debug.

To suppress the common advisory warnings without changing log level:

export TRANSFORMERS_NO_ADVISORY_WARNINGS=1

Warning

Setting TRANSFORMERS_VERBOSITY=debug in production will produce extremely high log volume including per-tensor shape information during model loading. Reserve debug level for isolated troubleshooting sessions only.

Caution

Enabling NETWORK_DEBUG_REPORT logs all outbound HTTP requests made by the library, which may include authentication tokens sent to the Hub. Ensure the NETWORK_DEBUG_REPORT_PATH output file is protected with appropriate filesystem permissions and is excluded from log aggregation pipelines.

6. Performance

6.1 Inference Optimization Checklist

6.2 Training Optimization

Gradient checkpointing (trades compute for memory):

training_args = TrainingArguments(
    output_dir="./output",
    gradient_checkpointing=True,
    gradient_accumulation_steps=8,
    per_device_train_batch_size=2,
    bf16=True,
)

Flash attention during training:

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16,
)

FSDP (Fully Sharded Data Parallel):

training_args = TrainingArguments(
    output_dir="./output",
    fsdp="full_shard auto_wrap",
    fsdp_config={"transformer_layer_cls_to_wrap": ["LlamaDecoderLayer"]},
)

6.3 Determinism

For reproducible results:

export FLASH_ATTENTION_DETERMINISTIC=1

Note that deterministic flash attention disables the non-deterministic tiling optimizations and will reduce throughput.

6.4 LR Scheduler Selection

Adafactor optimizer is also exported from transformers.optimization as a memory-efficient Adam alternative for large-scale training.

6.5 Benchmarking

The benchmark_v2/ directory contains a dedicated benchmarking framework for measuring inference and training throughput. Use this to establish baselines before and after optimization changes.

7. Security

Warning

Model weights loaded from untrusted sources can execute arbitrary code during deserialization when using the legacy .bin (pickle) format. Always prefer .safetensors format for models from untrusted or third-party sources. The safetensors library (required dependency >=0.4.3) provides a safe zero-copy loading path with no deserialization code execution.

Caution

The trust_remote_code=True parameter on from_pretrained() and pipeline() executes Python files downloaded from the model hub inside your process. Never set trust_remote_code=True for models from unverified publishers. Only use it for well-known model families from verified organizations when no native Transformers implementation exists.

7.1 Model Weight Security

Force safetensors-only loading:

model = AutoModelForCausalLM.from_pretrained(
    "org/model",
    use_safetensors=True,  # Raises error if safetensors not available
)

7.2 Hub Authentication

The library uses huggingface-hub for all Hub operations. Credentials are stored in ~/.cache/huggingface/token (or HF_HOME). Never commit this file or expose the token in environment variables accessible to untrusted code.

# Login
huggingface-cli login

# Or set token explicitly (avoid in shared environments)
export HF_TOKEN=hf_...

7.3 Serving Security

When running transformers serve on a network-accessible interface:

• The server has no built-in authentication. Place it behind a reverse proxy (nginx, Traefik) with auth middleware for any non-localhost deployment.
• Enable CORS middleware only for known origins.
• The /load_model and /reset endpoints can load arbitrary model names from the Hub — restrict access to these endpoints in multi-tenant environments.
• All requests include X-Request-ID for audit logging at the proxy layer.

7.4 Watermarking

For detecting AI-generated text, the generation subsystem includes two watermarking implementations:

Configure via WatermarkingConfig in GenerationConfig.

8. Developer Internals

8.1 Adding a New Model

The recommended workflow uses the modular model system:

transformers add-new-model-like <existing_model_name>

This invokes src/transformers/cli/add_new_model_like.py (ModelInfos class), which:

1. Prompts for the new model name and its base architecture
2. Creates src/transformers/models/{new_model}/modular_{new_model}.py
3. Scaffolds the configuration, modeling, tokenizer, and processing files by copying the base model structure

After implementing your changes in the modular file, convert to standalone:

python utils/modular_model_converter.py --model_type {new_model}

Register the model in Auto classes by adding entries to CONFIG_MAPPING_NAMES and the relevant MODEL_FOR_*_MAPPING_NAMES in src/transformers/models/auto/auto_mappings.py.

8.2 Repository Maintenance Tools (utils/)

The utils/ directory (64 files) contains tooling for maintaining the repository's internal consistency. These are run in CI and can be run locally:

Running consistency checks:

python utils/check_repo.py
python utils/check_copies.py
python utils/check_auto.py

8.3 PreTrainedModel Base Class

All PyTorch models inherit from PreTrainedModel (src/transformers/modeling_utils.py). Key methods:

8.4 Implementing a Custom Logits Processor

Logits processors allow arbitrary transforms to the output distribution at each generation step. Implement the LogitsProcessor interface:

from transformers.generation import LogitsProcessor
import torch

class BlockTokenProcessor(LogitsProcessor):
    def __init__(self, blocked_token_ids: list[int]):
        self.blocked = blocked_token_ids

    def __call__(
        self, input_ids: torch.LongTensor, scores: torch.FloatTensor
    ) -> torch.FloatTensor:
        scores[:, self.blocked] = float("-inf")
        return scores

# Use in generation
from transformers import LogitsProcessorList
processor_list = LogitsProcessorList([BlockTokenProcessor([1234, 5678])])
model.generate(inputs, logits_processor=processor_list)

8.5 Implementing a Custom Stopping Criterion

from transformers.generation import StoppingCriteria

class KeywordCriteria(StoppingCriteria):
    def __init__(self, keyword_ids: list[int], tokenizer):
        self.keyword_ids = keyword_ids

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> bool:
        for kw_id in self.keyword_ids:
            if kw_id in input_ids[0][-5:]:
                return True
        return False

8.6 Custom Trainer Callbacks

The Trainer callback system allows hooking into training events:

from transformers import TrainerCallback

class EarlyStoppingCallback(TrainerCallback):
    def __init__(self, patience: int = 3):
        self.patience = patience
        self._no_improve = 0

    def on_evaluate(self, args, state, control, metrics, **kwargs):
        metric = metrics.get("eval_loss")
        # Custom logic here
        if self._no_improve >= self.patience:
            control.should_training_stop = True
        return control

8.7 The mlinter Modeling Linter

utils/mlinter/ implements a custom linter enforcing TRF (Transformers Rules Framework) rules specific to model implementation patterns. Run it before submitting model PRs:

python utils/mlinter/run_mlinter.py src/transformers/models/your_model/

8.8 Testing Conventions

The test suite in tests/ is pytest-based with extensive custom infrastructure in testing_utils.py:

Custom decorators:

Running tests:

# All fast tests
pytest tests/ -x

# Include slow tests
RUN_SLOW=1 pytest tests/

# Specific model
pytest tests/models/bert/ -v

# Pipeline tests
pytest tests/pipelines/ -v

# With timeout override
PYTEST_TIMEOUT=1200 pytest tests/models/llama/

Generating dummy models for testing:

python utils/create_dummy_models.py --model_type bert --output_dir /tmp/dummy-bert

CI-optimized test selection (run only tests affected by a diff):

python utils/tests_fetcher.py --diff_with_last_commit

9. Integration Points

9.1 Logging Integrations (via Trainer)

Set report_to in TrainingArguments to enable:

9.2 PEFT Integration

Parameter-efficient fine-tuning via the peft library integrates directly with Trainer:

from peft import get_peft_model, LoraConfig, TaskType

peft_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    inference_mode=False,
    r=8,
    lora_alpha=32,
    lora_dropout=0.1,
)
model = get_peft_model(model, peft_config)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

9.3 ExecuTorch Integration

src/transformers/integrations/ includes ExecuTorch integration for on-device inference export. This enables exporting Transformers models to the ExecuTorch runtime for mobile and edge deployment.

9.4 SageMaker

The library detects SageMaker environments via the SM_* environment variables (see §3.4). The Trainer automatically adapts to SageMaker distributed training when TRAINING_JOB_ARN is set.

# SageMaker sets these automatically in training containers
TRAINING_JOB_ARN=arn:aws:sagemaker:us-east-1:123456789:training-job/my-job
SM_NUM_GPUS=8
SM_NUM_CPUS=32

9.5 Hub Webhooks and Push

# Push model after training
training_args = TrainingArguments(
    output_dir="./output",
    push_to_hub=True,
    hub_model_id="your-org/your-model-name",
    hub_strategy="every_save",  # or "end", "checkpoint", "all_checkpoints"
)

9.6 OpenAI-Compatible API Clients

When running transformers serve, any client that speaks the OpenAI API format can connect:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-required",  # Server has no auth by default
)

response = client.chat.completions.create(
    model="loaded-model",
    messages=[{"role": "user", "content": "Hello!"}],
)

10. Troubleshooting

10.1 CUDA Out of Memory

Error:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate X GiB...

Cause:

Model size exceeds available GPU VRAM, often compounded by KV-cache growth during generation or large batch sizes.

Solution:

# 1. Use quantization to reduce model footprint
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
    device_map="auto",
)

# 2. Enable gradient checkpointing for training
training_args = TrainingArguments(gradient_checkpointing=True)

# 3. Reduce batch size and accumulate gradients
training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
)

# 4. Free cache between inferences
import torch
torch.cuda.empty_cache()

Prevention:

Estimate memory requirements before loading: a float16 model uses ~2 bytes/parameter. A 7B model needs ~14 GB; use 4-bit quantization to bring this to ~4 GB.

10.2 Multi-GPU Cross-Device Tensor Errors

Error:

RuntimeError: Expected all tensors to be on the same device...

Cause:

With device_map="auto" and CPU offload, tensors may be on different devices. This is the root cause of issues like GitHub #45482 (Gemma4 with CPU offload).

Solution:

# Explicitly control layer placement
from accelerate import dispatch_model, infer_auto_device_map

device_map = infer_auto_device_map(
    model,
    max_memory={0: "20GiB", 1: "20GiB", "cpu": "30GiB"},
    no_split_module_classes=["LlamaDecoderLayer"],
)

Prevention:

When using CPU offload (device_map="auto" with limited GPU memory), test the specific model architecture with your hardware configuration before production deployment.

10.3 from_pretrained Fails with KeyError on Model Type

Error:

KeyError: 'qwen3_moe' is not a supported model type.

Cause:

The installed version of Transformers does not include the model architecture. This is the root cause of GitHub #45478 (Qwen3.5 Moe on >=5.4.0 in older installs).

Solution:

# Update to latest version
pip install --upgrade transformers

# Or install from source for bleeding-edge models
pip install git+https://github.com/huggingface/transformers

Prevention:

Pin transformers version in deployment manifests and test model loading as part of your CI pipeline.

10.4 tokenizers Version Conflict

Error:

ImportError: cannot import name 'XTokenizer' from 'tokenizers'

or silent behavioral differences in fast tokenizers.

Cause:

tokenizers is bounded at <=0.23.0. A newer version may introduce breaking API changes or behavioral differences.

Solution:

pip install "tokenizers>=0.22.0,<=0.23.0" --force-reinstall

Prevention:

Lock tokenizers version explicitly in requirements.txt or pyproject.toml.

10.5 No module named transformers.onnx

Error:

ModuleNotFoundError: No module named 'transformers.onnx'

Cause:

The transformers.onnx submodule was removed from the library (GitHub #22355, closed). ONNX export is now handled by the optimum library.

Solution:

pip install optimum[onnxruntime]
optimum-cli export onnx --model bert-base-uncased ./bert-onnx

Prevention:

Migrate all transformers.onnx usages to optimum before updating Transformers.

10.6 Zero Loss with num_labels=1 for Single-Label Classification

Error:

Training loss stays at 0.0 for sequence classification with num_labels=1.

Cause:

GitHub #45479: the single_label_classification problem type with num_labels=1 produces zero loss due to how the loss function is selected.

Solution:

Use num_labels=2 for binary classification and encode labels as 0/1, or set problem_type="regression" if your task is truly single-output regression.

Prevention:

When working with custom classification heads, verify loss values are non-zero in the first training step.

10.7 Stuck/Hanging Distributed Training

Error:

Training hangs indefinitely, no progress after a few steps.

Cause:

NCCL collective operation timeout, typically caused by a GPU process falling behind (OOM on one rank, uneven data distribution, or a dead process in the group).

Solution:

# Enable NCCL debug output
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL

# Check which rank is hanging
export NETWORK_DEBUG_REPORT=1
export NETWORK_DEBUG_REPORT_PATH=/tmp/network_debug.log

Prevention:

Set NCCL_TIMEOUT (via Accelerate config) to a value that allows catching hangs without waiting indefinitely. Use --max_steps with periodic checkpointing so partial progress is not lost.

11. Reference

11.1 Complete Logits Processors

11.2 Stopping Criteria

11.3 Quantization Backends

11.4 Cache Classes

11.5 Key Source File Index

11.6 Entry Points

11.7 Package Metadata