# Hugging Face Transformers User Guide

> **Version**: 5.6.0.dev0 | **License**: Apache 2.0 | **Python**: 3.10–3.14 | **Source**: `huggingface/transformers` @ `71560f4` (2026-04-17)

---

## 1. Quick Start

Get a model running in under two minutes with the `pipeline()` factory — the fastest path to working inference.

> [!TIP]
> `pipeline()` handles tokenization, model loading, and post-processing automatically. Start here before diving into lower-level APIs.

### Install and run your first pipeline

```bash
pip install transformers torch
```

```python
from transformers import pipeline

# Text generation
generator = pipeline("text-generation", model="gpt2")
result = generator("The future of AI is", max_new_tokens=50)
print(result[0]["generated_text"])
# The future of AI is bright, with breakthroughs in language understanding...

# Sentiment analysis
classifier = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
print(classifier("I love using Transformers!"))
# [{'label': 'POSITIVE', 'score': 0.9998}]
```

### Run a chat model in the terminal

```bash
pip install transformers[serving]
transformers serve --model meta-llama/Llama-3.2-1B-Instruct
```

Then in a second terminal:

```bash
transformers chat
```

You are now in an interactive Rich-based TUI connected to the local serve endpoint.

---

## 2. Installation

### System requirements

| Requirement | Minimum | Notes |
|---|---|---|
| Python | 3.10 | Tested through 3.14 |
| PyTorch | See `[torch]` extra | Recommended backend |
| RAM | 8 GB | Varies greatly by model size |
| Disk | 1 GB+ per model | Cached in HuggingFace Hub cache |

> [!IMPORTANT]
> PyTorch is **not** installed by `pip install transformers` alone. Use `pip install transformers[torch]` or install your framework separately. Without a deep-learning backend (PyTorch, TensorFlow, Flax, or JAX), model inference will not work.

### Base install

```bash
pip install transformers
```

Installs core library with hard dependencies: `huggingface-hub>=1.5.0,<2.0`, `numpy>=1.17`, `packaging>=20.0`, `pyyaml>=5.1`, `regex>=2025.10.22`, `tokenizers>=0.22.0,<=0.23.0`, `typer`, `safetensors>=0.4.3`, `tqdm>=4.27`.

### Install extras

Install only what you need. Each extra pulls in the relevant framework and supporting libraries.

| Extra | Command | What it adds |
|---|---|---|
| PyTorch | `pip install transformers[torch]` | PyTorch + accelerate |
| Vision | `pip install transformers[vision]` | torchvision + Pillow |
| Audio | `pip install transformers[audio]` | torchaudio + librosa + pyctcdecode + phonemizer (+ kenlm on Python <3.13) |
| Video | `pip install transformers[video]` | av (PyAV) |
| Serving | `pip install transformers[serving]` | FastAPI + uvicorn + openai + pydantic + rich + torch |
| SentencePiece | `pip install transformers[sentencepiece]` | sentencepiece tokenizer |
| Tiktoken | `pip install transformers[tiktoken]` | tiktoken tokenizer |
| Quantization kernels | `pip install transformers[kernels]` | optimum-benchmark kernels |
| Accelerate | `pip install transformers[accelerate]` | accelerate (multi-GPU, CPU offload) |
| DeepSpeed | `pip install transformers[deepspeed]` | deepspeed integration |
| SageMaker | `pip install transformers[sagemaker]` | SageMaker training |
| All | `pip install transformers[all]` | torch + vision + audio + video + kernels + timm + sentencepiece + tiktoken + chat_template + num2words + mistral-common |
| Dev | `pip install transformers[dev]` | all + testing + Japanese + sklearn |

> [!WARNING]
> On Python 3.13+, `kenlm` is excluded from the `[audio]` extra automatically. If you need `kenlm` for language-model-based ASR decoding, use Python 3.12 or earlier, or install `kenlm` from source.

### Install from source

```bash
pip install git+https://github.com/huggingface/transformers.git
```

For editable (development) install:

```bash
git clone https://github.com/huggingface/transformers
cd transformers
pip install -e ".[dev]"
```

### Verify installation

```bash
transformers env
```

Expected output (abbreviated):

```
- `transformers` version: 5.6.0.dev0
- Platform: Linux-5.15.0 x86_64
- Python version: 3.11.9
- PyTorch version (GPU?): 2.5.0 (True)
- `accelerate` version: 0.33.0
- `tokenizers` version: 0.22.0
```

---

## 3. Configuration

### Environment variables

These variables tune library behavior without touching code. Set them in your shell, `.env` file, or process environment before importing Transformers.

| Variable | Purpose | Default |
|---|---|---|
| `TRANSFORMERS_VERBOSITY` | Logging level (`debug`, `info`, `warning`, `error`, `critical`) | `warning` |
| `TRANSFORMERS_NO_ADVISORY_WARNINGS` | Set to `1` to suppress non-critical advisory warnings | unset |
| `TRANSFORMERS_DISABLE_TORCH_CHECK` | Set to `1` to skip PyTorch version compatibility check | unset |
| `USE_TORCH_XLA` | Enable (`1`) or disable (`0`) TPU/XLA support | `1` |
| `HF_MODULES_CACHE` | Custom directory for dynamically-loaded model modules | `~/.cache/huggingface/modules` |
| `FLASH_ATTENTION_DETERMINISTIC` | Set to `1` for deterministic flash attention (slower) | unset |

```bash
# Example: verbose logging + suppress advisory warnings
export TRANSFORMERS_VERBOSITY=info
export TRANSFORMERS_NO_ADVISORY_WARNINGS=1
python my_script.py
```

### Logging in code

```python
import transformers

transformers.logging.set_verbosity_info()
transformers.logging.enable_explicit_format()
```

### Model cache

By default models download to `~/.cache/huggingface/hub`. Override with:

```bash
export HF_HOME=/data/hf_cache
```

Or per-session:

```python
from transformers import AutoModel
model = AutoModel.from_pretrained("bert-base-uncased", cache_dir="/data/my_models")
```

### Download a model explicitly

```bash
transformers download bert-base-uncased
transformers download meta-llama/Llama-3.2-1B-Instruct --repo-type model
```

---

## 4. Core Usage

### 4.1 Auto classes

The `Auto*` family resolves the correct model/tokenizer class from a checkpoint name or local path. This is the recommended pattern for virtually all use cases.

```python
from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM

# Tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Encoder model (embeddings, classification backbone)
model = AutoModel.from_pretrained("bert-base-uncased")

# Causal LM (text generation)
lm = AutoModelForCausalLM.from_pretrained("gpt2")
```

Common `Auto*` classes:

| Class | Use case |
|---|---|
| `AutoTokenizer` | Any tokenizer |
| `AutoConfig` | Load/inspect config only |
| `AutoModel` | Base model (no head) |
| `AutoModelForCausalLM` | Text generation |
| `AutoModelForSequenceClassification` | Sentence / text classification |
| `AutoModelForTokenClassification` | NER, POS tagging |
| `AutoModelForSeq2SeqLM` | Translation, summarization |
| `AutoModelForQuestionAnswering` | Extractive QA |
| `AutoModelForMaskedLM` | Fill-mask |
| `AutoModelForSpeechSeq2Seq` | ASR (Whisper-style) |
| `AutoModelForImageClassification` | Image classification |

### 4.2 Tokenizers

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Single sentence
encoding = tokenizer("Hello, world!", return_tensors="pt")
# {'input_ids': tensor([[101, 7592, 1010, 2088, 999, 102]]),
#  'attention_mask': tensor([[1, 1, 1, 1, 1, 1]])}

# Batch with padding and truncation
batch = tokenizer(
    ["Short sentence.", "A much longer sentence that needs truncation or padding."],
    padding=True,
    truncation=True,
    max_length=32,
    return_tensors="pt",
)

# Decode back to text
tokenizer.decode(encoding["input_ids"][0])
# '[CLS] hello, world! [SEP]'
```

### 4.3 Running inference

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

inputs = tokenizer("Once upon a time", return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
# Once upon a time there was a young princess who lived in a...
```

### 4.4 Text generation and GenerationConfig

Control every aspect of generation through `GenerationConfig` or inline kwargs.

```python
from transformers import GenerationConfig

gen_config = GenerationConfig(
    max_new_tokens=200,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.1,
)

outputs = model.generate(**inputs, generation_config=gen_config)
```

Common generation parameters:

| Parameter | Type | Purpose |
|---|---|---|
| `max_new_tokens` | int | Maximum tokens to generate |
| `do_sample` | bool | Stochastic sampling vs greedy |
| `temperature` | float | Sampling temperature (lower = sharper) |
| `top_p` | float | Nucleus sampling threshold |
| `top_k` | int | Top-k sampling |
| `num_beams` | int | Beam search width |
| `repetition_penalty` | float | Penalize repeated tokens |
| `eos_token_id` | int/list | Stop token(s) |
| `pad_token_id` | int | Padding token |

### 4.5 Streaming output

Use `TextStreamer` for real-time printing or `TextIteratorStreamer` to consume tokens in a loop.

```python
from transformers import TextStreamer, AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

streamer = TextStreamer(tokenizer, skip_prompt=True)
inputs = tokenizer("The quick brown fox", return_tensors="pt")
model.generate(**inputs, max_new_tokens=100, streamer=streamer)
# tokens print to stdout as they are generated
```

For async / UI use:

```python
from transformers import TextIteratorStreamer
import threading

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True)

def generate():
    model.generate(**inputs, max_new_tokens=100, streamer=streamer)

thread = threading.Thread(target=generate)
thread.start()

for token_text in streamer:
    print(token_text, end="", flush=True)

thread.join()
```

`AsyncTextIteratorStreamer` is available for `async for` loops in async frameworks.

### 4.6 Loading models efficiently

> [!WARNING]
> Never pass `trust_remote_code=True` to `from_pretrained()` for models from unverified publishers. This parameter executes arbitrary Python code downloaded from the Hub inside your process. Only use it for well-known model families from verified organizations when no native Transformers implementation exists.

```python
# Load in half precision (saves ~50% VRAM)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", torch_dtype=torch.float16)

# Automatic device placement (requires accelerate)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", device_map="auto")

# Load to a specific device
model = AutoModelForCausalLM.from_pretrained("gpt2").to("cuda")
```

### 4.7 Saving and loading locally

```python
# Save
model.save_pretrained("./my_model")
tokenizer.save_pretrained("./my_model")

# Load back
model = AutoModelForCausalLM.from_pretrained("./my_model")
tokenizer = AutoTokenizer.from_pretrained("./my_model")
```

---

## 5. Common Tasks

### 5.1 Text generation

```python
from transformers import pipeline

generator = pipeline("text-generation", model="gpt2", device=0)  # device=0 for first GPU

outputs = generator(
    "In the beginning",
    max_new_tokens=100,
    num_return_sequences=3,
    do_sample=True,
    temperature=0.8,
)
for i, out in enumerate(outputs):
    print(f"--- Sequence {i+1} ---")
    print(out["generated_text"])
```

### 5.2 Text classification / sentiment analysis

`sentiment-analysis` is an alias for `text-classification`.

```python
from transformers import pipeline

classifier = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")

results = classifier([
    "This movie was fantastic!",
    "I would not recommend this product.",
    "The service was average.",
])

for r in results:
    print(r["label"], f"{r['score']:.3f}")
# POSITIVE 0.999
# NEGATIVE 0.997
# NEGATIVE 0.523
```

### 5.3 Token classification (NER)

```python
from transformers import pipeline

ner = pipeline("token-classification", model="dbmdz/bert-large-cased-finetuned-conll03-english", aggregation_strategy="simple")

entities = ner("Hugging Face was founded in New York by Clément Delangue.")
for e in entities:
    print(f"{e['word']:20} {e['entity_group']:10} {e['score']:.3f}")
# Hugging Face         ORG        0.998
# New York             LOC        0.999
# Clément Delangue     PER        0.995
```

`ner` is an alias for `token-classification`.

### 5.4 Fill-mask

```python
from transformers import pipeline

unmasker = pipeline("fill-mask", model="bert-base-uncased")
results = unmasker("The capital of France is [MASK].")
for r in results[:3]:
    print(f"{r['token_str']:15} score={r['score']:.4f}")
# paris           score=0.9978
# lyon            score=0.0005
# france          score=0.0003
```

### 5.5 Automatic speech recognition

```python
from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")

result = asr("path/to/audio.wav")
print(result["text"])
# "Hello, this is a test of speech recognition."

# With timestamps
result = asr("path/to/audio.wav", return_timestamps=True)
print(result["chunks"])
# [{'timestamp': (0.0, 2.5), 'text': 'Hello, this is a test'}, ...]
```

### 5.6 Audio classification

```python
from transformers import pipeline

classifier = pipeline("audio-classification", model="superb/wav2vec2-base-superb-ks")
result = classifier("path/to/audio.wav")
print(result)
# [{'score': 0.997, 'label': 'yes'}, {'score': 0.001, 'label': 'no'}, ...]
```

### 5.7 Image classification

```python
from transformers import pipeline
from PIL import Image

classifier = pipeline("image-classification", model="google/vit-base-patch16-224")

result = classifier("path/to/image.jpg")
for r in result[:3]:
    print(f"{r['label']:40} {r['score']:.4f}")
# tabby, tabby cat                         0.5312
# tiger cat                                0.2297
# Egyptian cat                             0.1090
```

### 5.8 Object detection

```python
from transformers import pipeline

detector = pipeline("object-detection", model="facebook/detr-resnet-50")
results = detector("path/to/image.jpg")
for obj in results:
    print(obj["label"], f"score={obj['score']:.3f}", obj["box"])
# cat score=0.998 {'xmin': 42, 'ymin': 71, 'xmax': 315, 'ymax': 474}
```

### 5.9 Image segmentation

```python
from transformers import pipeline

segmenter = pipeline("image-segmentation", model="facebook/detr-resnet-50-panoptic")
segments = segmenter("path/to/image.jpg")
for seg in segments:
    print(seg["label"], seg["score"])
```

### 5.10 Zero-shot classification

Classify text into arbitrary categories without task-specific training.

```python
from transformers import pipeline

classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")

result = classifier(
    "The new iPhone features a 48MP camera and improved battery life.",
    candidate_labels=["technology", "sports", "politics", "entertainment"],
)
print(result["labels"][0], result["scores"][0])
# technology 0.9923
```

### 5.11 Zero-shot image classification

```python
from transformers import pipeline

classifier = pipeline("zero-shot-image-classification", model="openai/clip-vit-large-patch14")

result = classifier(
    "path/to/image.jpg",
    candidate_labels=["a dog", "a cat", "a car", "a mountain"],
)
print(result[0])
# {'score': 0.941, 'label': 'a cat'}
```

### 5.12 Depth estimation

```python
from transformers import pipeline

depth = pipeline("depth-estimation", model="depth-anything/Depth-Anything-V2-Small-hf")
result = depth("path/to/image.jpg")
# result["depth"] — PIL Image of the depth map
# result["predicted_depth"] — raw tensor
```

### 5.13 Image-text to text (multimodal)

```python
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="llava-hf/llava-1.5-7b-hf")

result = pipe(
    images="path/to/image.jpg",
    text="<image>\nDescribe what you see in this image.",
    max_new_tokens=200,
)
print(result[0]["generated_text"])
```

### 5.14 Document question answering

```python
from transformers import pipeline

dqa = pipeline("document-question-answering", model="impira/layoutlm-document-qa")

result = dqa(
    image="path/to/invoice.jpg",
    question="What is the total amount?",
)
print(result[0]["answer"])
# $1,234.56
```

### 5.15 Video classification

```python
from transformers import pipeline

classifier = pipeline("video-classification", model="MCG-NJU/videomae-base-finetuned-kinetics")
result = classifier("path/to/video.mp4", top_k=3)
for r in result:
    print(r["label"], f"{r['score']:.3f}")
```

### 5.16 Feature extraction

Get raw embeddings from any encoder model.

```python
from transformers import pipeline

extractor = pipeline("feature-extraction", model="bert-base-uncased")
features = extractor("Hello, world!", return_tensors=True)
print(features.shape)  # [1, sequence_length, hidden_size]
```

### 5.17 Mask generation (SAM-style)

```python
from transformers import pipeline

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)
masks = generator("path/to/image.jpg", points_per_batch=64)
# masks["masks"] — list of boolean numpy arrays
```

### 5.18 Text to audio

```python
from transformers import pipeline

synth = pipeline("text-to-audio", model="facebook/musicgen-small")
output = synth("A cheerful acoustic guitar melody")
# output["audio"] — numpy array
# output["sampling_rate"] — int

import scipy.io.wavfile as wavfile
wavfile.write("output.wav", output["sampling_rate"], output["audio"])
```

### 5.19 Quantization for smaller memory footprint

Load quantized models to run large models on consumer hardware.

> [!TIP]
> `BitsAndBytesConfig` with `load_in_4bit=True` is the most common starting point for fitting large models into limited VRAM. Requires `bitsandbytes` package and CUDA.

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)
```

Supported quantization backends:

| Config class | Backend | Notes |
|---|---|---|
| `BitsAndBytesConfig` | bitsandbytes | 4-bit and 8-bit; requires CUDA |
| `GPTQConfig` | auto-gptq | Post-training quantization |
| `AwqConfig` | autoawq | Activation-aware quantization |
| `TorchAoConfig` | torchao | Native PyTorch quantization |
| `HqqConfig` | hqq | Half-quadratic quantization |
| `QuantoConfig` | quanto | Multi-backend quantization |

22+ quantization configs available — see the Admin Guide for the full list and configuration reference.

### 5.20 Training with Trainer

```python
from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    Trainer,
    TrainingArguments,
)
from datasets import load_dataset

model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

dataset = load_dataset("imdb")

def tokenize(examples):
    return tokenizer(examples["text"], truncation=True, padding="max_length", max_length=512)

tokenized = dataset.map(tokenize, batched=True)

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=16,
    warmup_steps=500,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    fp16=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["test"],
)

trainer.train()
trainer.save_model("./my_fine_tuned_model")
```

### 5.21 Learning rate schedulers

Pass `lr_scheduler_type` in `TrainingArguments`:

| Scheduler | Value |
|---|---|
| Linear decay | `"linear"` |
| Cosine decay | `"cosine"` |
| Cosine with restarts | `"cosine_with_restarts"` |
| Polynomial decay | `"polynomial"` |
| Constant | `"constant"` |
| Constant with warmup | `"constant_with_warmup"` |
| Inverse square root | `"inverse_sqrt"` |
| Reduce on plateau | `"reduce_on_plateau"` |
| Cosine with min LR | `"cosine_with_min_lr"` |
| WSD (warmup-stable-decay) | `"wsd"` |
| Greedy | `"greedy"` |

```python
training_args = TrainingArguments(
    ...,
    lr_scheduler_type="cosine",
    warmup_ratio=0.06,
    learning_rate=2e-5,
)
```

### 5.22 Data collators

| Collator | Use case |
|---|---|
| `DataCollatorWithPadding` | Classification, QA — dynamic padding |
| `DataCollatorForLanguageModeling` | Causal or masked LM pre-training |
| `DataCollatorForSeq2Seq` | Encoder-decoder (translation, summarization) |

```python
from transformers import DataCollatorWithPadding

collator = DataCollatorWithPadding(tokenizer=tokenizer)
# Pass as collate_fn to DataLoader or as data_collator to Trainer
```

### 5.23 Serving: OpenAI-compatible API

Start a local inference server:

```bash
pip install transformers[serving]
transformers serve --model gpt2 --port 8000
```

> [!NOTE]
> The built-in server has no authentication. For any deployment beyond localhost, place it behind a reverse proxy (nginx, Traefik, Caddy) with authentication middleware. The `/load_model` and `/reset` endpoints can load arbitrary models — restrict access in shared environments.

Available endpoints:

| Endpoint | Method | Purpose |
|---|---|---|
| `/v1/chat/completions` | POST | Chat completions (OpenAI-compatible) |
| `/v1/responses` | POST | Responses API |
| `/v1/audio/transcriptions` | POST | Audio transcription |
| `/v1/models` | GET | List loaded models |
| `/health` | GET | Server health check |
| `/load_model` | POST | Load a model at runtime |
| `/reset` | POST | Reset server state |

Use with any OpenAI-compatible client:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="gpt2",
    messages=[{"role": "user", "content": "What is machine learning?"}],
    max_tokens=200,
)
print(response.choices[0].message.content)
```

### 5.24 Chat CLI

Start an interactive chat session:

```bash
# Connect to local serve endpoint (default)
transformers chat

# Connect to HF Inference API
transformers chat --model meta-llama/Llama-3.2-1B-Instruct
```

In-session commands:

| Command | Purpose |
|---|---|
| `!help` | Show all commands |
| `!status` | Show current model and generation settings |
| `!clear` | Start a new conversation (clears history) |
| `!example {NAME}` | Load a preset example (`llama`, `code`, `helicopter`, `numbers`, `birds`, `socks`, `numbers2`) |
| `!set {ARG=VALUE}` | Change generation settings live (e.g. `!set temperature=0.9`) |
| `!save {NAME}` | Save the current chat to a named file |
| `!exit` | Close the chat |

### 5.25 Converting checkpoints

Convert PyTorch weights to TensorFlow:

```bash
transformers pt-to-tf --model bert-base-uncased --output-dir ./tf_model
```

Enable Git LFS for large model files:

```bash
transformers lfs-enable-largefiles .
```

Upload large files via multipart:

```bash
transformers lfs-multipart-upload ./large_file.bin
```

---

## 6. Troubleshooting

### Error: `CUDA out of memory`

**Cause**: The model and its activations exceed available GPU VRAM.

**Solution**:
1. Load in lower precision: `torch_dtype=torch.float16` or `torch_dtype=torch.bfloat16`
2. Use 4-bit quantization: `BitsAndBytesConfig(load_in_4bit=True)`
3. Enable device mapping: `device_map="auto"` (requires `accelerate`)
4. Reduce batch size or `max_new_tokens`
5. Use `torch.backends.cuda.enable_flash_sdp(True)` for attention memory savings

```python
model = AutoModelForCausalLM.from_pretrained(
    "large-model-name",
    torch_dtype=torch.float16,
    device_map="auto",
)
```

**Prevention**: Check model size against VRAM before loading. A 7B model in fp16 needs ~14 GB VRAM.

---

### Error: `No module named 'transformers.onnx'`

**Cause**: The `transformers.onnx` submodule was removed in a previous version.

**Solution**: Use `optimum` for ONNX export instead:

```bash
pip install optimum[exporters]
optimum-cli export onnx --model bert-base-uncased ./bert_onnx/
```

**Prevention**: Do not reference `transformers.onnx` in new code; it no longer exists in the library.

---

### Error: `from_pretrained` fails with `Qwen3.5 MoE` on transformers >= 5.4.0

**Cause**: Breaking change in MoE model loading in 5.4.0 (tracked in issue #45478).

**Solution**:
1. Pin an older version: `pip install "transformers==5.3.x"`
2. Or check the issue tracker for a patched version / workaround commit

**Prevention**: Pin exact versions for production deployments of MoE models until the fix is confirmed stable.

---

### Error: `cross-device tensor error` with CPU offload on Gemma 26B

**Cause**: Known issue with Gemma4 26B-A4B models when `cpu_offload` is enabled (tracked in issue #45482).

**Solution**:
- Disable CPU offload: remove `cpu_offload_gbytes` or set `device_map` without offload
- Use a GPU with sufficient VRAM for the full model
- Monitor the upstream issue #45482 for the official fix

**Prevention**: Test CPU offload configurations on smaller Gemma variants before scaling to 26B.

---

### Error: `Gemma4 31B-IT Multi-GPU CUDA OOM`

**Cause**: Multi-GPU tensor placement with large Gemma4 models can produce OOM even when total VRAM is sufficient (tracked in issue #45229).

**Solution**:
- Explicitly set a `max_memory` dict to control per-GPU allocation:
  ```python
  model = AutoModelForCausalLM.from_pretrained(
      "google/gemma-4-31b-it",
      device_map="auto",
      max_memory={0: "20GiB", 1: "20GiB"},
  )
  ```
- Use `torch_dtype=torch.bfloat16` to halve memory
- Apply 4-bit quantization via `BitsAndBytesConfig`

**Prevention**: Always specify `max_memory` when distributing across 2+ GPUs.

---

### Error: `problem_type="single_label_classification"` produces zero loss with `num_labels=1`

**Cause**: Known bug — setting `num_labels=1` with `single_label_classification` leads to degenerate zero loss (tracked in issue #45479).

**Solution**: Use `num_labels=2` for binary classification, or use `problem_type="regression"` with `num_labels=1` for regression tasks.

**Prevention**: Do not combine `num_labels=1` with `single_label_classification`; binary classification always requires `num_labels=2`.

---

### Error: `resize_token_embeddings` doesn't update decoder on t5gemma

**Cause**: Known issue — `resize_token_embeddings` does not propagate to `decoder.embed_tokens` in t5gemma models (tracked in issue #45335).

**Solution**: Manually resize both encoder and decoder embeddings:

```python
model.resize_token_embeddings(new_vocab_size)
model.decoder.embed_tokens = model.shared  # re-tie if needed
```

**Prevention**: After adding tokens to a t5gemma model, verify embedding sizes on both encoder and decoder sides.

---

### Error: `tokenizers` version mismatch

**Cause**: `tokenizers` must be `>=0.22.0,<=0.23.0`. Using a newer or older version may cause silent tokenization bugs.

**Solution**:
```bash
pip install "tokenizers>=0.22.0,<=0.23.0"
```

**Prevention**: Pin `tokenizers` in `requirements.txt`; do not let it upgrade freely.

---

### Warning: `Some weights were not initialized`

**Cause**: Normal when loading a base model and adding a task-specific head. The head weights are randomly initialized and require fine-tuning.

**Solution**: This is expected. Train the model before using it for inference.

**Prevention**: For inference only, always load a fine-tuned checkpoint, not a base model.

---

### Error: Pipeline device mismatch (`tensors on different devices`)

**Cause**: Inputs are on CPU but model is on GPU, or vice versa.

**Solution**:
```python
# Specify device explicitly when creating pipeline
pipe = pipeline("text-classification", model="...", device=0)  # GPU 0

# Or use device_map for large models
pipe = pipeline("text-generation", model="...", device_map="auto")
```

**Prevention**: Always set `device` or `device_map` when constructing pipelines on GPU systems.

---

### Error: `huggingface-hub` version outside bounds

**Cause**: `huggingface-hub` requires `>=1.5.0,<2.0`. Hub 2.0+ has breaking changes.

**Solution**:
```bash
pip install "huggingface-hub>=1.5.0,<2.0"
```

**Prevention**: Pin in requirements files; set upper bounds to prevent silent breaking upgrades.

---

### Slow generation / high latency

**Cause**: Model on CPU, or using eager attention instead of flash attention.

**Solution**:
1. Move model to GPU: `.to("cuda")` or `device_map="auto"`
2. Enable Flash Attention 2 (requires `flash-attn` package):
   ```python
   model = AutoModelForCausalLM.from_pretrained("...", attn_implementation="flash_attention_2")
   ```
3. Use `torch.compile()` for repeated inference
4. Reduce `max_new_tokens` if low latency is the priority

---

## 7. FAQ

**Q: What is the difference between `pipeline()` and using `AutoModel` directly?**

`pipeline()` is a high-level convenience wrapper that bundles a model, tokenizer, and post-processing logic into a single call. It is ideal for quick experiments and production inference when you do not need to inspect intermediate representations. `AutoModel` gives you direct access to model outputs (logits, hidden states, attention weights) and is necessary for custom training loops, feature extraction research, or any use case where you need control over pre/post-processing.

**Q: Do I need a GPU to use Transformers?**

No. All models run on CPU. However, large models (7B+ parameters) will be extremely slow on CPU — expect minutes per token for generation. For practical use of large models, a GPU with sufficient VRAM is strongly recommended. For CPU-only deployment, consider smaller distilled models (DistilBERT, DistilGPT-2, TinyLlama) or quantized models with a CPU-compatible quantization backend.

**Q: How do I pick the right model for my task?**

Use the Hugging Face Hub model page (`huggingface.co/models`) and filter by task. For most NLP tasks, start with a model that is already fine-tuned for that task rather than a base model. Example: for sentiment analysis use a model fine-tuned on SST-2 or financial sentiment datasets, not `bert-base-uncased`.

**Q: What is the difference between `AutoModelForCausalLM` and `AutoModelForSeq2SeqLM`?**

Causal LM (`AutoModelForCausalLM`) is a decoder-only architecture (GPT-2, LLaMA, Gemma, Mistral) that generates text auto-regressively — each token depends on all previous tokens. Seq2Seq LM (`AutoModelForSeq2SeqLM`) is an encoder-decoder architecture (T5, BART, mBART) where an encoder processes the full input before a decoder generates the output. Use causal LM for open-ended generation; use seq2seq for translation, summarization, or structured output tasks.

**Q: How do I run a model that requires authentication (gated models)?**

Log in with the Hugging Face CLI before loading:

```bash
pip install huggingface_hub
huggingface-cli login
```

Then `from_pretrained` will use your stored token automatically. You can also pass `token="hf_..."` directly to `from_pretrained`.

**Q: Can I run Transformers without internet access (air-gapped)?**

Yes. First download the model with `transformers download` or `snapshot_download` on a connected machine. Transfer the cached files to the target machine. Then load from the local path:

```python
model = AutoModelForCausalLM.from_pretrained("/path/to/local/model")
```

Set `HF_DATASETS_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1` to prevent any outbound requests.

**Q: What is the difference between `PreTrainedTokenizer` and `PreTrainedTokenizerFast`?**

`PreTrainedTokenizerFast` is backed by the Rust-based `tokenizers` library and is significantly faster — especially for batch tokenization of large datasets. `PreTrainedTokenizer` is a pure-Python fallback. Most modern models ship a fast tokenizer. `AutoTokenizer` selects the fast version automatically when available. Prefer fast tokenizers for any training or large-scale processing pipeline.

**Q: How do I use `Trainer` with multiple GPUs?**

Use `accelerate launch` instead of `python`:

```bash
pip install accelerate
accelerate config  # one-time setup
accelerate launch train.py
```

Or set `CUDA_VISIBLE_DEVICES` and let `Trainer` detect available GPUs automatically when `device_map` is not set. For distributed training across nodes, configure `accelerate` with DeepSpeed or FSDP.

**Q: Can I serve models with an OpenAI-compatible API?**

Yes. `transformers serve` starts a FastAPI server with `/v1/chat/completions`, `/v1/responses`, `/v1/audio/transcriptions`, `/v1/models`, `/health`, `/load_model`, and `/reset` endpoints. Install with `pip install transformers[serving]` and run `transformers serve --model <model-id>`. Any OpenAI-compatible client (Python `openai` SDK, curl, etc.) can connect to `http://localhost:8000/v1`.

**Q: How do I add a new model to the library?**

Use the scaffolding command:

```bash
transformers add-new-model-like --model-name MyModel --model-type existing_model_type
```

This generates boilerplate files following library conventions. See the Transformers contribution guide for full instructions on adding architectures.

**Q: What quantization method should I use for my use case?**

| Scenario | Recommended |
|---|---|
| CUDA GPU, max quality-to-size ratio | `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4")` |
| CUDA GPU, fast inference, pre-quantized | `AwqConfig` or `GPTQConfig` |
| CPU or MPS (Apple Silicon) | `TorchAoConfig` or `QuantoConfig` |
| Need reproducible hardware-agnostic quantization | `QuantoConfig` |

**Q: How do I print my environment to report a bug?**

```bash
transformers env
```

Include this output in every GitHub issue report. It captures the exact versions of Transformers, PyTorch, CUDA, tokenizers, and accelerate.

**Q: What Python versions are supported?**

Python 3.10 through 3.14. Python 3.9 and earlier are not supported as of v5.6.0.dev0. Note that the `kenlm` dependency in `transformers[audio]` is excluded on Python 3.13+.

**Q: How do I disable the tqdm progress bars?**

```python
from transformers.utils import logging
logging.disable_progress_bar()
```

Or set `TRANSFORMERS_NO_ADVISORY_WARNINGS=1` to suppress advisory-level output. For no output at all, set `TRANSFORMERS_VERBOSITY=error`.

**Q: Is there a way to cache model downloads to a custom location?**

Yes. Set `HF_HOME` environment variable before running:

```bash
export HF_HOME=/data/hf_cache
```

Or pass `cache_dir` to any `from_pretrained` call:

```python
model = AutoModel.from_pretrained("bert-base-uncased", cache_dir="/data/hf_cache")
```

---

_Generated May 2026 from huggingface/transformers at 71560f4. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._