# nanoGPT User Guide

nanoGPT is Andrej Karpathy's minimal, hackable reference implementation of GPT
for training and finetuning small-to-medium language models. The entire model
fits in a single ~330-line `model.py`, the training loop is one ~330-line
`train.py`, and configuration is a "poor man's configurator" that runs Python
config files and CLI overrides through `exec()` against module globals. The
repo is not a Python package — there is no `pip install nanogpt`. The intended
workflow is `git clone`, install the PyTorch ecosystem, then run scripts from
the repo root.

This guide covers everything a user needs to train, sample, finetune, and
benchmark with nanoGPT. The Shakespeare-character demo trains a fully working
baby GPT in under ten minutes on a single GPU and remains the fastest way to
verify your installation.

## 1. Quick Start

The fastest path from an empty directory to a trained, sampling baby GPT is
three commands plus a clone. Run all of them from the repository root.

```bash
git clone https://github.com/karpathy/nanoGPT.git
cd nanoGPT
pip install torch numpy transformers datasets tiktoken wandb tqdm
python data/shakespeare_char/prepare.py
python train.py config/train_shakespeare_char.py
python sample.py --out_dir=out-shakespeare-char
```

> [!TIP]
> If you are on a MacBook (no CUDA), append `--device=cpu --compile=False
> --eval_iters=20 --log_interval=1 --block_size=64 --batch_size=12
> --n_layer=4 --n_head=4 --n_embd=128 --max_iters=2000 --lr_decay_iters=2000
> --dropout=0.0` to the `train.py` invocation. The character-level Shakespeare
> recipe is intentionally tiny so it runs end-to-end on a laptop CPU in a
> reasonable amount of time.

### What just happened

| Step | Effect |
|---|---|
| `git clone` | Pulls the source tree (5 Python files at root + `config/` + `data/`). |
| `pip install` | Installs PyTorch, NumPy, HuggingFace Transformers, HuggingFace Datasets, tiktoken, wandb, tqdm. |
| `prepare.py` | Downloads `tinyshakespeare/input.txt`, builds a character-level vocabulary (65 unique chars), writes `train.bin`, `val.bin`, and `meta.pkl`. |
| `train.py` | Trains a 6-layer / 6-head / 384-embed model for 5,000 iterations (~10 minutes on a single A100, longer on consumer cards). |
| `sample.py` | Loads `out-shakespeare-char/ckpt.pt` and prints 10 samples of 500 tokens each. |

### Key features

- Single-file model in `model.py` (`LayerNorm`, `CausalSelfAttention`, `MLP`,
  `Block`, `GPTConfig`, `GPT`). Around 330 lines, fully readable in one
  sitting.
- Single-file training loop in `train.py` with cosine learning-rate decay,
  gradient accumulation, mixed-precision autocast, optional `torch.compile`,
  and DDP support out of the box.
- Loads HuggingFace GPT-2 weights (`gpt2`, `gpt2-medium`, `gpt2-large`,
  `gpt2-xl`) for finetuning or zero-shot evaluation via
  `GPT.from_pretrained()`.
- Flash Attention path activates automatically on PyTorch 2.0+
  (`scaled_dot_product_attention`).
- Optional Weights & Biases logging and `bench.py` micro-benchmarking script
  with `torch.profiler` integration.
- Five reference configs in `config/`: train GPT-2 124M, finetune Shakespeare,
  evaluate the four GPT-2 sizes, and the Shakespeare-char baby model.

## 2. Installation

nanoGPT is a clone-and-run repository. There is no PyPI package, no Docker
image in the repo, and no installation script — you clone the source, install
the PyTorch ecosystem, and run scripts directly.

### System requirements

| Resource | Recommended | Minimum |
|---|---|---|
| Python | 3.10+ | 3.8 |
| PyTorch | 2.0+ (Flash Attention path enabled) | 1.13 (manual attention fallback) |
| GPU | NVIDIA A100/A6000/4090 with bfloat16 | Single consumer GPU, or CPU/MPS for the baby model |
| RAM | 16 GB | 8 GB |
| Disk | 60 GB free for OpenWebText pretraining | 200 MB for Shakespeare-char |

> [!IMPORTANT]
> The full GPT-2 124M reproduction requires roughly **1 node × 8 × A100 40GB
> for ~5 days** (per the comment at the top of `config/train_gpt2.py`). The
> Shakespeare-char demo runs comfortably on a single consumer GPU — or a
> laptop CPU with the size overrides shown in Quick Start.

### Step 1: Clone the repository

```bash
git clone https://github.com/karpathy/nanoGPT.git
cd nanoGPT
```

### Step 2: Install dependencies

There is no `requirements.txt` shipped with this repo. The dependencies are
the ones imported by the scripts:

```bash
pip install torch numpy transformers datasets tiktoken wandb tqdm
```

| Package | Used by | Purpose |
|---|---|---|
| `torch` | every script | The model, optimizer, autocast, DDP, `torch.compile`. |
| `numpy` | `train.py`, `bench.py`, `data/*/prepare.py` | `np.memmap` over `train.bin`/`val.bin` for the data loader. |
| `transformers` | `model.py:GPT.from_pretrained` | Loads GPT-2 124M / medium / large / xl weights from HuggingFace. |
| `datasets` | `data/openwebtext/prepare.py` | HuggingFace `load_dataset("openwebtext")`. |
| `tiktoken` | `sample.py`, `data/shakespeare/prepare.py`, `data/openwebtext/prepare.py` | GPT-2 BPE encoder/decoder. |
| `wandb` | `train.py` (only if `wandb_log=True`) | Optional experiment tracking. |
| `tqdm` | `data/openwebtext/prepare.py` | Progress bar for the long tokenization step. |
| `requests` | `data/shakespeare*/prepare.py` | Downloads the tinyshakespeare corpus. Bundled with most Python distributions. |

> [!NOTE]
> If you only plan to run the Shakespeare demos and never touch OpenWebText,
> you can omit `datasets`. Conversely, if you never plan to load HuggingFace
> GPT-2 weights, you can omit `transformers`. The scripts only import what
> they need.

### Step 3: Verify the install

```bash
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
```

```text
2.5.1+cu121 True
```

If `torch.cuda.is_available()` returns `False` and you have an NVIDIA GPU,
reinstall PyTorch with the CUDA wheel that matches your driver — see
[pytorch.org](https://pytorch.org/get-started/locally/).

### Step 4: Prepare a dataset

Pick one of the three bundled datasets:

| Dataset | Prepare command | Output | Size |
|---|---|---|---|
| Shakespeare-char | `python data/shakespeare_char/prepare.py` | 65-char vocab, `train.bin` (~1M tokens), `val.bin`, `meta.pkl` | ~2 MB |
| Shakespeare (BPE) | `python data/shakespeare/prepare.py` | GPT-2 BPE, `train.bin` (301,966 tokens), `val.bin` (36,059 tokens) | ~700 KB |
| OpenWebText | `python data/openwebtext/prepare.py` | GPT-2 BPE, `train.bin` (~17 GB, ~9B tokens), `val.bin` (~8.5 MB, ~4M tokens) | ~54 GB cache + ~17 GB output |

> [!WARNING]
> `data/openwebtext/prepare.py` downloads roughly 54 GB into the HuggingFace
> cache and writes another ~17 GB of `train.bin`. It is single-shot but takes
> hours and a fast disk. Start with Shakespeare-char to confirm everything
> works.

### Step 5: Run the Shakespeare-char demo

```bash
python train.py config/train_shakespeare_char.py
```

You should see output like:

```text
tokens per iteration will be: 16,384
Initializing a new model from scratch
defaulting to vocab_size of GPT-2 to 50304 (50257 rounded up for efficiency)
number of parameters: 10.65M
num decayed parameter tensors: 26, with 10,727,424 parameters
num non-decayed parameter tensors: 13, with 4,992 parameters
using fused AdamW: True
compiling the model... (takes a ~minute)
step 0: train loss 4.2876, val loss 4.2823
iter 0: loss 4.2841, time 6829.40ms, mfu -100.00%
iter 10: loss 3.2447, time 23.65ms, mfu 5.46%
...
step 5000: train loss 0.6178, val loss 1.4974
```

When training finishes (~5,000 iters), a checkpoint will live at
`out-shakespeare-char/ckpt.pt`.

## 3. Configuration

nanoGPT uses a deliberately bare-bones configuration mechanism nicknamed the
"poor man's configurator." It lives in `configurator.py` and works like this:

1. `train.py`, `sample.py`, and `bench.py` declare default values as
   module-level globals.
2. They call `exec(open('configurator.py').read())`.
3. `configurator.py` walks `sys.argv[1:]`. If an arg has no `=`, it is treated
   as a Python config file and `exec()`'d into the calling module's globals.
   If an arg starts with `--key=value`, it overrides the matching global by
   parsing `value` with `ast.literal_eval`.
4. Any unknown key raises `ValueError: Unknown config key: {key}`.

### The override pattern

```bash
# config file only
python train.py config/train_shakespeare_char.py

# config file + CLI overrides (overrides win)
python train.py config/train_shakespeare_char.py --batch_size=32 --device=cpu

# CLI overrides only, all defaults from train.py
python train.py --batch_size=32 --compile=False
```

> [!IMPORTANT]
> Order matters. The config file is `exec()`'d **first**, then `--key=value`
> overrides are applied **after**. CLI flags always win over config-file
> values, which always win over the `train.py` defaults.

### Config file shape

A config file is just plain Python. It assigns values to the same names that
`train.py`/`sample.py`/`bench.py` declare as defaults. Example
(`config/train_shakespeare_char.py`):

```python
out_dir = 'out-shakespeare-char'
eval_interval = 250
eval_iters = 200
log_interval = 10

always_save_checkpoint = False
wandb_log = False
wandb_project = 'shakespeare-char'
wandb_run_name = 'mini-gpt'

dataset = 'shakespeare_char'
gradient_accumulation_steps = 1
batch_size = 64
block_size = 256

n_layer = 6
n_head = 6
n_embd = 384
dropout = 0.2

learning_rate = 1e-3
max_iters = 5000
lr_decay_iters = 5000
min_lr = 1e-4
beta2 = 0.99
warmup_iters = 100
```

### Most-used training settings

The full reference is in the Admin & Developer Guide. The settings below are
the ones users tune most often.

| Setting | Default in `train.py` | Meaning |
|---|---|---|
| `out_dir` | `'out'` | Checkpoint and log directory. Created if missing. |
| `dataset` | `'openwebtext'` | Folder name under `data/` containing `train.bin`/`val.bin`. |
| `init_from` | `'scratch'` | One of `'scratch'`, `'resume'`, `'gpt2'`, `'gpt2-medium'`, `'gpt2-large'`, `'gpt2-xl'`. |
| `batch_size` | `12` | Micro-batch size. With `gradient_accumulation_steps > 1` this is the per-step micro-batch. |
| `block_size` | `1024` | Context window in tokens. |
| `gradient_accumulation_steps` | `40` (5 × 8) | Number of micro-steps per optimizer step. |
| `n_layer` / `n_head` / `n_embd` | `12` / `12` / `768` | Model size (default GPT-2 124M). |
| `dropout` | `0.0` | `0.0` for pretraining, `~0.1+` for finetuning. |
| `bias` | `False` | Whether `nn.Linear` and `LayerNorm` use bias terms. |
| `learning_rate` | `6e-4` | Peak LR after warmup. |
| `min_lr` | `6e-5` | Floor LR after cosine decay. Karpathy recommends ≈`learning_rate / 10`. |
| `warmup_iters` | `2000` | Linear warmup iterations. |
| `lr_decay_iters` | `600000` | Iteration at which cosine decay reaches `min_lr`. Per Chinchilla, set ≈ `max_iters`. |
| `max_iters` | `600000` | Total training iterations. |
| `weight_decay` | `1e-1` | AdamW weight decay (only on 2D tensors). |
| `beta1` / `beta2` | `0.9` / `0.95` | AdamW betas. |
| `grad_clip` | `1.0` | Gradient norm clipping value. `0.0` disables. |
| `decay_lr` | `True` | Toggle the cosine LR scheduler. |
| `eval_interval` | `2000` | Iters between train/val loss evaluations + checkpoint write. |
| `eval_iters` | `200` | Number of batches averaged when computing eval loss. |
| `eval_only` | `False` | If `True`, run eval once and exit. |
| `always_save_checkpoint` | `True` | If `False`, only save when `val` loss improves. |
| `device` | `'cuda'` | `'cuda'`, `'cpu'`, `'cuda:0'`, `'mps'`. |
| `dtype` | `'bfloat16'` if supported else `'float16'` | Autocast dtype. `'float16'` activates `GradScaler` automatically. |
| `compile` | `True` | `torch.compile(model)` for ~10-30% speedup on PyTorch 2.0+. |
| `wandb_log` / `wandb_project` / `wandb_run_name` | `False` / `'owt'` / `'gpt2'` | Optional Weights & Biases logging. |
| `backend` | `'nccl'` | DDP backend (`'nccl'` for GPU, `'gloo'` for CPU/cross-platform). |

### Most-used sampling settings

| Setting | Default in `sample.py` | Meaning |
|---|---|---|
| `init_from` | `'resume'` | `'resume'` loads `out_dir/ckpt.pt`; `'gpt2*'` loads HuggingFace weights. |
| `out_dir` | `'out'` | Checkpoint directory (only used with `init_from='resume'`). |
| `start` | `'\n'` | Prompt. Use `"FILE:prompt.txt"` to read from a file. |
| `num_samples` | `10` | Number of samples to generate. |
| `max_new_tokens` | `500` | Tokens per sample. |
| `temperature` | `0.8` | `<1.0` more conservative, `>1.0` more random. |
| `top_k` | `200` | Truncate to top-k logits before sampling. |
| `seed` | `1337` | RNG seed. |
| `device` / `dtype` / `compile` | `'cuda'` / autodetect / `False` | Same semantics as `train.py`. |

> [!TIP]
> The sampling defaults are deliberately chatty (`num_samples=10`,
> `max_new_tokens=500`). Knock those down with
> `--num_samples=1 --max_new_tokens=100` for quick sanity checks.

## 4. Core Usage

The three scripts you will run day-to-day are `train.py`, `sample.py`, and
`bench.py`. They share the configurator, so everything below works with both
config-file syntax and `--key=value` overrides.

### Training (`train.py`)

`train.py` is a single while-loop that:

1. Resolves config (defaults → config file → CLI overrides).
2. Detects DDP from the `RANK` environment variable. If unset, runs single
   process.
3. Builds an `np.memmap` data loader over `data/{dataset}/train.bin` and
   `val.bin`.
4. Instantiates the model from one of three init paths: `'scratch'`,
   `'resume'`, or `'gpt2*'`.
5. Configures AdamW via `model.configure_optimizers` (2D tensors get weight
   decay, 1D tensors don't).
6. Optionally `torch.compile`s the model.
7. Wraps in DDP if `RANK >= 0`.
8. Runs the training loop with cosine LR scheduling, gradient accumulation,
   `bfloat16`/`float16` autocast, and optional `GradScaler` for `float16`.
9. Every `eval_interval` iterations, evaluates `train` and `val` loss over
   `eval_iters` batches and writes `ckpt.pt` if `val` improved (or always, if
   `always_save_checkpoint=True`).

### Single-GPU training

```bash
python train.py config/train_shakespeare_char.py
python train.py --batch_size=32 --compile=False
```

### Distributed training (DDP) — single node

```bash
torchrun --standalone --nproc_per_node=4 train.py config/train_gpt2.py
```

`torchrun` sets `RANK`, `LOCAL_RANK`, and `WORLD_SIZE` for each process.
`train.py` reads them, assigns each rank to `cuda:{LOCAL_RANK}`, picks the
rank-0 process as `master_process` (handles logging, eval, and checkpointing),
and divides `gradient_accumulation_steps` by `WORLD_SIZE`.

### Distributed training — multi-node

```bash
# master node (123.456.123.456)
torchrun --nproc_per_node=8 --nnodes=2 --node_rank=0 \
  --master_addr=123.456.123.456 --master_port=1234 train.py

# worker node
torchrun --nproc_per_node=8 --nnodes=2 --node_rank=1 \
  --master_addr=123.456.123.456 --master_port=1234 train.py
```

> [!NOTE]
> If your cluster lacks Infiniband, prepend `NCCL_IB_DISABLE=1` to the
> `torchrun` command on every node. This is documented in the docstring at
> the top of `train.py`.

### Resuming an interrupted run

```bash
python train.py config/train_gpt2.py --init_from=resume
```

`init_from='resume'` loads `out_dir/ckpt.pt`, restores `iter_num` and
`best_val_loss`, and forces `n_layer`, `n_head`, `n_embd`, `block_size`,
`bias`, `vocab_size` to match the checkpoint (other args like `dropout` can
still be tuned).

> [!IMPORTANT]
> Keep `out_dir` stable when resuming. The script reads `{out_dir}/ckpt.pt`
> directly. If you change `out_dir`, copy the checkpoint over first.

### Sampling (`sample.py`)

```bash
# from a checkpoint you trained
python sample.py --out_dir=out-shakespeare-char

# from a HuggingFace GPT-2 checkpoint
python sample.py --init_from=gpt2-xl

# with a prompt
python sample.py --out_dir=out-shakespeare-char --start="ROMEO:"

# from a prompt file
python sample.py --out_dir=out --start="FILE:prompt.txt" --num_samples=3
```

If the checkpoint's training config used a `dataset` with a `meta.pkl`
(character-level), `sample.py` loads `data/{dataset}/meta.pkl` and uses its
`stoi`/`itos` mapping. Otherwise it falls back to the GPT-2 BPE encoder via
`tiktoken.get_encoding("gpt2")`.

### Benchmarking (`bench.py`)

`bench.py` is a stripped-down `train.py` that runs a fixed number of forward,
backward, and optimizer steps and reports time-per-iteration plus model FLOPs
utilization (MFU).

```bash
python bench.py
python bench.py --compile=False --real_data=False --profile=True
```

| `bench.py` setting | Default | Meaning |
|---|---|---|
| `batch_size` | `12` | Same as in `train.py`. |
| `block_size` | `1024` | Same as in `train.py`. |
| `bias` | `False` | Same as in `train.py`. |
| `real_data` | `True` | If `False`, generate random `int` tensors instead of memmapping `train.bin`. |
| `seed` | `1337` | RNG seed. |
| `device` | `'cuda'` | Same as in `train.py`. |
| `dtype` | autodetect | Same as in `train.py`. |
| `compile` | `True` | Same as in `train.py`. |
| `profile` | `False` | If `True`, run `torch.profiler` and emit traces to `./bench_log` for TensorBoard. |

### Mixed precision — `bfloat16` vs `float16`

`train.py` autodetects: if the GPU supports bfloat16
(`torch.cuda.is_bf16_supported()`), it picks `'bfloat16'`; otherwise
`'float16'`. The difference matters:

- **bfloat16**: same exponent range as `float32`, no `GradScaler` needed. The
  cleaner default on Ampere (A100, A6000), Hopper (H100), and Ada (RTX 4090).
- **float16**: narrower exponent range, requires `torch.cuda.amp.GradScaler`,
  which `train.py` enables automatically when `dtype == 'float16'`.

> [!WARNING]
> If you force `--dtype=bfloat16` on a card that doesn't support it (anything
> pre-Ampere), training will silently fall back to slow paths or NaN out.
> Either let the autodetect run or explicitly set `--dtype=float16`.

### Checkpoint contents

A checkpoint is a Python dict pickled by `torch.save`:

```python
{
    'model':         state_dict,    # raw_model.state_dict()
    'optimizer':     state_dict,    # optimizer.state_dict()
    'model_args':    dict,          # n_layer, n_head, n_embd, block_size, bias, vocab_size, dropout
    'iter_num':      int,           # last completed iter
    'best_val_loss': float,         # for early stopping decisions
    'config':        dict,          # all of train.py's config_keys snapshot
}
```

Checkpoints land at `{out_dir}/ckpt.pt`. There is one checkpoint per
`out_dir` — the file is overwritten each save.

## 5. Common Tasks

### Task 1: Train the Shakespeare-char baby GPT in under 10 minutes

```bash
python data/shakespeare_char/prepare.py
python train.py config/train_shakespeare_char.py
python sample.py --out_dir=out-shakespeare-char --start="JULIET:"
```

This trains a 6-layer / 6-head / 384-embed model (~10.65M params) for 5,000
iterations at `batch_size=64`, `block_size=256`, character-level vocab
(65 unique chars). Final val loss lands around 1.5, which produces
recognizable mock-Shakespeare.

### Task 2: Finetune a public GPT-2 checkpoint on Shakespeare

```bash
python data/shakespeare/prepare.py
python train.py config/finetune_shakespeare.py
python sample.py --out_dir=out-shakespeare --start="To be or not to be,"
```

`config/finetune_shakespeare.py` sets `init_from='gpt2-xl'`,
`learning_rate=3e-5`, `decay_lr=False`, `max_iters=20`,
`gradient_accumulation_steps=32`, `always_save_checkpoint=False`. It only
saves when validation improves and finetunes at constant LR. Swap
`init_from` for `'gpt2'`, `'gpt2-medium'`, or `'gpt2-large'` if `gpt2-xl` is
too big for your GPU.

> [!TIP]
> For finetuning, bump `dropout` from `0.0` to `0.1` or so. The default
> `train.py` config sets it to `0.0` because that's optimal for pretraining,
> but a small amount of regularization helps when training on tiny corpora.

### Task 3: Sample from a HuggingFace GPT-2 checkpoint with no training

```bash
python sample.py \
  --init_from=gpt2-xl \
  --start="What is the answer to life, the universe, and everything?" \
  --num_samples=5 --max_new_tokens=100
```

`sample.py` notices `init_from.startswith('gpt2')`, calls
`GPT.from_pretrained('gpt2-xl', dict(dropout=0.0))`, and that helper pulls
weights from HuggingFace via `transformers.GPT2LMHeadModel.from_pretrained`,
then maps them onto nanoGPT's vanilla `nn.Linear` layout (transposing the
GPT-2 `Conv1D` weights).

### Task 4: Pretrain GPT-2 124M on OpenWebText across 8 GPUs

```bash
python data/openwebtext/prepare.py
torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py
```

`config/train_gpt2.py` is the reference script for "GPT-2 124M down to a very
nice loss of ~2.85 on 1 node of 8X A100 40GB" (≈5 days). It sets
`batch_size=12`, `block_size=1024`, `gradient_accumulation_steps=40`
(5 × 8 GPUs), `max_iters=600000`, `lr_decay_iters=600000`,
`weight_decay=1e-1`, with wandb logging on under project `'owt'`.

### Task 5: Log to Weights & Biases

```bash
pip install wandb
wandb login
python train.py config/train_gpt2.py --wandb_log=True \
  --wandb_project=my-experiments --wandb_run_name=run-001
```

When `wandb_log=True` and the process is `master_process`, `train.py` calls
`wandb.init` and logs `iter`, `train/loss`, `val/loss`, `lr`, and `mfu` at
each `eval_interval`.

> [!CAUTION]
> Never commit your wandb API key to source control. `wandb login` writes it
> to `~/.netrc`. If you must run on a shared box, use `wandb login --relogin`
> in your shell, not in a config file.

### Task 6: Scale the model to a custom dataset

```bash
mkdir -p data/mydata
# write your own data/mydata/prepare.py that produces train.bin / val.bin

python data/mydata/prepare.py

python train.py \
  --dataset=mydata \
  --out_dir=out-mydata \
  --n_layer=8 --n_head=8 --n_embd=512 \
  --block_size=512 \
  --batch_size=32 --gradient_accumulation_steps=4 \
  --learning_rate=3e-4 --min_lr=3e-5 \
  --max_iters=20000 --lr_decay_iters=20000 \
  --warmup_iters=200 \
  --eval_interval=500 --eval_iters=100
```

`train.py` reads `data/{dataset}/train.bin` and `val.bin` directly. If you
also drop a `meta.pkl` with `vocab_size`, the training run picks it up
automatically; otherwise nanoGPT defaults to `vocab_size=50304` (the GPT-2
vocabulary padded up from 50,257 to the nearest multiple of 64 for kernel
efficiency).

### Task 7: Prepare a custom dataset

The Shakespeare-char `prepare.py` is the cleanest template. The contract
`train.py` expects is two NumPy `uint16` files:

```python
import numpy as np

# encode your text however you want - char-level, BPE, sentencepiece, etc.
train_ids = ...  # list[int]
val_ids   = ...  # list[int]

train_ids = np.array(train_ids, dtype=np.uint16)
val_ids   = np.array(val_ids,   dtype=np.uint16)
train_ids.tofile('data/mydata/train.bin')
val_ids.tofile('data/mydata/val.bin')

# optional but recommended for char-level vocabs
import pickle
with open('data/mydata/meta.pkl', 'wb') as f:
    pickle.dump({'vocab_size': vocab_size, 'stoi': stoi, 'itos': itos}, f)
```

> [!NOTE]
> `np.uint16` caps your vocab at 65,536. The GPT-2 BPE vocab (50,257) and
> common BPE sizes fit comfortably. If you need a vocab >65K, switch to
> `np.uint32` and update the `dtype=` arguments in `train.py`'s `get_batch`
> and `bench.py`'s `get_batch` to match.

### Task 8: Recover or resume an interrupted run

If your training crashed or you Ctrl-C'd it:

```bash
python train.py config/train_gpt2.py --init_from=resume
```

`init_from='resume'` reads `{out_dir}/ckpt.pt`, restores `iter_num` and
`best_val_loss`, and continues. The optimizer state is also restored, so
the LR schedule picks up exactly where it left off.

> [!WARNING]
> If you change `n_layer`, `n_head`, `n_embd`, `block_size`, `bias`, or
> `vocab_size` between runs, `--init_from=resume` will overwrite your CLI
> overrides with the checkpoint's values to keep the architecture compatible.
> Other knobs like `dropout` or `learning_rate` can still be tuned freely.

## 6. Troubleshooting

### Error: `RuntimeError: Dataset scripts are no longer supported`

**Cause:** As of recent HuggingFace `datasets` releases, the `openwebtext`
dataset stopped publishing as a scripted dataset. This is tracked upstream
in GitHub issue #633.

**Solution:**

1. Pin an older `datasets` version: `pip install "datasets<2.20"`.
2. Or use a community mirror that still ships the parquet shards directly.
3. Or replace `data/openwebtext/prepare.py` with a hand-rolled tokenizer over
   any large web-text corpus you can supply yourself — the only contract
   `train.py` cares about is `train.bin` / `val.bin` as `uint16` token files.

**Prevention:** This is a moving target on the HuggingFace side. If you must
reproduce GPT-2 124M end-to-end, lock your `datasets` version in a venv.

### Error: `ValueError: Unknown config key: {key}`

**Cause:** You passed `--{key}=value` on the command line (or set it in a
config file) but `{key}` is not a global declared in `train.py`/`sample.py`/
`bench.py`. The configurator deliberately rejects typos so a misspelled flag
doesn't silently no-op.

**Solution:** Double-check the spelling against the tables in Section 3 of
this guide. Common gotchas: `lr` (the right name is `learning_rate`),
`epochs` (nanoGPT uses `max_iters`, not epochs), `dropout_rate` (the right
name is `dropout`).

**Prevention:** When in doubt, scan the `# default config values` block at
the top of `train.py` (or `sample.py` / `bench.py`) — the configurator only
recognizes names declared there.

### Error: `WARNING: using slow attention. Flash Attention requires PyTorch >= 2.0`

**Cause:** `model.py:CausalSelfAttention.__init__` checks
`hasattr(torch.nn.functional, 'scaled_dot_product_attention')`. On
PyTorch < 2.0 that attribute is missing, so the model falls back to a manual
QK^T softmax with a registered triangular causal mask buffer.

**Solution:** Upgrade to PyTorch 2.0 or newer:

```bash
pip install --upgrade torch
```

**Prevention:** PyTorch 2.0 has been stable for a while; pin
`torch>=2.0` in any requirements file you maintain downstream.

### Error: Training NaNs out within a few iterations

**Cause:** Common in `float16` autocast when gradients overflow before
`GradScaler` adapts; or when the learning rate is too high; or when
`gradient_accumulation_steps` is set so high that loss scaling drifts. Issue
#524 (Pretraining Divergence) is the long-running upstream thread for
discussion of nanoGPT NaN modes.

**Solution:**

1. Switch to `--dtype=bfloat16` if your GPU supports it (Ampere+).
2. Reduce `learning_rate` (try halving it).
3. Lower `gradient_accumulation_steps` until the issue stops, then bisect.
4. Verify `grad_clip=1.0` is in effect (the default).

**Prevention:** On Ampere/Hopper/Ada GPUs, prefer `bfloat16`. The Karpathy
default already does this when bfloat16 is supported.

### Error: `CUDA out of memory` during training

**Cause:** Your model + activations + optimizer state + gradients exceed
GPU memory.

**Solution:**

1. Lower `batch_size` and raise `gradient_accumulation_steps` proportionally
   to keep effective batch size constant.
2. Lower `block_size` (context length is quadratic in attention memory).
3. Drop `compile=True` — `torch.compile` increases peak memory during
   compilation.
4. Switch to a smaller model: lower `n_layer`, `n_embd`, or both.

**Prevention:** Use `bench.py --profile=True` on your target hardware before
launching the long run — the resulting traces under `./bench_log` show peak
GPU allocation per step.

### Error: Sampling outputs are gibberish even though training loss is low

**Cause 1:** Mismatched encoder. If you trained with character-level
encoding (`shakespeare_char` / a custom `meta.pkl`) but `sample.py` defaulted
to the GPT-2 BPE encoder, every "char" gets re-encoded as several BPE
tokens of unrelated semantics.

**Cause 2:** You passed `--init_from=gpt2*` to `sample.py` but pointed
`--out_dir` at a checkpoint trained from scratch. `init_from='gpt2*'`
ignores `out_dir`.

**Solution:**

1. For Cause 1: confirm `data/{dataset}/meta.pkl` exists and that
   `sample.py`'s `Loading meta from ...` print appears at startup.
2. For Cause 2: pick exactly one of `--init_from=resume --out_dir=...` or
   `--init_from=gpt2-xl`. They're mutually exclusive.

**Prevention:** Always sanity-check the first two lines of `sample.py`
output — they tell you exactly which encoder and which weights are being
used.

### Error: `connection refused` / DDP processes hang on multi-node setup

**Cause 1:** `master_addr` unreachable from worker nodes (firewall, wrong
IP).

**Cause 2:** Cluster lacks Infiniband and NCCL is trying to use it.

**Solution:**

1. Verify worker → master TCP reachability on `master_port`.
2. Prepend `NCCL_IB_DISABLE=1` to every `torchrun` invocation:
   `NCCL_IB_DISABLE=1 torchrun --nproc_per_node=8 ...`
3. For deeper diagnostics, set `NCCL_DEBUG=INFO`.

**Prevention:** Test single-node DDP first
(`torchrun --standalone --nproc_per_node=2 train.py`) before scaling out.

### Diagnostic decision tree

1. **Did the install verify?** `python -c "import torch; print(torch.__version__, torch.cuda.is_available())"` — if not, fix PyTorch first.
2. **Does `python data/shakespeare_char/prepare.py` produce `train.bin`?** If not, it's a network or write-permission issue.
3. **Does `python train.py config/train_shakespeare_char.py` make it past `step 0`?** If not, see the OOM and NaN sections above.
4. **Does `python sample.py --out_dir=out-shakespeare-char` produce readable output?** If not, see the gibberish-sampling section.

If all four pass, your installation is healthy and any failures from there
are configuration- or data-specific.

## 7. FAQ

**Q: Is nanoGPT a Python package I can `pip install`?**

No. nanoGPT is a clone-and-run repository. There is no `setup.py`, no
`pyproject.toml`, no PyPI release. You `git clone`, `pip install` the
PyTorch ecosystem manually, and run `python train.py` from the repo root.

**Q: What is the smallest GPU I can train GPT-2 124M on?**

`config/train_gpt2.py` was tuned for 8 × A100 40GB. To train 124M on a
single GPU you'll need to keep the effective batch size by raising
`gradient_accumulation_steps` and lowering `batch_size`. Even with that,
expect dozens of days of compute. The Shakespeare-char demo is the realistic
single-GPU and single-CPU target for hobbyists.

**Q: Why `vocab_size=50304` instead of the actual GPT-2 size of 50257?**

`train.py` rounds 50257 up to the nearest multiple of 64 (= 50304) because
"50304 % 64 == 0" lets the GPU run faster matmul kernels. The model wastes
47 unused embedding rows but in exchange every Linear layer hits an
efficient tiling. The `from_pretrained` path forces `vocab_size=50257`
because the OpenAI GPT-2 checkpoints were trained with that size — but the
default scratch path bumps it back up to 50304.

**Q: How do I disable `torch.compile`?**

Pass `--compile=False` on the CLI, or set `compile = False` in your config
file. `torch.compile` requires PyTorch 2.0 and adds a ~1-minute warmup. On
small models or quick experiments, it's often slower overall.

**Q: Why does the validation loss differ slightly between runs even with
fixed seed?**

`torch.manual_seed(1337 + seed_offset)` is called once at startup, but
non-determinism creeps in via cuDNN, async CUDA streams, and `torch.compile`
fusions. For bit-exact reproducibility, also set
`torch.use_deterministic_algorithms(True)` — but expect a slowdown.

**Q: Where do I look for the model architecture?**

Everything lives in `model.py` (~330 lines):

- `LayerNorm` — line 18
- `CausalSelfAttention` — line 29
- `MLP` — line 78
- `Block` — line 94
- `GPTConfig` (dataclass) — line 109
- `GPT` — line 118 (with `from_pretrained`, `configure_optimizers`,
  `estimate_mfu`, `crop_block_size`, `generate`)

**Q: Can I train on Apple Silicon?**

Yes for the small demos. Set `--device=mps --dtype=float16 --compile=False`
on the `train.py` and `sample.py` commands. `torch.compile` on MPS is
unstable, and `bfloat16` on MPS isn't supported on all releases.

**Q: How do I make sampling deterministic?**

`sample.py` already calls `torch.manual_seed(seed)` and
`torch.cuda.manual_seed(seed)` with `seed=1337` by default. Override with
`--seed=42` if you want a fresh stream. For top-p (nucleus) sampling, pass
`--top_k=None` and adjust `--temperature` instead — there is no native
top-p flag.

**Q: Where are the GPT-2 weights downloaded to?**

`GPT.from_pretrained` calls `transformers.GPT2LMHeadModel.from_pretrained`
which uses HuggingFace's standard cache (`~/.cache/huggingface/hub/` by
default, overridable via `HF_HOME`).

**Q: Is there an HTTP API or a CLI binary?**

No. nanoGPT is intentionally a script-only repo. There are no exposed
endpoints, no `argparse` CLI, no installable entry points. Every interaction
is `python train.py …` or `python sample.py …` from the repo root.

**Q: How do I cite nanoGPT?**

Per the LICENSE in the repo (MIT), you don't have to. If you want to,
referencing the GitHub URL and commit hash is the standard practice for
research code without a published paper.

---

_Generated April 2026 from karpathy/nanoGPT at 3adf61e. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._