nanochat User Guide

nanochat is the simplest experimental harness for training your own ChatGPT-style large language model (LLM) end to end on a single GPU node. One codebase carries you through every stage — tokenization, pretraining, fine-tuning, reinforcement learning, evaluation, efficient inference, and a ChatGPT-like web UI. You can train a GPT-2-capability model for roughly $48 in compute (~2 hours on an 8×H100 node) and then chat with it in your browser.

This guide shows you how to install nanochat, run the headline "speedrun" to build and talk to your own model, drive every command-line script, and troubleshoot the most common problems. It is written entirely from the source code at the documented commit, so every command, flag, and default reflects what the code actually does.

Note

nanochat is not a configurable LLM framework. It is a single, minimal, readable, hackable codebase designed to be forked and modified. There are no plugin systems, model factories, or giant config objects — just Python scripts you run in sequence.

Table of Contents

1. Quick Start
2. What You Need
3. Installation
4. The One Dial You Control: --depth
5. The Full Pipeline at a Glance
6. Everyday Usage: The Scripts
7. Talking to Your Model
8. Common Workflows
9. Tuning for Your Hardware
10. How It Works (Concepts)
11. Troubleshooting
12. FAQ
13. Command & Flag Reference

1. Quick Start

If you have a freshly booted 8×H100 GPU node, the entire pipeline lives in a single script. This trains a GPT-2-grade model and prepares it for chat in about three hours.

# 1. Clone the repository
git clone https://github.com/karpathy/nanochat.git
cd nanochat

# 2. Run the full speedrun (setup + tokenizer + pretrain + SFT + eval)
bash runs/speedrun.sh

Because the run takes hours, launch it inside a screen session so it survives a dropped SSH connection:

screen -L -Logfile runs/speedrun.log -S speedrun bash runs/speedrun.sh

When it finishes, serve the chat web UI and open the printed URL in your browser:

source .venv/bin/activate
python -m scripts.chat_web

Starting NanoChat Web Server
Temperature: 0.8, Top-k: 50, Max tokens: 512
Server ready at http://localhost:8000

Tip

On a cloud box, connect using the node's public IP and the port, e.g. http://209.20.xxx.xxx:8000/. localhost only works if you are on the machine itself or have tunneled the port.

Important

A speedrun-grade model is a ~4×10¹⁹ FLOPs capability model — "a bit like talking to a kindergartener." It can answer simple factual questions and will happily hallucinate. This is expected; nanochat exists to teach the full training stack, not to compete with frontier models.

2. What You Need

Hardware

Warning

Training real LLMs requires GPU compute and money. You will not get far on a MacBook. Treat CPU/MPS runs as a way to exercise the code paths, not to produce a usable model.

Caution

If your GPUs have less than 80 GB of VRAM, the default settings will run out of memory (OOM). Reduce --device-batch-size (from 32 down to 16, 8, 4, 2, or 1) until training fits. See Tuning for Your Hardware.

Software

• Python ≥ 3.10 (the project pins torch==2.9.1, targeting CUDA 12.8 or CPU).
• uv for dependency management (the setup scripts install it automatically if missing).
• A Linux environment for GPU training. macOS is supported only for the small CPU/MPS demo.

3. Installation

nanochat uses uv for dependency management. Pick the install method that matches your hardware.

CUDA (A100/H100/etc.)

uv sync --extra gpu
source .venv/bin/activate

CPU-only / Apple Silicon (MPS)

uv sync --extra cpu
source .venv/bin/activate

Development (adds test/plotting tools)

uv sync --extra gpu --group dev
source .venv/bin/activate

The cpu and gpu extras are mutually exclusive — they pull torch from different package indexes (download.pytorch.org/whl/cpu vs. .../cu128). Pick exactly one.

Note

If uv is not installed, the setup scripts run this for you:

curl -LsSf https://astral.sh/uv/install.sh | sh

You can also run it yourself before uv sync.

The dev group adds pytest, matplotlib, ipykernel, python-dotenv, and transformers (the last is needed to evaluate HuggingFace baselines like GPT-2).

Verify your install

After activating the virtual environment, confirm the package imports and check which device and precision nanochat will auto-detect:

python -c "import nanochat, torch; from nanochat.common import COMPUTE_DTYPE, COMPUTE_DTYPE_REASON; print(torch.__version__); print(COMPUTE_DTYPE, '-', COMPUTE_DTYPE_REASON)"

2.9.1+cu128
torch.bfloat16 - auto-detected: CUDA SM 90 (bf16 supported)

4. The One Dial You Control: --depth

The single most important concept in nanochat is that you describe the model you want with one integer: --depth, the number of transformer layers.

Everything else — the model width, number of attention heads, learning rates, training horizon (how many tokens to train on), batch size, and weight decay — is calculated automatically to produce a compute-optimal model. You ask for a bigger or smaller model; the code figures out the rest.

# A 12-layer model — the author's favorite scale for quick experiments
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- --depth=12

Tip

When you change something in the code and want to know if it helped, re-run a d12 (or d16) and compare. It's the fastest meaningful signal.

Important

model_dim = depth × 64 (the --aspect-ratio, default 64), rounded up to a clean multiple of the attention head dimension. You normally never set the width directly — increasing --depth widens the model for you.

5. The Full Pipeline at a Glance

nanochat builds a chat model in stages. Each stage is one script you run, and each consumes the artifacts of the previous one. All intermediate files (datasets, tokenizer, checkpoints, reports) are written under a single base directory (default ~/.cache/nanochat).

   download data        train tokenizer        PRETRAIN              FINE-TUNE            (optional) RL          CHAT
 ┌────────────────┐   ┌────────────────┐   ┌──────────────┐   ┌──────────────────┐   ┌──────────────┐   ┌──────────────┐
 │ nanochat.dataset│──▶│ scripts.tok_train│─▶│scripts.base_ │──▶│ scripts.chat_sft │──▶│scripts.chat_rl│──▶│scripts.chat_ │
 │   (parquet      │   │  (BPE, GPT-4     │   │  train       │   │ (teach chat,     │   │ (GRPO on      │   │  cli / web   │
 │    shards)      │   │   style)         │   │ (base model) │   │  tools, MC)      │   │  GSM8K math)  │   │              │
 └────────────────┘   └────────────────┘   └──────────────┘   └──────────────────┘   └──────────────┘   └──────────────┘
        │                                          │                    │                     │
        │                                    base_eval             chat_eval             chat_eval
        │                                  (CORE, bpb,           (ARC, MMLU,            (re-measure)
        ▼                                   samples)              GSM8K, ...)
  tok_eval (compression)

A checkpoint produced by one stage feeds the next via a model source:

• base — the pretrained base model (output of base_train).
• sft — the supervised-fine-tuned chat model (output of chat_sft).
• rl — the reinforcement-learned chat model (output of chat_rl).

Chat and evaluation scripts select which one to load with -i/--source (e.g. -i sft).

Note

All stage outputs are organized by a model tag, which defaults to d<depth> (e.g. d20). If you run multiple experiments, pass --model-tag to keep them separate.

6. Everyday Usage: The Scripts

All scripts are run as Python modules from the repository root. For multi-GPU training, prefix with torchrun and note the -- separator that divides torchrun's own arguments from the script's arguments:

torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- --depth=24 --fp8
#                                                              ^^ everything after -- goes to base_train

For single-GPU (or CPU/MPS) runs, drop torchrun entirely:

python -m scripts.base_train --depth=24

Important

The -- after the module name is required with torchrun. Without it, your --depth, --run, etc. are swallowed by torchrun instead of reaching the script.

6.1 Download the pretraining data — nanochat.dataset

Downloads pretraining shards (parquet files) from the ClimbMix-400B dataset on demand. The validation shard is always downloaded automatically in addition to the train shards you request.

# Download 8 shards (~2B characters, enough to train the tokenizer)
python -m nanochat.dataset -n 8

# Download ~170 shards (enough for GPT-2 capability pretraining)
python -m nanochat.dataset -n 170

Downloading 9 shards using 4 workers...
Target directory: /root/.cache/nanochat/base_data_climbmix
...
Done! Downloaded: 9/9 shards to /root/.cache/nanochat/base_data_climbmix

Each shard is ~250M characters (~100 MB compressed). Downloads are resumable — already-present shards are skipped, and failed downloads retry with exponential backoff.

Tip

You can kick off a large download in the background while the tokenizer trains, exactly as the speedrun does:

python -m nanochat.dataset -n 170 &
python -m scripts.tok_train
wait

6.2 Train the tokenizer — scripts.tok_train

Trains a byte-pair-encoding (BPE) tokenizer in the style of the GPT-4 tokenizer, using the Rust-backed rustbpe library. Saves to ~/.cache/nanochat/tokenizer/.

python -m scripts.tok_train --max-chars=2000000000 --vocab-size=32768

After training, the script runs a sanity round-trip (encode → decode must match) and caches a token-bytes table used later for the bits-per-byte metric.

6.3 Evaluate the tokenizer — scripts.tok_eval

Reports your tokenizer's compression ratio against the GPT-2 and GPT-4 tokenizers across several text types (news, Korean, code, math, science, and held-out training/validation text). Takes no arguments.

python -m scripts.tok_eval

Vocab sizes:
GPT-2: 50257
GPT-4: 100277
Ours: 32768

Comparison with GPT-2:
Text Type  Bytes    Tokens  Ratio   Tokens  Ratio   Diff %
news       ...      ...     ...     ...     ...     +x.x%   Ours
...

Higher compression ratio (more bytes per token) is better.

6.4 Pretrain the base model — scripts.base_train

The compute-heavy stage. Trains the GPT transformer on the pretraining data. This is what the leaderboard measures.

# The speedrun's pretraining command (GPT-2 grade, d24, with FP8)
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=24 --target-param-data-ratio=8 --device-batch-size=16 --fp8 --run=my_run

step 00000/21500 (0.00%) | loss: 10.981 | lrm: 0.03 | dt: 412.55ms | tok/sec: 2,541,872 | bf16_mfu: 41.20 | ...
...
Step 21500 | Validation bpb: 0.748331
Step 21500 | CORE metric: 0.2585

This script has the largest set of flags. The most important ones:

See the full list in the Command & Flag Reference.

Tip

The training horizon is chosen by precedence: --num-iterations (if set) → --target-flops (if set) → --target-param-data-ratio (the common default). You only set one.

6.5 Evaluate the base model — scripts.base_eval

Evaluates a base model on three things (any subset, comma-separated via --eval): the CORE metric, bits per byte (bpb) on train/val splits, and text samples.

# Evaluate your nanochat d24 model on 8 GPUs
torchrun --standalone --nproc_per_node=8 -m scripts.base_eval -- --model-tag d24 --device-batch-size=16

# Evaluate the real GPT-2 (124M) baseline for comparison (requires the dev group)
torchrun --nproc_per_node=8 -m scripts.base_eval -- --hf-path openai-community/gpt2

# Quick approximate single-GPU evaluation
python -m scripts.base_eval --model-tag d24 --device-batch-size=16 --max-per-task=100 --split-tokens=524288

The CORE results are written to a CSV under ~/.cache/nanochat/base_eval/.

6.6 Fine-tune for chat — scripts.chat_sft

Supervised fine-tuning (SFT) teaches the base model the chat format: conversation special tokens, tool use, and multiple-choice answering. It trains on a mixture that includes SmolTalk conversations, synthetic identity conversations, MMLU, GSM8K, and spelling tasks.

torchrun --standalone --nproc_per_node=8 -m scripts.chat_sft -- --device-batch-size=16 --run=my_run

Note

Most SFT hyperparameters (batch sizes, learning rates) are inherited from the pretrained checkpoint unless you override them. The console logs each value it inherited or overrode.

The identity conversations give your model its "personality." The speedrun downloads a ready-made set:

curl -L -o $NANOCHAT_BASE_DIR/identity_conversations.jsonl \
  https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonl

6.7 Evaluate the chat model — scripts.chat_eval

Runs the chat model against a suite of tasks: ARC-Easy, ARC-Challenge, MMLU, GSM8K, HumanEval, and SpellingBee. When all tasks run, it also reports the ChatCORE metric (mean centered accuracy, 0 = random baseline, 1 = perfect).

# Evaluate the SFT model on all tasks
torchrun --nproc_per_node=8 -m scripts.chat_eval -- -i sft

# Evaluate a single task
python -m scripts.chat_eval -i sft -a ARC-Easy

# Multiple specific tasks (pipe-separated)
python -m scripts.chat_eval -i sft -a "MMLU|GSM8K"

6.8 Reinforcement learning (optional) — scripts.chat_rl

Improves math ability via a simplified GRPO/REINFORCE loop on GSM8K. This stage is optional and is not part of the default speedrun.

torchrun --standalone --nproc_per_node=8 -m scripts.chat_rl -- --run=my_run

The RL model is saved under the rl source, which you then load with chat_cli -i rl, chat_web -i rl, or chat_eval -i rl.

7. Talking to Your Model

Once you have an SFT (or RL) checkpoint, there are two ways to chat.

7.1 Command line — scripts.chat_cli

A simple interactive REPL in your terminal. Intended for single-GPU use.

# Interactive mode
python -m scripts.chat_cli

# Single-shot: ask one question, get one answer, exit
python -m scripts.chat_cli -p "What is the capital of France?"

NanoChat Interactive Mode
--------------------------------------------------
Type 'quit' or 'exit' to end the conversation
Type 'clear' to start a new conversation
--------------------------------------------------

User: Why is the sky blue?

Assistant: The sky appears blue because ...

In interactive mode, type clear to reset the conversation or quit/exit to leave.

Tip

nanochat models sometimes respond better if you greet them ("Hi") before asking your question. Try "What is the capital of France?" — a working speedrun model should answer Paris, and may even know the sky is blue.

7.2 Web UI — scripts.chat_web

A ChatGPT-style browser interface served by FastAPI. It distributes requests across multiple GPUs (one model replica per GPU).

# Single GPU (default)
python -m scripts.chat_web

# Spread across 4 GPUs
python -m scripts.chat_web --num-gpus 4

Open the printed URL in your browser. The server exposes:

Caution

The web server has no authentication and binds to 0.0.0.0 (all interfaces) by default. On a public cloud box, anyone who can reach the port can use your model. See the Admin Guide for hardening guidance before exposing it.

8. Common Workflows

8.1 Train and talk to a GPT-2-grade model (the speedrun)

This is the recommended first experience. Everything is in one script.

• [ ] Boot a fresh 8×H100 (or 8×A100) node.
• [ ] git clone the repo and cd into it.
• [ ] bash runs/speedrun.sh (ideally inside screen).
• [ ] Wait ~3 hours; watch runs/speedrun.log if you used screen logging.
• [ ] source .venv/bin/activate && python -m scripts.chat_web.
• [ ] Open the public URL and chat.

Internally the speedrun runs, in order: data download → tokenizer train/eval → base_train --depth=24 --fp8 → base_eval → download identity data → chat_sft → chat_eval -i sft → generate the run report.

8.2 Run a quick experiment (research loop)

When you want to test a code change quickly, a d12 model pretrains in ~5 minutes on 8 GPUs:

OMP_NUM_THREADS=1 torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=12 \
    --run="d12" \
    --model-tag="d12" \
    --core-metric-every=999999 \
    --sample-every=-1 \
    --save-every=-1

This logs to wandb (run name d12), evaluates CORE only on the final step, and skips intermediate sampling and checkpointing. Watch val_bpb and core_metric in wandb to judge whether your change helped.

8.3 Try it on a Mac / CPU (no GPU)

runs/runcpu.sh shrinks the model dramatically so a tiny run completes in a few tens of minutes. It trains a 4–6 layer model, runs a short SFT, and lets you chat.

bash runs/runcpu.sh

Warning

Results will be poor — this exists to exercise the code, not to produce a usable model. You may prefer to copy-paste the commands from the script one at a time.

8.4 Give your model a custom personality

Personality comes from the synthetic identity conversations mixed into SFT. To customize it, generate your own conversations (see dev/gen_synthetic_data.py) and point SFT at your file via the identity-conversations data path. The author's guides in the repo Discussions walk through this end to end.

8.5 Resume an interrupted pretraining run

If a long run was interrupted and you saved checkpoints (--save-every), resume from the last saved step:

torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=24 --model-tag=d24 --resume-from-step=10000

This reloads the model weights, optimizer state, and dataloader position from the checkpoint.

9. Tuning for Your Hardware

The single most common adjustment is fitting training into your VRAM. The lever is --device-batch-size.

# Default is 32. If you OOM, step it down:
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- --depth=24 --device-batch-size=16
# still OOM? try 8, then 4, 2, 1

Reducing --device-batch-size does not change your effective batch size or results — nanochat compensates with gradient accumulation to reach the same total batch size. It only changes how much memory each forward/backward pass uses (and makes each step a little slower).

Tip

The runs/miniseries.sh and runs/scaling_laws.sh scripts already encode sensible --device-batch-size reductions at larger depths (e.g. 16 for depth ≥ 20, 8 for depth ≥ 28). Use them as a reference.

Precision (dtype)

nanochat manages precision explicitly through a single global compute dtype, auto-detected from your hardware:

Override with the NANOCHAT_DTYPE environment variable:

NANOCHAT_DTYPE=float32 python -m scripts.chat_cli -p "hello"          # force fp32
NANOCHAT_DTYPE=bfloat16 torchrun --nproc_per_node=8 -m scripts.base_train  # force bf16

Note

Flash Attention 3 (the fast attention path) only activates on Hopper (H100/H200) GPUs running in bf16. On other GPUs nanochat automatically falls back to PyTorch's SDPA attention — correct, just slower. On the SDPA fallback, prefer --window-pattern L (full context) for best utilization.

Single GPU

Omit torchrun and run with plain python. The code detects the single-process setting and uses gradient accumulation. Expect roughly 8× the wall-clock time of an 8-GPU node.

10. How It Works (Concepts)

You don't need this section to use nanochat, but it helps you read the logs and tune intelligently.

Tokenizer. A GPT-4-style BPE tokenizer (vocab 32768) trained on your data with the Rust rustbpe library, then used at inference via tiktoken for speed.

Base model (pretraining). A from-scratch GPT transformer with rotary positional embeddings, QK-norm, ReLU² MLPs, untied embedding/unembedding weights, Group-Query Attention, and a sliding-window attention pattern. It learns to predict the next token on a large web text corpus (ClimbMix). The headline quality metric is the DCLM CORE score; the loss is reported as bits per byte (bpb), which is invariant to vocabulary size.

Compute-optimal scaling. From your --depth, nanochat derives the model width and uses scaling-law relationships (Chinchilla-style data:param ratio, batch-size and learning-rate scaling from the Power Lines and μP literature) to set the training horizon and hyperparameters automatically.

SFT (chat fine-tuning). Teaches the model the conversation protocol using special tokens (<|user_start|>, <|assistant_start|>, etc.), how to call a calculator tool, and how to answer multiple-choice questions — by training on a curated mixture (SmolTalk, identity, MMLU, GSM8K, spelling tasks).

RL (optional). A simplified GRPO/REINFORCE loop that samples multiple answers per GSM8K problem, rewards correct ones, and nudges the model toward higher accuracy.

Inference Engine. An efficient KV-cache generator that prefills the prompt once, then decodes new tokens. It also runs a small tool-use state machine: when the model emits a <|python_start|> … <|python_end|> block, the Engine evaluates the expression with a safe calculator and feeds the result back as <|output_start|> … <|output_end|> tokens.

The report. During a speedrun, each stage appends to a markdown report under the base directory, capturing system info, configs, metrics, and sample outputs — a complete record of the run.

11. Troubleshooting

Error: No dataset parquet files found, did you run dataset.py?

Cause: Training or tokenizer scripts can't find pretraining shards in the base directory. Solution: Download data first: python -m nanochat.dataset -n 8 (tokenizer) or -n 170 (full pretrain). Prevention: Always run the dataset download step before tokenizer/base training. The speedrun does this for you.

Warning: DATASET UPGRADE REQUIRED

Cause: nanochat switched from FinewebEdu-100B to ClimbMix-400B (March 4 2026). After a git pull, your old data directory no longer matches. Solution: Re-download and re-train the tokenizer on the new data:

python -m nanochat.dataset -n 170
python -m scripts.tok_train

Prevention: Re-run the data and tokenizer steps whenever you pull a dataset-changing update.

Out of memory (CUDA OOM) during training

Cause: --device-batch-size is too large for your VRAM (common below 80 GB). Solution: Lower --device-batch-size: 32 → 16 → 8 → 4 → 2 → 1. Results are unchanged (gradient accumulation compensates). Prevention: At larger depths, start lower (e.g. 16 for depth ≥ 20). The provided run scripts encode good defaults.

Warning: Flash Attention 3 not available, using PyTorch SDPA fallback

Cause: You're not on a Hopper (H100/H200) GPU in bf16. FA3 kernels are Hopper-only. Solution: None needed — training is correct, just slower. If you see a further warning about sliding windows under SDPA, add --window-pattern L. Prevention: Use H100-class GPUs in bf16 for the fastest path; otherwise accept the fallback.

AssertionError about total batch size divisibility

Cause: --total-batch-size isn't divisible by device_batch_size × max_seq_len × num_gpus. Solution: Use a power-of-two total batch size (e.g. 524288) or let it auto-compute (--total-batch-size=-1), and keep --device-batch-size a power of two. Prevention: Prefer the auto-computed batch size unless you have a specific reason to override.

AssertionError: Your PyTorch installation is not configured for CUDA but device_type is 'cuda'

Cause: You installed the CPU extra (uv sync --extra cpu) but are requesting CUDA. Solution: Reinstall with the GPU extra: uv sync --extra gpu. (The extras are mutually exclusive.) Prevention: Match the extra to your hardware at install time.

FileNotFoundError: No checkpoints found in ...

Cause: A chat/eval script tried to load a model stage that hasn't been trained yet (e.g. running chat_cli before chat_sft). Solution: Run the prerequisite stage first, or point --model-tag/-i source at a checkpoint that exists. Prevention: Follow the pipeline order: base → sft → (rl) before chatting.

The web UI loads but I can't reach it from my laptop

Cause: You're using localhost from a remote machine, or the cloud firewall blocks the port. Solution: Use the node's public IP and the port (e.g. http://<public-ip>:8000/), and open the port in your provider's firewall/security group. Prevention: Note the exact URL the server prints and confirm the port is reachable.

Chat responses look like gibberish / the model is "dumb"

Cause: A speedrun-grade model is genuinely tiny (~4×10¹⁹ FLOPs). Low quality is expected. Solution: Train a deeper model (--depth=26), train on more data, or simply accept the toy-scale quality. Prevention: Set expectations — nanochat teaches the stack; it is not a frontier model.

assert decoded == test_text fails during tokenizer training

Cause: A round-trip encode/decode mismatch — usually a corrupted or incompatible tokenizer build. Solution: Re-run scripts.tok_train cleanly; ensure rustbpe/tiktoken installed correctly via uv sync. Prevention: Don't hand-edit tokenizer artifacts in the base directory.

Calculator/tool output never appears in answers

Cause: The model only invokes the calculator when it emits a Python tool block, and the calculator only accepts simple math or .count(...) expressions (it refuses **, imports, and other unsafe patterns). Solution: This is by design and safe. Ask arithmetic or letter-counting questions (e.g. "how many r in strawberry") to see tool use. Prevention: None needed.

12. FAQ

Q: How much does it cost to train a model? A GPT-2-grade speedrun is ~2 hours on an 8×H100 node. At ~$3/GPU/hr (~$24/hr for the node) that's roughly $48; on a spot instance it can be closer to ~$15.

Q: Do I need 8 GPUs? No. Any stage runs on a single GPU — just omit torchrun. It takes about 8× longer and produces nearly identical results via gradient accumulation.

Q: Can I run this on my MacBook? Yes, for a tiny educational demo via runs/runcpu.sh. You will not get a usable model. Real training needs GPUs.

Q: What's the difference between the base, sft, and rl models? base is the pretrained next-token predictor. sft is fine-tuned to chat, use tools, and answer multiple choice. rl further sharpens math via reinforcement learning. Chat scripts pick one with -i/--source.

Q: What does --depth actually control? The number of transformer layers. Everything else (width, heads, learning rates, training length, batch size, weight decay) is derived automatically to keep the model compute-optimal.

Q: Where are all my files stored? Under ~/.cache/nanochat by default. Override with the NANOCHAT_BASE_DIR environment variable. This holds the dataset, tokenizer, checkpoints, eval results, and reports.

Q: What is "CORE" and "bpb"? CORE is the DCLM benchmark score (the headline capability metric; GPT-2's is 0.256525). bpb is bits-per-byte validation loss — a vocab-size-invariant loss you watch during training.

Q: How do I log training to Weights & Biases? Run wandb login once, then pass --run=<name> to a training script. The default --run=dummy disables wandb.

Q: Why are there two attention paths? Flash Attention 3 is used on Hopper GPUs (fastest). Everywhere else (Ada, Blackwell, MPS, CPU), nanochat falls back to PyTorch SDPA automatically. Results are equivalent; speed differs.

Q: Can I evaluate the real GPT-2 to compare? Yes: torchrun --nproc_per_node=8 -m scripts.base_eval -- --hf-path openai-community/gpt2. This requires the dev group (for transformers).

Q: How do I give my model a personality? Mix custom synthetic "identity conversations" into SFT. See dev/gen_synthetic_data.py and the repo's Discussions guides on infusing identity.

Q: Is the chat web server safe to expose publicly? It has basic abuse limits (message/length/temperature caps) but no authentication and binds to all interfaces. Don't expose it to the open internet without putting it behind your own auth/proxy. See the Admin Guide.

Q: Can I train a bigger-than-GPT-2 model? Yes — increase --depth beyond 26. You'll likely need to reduce --device-batch-size to fit memory, and training will take longer.

Q: What's the leaderboard? A community "GPT-2 speedrun" leaderboard tracking the wall-clock time to reach GPT-2 CORE capability on an 8×H100 node. runs/speedrun.sh is always the reference recipe. See dev/LEADERBOARD.md.

Q: How do I resume a crashed run? Train with --save-every=N so checkpoints exist, then restart with --resume-from-step=<step> and the same --model-tag.

Q: Where can I ask deeper questions about the code? The repo recommends DeepWiki, the GitHub Discussions tab, and the #nanochat Discord channel.

13. Command & Flag Reference

Module entry points

Run scripts

base_train — full flag list

Note

For chat_sft, chat_rl, chat_eval, base_eval, chat_cli, and chat_web flag lists, see the relevant sections above (6.4–7.2). Flags marked "inherited" default to the value baked into the checkpoint being loaded.