# autoresearch User Guide

`autoresearch` is a deliberately minimal, single-GPU LLM pretraining sandbox built by Andrej Karpathy as a derivative of [`nanochat`](https://github.com/karpathy/nanochat). The whole point of the repo is to be **edited autonomously by a coding agent** (Claude Code, OpenAI Codex, or similar) running an overnight loop of: edit `train.py`, run a 5-minute experiment, keep or revert based on a single validation metric (`val_bpb`), and try the next idea.

This guide takes you from a fresh GPU box to a working autonomous research loop. It is intentionally short, because the codebase is intentionally short — two Python files (`prepare.py`, `train.py`), one markdown file (`program.md`) that programs the agent, plus an analysis notebook.

> [!IMPORTANT]
> `autoresearch` is a research toy. There are no model checkpoints, no multi-GPU support, no CLI flags on the training script, and no API. Every "knob" the agent touches is a module-level constant inside `train.py`. If you are looking for a production training framework, look elsewhere.

---

## 1. Quick Start

On any single-GPU NVIDIA Linux box with Python 3.10+ and [`uv`](https://docs.astral.sh/uv/) installed, four commands stand the project up end-to-end:

```bash
git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync
uv run prepare.py
uv run train.py
```

The first three commands fetch the repo and install dependencies (PyTorch `2.9.1` on the `cu128` index, plus eight other packages). `uv run prepare.py` downloads 11 Parquet data shards from Hugging Face and trains a BPE tokenizer (~40 s). `uv run train.py` then trains an 80M-parameter GPT for exactly 5 minutes of steady-state wall-clock time and prints a summary.

Expected tail of the final run:

```text
---
val_bpb:          0.997900
training_seconds: 300.1
total_seconds:    325.9
peak_vram_mb:     45060.2
mfu_percent:      39.80
total_tokens_M:   499.6
num_steps:        953
num_params_M:     50.3
depth:            8
```

If you see a `val_bpb:` line at the end, the rig works and you can hand the project off to an autonomous agent (see Section 4).

> [!TIP]
> The fastest way to confirm a fresh setup is healthy is to run `uv run prepare.py --num-shards 2` first. That downloads only the two shards needed to train the tokenizer (~30 s of network), proves your toolchain works, and gives the training run something to read. You can scale up to `--num-shards 10` afterwards.

> [!NOTE]
> Every command in this guide is shown as `uv run <script>.py`. That wrapper executes inside the project virtualenv automatically. If you prefer, you can `source .venv/bin/activate` once and drop the `uv run` prefix.

---

## 2. Installation

### 2.1 Hardware and platform requirements

| Requirement | Detail |
|---|---|
| GPU | A single NVIDIA GPU, Hopper (H100/H200) or Ampere (A100, RTX 30/40 series). The script hardcodes `torch.device("cuda")`. |
| VRAM | ~44 GB peak at the default config. Smaller cards require batch-size cuts (Section 5.4). |
| Python | `>= 3.10` (pinned in `.python-version`). |
| Disk | ~5 GB for the default 10 training shards plus the pinned validation shard. |
| Network | Outbound HTTPS to `huggingface.co` (data + FA3 kernels) and `pytorch.org` (wheels). |

> [!WARNING]
> There is **no CPU, MPS, ROCm, or multi-GPU code path** in this repository. Running on a Mac, AMD GPU, or Windows-without-CUDA will fail at `torch.cuda.get_device_capability()` very early. Use one of the community forks listed in Section 5.5 instead of filing a bug.

### 2.2 Software stack

`pyproject.toml` pins the following minimums:

| Package | Minimum | Role |
|---|---|---|
| `torch` | `2.9.1` | Core framework, pinned (uses `pytorch-cu128` index) |
| `kernels` | `>= 0.11.7` | Pulls Flash Attention 3 from Hugging Face Hub at startup |
| `pyarrow` | `>= 21.0.0` | Reads Parquet data shards |
| `tiktoken` | `>= 0.11.0` | Encoding used at runtime |
| `rustbpe` | `>= 0.1.0` | Trains the BPE tokenizer |
| `requests` | `>= 2.32.0` | Downloads data shards |
| `numpy` | `>= 2.2.6` | Numerics |
| `pandas` | `>= 2.3.3` | Used by `analysis.ipynb` |
| `matplotlib` | `>= 3.10.8` | Used by `analysis.ipynb` |

> [!IMPORTANT]
> The PyTorch wheel comes from the `pytorch-cu128` index, which means your driver must support CUDA 12.8. Verify with `nvidia-smi` before `uv sync`. Older drivers will sync successfully but fail at `import torch` during `prepare.py`.

### 2.3 Install `uv`

=== "macOS / Linux"

    ```bash
    curl -LsSf https://astral.sh/uv/install.sh | sh
    ```

=== "Windows (PowerShell)"

    ```powershell
    powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
    ```

=== "Already installed"

    ```bash
    uv --version
    ```

### 2.4 Clone and install dependencies

```bash
git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync
```

Expected output (abridged):

```text
Resolved 47 packages in 1.24s
Prepared 9 packages in 3.2s
Installed 47 packages in 11s
```

`uv sync` reads `pyproject.toml`, resolves `uv.lock`, and creates `.venv/` with the CUDA 12.8 PyTorch build.

### 2.5 Verify the GPU is visible

```bash
uv run python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
```

Expected output:

```text
True NVIDIA H100 80GB HBM3
```

If this prints `False`, fix your CUDA driver before continuing — every later command depends on the GPU.

---

## 3. Configuration

`autoresearch` has **no configuration files**, no environment variables, and no CLI flags on `train.py`. Configuration lives in two places:

1. **`prepare.py`** — fixed constants that define the evaluation harness (treat as read-only).
2. **`train.py`** — module-level hyperparameters near the bottom of the file (intended to be edited).

The agent edits `train.py` only. `program.md` (the agent prompt) declares `prepare.py` off-limits because it owns the metric.

### 3.1 The two CLI flags that exist (`prepare.py` only)

| Flag | Type | Default | Effect |
|---|---|---|---|
| `--num-shards` | `int` | `10` | Number of training shards to download. `-1` means all 6,542 shards. The validation shard (`shard_06542.parquet`) is always pinned and always downloaded in addition. |
| `--download-workers` | `int` | `8` | Parallel download workers. Reduce for slow or rate-limited connections. |

### 3.2 Fixed constants in `prepare.py` (do not edit)

These are the evaluation harness. The agent's `program.md` rule set forbids touching them, because changing any of them silently invalidates comparisons across runs.

| Constant | Value | Role |
|---|---|---|
| `MAX_SEQ_LEN` | `2048` | Context length for both training and eval |
| `TIME_BUDGET` | `300` | Seconds of steady-state training per experiment |
| `EVAL_TOKENS` | `40 * 524288` (≈21 M) | Tokens scored by the final `evaluate_bpb` pass |
| `VAL_SHARD` | `6542` | Pinned validation shard index |
| `VOCAB_SIZE` | `8192` | BPE vocabulary, including 4 reserved special tokens |
| `BASE_URL` | `https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle/resolve/main` | Dataset source |
| `BOS_TOKEN` | `<\|reserved_0\|>` | Beginning-of-sequence token used by the dataloader |

### 3.3 Hyperparameters in `train.py` (the knobs the agent turns)

These live in a single block under the comment `# Hyperparameters (edit these directly, no CLI flags needed)`:

| Constant | Default | Effect |
|---|---|---|
| `DEPTH` | `8` | Number of transformer layers |
| `ASPECT_RATIO` | `64` | `model_dim = DEPTH * ASPECT_RATIO`, rounded to a multiple of `HEAD_DIM` |
| `HEAD_DIM` | `128` | Per-head dimension. `n_head = model_dim // HEAD_DIM` |
| `WINDOW_PATTERN` | `"SSSL"` | Per-layer attention window pattern: `L`=full (2048), `S`=half (1024). The last layer is always overridden to `L`. |
| `TOTAL_BATCH_SIZE` | `2**19` (524,288) | Tokens per optimizer step |
| `DEVICE_BATCH_SIZE` | `128` | Per-device batch size; reduce on OOM |
| `EMBEDDING_LR` | `0.6` | AdamW LR for token + value embeddings |
| `UNEMBEDDING_LR` | `0.004` | AdamW LR for `lm_head` |
| `MATRIX_LR` | `0.04` | Muon LR for 2-D matrix params inside transformer blocks |
| `SCALAR_LR` | `0.5` | AdamW LR for per-layer scalars (`x0_lambdas`); `resid_lambdas` uses `SCALAR_LR * 0.01` |
| `WEIGHT_DECAY` | `0.2` | Cautious weight decay for Muon; decays linearly to 0 over training |
| `ADAM_BETAS` | `(0.8, 0.95)` | `(beta1, beta2)` for AdamW groups |
| `WARMUP_RATIO` | `0.0` | Fraction of `TIME_BUDGET` for LR warmup |
| `WARMDOWN_RATIO` | `0.5` | Fraction of `TIME_BUDGET` for linear LR cooldown |
| `FINAL_LR_FRAC` | `0.0` | Final-LR multiplier, relative to initial |

> [!NOTE]
> Learning rates are tuned at `n_embd = 768` (`DEPTH=8`, `ASPECT_RATIO=64`, `HEAD_DIM=128`). When you change model size, `setup_optimizer` rescales all AdamW LRs by `(model_dim / 768) ** -0.5`, and Muon's per-parameter step adds a `max(1.0, rows / cols) ** 0.5` factor. You generally do not need to compensate by hand.

> [!TIP]
> Because every knob is a constant, every experiment is fully described by a `git diff`. That is the design — it lets the agent (and you) read run history just by reading commit messages.

---

## 4. Core Usage

`autoresearch` runs in two modes:

1. **Manual** — you run `uv run train.py`, read the output, hand-edit constants, re-run. Useful for sanity checks.
2. **Autonomous** — you point an AI coding agent (Claude Code, OpenAI Codex) at `program.md` and let it iterate overnight.

The autonomous mode is what the project was built for; the manual mode is mostly a debugging tool.

### 4.1 One-time data and tokenizer prep

Before any training, you need data shards on disk and a trained tokenizer. Both are produced by `prepare.py` and cached in `~/.cache/autoresearch/`:

```bash
uv run prepare.py
```

Default behaviour:

1. Download 10 training shards plus `shard_06542.parquet` (validation) from `karpathy/climbmix-400b-shuffle`.
2. Train a BPE tokenizer with `rustbpe` on up to 1 GB of training text.
3. Convert merges into a `tiktoken.Encoding` and pickle to `~/.cache/autoresearch/tokenizer/tokenizer.pkl`.
4. Compute and save a `token_bytes.pt` lookup tensor used for the `val_bpb` metric.
5. Run a tokenizer round-trip sanity check on `"Hello world! Numbers: 123. Unicode: 你好"`.

Expected output (trimmed):

```text
Cache directory: /home/you/.cache/autoresearch

Data: downloading 11 shards (0 already exist)...
  Downloaded shard_00000.parquet
  ...
  Downloaded shard_06542.parquet
Data: 11/11 shards ready at /home/you/.cache/autoresearch/data

Tokenizer: training BPE tokenizer...
Tokenizer: trained in 38.2s, saved to /home/you/.cache/autoresearch/tokenizer/tokenizer.pkl
Tokenizer: building token_bytes lookup...
Tokenizer: saved token_bytes to /home/you/.cache/autoresearch/tokenizer/token_bytes.pt
Tokenizer: sanity check passed (vocab_size=8192)

Done! Ready to train.
```

> [!TIP]
> `prepare.py` is **idempotent**. Re-running it skips already-downloaded shards and skips tokenizer training if both `tokenizer.pkl` and `token_bytes.pt` already exist. It is safe to re-run after a crash.

> [!IMPORTANT]
> `tokenizer.pkl` and `token_bytes.pt` are a pair. If only one is present, `prepare.py` will retrain from scratch. Do not delete just one — either keep both or wipe the whole `~/.cache/autoresearch/tokenizer/` directory.

### 4.2 Manual training run

After `prepare.py` succeeds, train manually with:

```bash
uv run train.py
```

The script:

1. Loads the tokenizer from `~/.cache/autoresearch/tokenizer/`.
2. Builds a GPT model from the `train.py` constants (no CLI flags).
3. Compiles with `torch.compile(dynamic=False)` — the first ~10 steps are excluded from the timing budget.
4. Trains for exactly `TIME_BUDGET = 300` seconds of steady-state wall-clock.
5. Runs `evaluate_bpb` over `EVAL_TOKENS` of the pinned validation shard.
6. Prints a final summary block.

A typical streaming log:

```text
Vocab size: 8,192
Model config: {'sequence_len': 2048, 'vocab_size': 8192, 'n_layer': 8, 'n_head': 6, 'n_kv_head': 6, 'n_embd': 768, 'window_pattern': 'SSSL'}
Parameter counts:
  wte                     : 6,291,456
  value_embeds            : 25,165,824
  lm_head                 : 6,291,456
  transformer_matrices    : 42,467,328
  scalars                 : 16
  total                   : 80,216,080
Estimated FLOPs per token: 6.03e+08
Scaling AdamW LRs by 1/sqrt(768/768) = 1.000000
Time budget: 300s
Gradient accumulation steps: 2

step 00000 (0.0%) | loss: 9.123456 | lrm: 1.00 | dt: 1034ms | tok/sec: 507,123 | mfu: 39.8% | epoch: 1 | remaining: 300s
...
step 00953 (100.0%) | loss: 1.892100 | lrm: 0.00 | ...

---
val_bpb:          0.997900
...
```

> [!TIP]
> Redirect the full log to a file with `uv run train.py > run.log 2>&1`. Then `grep "^val_bpb:" run.log` reads only the metric. This is exactly what the autonomous loop does so the streaming progress lines do not flood the agent's context.

### 4.3 Autonomous loop

In a checkout you do not mind being mutated, launch your coding agent with permissions disabled (so it can edit `train.py` and shell-execute without prompting), then say something like:

```text
Hi, have a look at program.md and let's kick off a new experiment! Let's do the setup first.
```

The agent will read `program.md`, agree on a run tag (e.g. `mar5`), create a branch `autoresearch/mar5`, verify data exists, and start looping. Each loop iteration:

1. Inspect git state.
2. Edit `train.py` with one experimental idea.
3. `git commit` the change.
4. `uv run train.py > run.log 2>&1`.
5. `grep "^val_bpb:\|^peak_vram_mb:" run.log` to read metrics.
6. Append a row to `results.tsv` (kept untracked).
7. If `val_bpb` improved (lower is better), advance the branch. Otherwise `git reset` and try the next idea.

Expected throughput on an H100: roughly **12 experiments per hour**, or about **100 experiments per 8-hour overnight session**.

> [!IMPORTANT]
> `program.md` instructs the agent never to pause for confirmation once the loop starts. To stop, kill the agent process — it will not stop on its own. This is intentional, so you can leave it running unattended overnight.

> [!CAUTION]
> The agent is given autonomous write access to the working tree and shell. It runs `git reset` regularly and edits `train.py` freely. Only run it inside a checkout you are willing to lose. Never aim it at a branch with work you care about.

### 4.4 The `program.md` "skill file"

`program.md` is the **human-editable** programming surface for the agent. The default `program.md` shipped with the repo is a minimal baseline — one agent, one loop, one metric. Typical evolutions:

- Tell the agent to focus on a specific region of the search space first.
- Add minimum-improvement thresholds before keeping a change.
- Constrain edits to architecture-only or optimizer-only.
- Spawn parallel agents on different branches (`autoresearch/mar5-gpu0`, `autoresearch/mar5-gpu1`, …) with different `program.md` strategies per GPU.

> [!NOTE]
> `program.md` is part of the repo. If you change it, commit the change so future runs are reproducible. It is the only file other than `train.py` that meaningfully drives experimental behaviour.

---

## 5. Common Tasks

### 5.1 Smoke-test a fresh install

```bash
uv run prepare.py --num-shards 2
uv run train.py > run.log 2>&1
grep "^val_bpb:" run.log
```

The smaller `--num-shards 2` value still satisfies the `len(parquet_files) >= 2` precondition for tokenizer training and shaves several minutes off the first prep. Expected output:

```text
val_bpb:          0.997900
```

(The exact number depends on your hardware. On slower GPUs you fit fewer steps in 300 s, so `val_bpb` will be higher.)

### 5.2 Resume a crashed agent loop

If your agent process exited or was killed mid-experiment on branch `autoresearch/mar5`:

```bash
git checkout autoresearch/mar5
# Restart your agent and re-prompt it with program.md.
```

`results.tsv` is intentionally untracked, so it survives `git reset`/`git checkout` and your historical log is preserved across restarts.

### 5.3 Start a fresh tagged run

The branch name `autoresearch/<tag>` must not already exist. Pick a new tag:

```bash
git checkout master
git pull
# Prompt the agent with a new tag, e.g. "mar5b" or "mar5-v2".
```

### 5.4 Tune for smaller GPUs

The default config peaks at ~44 GB VRAM. To run on a 12/16/24 GB card:

| Knob | File | Default | Try |
|---|---|---|---|
| `DEPTH` | `train.py` | `8` | `4` or `2` |
| `DEVICE_BATCH_SIZE` | `train.py` | `128` | Step down until OOM clears, then back up |
| `TOTAL_BATCH_SIZE` | `train.py` | `2**19` | `2**14` or lower (must remain a multiple of `DEVICE_BATCH_SIZE * MAX_SEQ_LEN`) |
| `WINDOW_PATTERN` | `train.py` | `"SSSL"` | `"L"` for short sequences |
| `MAX_SEQ_LEN` | `prepare.py` | `2048` | `512` or `256` (forces full re-prep) |
| `VOCAB_SIZE` | `prepare.py` | `8192` | `4096` or smaller (forces full re-prep) |
| `EVAL_TOKENS` | `prepare.py` | `40 * 524288` | Reduce 10–100× so eval finishes quickly |

> [!WARNING]
> Reducing `MAX_SEQ_LEN` or `VOCAB_SIZE` requires editing `prepare.py`, which `program.md` declares off-limits to the agent. Do these edits **yourself, manually, before** starting the agent. After editing `prepare.py`, wipe `~/.cache/autoresearch/tokenizer/` so the new vocab size is rebuilt.

### 5.5 Switch to a community fork for non-CUDA hardware

The README lists these forks for hardware the upstream repo does not support:

| Fork | Target |
|---|---|
| [`miolini/autoresearch-macos`](https://github.com/miolini/autoresearch-macos) | macOS |
| [`trevin-creator/autoresearch-mlx`](https://github.com/trevin-creator/autoresearch-mlx) | macOS (MLX backend) |
| [`jsegov/autoresearch-win-rtx`](https://github.com/jsegov/autoresearch-win-rtx) | Windows (consumer RTX) |
| [`andyluo7/autoresearch`](https://github.com/andyluo7/autoresearch) | AMD |

There is also an open request (#519) to recognize an Ascend-compatible fork in the README's notable-forks list.

### 5.6 Compare two experiment branches

Each `autoresearch/<tag>` branch carries its own untracked `results.tsv`. To diff them, copy out before switching:

```bash
git checkout autoresearch/mar5
cp results.tsv /tmp/mar5.tsv
git checkout autoresearch/mar5b
cp results.tsv /tmp/mar5b.tsv
diff /tmp/mar5.tsv /tmp/mar5b.tsv
```

For a richer comparison, use `analysis.ipynb` — `pandas` and `matplotlib` are already in the environment from `uv sync`.

---

## 6. Troubleshooting

The training loop prints progress on a single carriage-returning line, so failures usually surface either as a Python traceback or as the literal string `FAIL`. Below are the failure modes confirmed in the source.

### Error: `torch.cuda.is_available()` returns False

**Cause:** PyTorch cannot see a CUDA device. Either no GPU is attached, the NVIDIA driver is missing, or the driver is older than what the pinned `torch==2.9.1+cu128` wheel requires.

**Solution:** Install or update the NVIDIA driver. Verify with `nvidia-smi`. Do not downgrade PyTorch — the Muon and AdamW fused steps rely on `torch.compile` behaviour from a recent version.

**Prevention:** Keep your driver current. If you are on a shared cluster, check that `CUDA_VISIBLE_DEVICES` is not set to the empty string in your shell.

### Error: kernel-loading failure inside `kernels.get_kernel(...)`

**Cause:** `train.py` picks `varunneal/flash-attention-3` for Hopper (`(9, 0)`) and `kernels-community/flash-attn3` for everything else. If neither builds for your GPU, the import fails.

**Solution:** Run `nvidia-smi --query-gpu=compute_cap --format=csv,noheader`. If it reports below sm_80 (Ampere), this repo cannot run on your hardware. Use a fork from Section 5.5 or switch GPUs.

**Prevention:** This repo targets H100 first, A100 second. Older datacenter GPUs (V100) and pre-Ampere consumer cards (RTX 20-series and earlier) are not supported.

### Error: `ValueError: Invalid input type: <class '...'>`

**Cause:** Raised at `prepare.py:241` (line numbers approximate). Your code called `Tokenizer.encode(x)` where `x` is neither `str` nor `list`. The dataloader passes lists, so this only appears in custom code that calls `tokenizer.encode` with a generator, numpy array, or similar.

**Solution:** Coerce to `str` or `list[str]` before passing.

**Prevention:** Stick to the documented input types.

### Error: `Tokenizer: need at least 2 data shards (1 train + 1 val). Download more data first.`

**Cause:** `prepare.py` requires at least 2 parquet files to train. You ran `--num-shards 0`, or your downloads all failed.

**Solution:**

```bash
uv run prepare.py --num-shards 2
```

**Prevention:** Default `--num-shards 10` already protects you. Avoid setting it below 2.

### Error: `AssertionError: No parquet files found. Run prepare.py first.`

**Cause:** `train.py` opened a dataloader before any data was on disk.

**Solution:** Run `uv run prepare.py` before `uv run train.py`. Required after every fresh checkout or after wiping `~/.cache/autoresearch/`.

**Prevention:** Always run `prepare.py` first on a new machine. The Quick Start in this guide does this.

### Error: download retries exhausted; some `shard_NNNNN.parquet` files missing

**Cause:** `download_single_shard` retries 5 times with exponential backoff (`2^attempt` seconds — so 2, 4, 8, 16 s between attempts). Hugging Face rate limits or 5xx errors can exhaust those attempts. Issue #215 tracks hardening this further (e.g. checksumming the cached shards).

**Solution:** Re-run `prepare.py`; only missing shards are re-fetched. For large jobs, lower the worker count to avoid rate-limit triggers:

```bash
uv run prepare.py --num-shards 10 --download-workers 2
```

**Prevention:** Use `--download-workers 2` for `--num-shards -1` (full dataset).

### Error: training prints `FAIL` and exits with code 1

**Cause:** A guard at the end of each step: if `train_loss` is NaN or greater than 100, the run prints `FAIL` and exits. Triggered by numerically unstable hyperparameters (LR far too high, broken normalization, etc.).

**Solution:** This is the intended behaviour in autonomous mode — the agent reads an empty `val_bpb` line, marks the experiment as `crash`, and `git reset`s. If you are running manually, lower the offending LR or revert the change.

**Prevention:** Make small LR moves; reserve aggressive architecture changes for cases where the baseline is stable.

### `peak_vram_mb` climbs across long runs

**Cause:** Python's GC is **disabled** after step 0 (`gc.disable()` in the training loop) to avoid GC-induced stalls, with a manual `gc.collect()` every 5,000 steps. At default config, 5-minute runs finish in ~1,000 steps so this never matters — but if you cut batch size dramatically you may run far more steps without ever hitting a manual GC.

**Solution:** Either raise `DEVICE_BATCH_SIZE` (fewer steps), or change the `(step + 1) % 5000 == 0` condition in `train.py` to a smaller interval.

**Prevention:** Don't increase total step count by an order of magnitude without also tightening the manual GC schedule.

### `mfu_percent` very low (< 20%) on H100

**Cause:** On H100 the expected MFU is roughly 40%. Much lower usually means the FA3 kernel didn't load and attention fell back to a slower path, or `torch.compile` is recompiling because of dynamic shapes the agent introduced.

**Solution:** Check the startup log for `kernels` lines indicating which FA3 backend loaded. If it failed, run `uv sync --reinstall` and retry. If `torch.compile` is recompiling repeatedly, find the agent commit that introduced dynamic shapes and revert.

**Prevention:** Keep tensor shapes static. The fused steps are decorated `@torch.compile(dynamic=False, fullgraph=True)`; do not relax that.

### Tokenizer hangs at the sanity-check assertion

**Cause:** The end of `train_tokenizer()` does `enc.decode(enc.encode_ordinary("Hello world! Numbers: 123. Unicode: 你好")) == ...`. A hang there usually means `rustbpe` failed to link or `tiktoken` failed to import — not a real round-trip failure.

**Solution:** `uv sync --reinstall`.

**Prevention:** Don't mix `pip install` with `uv sync` in the same virtualenv.

### Issue #35 — "Why not fixed FLOPs budget?"

**Status:** Open. The chosen design fixes wall-clock time rather than FLOPs, which means slower hardware sees fewer tokens and reaches a higher `val_bpb` than fast hardware on identical code. Discussion is in the upstream issue; the choice is intentional and tied to the autonomous-loop comparability story (Section 7 FAQ).

---

## 7. FAQ

**Q: Can I run this on a MacBook or M-series chip?**

Not directly. The code hardcodes `torch.device("cuda")` and Flash Attention 3. Use [`miolini/autoresearch-macos`](https://github.com/miolini/autoresearch-macos) or [`trevin-creator/autoresearch-mlx`](https://github.com/trevin-creator/autoresearch-mlx).

**Q: Can I run this on AMD?**

Not directly. Use [`andyluo7/autoresearch`](https://github.com/andyluo7/autoresearch).

**Q: Can I run multiple GPUs?**

No. The repo is explicitly single-GPU. The parent `nanochat` repo supports distributed training; use that if you need DDP/FSDP.

**Q: Why is the training time hardcoded to 5 minutes?**

So that every experiment uses the same wall-clock budget on a given machine. Tradeoffs between depth, batch size, and step count become directly comparable. The downside is that results are not directly comparable to other people on different hardware — but that is not the goal. The goal is to give an autonomous agent apples-to-apples decisions on **your** GPU.

**Q: Why `val_bpb` and not `val_loss`?**

Bits-per-byte is independent of vocabulary size. If the agent ever changed `VOCAB_SIZE` (which `prepare.py` rules out), cross-entropy in nats would no longer be a fair metric. BPB converts via `loss / (ln(2) * total_bytes)`, which is comparable across tokenizer designs.

**Q: Why is `prepare.py` off-limits to the agent?**

Because it owns the metric. It defines `evaluate_bpb`, the pinned validation shard, and the fixed time budget. Letting the agent edit any of those is letting it improve the metric by changing the metric.

**Q: Why does `train.py` have no CLI flags?**

Because the agent's interaction model is "edit source, commit, run". CLI flags would be a second source of truth. Module-level constants mean `git diff` is the complete description of an experiment.

**Q: Can I run two agents in parallel on one GPU?**

Not recommended — they would contend for VRAM and `torch.compile` cache. The intended pattern is one agent per GPU. On a multi-GPU box, give each agent its own checkout (or git worktree) and its own branch tag.

**Q: Where are model checkpoints saved?**

They aren't. `train.py` trains for 5 minutes, evaluates, and exits. No checkpoints, no intermediate saves. The agent is optimizing a metric, not producing a deployable model.

**Q: The dataset is 6,542 shards. Do I need all of it?**

No. With the default 5-minute budget on H100 you process ~500M tokens, which is less than one shard's worth. `--num-shards 10` is plenty of headroom. Use `--num-shards -1` only when training a much larger model for much longer.

**Q: Does the tokenizer need to be retrained per experiment?**

No, and you can't — the tokenizer lives in `prepare.py`, which the agent cannot edit. The tokenizer is a fixed artifact of the eval harness; every experiment sees the same encoding.

**Q: Can I commit `results.tsv`?**

You can, but `program.md` asks the agent to keep it untracked so it survives `git reset`. If you want to share results, copy the file to a gist or a separate results repo.

**Q: Is there a license?**

`README.md` lists MIT at the bottom. Issue #523 (closed) and #395 (closed) tracked the request to add a separate `LICENSE` file — check the current repo state if you need a file-level licence artefact.

**Q: What is `WINDOW_PATTERN = "SSSL"`?**

A per-layer attention window pattern. `L` is full context (`MAX_SEQ_LEN = 2048`). `S` is half (`1024`). `SSSL` repeats across layers; the very last layer is always overridden to `L` so the final readout sees the full context. Changing this is one of the more impactful architecture knobs the agent can turn.

**Q: Can the agent modify the optimizer?**

Yes. The `MuonAdamW` class is defined inside `train.py`, so it is fair game. Optimizer changes are explicitly allowed by `program.md`.

**Q: The agent keeps reverting crashed runs — is that normal?**

Yes. Crashes (`FAIL`, OOM, Python exceptions) are logged with `status=crash` and reverted. It is only a bug if it happens on every run from the baseline, which means your environment is broken — re-do Section 2.

**Q: How do I know my run actually hit the time budget?**

Check `training_seconds` in the summary block. It should be very close to `300.0`. Much less means the run exited early (usually `FAIL`). Much more is unreachable in normal flow because the loop exit condition is `total_training_time >= TIME_BUDGET`.

**Q: Why are the first ~10 steps not counted?**

The training loop guards `if step > 10` before adding to `total_training_time`. `torch.compile` does graph capture and autotuning during those steps, which takes seconds. Excluding them keeps the measured "training time" reflective of steady state, not compilation.

---

## 8. Where to Look Next

For internals — how the model, optimizer, and dataloader actually work; how to scale the autonomous loop across multiple GPUs; security considerations for letting an agent edit your code overnight — see the companion **`karpathy__autoresearch__AdminGuide.md`** in this directory.

---

_Generated April 2026 from karpathy/autoresearch at 228791f. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._