# d2l-zh Administrator & Developer Guide

*Administrator, maintainer, and contributor documentation for the Chinese edition of **Dive into Deep Learning** (动手学深度学习) — covering build pipeline, multi-framework code extraction, LaTeX post-processing, CI infrastructure, and fork strategy.*

---

## 1. Architecture Overview

### 1.1 What this repo is, operationally

`d2l-ai/d2l-zh` is not a conventional software project. It is an **open, reproducible textbook**: source Markdown chapters that contain executable code cells, which are compiled by a custom book-build tool (`d2lbook`) into:

- A browsable HTML website (`https://zh-v2.d2l.ai/`)
- Four framework-specific PDFs (MXNet, PyTorch, TensorFlow, PaddlePaddle)
- A downloadable ZIP of Jupyter notebooks
- A `pip`-installable helper Python package (`d2l`) generated from annotated code cells

Every chapter renders four parallel implementations of the same deep-learning concepts — one per supported framework — from **a single Markdown source**. The `d2lbook` tool handles the fan-out. This design is what makes the repository operationally unusual and what most of this guide addresses.

### 1.2 The two surfaces

`d2l-zh` ships two distinct artifacts from one source tree:

```
                         source tree
                             │
        ┌────────────────────┼────────────────────┐
        │                                         │
  chapter_*/                                  d2l/
  (Markdown + code cells)                     (Python package)
        │                                         │
        ▼                                         ▼
 ┌──────────────────┐                     ┌──────────────────┐
 │   d2lbook        │                     │   setup.py       │
 │   - evaluates    │                     │   - packages     │
 │   - renders HTML │                     │   - publishes    │
 │   - renders PDF  │                     │   - versioned    │
 │   - zips nb's    │                     │   - pinned deps  │
 └────────┬─────────┘                     └────────┬─────────┘
          │                                        │
          ▼                                        ▼
  zh-v2.d2l.ai/*                           pip install d2l
  zh-v2.d2l.ai/d2l-zh.pdf                  (4 frameworks, one wheel)
  zh-v2.d2l.ai/d2l-zh.zip
```

The two are linked: the `d2l` Python package is **partially generated** from the chapter code cells via the `#@save` annotation. A reusable function defined inline in a chapter (e.g., `load_data_fashion_mnist`) is tagged with `#@save` and automatically collected by `d2lbook build lib` into `d2l/<framework>.py`. Readers then `import d2l.torch as d2l` and call `d2l.load_data_fashion_mnist(...)` from their own notebooks.

This loop (book code ↔ library code) is the core architectural idea:

1. The book **teaches** the function.
2. The book **uses** the function in later chapters.
3. The `d2l` package **exports** the function to the reader's own code.

> [!NOTE]
> You cannot refactor a `#@save`-annotated function purely as a library maintainer. The function's name, signature, and semantics are published to readers both textually (in the book) and programmatically (via pip). Breaking changes require a version bump and coordinated edits to many chapters.

### 1.3 Component inventory

| Component | Path | Role |
|-----------|------|------|
| Book content | `chapter_*/` | 19 chapters of Markdown + executable code cells |
| Translation refs | `chapter_*/*_origin.md` | English source snapshots for sync |
| Python package | `d2l/` | Four framework backends (mxnet, torch, tensorflow, paddle) |
| Package metadata | `setup.py` | Defines `d2l` on PyPI |
| Build config | `config.ini` | `d2lbook` master switchboard |
| LaTeX post-proc | `static/post_latex/main.py` | Chinese typography fixes |
| CI entry | `ci/submit-job.py` | AWS Batch job submission |
| Docker images | `ci/docker/` | Per-framework containers |
| Diagrams (source) | `graffle/` | OmniGraffle originals |
| Diagrams (rendered) | `img/` | PNG/SVG used by chapters |
| Web assets | `static/` | Logo, favicon, frontpage |
| Bibliography | `d2l.bib` | BibTeX for PDF references |
| Style guides | `STYLE_GUIDE.md`, `TERMINOLOGY.md` | Formatting and translation consistency |
| Contrib | `contrib/` | Draft chapters (excluded from build) |

### 1.4 Technology stack

| Layer | Technology |
|-------|-----------|
| Source format | Markdown + `d2lbook` directives (`:begin_tab:`, `#@tab`, `#@save`, `:numref:`) |
| Build tool | `d2lbook` (PyPI, reads `config.ini`) |
| HTML renderer | Sphinx (pulled in by `d2lbook`) |
| PDF chain | Sphinx → LaTeX (CTEX, Source Han fonts) → XeLaTeX → PDF |
| CI orchestration | GitHub Actions (`pull_request_target`) |
| CI compute | AWS Batch (GPU for eval, CPU for assembly) |
| Container registry | AWS ECR |
| Hosting | S3 + CloudFront (`zh-v2.d2l.ai`) |
| Package distribution | PyPI (`pip install d2l`) |

### 1.5 At-a-glance stats

| Property | Value |
|----------|-------|
| Repository type | Executable textbook (Markdown + code cells) |
| Source commit documented | `e6b18cc` (2023-08-17) |
| Total source files | 819 |
| Book version | `2.0.0` |
| Python package version | `d2l 2.0.0` |
| License (code / content) | MIT-0 (code), CC-BY-SA-4.0 (content) |
| Supported frameworks | MXNet, PyTorch, TensorFlow, PaddlePaddle |
| Python package deps (pinned) | `jupyter==1.0.0`, `numpy==1.21.5`, `matplotlib==3.5.1`, `requests==2.25.1`, `pandas==1.2.4` |
| Build tool | `d2lbook` (external, installed from PyPI) |
| CI | GitHub Actions → AWS Batch (ECR-hosted Docker) |
| Deployment | S3 + CloudFront (`zh-v2.d2l.ai`), four PDFs, notebook ZIP |
| Origin repo | `d2l-ai/d2l-en` (English source) |

### 1.6 Build pipeline (end to end)

```
   source .md (with :begin_tab:<fw> code cells)
        │
        │  d2lbook build eval --tab <fw>  ← runs per framework on GPU
        ▼
   _build/eval_<fw>/*.md (with evaluated outputs embedded)
        │
        │  d2lbook build html  (Sphinx HTML theme)
        │  d2lbook build pdf --tab <fw>  (Sphinx LaTeX → post_latex/main.py → xelatex)
        │  d2lbook build pkg  (collects #@save cells into d2l/<fw>.py)
        ▼
   _build/html/  ,  _build/pdf/d2l-zh.pdf  ,  d2l-*.whl
        │
        │  d2lbook deploy html pdf pkg colab sagemaker slides --s3 s3://zh-v2.d2l.ai/
        ▼
   CloudFront → https://zh-v2.d2l.ai/
```

### 1.7 CI architecture

```
GitHub push/PR
     │
     ▼
.github/workflows/ci.yml  (GitHub Actions, ubuntu-latest runner)
     │
     │  for each framework (torch, tf, mxnet, paddle):
     │    use ./.github/actions/submit-job → ci/submit-job.py → boto3 AWS Batch
     │
     ▼
AWS Batch  (D2L-CI-GPU queue,  job definitions d2l-ci-zh-gpu-<fw>:N)
     │
     │  each GPU job:
     │    pull ECR image (e.g. d2l-zh-torch)
     │    d2l_job.sh clones repo, restores S3 cache, runs build_<fw>.sh
     │    build_<fw>.sh: d2lbook build eval --tab <fw>; slides; upload cache
     │
     ▼
After all four GPU jobs succeed → build_and_deploy (CPU queue, D2L-Builder image)
     │
     │  d2lbook build pdf  (mxnet default)
     │  d2lbook build pdf --tab pytorch
     │  d2lbook build html
     │  if JOB_NAME == d2l-zh/release:
     │      d2lbook build pkg
     │      d2lbook deploy ... --s3 s3://zh-v2.d2l.ai/
     │  else:
     │      d2lbook deploy html pdf --s3 s3://preview.d2l.ai/<JOB_NAME>/
     │
     ▼
Live site / PR preview comment with preview URL
```

Three key trade-offs baked in:

- **Evaluation is cached in S3** (`s3://preview.d2l.ai/<cache_dir>/<repo>-<branch>/_build/`). A chapter only re-evaluates if its modification time differs from the cached copy. `git timesync` in `d2l_job.sh` restores modification times from git metadata because cloning resets them.
- **CI is split CPU/GPU by function.** Per-framework eval runs on GPU (torch/tf/mxnet/paddle); final HTML/PDF assembly runs on CPU. Separation keeps GPU minutes minimal.
- **The `SAFE_TO_USE_SCRIPT` parameter gates untrusted PRs** (see §7 Security). For PRs from forks without write permission, the CI overwrites the submitted `workflow_scripts/` with the master copy to prevent code execution via PR.

### 1.8 Red flags a new maintainer should know immediately

- **MXNet is end-of-life upstream.** Apache archived MXNet in late 2023. The `d2l/mxnet.py` backend and MXNet PDF builds will continue working for historical/reference purposes, but MXNet is not a target for new feature development.
- **`d2l==0.17.6` is hardcoded in CI build scripts**, not `d2l==2.0.0`. This is deliberate — the CI installs an old published d2l package for legacy compatibility while the source tree in the repo *becomes* the next published d2l package.
- **Code cells tagged `#@save` are load-bearing.** Removing one changes the public API of the `d2l` package. Never delete a `#@save` cell without checking downstream usage.
- **Framework tabs must stay in sync.** Every `:begin_tab:<framework>` block must have equivalent implementations across all four frameworks, or `d2lbook build tabcheck` fails in CI.
- **The CI pipeline requires AWS Batch with GPU compute environments.** A fork that wants the full build pipeline will need to replicate this infrastructure or replace it.

---

## 2. System Requirements

### 2.1 For building locally (contributor)

| Tool | Purpose | Notes |
|------|---------|-------|
| Python ≥ 3.5 | All tooling | 3.9 recommended in practice |
| `d2lbook` | Book build tool | Install via pip (see INFO.md) |
| One framework (MXNet, PyTorch, TF, or Paddle) | For `d2lbook build eval` | Only needed if you evaluate notebooks; otherwise not required for prose edits |
| Jupyter | `d2lbook` dependency | `pip install jupyter==1.0.0` per setup.py |
| Sphinx | HTML/LaTeX backend (pulled in by `d2lbook`) | Not a direct install |
| XeLaTeX + CTEX + Source Han fonts | PDF build only | `sudo apt install texlive-full texlive-xetex fonts-wqy-microhei` on Debian-family, plus manual Source Han install |
| OmniGraffle | Diagram editing only | macOS-only; graffle/ source is not required to build the book |
| Git | Source control | Standard |

### 2.2 For running CI (platform operator)

| Service | Use |
|---------|-----|
| AWS Batch | GPU + CPU compute environments |
| AWS ECR | Hosts the four Docker images (d2l-zh-torch, -tf, -mxnet, -paddle, d2l-builder) |
| AWS S3 | Build cache (`preview.d2l.ai/ci_cache_*`), preview site (`preview.d2l.ai/<JOB>/`), production site (`zh-v2.d2l.ai/`) |
| AWS IAM OIDC | GitHub Actions → AWS credentials (see `id-token: write` in `ci.yml`) |
| GitHub Actions | Workflow dispatch (`ubuntu-latest` runners for orchestration only; heavy work runs on Batch) |

### 2.3 Framework version matrix (as of commit `e6b18cc`)

Determined from ECR Dockerfiles and `print_versions.py`:

| Framework | Base image | Installed version | Extras |
|-----------|------------|-------------------|--------|
| PyTorch | `nvcr.io/nvidia/pytorch:22.06-py3` (PyTorch 1.12, May 2022) | `torch`, `torchvision`, `gym`, `gpytorch`, `syne_tune` | GPU, CUDA included |
| TensorFlow | (TF image, see `Dockerfile.d2l-zh-tf`) | `tensorflow`, `tensorflow_probability` | GPU |
| MXNet | (MXNet image) | `mxnet` (legacy, frozen) | GPU |
| Paddle | (Paddle image) | `paddle` | GPU |
| Builder | (CPU base) | `d2lbook==1.0.0` | CPU only, used to assemble HTML/PDF |

### 2.4 Reader-facing runtime

The `d2l` package pins:

```text
jupyter==1.0.0
numpy==1.21.5
matplotlib==3.5.1
requests==2.25.1
pandas==1.2.4
```

These are the pins **for the helper package**, not for readers' framework choice. The tight pins exist because the book's evaluated notebook outputs were generated against these versions; newer versions may produce different tensor shapes, warning messages, or plot styling that diverges from the printed/rendered book.

---

## 3. Configuration Reference

### 3.1 `config.ini` — the master switchboard

`d2lbook` reads `config.ini` for every build action. The file is organized into sections, each controlling a pipeline stage.

> [!IMPORTANT]
> `config.ini` is the single most operationally significant file in the repository. Changes here affect all four framework builds, HTML output, PDF rendering, deployment targets, and the `d2l` package itself. Always test a full build cycle after modifying it.

#### `[project]`

| Key | Value at `e6b18cc` | Effect |
|-----|-------------------|--------|
| `name` | `d2l-zh` | Output filenames (`d2l-zh.pdf`, `d2l-zh.zip`) |
| `title` | `动手学深度学习` | Cover title in HTML & PDF |
| `author` | Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola | Title page |
| `copyright` | 2022, All authors. Licensed under CC-BY-SA-4.0 and MIT-0. | Colophon |
| `release` | `2.0.0` | Edition identifier |
| `lang` | `zh` | Language tag used in `d2lbook deploy` S3 paths (`${lang}-v2.d2l.ai/`) |

#### `[translation]`

| Key | Value | Effect |
|-----|-------|--------|
| `origin_repo` | `d2l-ai/d2l-en` | Upstream English source repo |
| `origin_lang` | `en` | Source language |
| `translator` | `aws` | Translation provider tag |

#### `[build]`

| Key | Effect |
|-----|--------|
| `notebooks = *.md */*.md` | Markdown files at repo root and one level down are evaluated as Jupyter notebooks. |
| `resources = img/ d2lzh/ d2l.bib setup.py` | Copied verbatim into `_build/`. Note `d2lzh/` is legacy — the current package is `d2l/`. |
| `exclusions = */*_origin.md README.md STYLE_GUIDE.md INFO.md CODE_OF_CONDUCT.md CONTRIBUTING.md contrib/*md` | Skipped. |
| `eval_notebook = True` | Execute notebooks during build (set False for a fast structure-only build). |
| `tabs = mxnet, pytorch, tensorflow, paddle` | The four frameworks that fan out from each source chapter. |
| `sphinx_configs` | Inline Sphinx config: numbering format (`图%s`, `表%s`, `%s节`), full LaTeX preamble with CTEX, Source Han fonts, fancy page style, custom verbatim colors. |

`exclusions` is where you add a chapter-in-progress that should not yet compile.

#### `[html]`

`header_links` defines the navbar (PDF download, Jupyter ZIP, course link, GitHub, English edition). `favicon` and `html_logo` point to `static/` assets.

#### `[pdf]`

| Key | Effect |
|-----|--------|
| `post_latex = ./static/post_latex/main.py` | Post-processor run on the Sphinx-generated `.tex` before `xelatex`. |
| `latex_logo = static/logo.png` | Title page logo. |
| `bibfile = d2l.bib` | BibTeX file for references. |

#### `[library]` and `[library-<framework>]`

- `[library] version_file = d2l/__init__.py` — d2lbook reads `__version__` from this file.
- `[library-<fw>] lib_file = d2l/<fw>.py` — the file where `#@save` cells are collected.
- `[library-<fw>] lib_name = <name>` — the import alias (`np` for mxnet, `torch` for torch, `tf` for tensorflow, `paddle` for paddle).
- `simple_alias`, `fluent_alias`, `alias`, `reverse_alias` — four classes of rewrite rules that transform `d2l.foo(...)` calls in Markdown code cells into framework-native calls.

#### `[deploy]`

- `other_file_s3urls` — legacy ZIPs to include in release artifacts (the 1.0, 1.1, 2.0.0 notebook bundles).
- `google_analytics_tracking_id = UA-96378503-2` — embedded in generated HTML.

#### `[colab]` and `[sagemaker]`

Define derivative notebook repos for each framework:

| Framework | Colab repo | SageMaker repo |
|-----------|------------|----------------|
| mxnet | `d2l-ai/d2l-zh-colab` | `d2l-ai/d2l-zh-sagemaker` |
| pytorch | `d2l-ai/d2l-zh-pytorch-colab` | `d2l-ai/d2l-zh-pytorch-sagemaker` |
| tensorflow | `d2l-ai/d2l-zh-tensorflow-colab` | `d2l-ai/d2l-zh-tensorflow-sagemaker` |
| paddle | `d2l-ai/d2l-zh-paddle-colab` | `d2l-ai/d2l-zh-paddle-sagemaker` |

These are **push-only targets**. Never edit them directly; they are regenerated every release.

#### `[slides]`

| Key | Value |
|-----|-------|
| `github_repo = pytorch, d2l-ai/d2l-zh-pytorch-slides` | Slide-deck output repo (PyTorch only) |

### 3.2 Environment variables affecting CI

| Variable | Source | Effect |
|----------|--------|--------|
| `SOURCE_REF` | `submit-job.py --source-ref` | Git ref to check out (defaults to `master`). |
| `WORK_DIR` | `--work-dir` | Directory to `cd` into before running `COMMAND`. |
| `SAFE_TO_USE_SCRIPT` | `--safe-to-use-script` | When false, `d2l_job.sh` discards PR-supplied workflow scripts and uses master's. |
| `ORIGINAL_REPO` | `--original-repo` (default `d2l-zh`) | Used to construct clone URL. |
| `ORIGINAL_ORG` | positional arg 9 in `d2l_job.sh` (default `d2l-ai`) | Hardcoded in production to prevent org-swap attacks. |
| `DISABLE_CACHE` | read inside `build_*.sh` | When `true`, skips S3 cache restore (forces full re-evaluation). |

### 3.3 Local configuration overrides

`d2lbook` does not natively support a `config.local.ini` override file. To do per-environment config:

- **Option A:** Checkout a separate branch with a modified `config.ini` (ephemeral, never push).
- **Option B:** Use shell-level env vars consumed by `build_*.sh` (e.g., `DISABLE_CACHE=true`).

There is no per-user personal-config file; the canonical `config.ini` is always the one in the repo.

---

## 4. Deployment & Scaling

### 4.1 Branch model

| Branch | Purpose |
|--------|---------|
| `master` | Default. PRs land here. CI runs on every push; preview-only deploys. |
| `release` | Fast-forward-only from master when a release is cut. Triggers production deploy + `d2lbook build pkg`. |

### 4.2 Deployment flow

`build_and_deploy.sh` branches on `JOB_NAME`:

```bash
if [[ "$JOB_NAME" == "$REPO_NAME/release" ]]; then
  d2lbook build pkg
  d2lbook deploy html pdf pkg colab sagemaker slides --s3 "s3://${LANG}-v2.d2l.ai/"
else
  d2lbook deploy html pdf --s3 "s3://preview.d2l.ai/${JOB_NAME}/"
fi
```

**Release path** (pushed to `release` branch):
- Builds the `d2l` wheel (`d2lbook build pkg`).
- Deploys HTML, PDFs, wheel, Colab notebooks, SageMaker notebooks, and slides to `s3://zh-v2.d2l.ai/`.
- CloudFront fronts the S3 bucket; updates are live within minutes (subject to CF cache invalidation).

**Preview path** (any other branch or PR):
- Deploys only HTML and PDFs to `s3://preview.d2l.ai/<JOB_NAME>/`.
- GitHub Actions posts a comment on the PR with the preview URL.

> [!TIP]
> Use `[skip frameworks]` in commit messages for prose-only PRs where re-evaluating notebooks is wasteful. This skips all four GPU evaluation jobs and goes straight to the build-and-deploy assembly using cached outputs.

### 4.3 Cutting a release

1. Bump `d2l/__init__.py` `__version__` on `master` (semantic version — major if API breaks, minor for new chapters/`#@save` functions, patch for prose/figures).
2. Merge all pending content PRs into `master`.
3. Verify master CI is green across all four frameworks and the builder job.
4. `git checkout release && git merge --ff-only master && git push` (fast-forward only; never force-push `release`).
5. Release-branch CI triggers production deploy.
6. (External, manual) Publish the wheel to PyPI with `twine upload`.
7. Tag the repo: `git tag v2.0.0 && git push --tags`.

### 4.4 Derivative notebook repos

Four Colab repos and four SageMaker repos are pushed on every release. These are **regenerated** from the build output. Do not cherry-pick into them. If an external contributor PRs into one of these repos, redirect them to `d2l-zh` itself.

### 4.5 Legacy ZIPs

`config.ini [deploy] other_file_s3urls` lists three legacy bundles:

```text
s3://d2l-webdata/releases/d2l-zh/d2l-zh-1.0.zip
s3://d2l-webdata/releases/d2l-zh/d2l-zh-1.1.zip
s3://d2l-webdata/releases/d2l-zh/d2l-zh-2.0.0.zip
```

Do not delete on rotation — they are referenced by external courses, papers, and books.

> [!NOTE]
> Deployed URLs: `https://zh-v2.d2l.ai/` (live HTML), `https://zh-v2.d2l.ai/d2l-zh.pdf` (MXNet PDF), `https://zh-v2.d2l.ai/d2l-zh-pytorch.pdf` (PyTorch PDF), `https://zh-v2.d2l.ai/d2l-zh.zip` (notebook ZIP), `https://preview.d2l.ai/d2l-zh/PR-<N>-<SHA>/` (PR previews).

---

## 5. Operations

### 5.1 Local build

From `INFO.md`:

```bash
# HTML
make html

# PDF (requires XeLaTeX + Chinese fonts)
make pdf
```

`make` targets map to `d2lbook build` commands. A typical local workflow for prose-only edits:

```bash
d2lbook build html          # no eval, for fast iteration
d2lbook build eval --tab pytorch   # eval only PyTorch tab
d2lbook build pdf --tab pytorch    # PDF for PyTorch tab
```

To build everything the way CI does:

```bash
# per framework
d2lbook build eval --tab mxnet
d2lbook build eval --tab pytorch
d2lbook build eval --tab tensorflow
d2lbook build eval --tab paddle

# final assembly
d2lbook build html
d2lbook build pdf          # default tab (mxnet)
d2lbook build pdf --tab pytorch
d2lbook build pkg
```

Output lands in `_build/`.

> [!TIP]
> For a single-chapter rebuild: `d2lbook build eval --tab pytorch --md chapter_linear-networks/*`. `d2lbook` incrementally evaluates based on modification time — editing one `.md` and rebuilding only runs that file.

### 5.2 Sanity checks

Run during CI before per-framework evaluation:

```bash
d2lbook build outputcheck tabcheck
```

- `outputcheck`: ensures no Jupyter cell output has been committed accidentally (the book's outputs are always regenerated by CI).
- `tabcheck`: ensures every `:begin_tab:<fw>` section has a matching implementation for all frameworks declared in `[build] tabs`.

A `tabcheck` failure means you added or edited a multi-framework code section without updating all four tabs.

### 5.3 CI entry points

Five jobs, four parallel + one gated:

1. `build_torch` → AWS Batch → `build_pytorch.sh` (evaluate PyTorch tab)
2. `build_tf` → AWS Batch → `build_tf.sh`
3. `build_mxnet` → AWS Batch → `build_mxnet.sh`
4. `build_paddle` → AWS Batch → `build_paddle.sh`
5. `build_and_deploy` (needs all four above) → AWS Batch → `build_and_deploy.sh` (HTML + PDFs + conditional deploy)

> [!NOTE]
> Commit message skip tags: `[skip torch]`, `[skip tf]`, `[skip mxnet]`, `[skip paddle]`, `[skip frameworks]` (all four evals), `[skip builder]` (final assembly).

### 5.4 `ci/submit-job.py` — Batch job submission

CLI flags (argparse):

| Flag | Type | Default | Meaning |
|------|------|---------|---------|
| `--profile` | str | None | AWS profile name |
| `--region` | str | `us-west-2` | AWS region |
| `--name` | str | `d2l-ci` | AWS Batch job name |
| `--job-type` | str | `ci-cpu` | Queue/definition selector |
| `--source-ref` | str | `master` | Git ref to check out |
| `--work-dir` | str | `.` | Directory inside repo |
| `--command` | str | `git rev-parse HEAD \| tee stdout.log` | Command executed inside container |
| `--remote` | str | `https://github.com/d2l-ai/d2l-zh` | Git remote |
| `--safe-to-use-script` | flag | false | Trust PR-supplied workflow_scripts? |
| `--original-repo` | str | `d2l-zh` | Repository name |
| `--wait` | flag | false | Block until Batch job finishes |
| `--timeout` | int | `7200` | Job timeout in seconds (2 hours) |

Job type → queue mapping:

| `--job-type` | Queue | Definition |
|--------------|-------|------------|
| `ci-cpu` | `D2L-CI-CPU` | `d2l-ci-cpu-builder:2` |
| `ci-gpu-torch` | `D2L-CI-GPU` | `d2l-ci-zh-gpu-torch:1` |
| `ci-gpu-tf` | `D2L-CI-GPU` | `d2l-ci-zh-gpu-tf:1` |
| `ci-gpu-mxnet` | `D2L-CI-GPU` | `d2l-ci-zh-gpu-mxnet:1` |
| `ci-gpu-paddle` | `D2L-CI-GPU` | `d2l-ci-zh-gpu-paddle:1` |

### 5.5 Inside the Batch container — `d2l_job.sh`

`ci/docker/d2l_job.sh` is the container entrypoint:

```bash
git clone https://github.com/"$ORIGINAL_ORG"/"$ORIGINAL_REPO".git
cp -R "$ORIGINAL_REPO"/.github/workflow_scripts .

cd "$ORIGINAL_REPO"
git remote set-url origin $REMOTE
git fetch origin $SOURCE_REF:working
git checkout working

# Restore mtimes from git so S3 cache keys are stable
git timesync *.md **/*.md

# Untrusted PR guard
if [[ ${SAFE_TO_USE_SCRIPT,,} != true ]]; then
    rm -rf .github/workflow_scripts
    mv ../workflow_scripts .github/
fi

cd $WORK_DIR
/bin/bash -o pipefail -c "eval $COMMAND"
```

Key points:
- **`git timesync`** resets file modification times after a fresh clone so the S3 evaluation cache is valid.
- **Workflow script replacement** prevents untrusted PRs from injecting code into build scripts.

### 5.6 Cache operations

Two caches:

1. **S3 evaluation cache** (`s3://preview.d2l.ai/ci_cache_{pr,push}/<repo>-<branch>/_build/`): per-framework evaluated `.md` files. Keyed by file mtime.
2. **Docker layer cache** (ECR): built by `.github/workflows/build-docker.yml` on demand.

To invalidate: `.github/workflows/clear-cache.yml` is a manual workflow dispatch.

---

## 6. Performance

### 6.1 Build time optimization

The primary performance lever is the **S3 evaluation cache**. Chapters only re-evaluate when their modification time changes. `git timesync` ensures clones restore original mtimes.

| Optimization | Effect |
|---|---|
| S3 cache hit | Skip chapter eval entirely (saves GPU minutes) |
| `[skip <fw>]` tags | Skip specific framework evals for prose-only changes |
| `[skip frameworks]` | Skip all evals, use cached outputs for HTML/PDF assembly |
| Split CPU/GPU | GPU only used for notebook execution; assembly runs on cheap CPU |

### 6.2 Reader-side performance

The `d2l` helper library uses `get_dataloader_workers()` to set data-loading parallelism:

```python
def get_dataloader_workers():
    return 0 if sys.platform.startswith('win') else 4
```

On Windows, multi-process data loading is disabled (returns 0 workers) because Windows lacks fast fork. On Linux/macOS, 4 worker processes are used.

---

## 7. Security

### 7.1 Untrusted PR handling — `SAFE_TO_USE_SCRIPT`

The primary threat model: a contributor submits a PR from a fork that modifies `.github/workflow_scripts/build_torch.sh` to exfiltrate AWS credentials or deploy malicious content.

Mitigation in `d2l_job.sh`:

```bash
if [[ ${SAFE_TO_USE_SCRIPT,,} != true ]]; then
    rm -rf .github/workflow_scripts
    mv ../workflow_scripts .github/   # master's copy
fi
```

`SAFE_TO_USE_SCRIPT` should be set to `true` only when the PR author has write access to `d2l-ai/d2l-zh`.

> [!WARNING]
> Never edit `d2l_job.sh` to bypass the `SAFE_TO_USE_SCRIPT` guard for convenience. If a PR fails for lack of access, rebase it from a trusted branch inside the org. Removing this guard exposes AWS credentials to arbitrary code execution via PR.

### 7.2 `pull_request_target` — know what it means

`ci.yml` uses `pull_request_target` (not `pull_request`). This event fires with the secrets and permissions of the **target repository**, and checks out the **base branch** by default. The `d2l-zh` workflow checks out the PR's SHA into the Batch job (`$SOURCE_REF`) but executes the build scripts from master (via `SAFE_TO_USE_SCRIPT=false`). The GitHub Actions runner itself only does orchestration — it never runs PR code.

### 7.3 Organization pinning

```bash
ORIGINAL_ORG=${9:-'d2l-ai'}
# TODO @anirudhdagar: hardcode ORIGINAL_ORG
```

The `d2l-infra` repo defines the Batch job definition arguments and does not expose position 9 as a configurable input — so in production `ORIGINAL_ORG` is effectively pinned to `d2l-ai`. The TODO flags the desire to remove the default and make it fail loudly if not set.

> [!CAUTION]
> The default fallback in `ORIGINAL_ORG=${9:-'d2l-ai'}` means an attacker who controls the Batch job parameter 9 could redirect the clone. Production is safe because the Batch job definition doesn't expose this parameter, but the code-level guard should be hardened by removing the default.

### 7.4 Secret handling

The workflow uses OIDC (`permissions: id-token: write`) to assume an AWS role without long-lived secrets stored in GitHub. No AWS access keys in the repo. The IAM role attached to the Batch job has scoped access to:
- The CI cache S3 bucket
- The ECR repositories for the framework images
- The `preview.d2l.ai` S3 bucket

Release deploys to `zh-v2.d2l.ai` run under a potentially broader IAM role. Audit quarterly.

### 7.5 No telemetry from reader code

The shipped `d2l` package contains no telemetry. `setup.py` pins five widely-used packages and declares nothing network-facing beyond `requests`. Readers running the notebooks locally never phone home.

Google Analytics `UA-96378503-2` is injected into the HTML build only (`config.ini [deploy]`). It tracks web visits to `zh-v2.d2l.ai`, not reader code execution.

### 7.6 Content licensing

- Code (`d2l/`, chapter code cells, build scripts): MIT-0 per `setup.py`.
- Text and diagrams: CC-BY-SA-4.0 per `config.ini [project] copyright`.

Forks that modify content must retain attribution and share-alike under CC-BY-SA-4.0.

---

## 8. Developer Internals

### 8.1 The d2l Python package

#### Package layout

```
d2l/
├── __init__.py        # Holds __version__ = "2.0.0"
├── mxnet.py           # MXNet backend (import d2l.mxnet as d2l)
├── torch.py           # PyTorch backend
├── tensorflow.py      # TensorFlow backend
└── paddle.py          # PaddlePaddle backend
```

Each backend file is **generated** (or manually synchronized) so that all four expose the same Python API surface.

#### `#@save` — how book code becomes library code

In any chapter, a code cell annotated with `#@save` is automatically collected into the library:

```python
#@save
def load_data_fashion_mnist(batch_size, resize=None):
    """Download the Fashion-MNIST dataset and load it into memory."""
    # ...
```

When `d2lbook build pkg` runs, it scans all chapters, extracts every `#@save`-annotated definition, and writes them to `d2l/<framework>.py` (per tab).

> [!IMPORTANT]
> Do not add `#@save` casually. Each annotation is an implicit public API commitment. Any rename or signature change is a breaking change for reader notebooks, later chapters, and downstream forks. The version in `d2l/__init__.py` must be bumped for API changes.

#### Alias system — how `d2l.foo(...)` becomes framework-native

Four classes of alias rules live in `config.ini [library-<fw>]`:

**`simple_alias`** — direct attribute mapping:
- `d2l.ones(...)` → `np.ones(...)` (MXNet) / `torch.ones(...)` (PyTorch)
- `d2l.tensor(...)` → `np.array(...)` (MXNet) / `torch.tensor(...)` (PyTorch)

**`fluent_alias`** — method calls on the first argument:
- `d2l.reshape(x, shape)` → `x.reshape(shape)` (PyTorch)
- `d2l.numpy(x)` → `x.detach().numpy()` (PyTorch) / `x.asnumpy()` (MXNet)

**`alias`** — arbitrary Python expressions:
- `d2l.nn_Module` → `nn.Block` (MXNet) / `nn.Module` (PyTorch) / `tf.keras.Model` (TF) / `nn.Layer` (Paddle)

**`reverse_alias`** — regex substitutions on evaluated output.

#### Version pinning

`config.ini [library] version_file = d2l/__init__.py` tells `d2lbook` where the version lives. `d2lbook build pkg` reads this, tags the wheel, and publishes with that version. Do not edit `setup.py`'s version directly; it reads from `d2l.__version__`.

### 8.2 LaTeX post-processing (`static/post_latex/main.py`)

Chinese typography requires fixes that Sphinx's LaTeX writer does not handle natively. The script mutates the `.tex` file in place with three transformations:

#### `_unnumber_chaps_and_secs(lines)`

```python
NUM_UNNUMBERED_CHAPS = 3   # Preface, Installation, Notation
TOC2_START_CHAP_NO   = 5   # Preliminaries onward
```

- First three `\chapter{...}` → `\chapter*{...}` (unnumbered but in TOC).
- All sections within those chapters are likewise starred.
- From chapter 5 onward, TOC depth drops to 2 (suppresses subsections in PDF TOC).
- `\section{小结...`, `\section{练习...` are unnumbered everywhere (summaries and exercises).

#### `_sec_to_chap(lines)`

Regex rewrites `{Section \ref{\detokenize{...chapter_.../index:...}}}` to `Chapter \ref`. Uses the `regex` module for recursive balanced braces.

#### Commented-out functions

`_edit_titlepage` (blank date on title page) and `_delete_discussions_title` (remove forum QR codes) are both disabled. Dead code — either uncomment or delete.

### 8.3 Content conventions

These conventions affect build correctness, not just style:

**Framework tabs:** every framework-specific code block is wrapped in `:begin_tab:<fw>` / `:end_tab:` directives. `d2lbook build tabcheck` fails if any tab is missing.

**`#@save` rules:**
- Always place `#@save` at the top of the cell, first line after the opening fence.
- Don't rename `#@save`-marked functions in later chapters — collisions silently overwrite.

**Cross-references:**
- Section labels: `sec_<name>`. Referenced with `:numref:`.
- Chapter labels: `chap_<name>` on `index.md`. Referenced with `:numref:`.
- Figure labels: `fig_<name>`. Must use `:numref:` for "图 3.2" style numbering.
- Equation labels: `eq_<name>`. Must use `:eqref:`.

**No Jupyter outputs in source:** `d2lbook build outputcheck` fails if any `.md` contains embedded notebook output.

### 8.4 Translation workflow

| File | Role |
|------|------|
| `chapter_<x>/<y>.md` | Chinese translated content (built). |
| `chapter_<x>/<y>_origin.md` | Snapshot of the English source at the time of last translation. Excluded from build. |

When upstream `d2l-ai/d2l-en` updates a section:
1. Translator pulls the new English version.
2. Diffs against the local `_origin.md` to see what changed.
3. Applies the diff to the Chinese `.md`.
4. Overwrites `_origin.md` with the new English source.
5. Opens a PR.

### 8.5 Diagrams — OmniGraffle

Diagrams live in `graffle/` (source) and `img/` (rendered PNG/SVG). OmniGraffle is macOS-only. Fork operators without macOS can replace with draw.io or Mermaid — commit both source and rendered image.

### 8.6 Class hierarchy (126 classes)

The `d2l` package across all four backends defines these major class families:

| Class | Purpose | Backends |
|-------|---------|----------|
| `Timer` | Elapsed time measurement | All 4 |
| `Accumulator` | Running sums across minibatches | All 4 |
| `Animator` | Live-updating notebook plots | All 4 |
| `Vocab` | Token→index vocabulary | All 4 |
| `SeqDataLoader` | Sequence data iterator | All 4 |
| `RNNModelScratch` / `RNNModel` | From-scratch and framework RNN wrappers | All 4 |
| `Encoder` / `Decoder` / `EncoderDecoder` | Seq2seq abstractions | All 4 |
| `Seq2SeqEncoder` | GRU-based encoder | All 4 |
| `AdditiveAttention` / `DotProductAttention` | Attention scoring functions | All 4 |
| `MultiHeadAttention` | Multi-head attention layer | All 4 |
| `PositionalEncoding` | Sinusoidal position embedding | All 4 |
| `PositionWiseFFN` / `AddNorm` | Transformer sub-layers | All 4 |
| `EncoderBlock` / `TransformerEncoder` | Transformer building blocks | All 4 |
| `Residual` | ResNet residual block | All 4 |
| `BERTEncoder` / `MaskLM` / `NextSentencePred` / `BERTModel` | BERT pre-training | All 4 |
| `BananasDataset` / `VOCSegDataset` | CV datasets | All 4 |
| `TokenEmbedding` | Pre-trained word vectors | All 4 |
| `SNLIDataset` | NLI dataset | All 4 |
| `Benchmark` | Context-manager timer | All 4 |

Each class exists independently in each backend file with framework-specific base classes (e.g., `nn.Module` for PyTorch, `nn.Block` for MXNet, `tf.keras.Model` for TF, `nn.Layer` for Paddle).

---

## 9. Integration Points

### 9.1 External dependencies

| System | Integration | Direction |
|--------|-------------|-----------|
| AWS Batch | CI execution | Push (submit jobs) |
| AWS ECR | Docker images | Pull (containers) |
| AWS S3 | Cache + hosting | Read/write |
| CloudFront | CDN for zh-v2.d2l.ai | Read (serves site) |
| PyPI | `d2l` package distribution | Push (publish) |
| GitHub Actions | CI orchestration | Bidirectional |
| `d2l-ai/d2l-en` | Upstream English source | Pull (translation sync) |
| `d2lbook` (PyPI) | Build tool | Pull (dependency) |
| Colab/SageMaker repos | Derivative notebooks | Push (generated) |

### 9.2 Forking — Running your own edition

**What you can reuse without infrastructure:**
- The entire `chapter_*/` tree — edit prose.
- `config.ini` — change project/deploy settings.
- `static/post_latex/main.py` — edit the `UNNUMBERED` set for your language.
- `setup.py` — change package name if forking the library.

**What requires infrastructure replacement:**
- AWS Batch + ECR + S3 (your own accounts and Docker images).
- CI job definitions (create equivalents in your AWS account).
- Domain + CloudFront.

**Alternative infrastructures:**
- GitHub Actions with GPU runners (replace `submit-job` with direct workflow).
- Self-hosted Jupyter cluster for evaluation.
- CPU-only build (skip `eval_notebook`; prose only).

**What NOT to change without coordination:**
- The `d2l` package name on PyPI (owned by upstream). Pick a different name for forks.
- Colab/SageMaker repo targets — repoint or blank out.
- Google Analytics ID — replace with yours or remove.

---

## 10. Troubleshooting

### Common CI failures

| Symptom | Cause | Fix |
|---------|-------|-----|
| `tabcheck` fails with "tab X missing in file Y" | New section added without all framework tabs | Add the missing tab; rerun CI |
| PyTorch eval OOMs | Added a chapter with a larger model | Smaller batch size in the cell, or request a larger GPU instance |
| Cache hit rate near zero | `git timesync` didn't run or failed | Check d2l_job.sh output; ensure `git-timesync` binary is in the image |
| XeLaTeX PDF build fails with "Font undefined" | Source Han fonts missing from builder image | Rebuild `d2l-builder` via `build-docker.yml` |
| "no such file d2l/mxnet.py" during eval | MXNet tab skipped but later code imports it | Remove `[skip mxnet]` |
| Preview shows stale content | CloudFront edge cache | Manually invalidate or wait TTL |

> [!WARNING]
> `d2lbook build pkg` produces its API from evaluated output, not raw source. Running `d2lbook build pkg` without first running `d2lbook build eval --tab <fw>` will produce an incomplete or outdated library. Always eval before pkg.

### `d2lbook build pkg` produces wrong API

- Check that the `#@save` annotation is on the first line of the cell. A blank line before it defeats extraction.
- Check that `config.ini [library-<fw>] lib_file` points to the right file.
- Re-run `d2lbook build pkg` after `d2lbook build eval --tab <fw>`.

### When the English source has diverged significantly

Translation-workflow drift: after a major upstream refactor, every chapter may have diverging `_origin.md` files.

1. `git log --oneline upstream/master -- chapter_<x>/<y>.md` in the `d2l-en` checkout.
2. Translate incrementally per commit, not per file, if the drift is multi-year.
3. If a chapter was removed upstream, flag for removal and open a PR to drop both files.

> [!NOTE]
> Graffle source without macOS: OmniGraffle is macOS-only. Use a cloud Mac (MacStadium), or replace the diagram with draw.io/Mermaid/TikZ. Commit the new source alongside the rendered image; update `STYLE_GUIDE.md` if this becomes policy. Do not commit half-converted `.graffle` files.

### CI troubleshooting flowchart

```
CI failed?
├── One framework job failed → check build_<fw>.sh logs in AWS Batch
│   ├── "CUDA out of memory" → reduce batch_size in offending chapter
│   ├── "eval timed out" → split chapter or raise timeout in ci/submit-job.py
│   ├── "tabcheck failed"  → missing tab block; add it
│   └── "outputcheck failed" → strip committed outputs, re-commit
├── All four framework jobs green, build_and_deploy failed →
│   ├── "xelatex error" → new Chinese character without font coverage
│   │                      or mis-balanced LaTeX env from Markdown typo
│   ├── "Sphinx reference not found" → broken :numref: or :ref: label
│   └── "deploy to S3 failed" → IAM role perms; check OIDC assume-role
└── All jobs green, preview shows wrong content →
    ├── CloudFront cache (wait TTL or manual invalidation)
    └── Browser cache (hard reload)
```

---

## 11. Known Issues & Tech Debt

### 11.1 MXNet is end-of-life

Apache archived MXNet in late 2023. `nvcr.io/mxnet` images are no longer updated. The `d2l/mxnet.py` backend works with pinned versions but will degrade as CUDA drivers advance.

Mitigation options: freeze MXNet builds, remove from `[build] tabs`, or maintain a forked image indefinitely.

### 11.2 `d2l==0.17.6` hardcoded in build scripts

CI installs an old published `d2l` for bootstrap. The source tree builds the *next* `d2l` via `d2lbook build pkg`. When the source tree's exports diverge from `0.17.6`, compatibility issues arise.

**Fix:** use `d2l==$(python -c "from d2l import __version__; print(__version__)")` or lock to whatever was last published.

### 11.3 `ORIGINAL_ORG` default in `d2l_job.sh`

The default fallback should be removed and hardcoded to prevent potential org-swap attacks.

### 11.4 Dead code in `main.py`

`_delete_discussions_title` and `_edit_titlepage` are commented out. Either uncomment or delete.

### 11.5 `numpy==1.21.5` pin is stale

NumPy 1.21.x is past its maintenance window. Readers get conflicts with newer ML libraries. Update pin to a supported version, retest all notebooks.

### 11.6 No test suite

`d2l/` has no tests. Correctness is verified only by CI producing a successful build. Low-hanging fruit: add `tests/test_d2l_api.py` that imports each backend and asserts every expected function exists.

### 11.7 `contrib/to-rm-mx-contrib-text` marked for removal

Directory name literally says "to-rm". Removing it is a mechanical PR.

### 11.8 Brittle Dockerfile with pinned 2022 base image

`Dockerfile.d2l-zh-torch` uses `nvcr.io/nvidia/pytorch:22.06-py3` from May 2022. Accumulates CVEs. Set a monthly calendar reminder to rebuild or automate via cron workflow.

---

## 12. Quick Reference

### Command cheat sheet

```bash
# Local: fast iteration on prose (no eval)
d2lbook build html

# Local: single-framework PDF
d2lbook build eval --tab pytorch && d2lbook build pdf --tab pytorch

# Local: package build (all frameworks)
for fw in mxnet pytorch tensorflow paddle; do d2lbook build eval --tab $fw; done
d2lbook build pkg

# Sanity
d2lbook build outputcheck tabcheck

# CI: submit a Batch job manually (AWS CLI configured)
python ci/submit-job.py \
  --job-type ci-gpu-torch \
  --source-ref refs/pull/500/head \
  --safe-to-use-script \
  --command "chmod +x .github/workflow_scripts/build_pytorch.sh && .github/workflow_scripts/build_pytorch.sh d2l-zh master ci_cache_pr" \
  --wait

# CI: force preview rebuild bypassing cache
DISABLE_CACHE=true ./.github/workflow_scripts/build_pytorch.sh d2l-zh master ci_cache_pr
```

### File map

| Concern | Files |
|---------|-------|
| Book content | `chapter_*/**.md`, `index.md` |
| Translation state | `chapter_*/*_origin.md` |
| Python package | `d2l/{__init__,mxnet,torch,tensorflow,paddle}.py` |
| Package deps | `setup.py` |
| Build config | `config.ini` |
| Bibliography | `d2l.bib`, `chapter_references/` |
| Diagrams | `graffle/**.graffle` (source), `img/**.{png,svg}` (rendered) |
| Web assets | `static/` (logo, favicon, frontpage) |
| LaTeX post-proc | `static/post_latex/main.py` |
| CI workflow | `.github/workflows/ci.yml` |
| CI scripts | `.github/workflow_scripts/*.sh` |
| Batch submit | `ci/submit-job.py` |
| Docker images | `ci/docker/Dockerfile.d2l-*`, `d2l_job.sh`, `entry.sh`, `print_versions.py`, `git-timesync`, `login_ecr.sh` |
| Style / terms | `STYLE_GUIDE.md`, `TERMINOLOGY.md`, `INFO.md` |

### Entry points reference

| Entry point | Location | Purpose |
|-------------|----------|---------|
| `setup.py` | Repo root | `pip install -e .` or `pip install d2l` |
| `static/post_latex/main.py` | — | LaTeX post-processor invoked by `d2lbook` |
| `ci/submit-job.py` | CI | AWS Batch job submitter |
| `ci/docker/d2l_job.sh` | Docker images | Container ENTRYPOINT |
| `ci/docker/entry.sh` | Docker images | Calls `print_versions.py` then drops into bash |
| `ci/docker/print_versions.py` | Docker images | Diagnostic banner at job start |

### Useful external projects

| Project | Purpose |
|---------|---------|
| `d2l-ai/d2l-en` | English source of truth |
| `d2l-ai/d2lbook` | The book build tool (external dependency) |
| `d2l-ai/d2l-{zh,en}-{pytorch,tensorflow,mxnet,paddle}-colab` | Derivative Colab repos |
| `d2l-ai/d2l-{zh,en}-{pytorch,tensorflow,mxnet,paddle}-sagemaker` | Derivative SageMaker repos |
| `d2l-ai/d2l-zh-pytorch-slides` | Slide decks |

---

_Generated May 2026 from d2l-ai/d2l-zh at e6b18cc. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._