DocCompiler: agents that write user and admin guides

1Problem

Open-source docs are often thin or stale, and AI-written docs invent commands and config keys. I wanted two guides per repository, built from the source, in one fixed format.

For context: PANW's Aug 2026 release announcing its Frontier AI Critical Defense Program notes that PANW used frontier AI models to "uncover more than 14,000 previously unknown vulnerabilities in open source software" (Palo Alto Networks press release, Aug 2026). DocCompiler is not part of that program and does no vulnerability work. It is on the documentation side.

2What I built

A deterministic step builds a briefing for each repository from its source, with no LLM involved: entry points, config ground truth, a module index and real issues. Agents write a User Guide and an Admin Guide from it under written rules: the briefing wins, never invent, and if extraction found nothing, don't write about it. Where the briefing fell short, the agents also read the cloned source, so not every statement traces back to the briefing.

3How AI was used

Claude Code agents did all the writing, in 9 agent roles (worker, reviewer, monitor and others) on schedules. Workers share a learnings file: 483 lessons from 434 repositories. A reviewer agent proposes prompt changes, and a person applies them. The idea was mine; I ran the deploys by hand and I decide on every prompt change.

4Results

572 repositories from 497 owners, mostly April to June 2026, when scheduled agents ran mostly unattended (72, 242 and 251 repositories a month). That is 572 User Guides and 568 Admin Guides. Every Admin Guide has a Security section (568 of 568), 116 have a hardening heading, and 109 mention a threat model. The guides are substantial: the median User Guide is about 5,200 words and the median Admin Guide about 5,600, about 6.7 million words in all.

These are unofficial, AI-generated guides, not reviewed or endorsed by the projects they describe. Most of them are in the read-only guide library.

5Status and limits

Dormant proof of concept. Accuracy is checked by sampling, one repository per review. The last three scored 49, 52 and 55 out of 70; with one repository each, that is not a trend. The latest review checked about 30 claims against upstream source and found 1 factual error, 1 evasive answer and 1 minor issue.

Fewer than 1 in 10 guides cite source files by line. On very large monorepos the briefing can miss key parts: in one repository of over 28,000 files it missed the main command-line entry point. The reviewer's prompt suggestions wait on me; the framework is unchanged since May 2026.

6What I learned