# Screenshot to Code — Administrator & Developer Guide

A source-based reference for operators, architects, and contributors maintaining `abi/screenshot-to-code`. Covers the FastAPI backend, the Vite/React frontend, the agentic LLM pipeline (OpenAI / Anthropic / Gemini), the prompt-construction layer, deployment topology, security model, and every configurable surface.

## 1. Architecture Overview

`screenshot-to-code` is a two-tier app: a **FastAPI Python backend** that hosts a single WebSocket route plus a handful of REST routes, and a **React/Vite frontend** that drives the UI and streams updates. The entire stack runs locally; the only outbound traffic is to LLM providers and (optionally) ScreenshotOne and Replicate.

```
┌────────────────────────────────────────────────────────────────────────┐
│                           Browser (frontend)                           │
│   React 18 + Vite + Tailwind + Radix UI + Zustand                      │
│   ┌─────────────────────────────────────────────────────────────────┐  │
│   │ App.tsx (state machine: INITIAL → CODING → CODE_READY)          │  │
│   │ generateCode.ts ──── WebSocket ───► ws://host:7001/generate-code│  │
│   │ Zustand stores: app-store, project-store                        │  │
│   │ usePersistedState (localStorage: 'setting', 'app-theme')        │  │
│   └─────────────────────────────────────────────────────────────────┘  │
└─────────────────┬──────────────────────────────────────────────────────┘
                  │  WebSocket JSON frames
                  ▼
┌────────────────────────────────────────────────────────────────────────┐
│                      FastAPI Backend (port 7001)                       │
│   main.py: CORS allow_origins=['*'], 4 routers                         │
│   ┌─────────────────────────────────────────────────────────────────┐  │
│   │ /generate-code  (WebSocket)  ◄── Pipeline of 6 middlewares      │  │
│   │ /api/screenshot (POST)       ◄── ScreenshotOne proxy            │  │
│   │ /                  (GET)     ◄── healthz HTML                   │  │
│   │ /eval_input_files, /evals, /run_evals, /run_evals_stream, ...   │  │
│   └─────────────────────────────────────────────────────────────────┘  │
│              │                                                         │
│              ▼ (CodeGenerationMiddleware → AgenticGenerationStage)     │
│   ┌─────────────────────────────────────────────────────────────────┐  │
│   │ ProviderSession (Protocol)                                       │  │
│   │   ├── OpenAIProviderSession        (openai 2.16.0)              │  │
│   │   ├── AnthropicProviderSession     (anthropic ^0.84.0)          │  │
│   │   └── GeminiProviderSession        (google-genai ^1.16.1)       │  │
│   │ Tools: create_file, edit_file, generate_images,                 │  │
│   │        remove_background, retrieve_option                       │  │
│   └─────────────────────────────────────────────────────────────────┘  │
└─────────────────┬──────────────────────────────────────────────────────┘
                  │
   ┌──────────────┼─────────────────────────────────────────────────┐
   │              │                                                 │
   ▼              ▼                                                 ▼
┌────────┐   ┌────────────┐                                ┌──────────────┐
│ OpenAI │   │ Anthropic  │                                │ Gemini       │
│  API   │   │   API      │                                │ google-genai │
└────────┘   └────────────┘                                └──────────────┘
                                                                    │
                                       ┌────────────────┐ ┌─────────┴────────┐
                                       │  Replicate     │ │ ScreenshotOne    │
                                       │ flux-2-klein   │ │ /take            │
                                       └────────────────┘ └──────────────────┘
```

> [!NOTE]
> The backend exposes **no OpenAPI / Swagger UI** — `FastAPI(openapi_url=None, docs_url=None, redoc_url=None)` in `backend/main.py`. This is by design (reduces attack surface for the cloud build). All routes are documented in this guide instead.

### Component inventory

| Component | Path | Role |
|-----------|------|------|
| FastAPI app | `backend/main.py` | Entry point. Mounts 4 routers, configures permissive CORS, prints debug-mode banner on startup. |
| Pipeline orchestrator | `backend/routes/generate_code.py` | The single WebSocket route `/generate-code`. Implements 6 middleware stages and 5 logical stages (parameter extraction, model selection, prompt creation, agentic generation, post-processing). |
| Provider abstraction | `backend/agent/providers/{base,factory,openai,anthropic,gemini}.py` | Per-provider `stream_turn()` implementations behind a common `ProviderSession` Protocol. Handles tool serialization, streaming, and tool-result feedback. |
| Agent engine | `backend/agent/engine.py` | Tool-calling loop with `max_steps = 20`. Streams thinking/assistant/tool deltas to the frontend, manages file-state across turns. |
| Tool runtime | `backend/agent/tools/runtime.py` | Executes `create_file`, `edit_file` (string-replacement edits + unified diff), `generate_images`, `remove_background`, `retrieve_option`. |
| Prompt assembly | `backend/prompts/{pipeline,plan,policies,system_prompt,create/*,update/*}.py` | Builds the per-request prompt based on stack + input mode + create/update + history. |
| Image generation | `backend/image_generation/{core,generation,replicate}.py` | DALL-E 3 (OpenAI) and Flux 2 Klein (Replicate) async clients. |
| Screenshot capture | `backend/routes/screenshot.py` | Wraps ScreenshotOne `/take` into a single endpoint. |
| Eval harness | `backend/evals/`, `backend/routes/evals.py` | Multi-model batch evaluation, pairwise/best-of-N comparisons, NDJSON streaming progress. |
| Frontend state | `frontend/src/store/{app-store,project-store}.ts` | Two Zustand stores. `app-store` = ephemeral UI state, `project-store` = commits / variants / assets. |
| Frontend WS client | `frontend/src/generateCode.ts` | Opens the WS, handles 12 message types, dispatches callbacks. |
| Variant rendering | `frontend/src/components/variants/Variants.tsx` | Live-throttled iframe thumbnails for each variant; keyboard shortcut `Alt+1..9`. |

### Technology stack

| Layer | Library | Version | Purpose |
|-------|---------|---------|---------|
| Backend ASGI | `fastapi` | `^0.115.6` | Routing, dependency injection, WebSocket. |
| Backend server | `uvicorn` | `^0.25.0` | ASGI server (`reload=True` in dev). |
| WebSockets | `websockets` | `^14.1` | FastAPI WebSocket transport. |
| OpenAI client | `openai` | `2.16.0` (pinned) | GPT 4.1 / 5.2 / 5.3 / 5.4 + DALL-E 3. |
| Anthropic client | `anthropic` | `^0.84.0` | Claude Opus / Sonnet via streaming + tool use. |
| Gemini client | `google-genai` | `^1.16.1` | Gemini 3 Flash & 3.1 Pro. |
| Video processing | `moviepy` | `^1.0.3` | Frame extraction for video mode. |
| Image manipulation | `pillow` | `^10.3.0` | Resize / format conversion before sending to providers. |
| HTTP client | `httpx` | `^0.28.1` | Replicate polling. |
| Async HTTP | `aiohttp` | `^3.9.5` | ScreenshotOne capture. |
| Validation | `pydantic` | `^2.10` | Request/response models in routes. |
| Observability | `langfuse` | `^3.0.2` | Optional turn-input tracing. |
| Env loader | `python-dotenv` | `^1.0.0` | Loads `backend/.env`. |
| HTML cleanup | `beautifulsoup4` | `^4.12.2` | Post-processing `<html>...</html>` extraction. |
| Frontend UI | `react` / `react-dom` | `^18.2.0` | Strict-mode app, no SSR. |
| Build tool | `vite` | `^4.4.5` | Dev server + bundler. |
| Routing | `react-router-dom` | `^6.20.1` | `/`, `/evals/*` routes. |
| State | `zustand` | `^4.5.2` | App + project stores. |
| Headless UI | `@radix-ui/*` | per-package | Dialog, Select, Popover, Tabs, etc. |
| Styling | `tailwindcss` | `^3.3.5` | Default utility CSS. |
| Code editor | `codemirror` | `^6.0.1` + `@codemirror/lang-html ^6.4.6` | In-app HTML editing. |
| Drag-drop upload | `react-dropzone` | `^14.2.3` | Image / video upload. |
| Toasts | `react-hot-toast` | `^2.4.1` | Error / success notifications. |
| Variant IDs | `nanoid` | `^5.0.7` | Stable IDs for commits + assets. |

### Data flow — image-mode create generation

```
1. User drops PNG → ImageUpload converts to data URL.
2. App.tsx::doCreate() calls registerAssetIds() to store the image in projectStore.assetsById.
3. doGenerateCode() builds GenerationRequest, opens WebSocket to /generate-code.
4. Frontend sends JSON: { generationType, inputMode, prompt: {text, images, videos},
   generatedCodeConfig, openAiApiKey, anthropicApiKey, geminiApiKey, ... }
5. WebSocketSetupMiddleware → accept().
6. ParameterExtractionMiddleware → ExtractedParams (validated stack, input_mode, keys).
7. StatusBroadcastMiddleware → "variantCount" + N "Generating code..." status messages.
8. PromptCreationMiddleware → build_create_prompt_from_input(image, ...) → list[ChatMessage].
9. CodeGenerationMiddleware → ModelSelectionStage picks 4 models → AgenticGenerationStage
   spawns 4 asyncio tasks, each with its own AgentEngine + ProviderSession.
10. Each agent loop:
     a. session.stream_turn(on_event) → fires assistant_delta / thinking_delta / tool_call_delta.
     b. If tool calls: execute via AgentToolRuntime, send toolStart + toolResult frames.
     c. Loop up to max_steps=20 turns.
11. Variant completes → setCode + variantComplete frames sent.
12. PostProcessingMiddleware → cleanup. WebSocket closes (code 1000 = success, 4332 = app error,
    4333 = user cancel).
```

> [!NOTE]
> The pipeline is intentionally a **classic middleware chain**, not a DAG. Each middleware has a single `process(context, next_func)` signature. `Pipeline.execute()` reverses the list and wraps each middleware to compose the chain.

---

## 2. System Requirements

### Backend

| Item | Requirement |
|------|-------------|
| Python | `^3.10` per `backend/pyproject.toml` |
| Package manager | Poetry (lockfile is `poetry.lock`) |
| Memory | 1-2 GB for typical workloads |
| Outbound network | api.openai.com, api.anthropic.com, generativelanguage.googleapis.com, api.replicate.com, api.screenshotone.com |
| Open ports | 7001 (uvicorn) |

### Frontend

| Item | Requirement |
|------|-------------|
| Node.js | `>=14.18.0` (recommended 22 LTS) |
| Package manager | Yarn 1.x |
| Build tool | Vite 4.x |
| Dev port | 5173 |

### Container runtime

| Item | Requirement |
|------|-------------|
| Docker | 20.10+ |
| Compose | Docker Compose v2 plugin |
| Files | `docker-compose.yml` (root), `backend/Dockerfile`, `frontend/Dockerfile` |

> [!IMPORTANT]
> The backend Dockerfile copies `backend/.env` at build time only if `python-dotenv` is invoked at runtime (`load_dotenv()` in `main.py`). You should bind-mount `.env` (or `--env-file`) into the container rather than baking secrets into the image.

---

## 3. Configuration Reference

### 3.1 Configuration files

| File | Format | Location | Purpose |
|------|--------|----------|---------|
| `backend/.env` | KEY=VALUE | Local only (gitignored) | Backend secrets. Loaded by `python-dotenv`. |
| `backend/pyproject.toml` | TOML | Repo | Python deps + Poetry config. |
| `backend/pyrightconfig.json` | JSON | Repo | Type-check config. Excludes `image_generation.py`. |
| `backend/pytest.ini` | INI | Repo | Pytest discovery. |
| `frontend/.env.example` | KEY=VALUE | Repo (template) | Frontend env template. |
| `frontend/.env` | KEY=VALUE | Local only (gitignored) | Frontend overrides for `VITE_*` vars. |
| `frontend/.env.jest.example` | KEY=VALUE | Repo (template) | Jest test env template. |
| `frontend/components.json` | JSON | Repo | shadcn/ui config (`new-york` style, Tailwind base color `slate`). |
| `frontend/tailwind.config.js` | JS | Repo | Tailwind theme (custom button color `#ffd803`, dark-mode class strategy). |
| `frontend/vite.config.ts` | TS | Repo | Vite build + Plausible script injection. |
| `docker-compose.yml` | YAML | Repo | Two-service orchestration. |

### 3.2 Backend environment variables

All read in `backend/config.py`. Loaded via `os.environ.get(...)` at module import after `load_dotenv()`.

| Variable | Type | Default | Description | Required? |
|----------|------|---------|-------------|-----------|
| `OPENAI_API_KEY` | str | `None` | OpenAI key for GPT and DALL-E. | One of OpenAI/Anthropic/Gemini |
| `ANTHROPIC_API_KEY` | str | `None` | Anthropic key for Claude. | One of OpenAI/Anthropic/Gemini |
| `GEMINI_API_KEY` | str | `None` | Google Gemini key. Required for video mode. | One of OpenAI/Anthropic/Gemini |
| `OPENAI_BASE_URL` | str | `None` | Override OpenAI base URL (proxy/gateway). Disabled in prod (`IS_PROD=true`) for security. | No |
| `REPLICATE_API_KEY` | str | `None` | Replicate key for Flux 2 Klein image generation. | If using Flux for image gen |
| `IS_DEBUG_ENABLED` | bool | `False` | Enables debug-only websocket frames (`variantModels`) and verbose logging. | No |
| `DEBUG_DIR` | str | `""` | Directory for debug file dumps when enabled. | If `IS_DEBUG_ENABLED=true` |
| `IS_PROD` | bool | `False` | Enables hosted-version features (Stripe credit notice, hides custom OpenAI base URL). | No |

Additional env vars used by helper scripts:

| Variable | Type | Default | Used by | Description |
|----------|------|---------|---------|-------------|
| `OPENAI_TURN_INPUT_CONSOLE` | str | `""` | `backend/fs_logging/openai_turn_inputs.py` | If non-empty, prints OpenAI turn-input HTML reports to console. |
| `LOGS_PATH` | str | `""` | `backend/fs_logging/openai_turn_inputs.py` | Override directory where HTML reports are written. |
| `REPLICATE_API_TOKEN` | str | `""` | `backend/run_image_generation_evals.py` | Alias used by the eval script. Same value as `REPLICATE_API_KEY`. |

### 3.3 Frontend environment variables (Vite)

All prefixed with `VITE_` and read via `import.meta.env`. See `frontend/src/config.ts`.

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `VITE_WS_BACKEND_URL` | str | `ws://127.0.0.1:7001` | WebSocket endpoint for `/generate-code`. |
| `VITE_HTTP_BACKEND_URL` | str | `http://127.0.0.1:7001` | HTTP endpoint for `/api/screenshot`, `/run_evals`, etc. |
| `VITE_IS_DEPLOYED` | bool-string | `false` | When `"true"`, enables cloud-only UI (PicoBadge, ToS dialog, Plausible analytics tag, hides OpenAI Base URL). |
| `VITE_PICO_BACKEND_FORM_SECRET` | str | `null` | Used for hosted-version Stripe credit form. Self-hosters can ignore. |

### 3.4 In-browser settings (localStorage)

Persisted via `usePersistedState` hook (see `frontend/src/hooks/usePersistedState.ts`).

| Key | Default | Description |
|-----|---------|-------------|
| `setting.openAiApiKey` | `null` | Overrides backend `OPENAI_API_KEY`. |
| `setting.openAiBaseURL` | `null` | Overrides backend `OPENAI_BASE_URL` (only if not `IS_PROD`). |
| `setting.anthropicApiKey` | `null` | Overrides backend `ANTHROPIC_API_KEY`. |
| `setting.geminiApiKey` | `null` | Overrides backend `GEMINI_API_KEY`. |
| `setting.screenshotOneApiKey` | `null` | Browser-side ScreenshotOne key — sent as `apiKey` in `/api/screenshot` body. |
| `setting.isImageGenerationEnabled` | `true` | Toggles whether `generate_images` tool is offered to the agent. |
| `setting.editorTheme` | `EditorTheme.COBALT` | CodeMirror theme. `cobalt` or `espresso`. |
| `setting.generatedCodeConfig` | `Stack.HTML_TAILWIND` | Default stack. |
| `setting.codeGenerationModel` | `CodeGenerationModel.CLAUDE_4_5_OPUS_2025_11_01` | Default model. |
| `setting.isTermOfServiceAccepted` | `false` | Cloud-only ToS dialog state. |
| `app-theme` | `AppTheme.SYSTEM` | UI theme: `system`, `light`, or `dark`. |

> [!IMPORTANT]
> Keys in `localStorage` survive across tabs and reloads. Anyone with physical access to the browser profile can read them. For shared workstations, prefer backend `.env`.

### 3.5 Hard-coded constants

| Constant | File | Value | Purpose |
|----------|------|-------|---------|
| `NUM_VARIANTS` | `backend/config.py` | `4` | Image/text create variant count. |
| `NUM_VARIANTS_VIDEO` | `backend/config.py` | `2` | Video mode variant count. |
| `max_steps` | `backend/agent/engine.py:152` | `20` | Max tool-calling turns per variant. |
| `REPLICATE_BATCH_SIZE` | `backend/image_generation/generation.py` | `20` | Concurrent Flux jobs per batch. |
| `MAX_POLLS` | `backend/image_generation/replicate.py` | `100` | Max Replicate prediction polls. |
| `POLL_INTERVAL_SECONDS` | `backend/image_generation/replicate.py` | `0.1` | Poll cadence. |
| `FLUX_MODEL_PATH` | `backend/image_generation/replicate.py` | `black-forest-labs/flux-2-klein-4b` | Flux 2 model slug. |
| `REMOVE_BACKGROUND_VERSION` | `backend/image_generation/replicate.py` | `a029dff38972b5fda4ec5d75d7d1cd25aeff621d2cf4946a41055d7db66b80bc` | Background-remover Replicate version hash. |
| `APP_ERROR_WEB_SOCKET_CODE` | `backend/ws/constants.py` | `4332` | Custom WS close code for backend-side errors. |
| `USER_CLOSE_WEB_SOCKET_CODE` | `frontend/src/constants.ts` | `4333` | Custom WS close code for user-cancelled generations. |
| `DEFAULT_VIDEO_FPS` | `backend/agent/providers/gemini.py` | `10` | Frames per second extracted for Gemini video mode. |

### 3.6 CLI flags

The only CLI surface is `backend/start.py` (a development convenience for `uvicorn`).

| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `--port` | int | `7001` | Port for `uvicorn.run("main:app", reload=True, port=<port>)`. |

```bash
poetry run python start.py --port 7002
```

---

## 4. Deployment & Scaling

### 4.1 Docker Compose (single host)

```yaml
# docker-compose.yml (excerpt; edit freely)
services:
  backend:
    build: ./backend
    ports:
      - "7001:7001"
    env_file:
      - ./backend/.env
  frontend:
    build: ./frontend
    ports:
      - "5173:5173"
```

**Bring up:**

```bash
docker compose up -d --build
docker compose logs -f backend
```

**Tear down:**

```bash
docker compose down
```

### 4.2 Reverse-proxy deployment (production-ish)

If you want a single TLS endpoint:

```nginx
# /etc/nginx/sites-available/screenshot-to-code
server {
    listen 443 ssl http2;
    server_name s2c.example.com;
    # ... TLS config ...

    # Frontend (static or dev preview)
    location / {
        proxy_pass http://localhost:5173;
    }

    # Backend HTTP
    location /api/ {
        proxy_pass http://localhost:7001;
    }

    # Backend WebSocket — required upgrade headers
    location /generate-code {
        proxy_pass http://localhost:7001;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 600s;        # generation can be long
        proxy_send_timeout 600s;
    }
}
```

Then set in `frontend/.env`:

```bash
VITE_WS_BACKEND_URL=wss://s2c.example.com/generate-code
VITE_HTTP_BACKEND_URL=https://s2c.example.com/api
```

> [!WARNING]
> Default CORS in `backend/main.py` is `allow_origins=["*"]`. For production deployments behind a reverse proxy, harden it to your specific origin: `allow_origins=["https://s2c.example.com"]`.

### 4.3 Scaling considerations

The backend is **stateless** between WebSocket sessions — no shared in-memory state across requests. You can horizontally scale uvicorn workers with `uvicorn main:app --workers N` or run multiple containers behind a sticky-session-aware load balancer (sticky sessions matter because the entire generation lifecycle lives on one socket).

Bottlenecks in practice:

1. **Provider rate limits.** Each variant = 1 LLM session; image-create runs 4 in parallel. Heavy concurrency hits provider TPM caps fast.
2. **moviepy frame extraction.** Video mode is CPU-bound during the moviepy pass. Single-process by default.
3. **Replicate polling.** Each `generate_images` call spins up a polling loop (100 polls × 100ms = 10s ceiling).

> [!TIP]
> If you're running multiple uvicorn workers, beware the `print(...)` debug statements in the codebase — they interleave between workers. Pipe stdout through a structured logger if you need correlation.

---

## 5. Operations

### 5.1 Routine procedures

- [ ] Deploy: `docker compose up -d --build`
- [ ] Verify backend: `curl http://localhost:7001/` returns the green-banner HTML
- [ ] Verify frontend: open `http://localhost:5173`, run a quick HTML+Tailwind generation
- [ ] Sync `.env`: ensure `OPENAI_API_KEY` (or peer keys) match the latest provider rotation
- [ ] Backend tests: `cd backend && poetry run pytest` (uses `pytest.ini`)
- [ ] Frontend tests: `cd frontend && yarn test` (Jest, `jest.config.js`)
- [ ] E2E QA test (Puppeteer): `cd frontend && yarn test:qa`

### 5.2 Logs

The backend writes to stdout. Notable lines:

| Pattern | Source | Meaning |
|---------|--------|---------|
| `Backend startup complete. Debug mode is ENABLED/DISABLED.` | `main.py:log_debug_mode` | Startup banner. |
| `Incoming websocket connection...` | `WebSocketCommunicator.accept` | New `/generate-code` socket accepted. |
| `Received params` | `WebSocketCommunicator.receive_params` | Initial JSON received. |
| `Generating {stack} code in {input_mode} mode` | `ParameterExtractionMiddleware.process` | Validated payload. |
| `Variant models:` followed by 4 lines | `ModelSelectionStage.select_models` | Which models will run. |
| `Status (variant N): Generating code...` | `WebSocketCommunicator.send_message` | Status broadcast. |
| `Variant N complete` | `send_message` | Variant succeeded. |
| `Variant N error: ...` | `send_message` | Variant failed; error sent to client. |
| `Error in variant N:` + traceback | `AgenticGenerationStage._run_variant` | Unexpected exception during a variant. |
| `WebSocket closed by client, skipping message: ...` | `WebSocketCommunicator.send_message` | Client-side disconnect. |
| `[TOKEN USAGE] provider=... model=... | input=... output=... cost=$...` | Provider sessions | Per-turn token + cost summary. |
| `Image generation time: NN.NN seconds` | `image_generation/generation.py` | Image-gen latency. |

> [!NOTE]
> Set `IS_DEBUG_ENABLED=true` to also receive the `variantModels` WebSocket frame (the names of the 4 selected models) on the client side, useful for live A/B observation.

### 5.3 Monitoring runbooks

**RB-1: User reports "Error generating code. Please contact support."**

1. Inspect backend logs for the most recent block — look for `[GENERATE_CODE] Unexpected error: ...`.
2. If repeated traces show provider auth failures, rotate the relevant API key.
3. If traces show `Agent exceeded max tool turns`, the model got stuck in a loop. Lower expectations, simplify input, or bump `max_steps` in `agent/engine.py:152` (with caution — costs scale linearly).

**RB-2: All variants timing out**

1. Check provider status pages.
2. Try a single-provider mode (set only `OPENAI_API_KEY`) to bypass `ModelSelectionStage` fallback chains.
3. Check egress firewall — corporate proxies sometimes block streaming completions.

**RB-3: Image generation failing with `Inference timed out`**

1. Replicate polling exhausted at `MAX_POLLS=100` × `POLL_INTERVAL_SECONDS=0.1` = 10 seconds. Either Flux is slow today or the prompt is rejected.
2. Look for `Inference errored out: <message>` lines first.
3. Temporarily disable image generation in user Settings to unblock generations.

**RB-4: Eval streaming endpoint stuck**

1. `/run_evals_stream` emits NDJSON via `StreamingResponse`. If clients hang, the most likely cause is a model thread silently dying.
2. Restart the backend. Re-run with the `--diff_mode` flag to skip already-completed inputs.

### 5.4 Backup & restore

Nothing the backend writes is durable user state. The only on-disk artefact is `backend/evals/results/<timestamp>/` for batched evaluation outputs. Back these up like any other artefact directory; they are safe to delete.

The frontend has zero server-side state. User project history lives in browser memory only and is lost on page refresh (except for settings, which persist in localStorage). This is intentional — it's a local sandbox tool.

### 5.5 Upgrades

```bash
git pull
cd backend && poetry install --no-interaction
cd ../frontend && yarn install --frozen-lockfile && yarn build
docker compose up -d --build
```

> [!TIP]
> Pin `openai==2.16.0` in any forks; the backend explicitly depends on this version (changes to the Responses API in newer versions break the OpenAI provider session).

---

## 6. Performance

### 6.1 Latency profile (typical, single user)

| Stage | Median | P95 |
|-------|--------|-----|
| WebSocket accept + param parse | <50ms | 100ms |
| Prompt construction | <20ms | 50ms |
| First token from provider | 0.8-2s | 5s |
| Full HTML generation (single variant, ~3K LOC) | 8-25s | 60s |
| `generate_images` tool call (Flux) | 3-8s | 12s |
| `generate_images` tool call (DALL-E 3) | 4-10s | 18s |
| Background removal | 3-6s | 10s |

### 6.2 Cost model

Pricing tables ship in `backend/agent/providers/pricing.py` keyed by API model name. Excerpt:

| Model | Input ($/M tok) | Output ($/M tok) | Cache read | Cache write |
|-------|-----------------|------------------|-----------|-------------|
| `gpt-4.1-2025-04-14` | 2.00 | 8.00 | 0.50 | — |
| `gpt-5.2-codex` | 1.75 | 14.00 | 0.4375 | — |
| `gpt-5.3-codex` | 1.75 | 14.00 | 0.4375 | — |
| `gpt-5.4-2026-03-05` | 2.50 | 15.00 | 0.25 | — |
| `claude-sonnet-4-5-20250929` | 3.00 | 15.00 | 0.30 | 3.75 |
| `claude-sonnet-4-6` | 3.00 | 15.00 | 0.30 | 3.75 |
| `claude-opus-4-5-20251101` | 5.00 | 25.00 | 0.50 | 6.25 |
| `claude-opus-4-6` | 5.00 | 25.00 | 0.50 | 6.25 |
| `gemini-3-flash-preview` | 0.50 | 3.00 | 0.05 | — |
| `gemini-3.1-pro-preview` | 2.00 | 12.00 | 0.20 | — |

Cost per generation is the sum of all 4 variants' `TokenUsage.cost(pricing)` calculations. The `TokenUsage` dataclass (in `agent/providers/token_usage.py`) carries `input`, `output`, `cache_read`, `cache_write`, `total` and exposes `accumulate(other)` and `cost(pricing)`.

### 6.3 Tuning levers

| Lever | Where | Effect |
|-------|-------|--------|
| Reduce variant count | `NUM_VARIANTS` in `config.py` | Linear cost/time reduction. |
| Use cheaper default models | `model_choice_sets.py` | Swap Codex high → Flash minimal for example. |
| Disable image generation | UI Settings or `is_image_generation_enabled` flag | Skips Flux/DALL-E spending. |
| Lower max_steps | `agent/engine.py:152` | Prevents long tool-call loops. |
| Use cache-friendly prompts | Prompt structure in `prompts/` | Anthropic and Gemini cache reads can save ~80% on prompt tokens. |
| Disable thinking | Switch to `MINIMAL` / `LOW` reasoning effort | Massive output-token savings on Codex/Gemini. |

### 6.4 Caching

There is no application-layer caching. Provider-side caching is implicit:

- **Anthropic prompt caching** — automatic on identical prompt prefixes; reflected in `cache_read` and `cache_write` token counts.
- **OpenAI prompt caching** — automatic via the Responses API.
- **Gemini caching** — automatic for repeated content.

The system prompt (in `prompts/system_prompt.py`) is identical across requests, so its tokens are typically cache-resident on Anthropic.

---

## 7. Security

### 7.1 Threat model

| Asset | Threat | Mitigation |
|-------|--------|-----------|
| API keys (provider) | Disclosure via logs / repo / browser leak | `.env` gitignored; UI keys live in localStorage only; never persisted server-side. |
| API keys (provider) | Misuse via in-browser client paste | UI explicitly labels: *"Only stored in your browser. Never stored on servers."* |
| Generated HTML | XSS in preview iframe | Iframe `sandbox="allow-scripts allow-same-origin"` (variants thumbnail). Main preview iframe in `PreviewComponent` runs in a same-origin sandbox by design — generated code is treated as user-trusted. |
| WebSocket | Unauthenticated remote use | No built-in auth. Rely on network-level controls (firewall, reverse-proxy basic auth). |
| Eval routes (`/run_evals`, `/best-of-n-evals`) | Path traversal via `folder` query param | The route accepts arbitrary filesystem paths; only safe behind a trusted network. |
| OpenAI proxy | Leakage via attacker-controlled `OPENAI_BASE_URL` | When `IS_PROD=true`, the in-browser settings dialog hides the Base URL field and the backend ignores user-supplied `openAiBaseURL`. |

### 7.2 Authentication

**The backend has no authentication.** It is designed for self-hosting on `localhost` or behind a trusted network. If you expose it publicly:

> [!CAUTION]
> Adding TLS does not equal adding auth. With CORS at `*` and no per-request authentication, a publicly-reachable backend lets anyone invoke generations against your provider keys. Always run behind a reverse-proxy authenticator (basic auth, OIDC, IP allow-list) or a VPN.

> [!WARNING]
> The `/run_evals` and `/best-of-n-evals` routes accept arbitrary filesystem paths via the `folder` query parameter and will read any HTML/PNG it finds. Do not expose these to untrusted users — see community issue [#579](https://github.com/abi/screenshot-to-code/issues/579) for the security audit reference (CORS wildcard + `eval()` mention).

### 7.3 Input validation

`ParameterExtractionStage` (in `backend/routes/generate_code.py`) validates:

- `generatedCodeConfig` ∈ `Stack` literal (`html_css`, `html_tailwind`, `react_tailwind`, `bootstrap`, `vue_tailwind`, `ionic_tailwind`); else `ValueError`.
- `inputMode` ∈ `InputMode` literal (`image`, `video`, `text`); else `ValueError`.
- `generationType` ∈ `["create", "update"]`; else `ValueError`.

URLs sent to ScreenshotOne are normalized in `backend/routes/screenshot.py::normalize_url`:

- No scheme → prepended with `https://`.
- Scheme of `http`/`https` → kept as-is.
- Scheme is anything else (`ftp`, `file`, etc.) → `ValueError("Unsupported protocol")`.
- `domain:port` form → treated as `https://domain:port`.

> [!IMPORTANT]
> The screenshot route does **not** sanitize against SSRF. A malicious user with backend access could pass `https://169.254.169.254/...` to read cloud metadata if the backend runs on AWS/GCP/Azure. Run with appropriate network restrictions or add an allow-list for outbound URLs.

### 7.4 Secret handling

| Secret | Storage | Lifetime | Logged? |
|--------|---------|----------|---------|
| `OPENAI_API_KEY` (env) | `backend/.env` | Process lifetime | No (key never printed). |
| `OPENAI_API_KEY` (UI) | Browser localStorage | Until user clears | No (frontend never logs to remote). |
| Same for Anthropic / Gemini / Replicate / ScreenshotOne | Same as above | Same | No |
| Per-request keys | Process memory in WebSocket lifetime | Until socket close | No. The backend prints "Using X from environment variable" / "Using X from client-side settings dialog" but never the value. |

### 7.5 Dependency provenance

The backend uses Poetry with `poetry.lock` for deterministic installs. The frontend uses `yarn.lock` and `package-lock.json`. Audit in CI:

```bash
poetry export -f requirements.txt | safety check
yarn audit --level high
```

### 7.6 PDF / Markdown safety in evals

`/openai-input-compare` accepts arbitrary JSON, parsed into structured payloads via `OpenAIInputComparison` types. Bad JSON returns 400; otherwise the payload is iterated structurally — there is **no shell-out and no `eval()`** in the implementation despite the surface name (community issue [#579](https://github.com/abi/screenshot-to-code/issues/579) refers to historical concern about `eval()` use; current code does not use `eval()`).

---

## 8. Developer Internals

### 8.1 Repository structure

```
screenshot-to-code/
├── backend/                       # FastAPI service
│   ├── agent/                     # Agent loop + provider adapters
│   │   ├── engine.py              # AgentEngine (max_steps=20 tool loop)
│   │   ├── runner.py              # Agent (alias for AgentEngine)
│   │   ├── state.py               # AgentFileState, message-derived seeding
│   │   ├── providers/             # Per-provider streaming sessions
│   │   │   ├── base.py            # ProviderSession Protocol, StreamEvent
│   │   │   ├── factory.py         # create_provider_session(model, ...)
│   │   │   ├── openai.py          # OpenAIProviderSession (Responses API)
│   │   │   ├── anthropic/         # AnthropicProviderSession + image helpers
│   │   │   ├── gemini.py          # GeminiProviderSession + tool serialization
│   │   │   ├── pricing.py         # MODEL_PRICING dict
│   │   │   ├── token_usage.py     # TokenUsage dataclass
│   │   │   └── types.py           # Backwards-compat aliases
│   │   └── tools/                 # Tool definitions + runtime
│   │       ├── definitions.py     # canonical_tool_definitions(...)
│   │       ├── runtime.py         # AgentToolRuntime (executes tools)
│   │       ├── parsing.py         # JSON-arg + partial-arg extraction
│   │       └── summaries.py       # Pretty-printed tool input/output
│   ├── codegen/utils.py           # extract_html_content() regex extractor
│   ├── debug/DebugFileWriter.py   # IS_DEBUG_ENABLED scratch writer
│   ├── evals/                     # Multi-model batch eval harness
│   │   ├── core.py                # generate_code_for_image(image, stack, model)
│   │   ├── runner.py              # run_image_evals(...) + count_pending_eval_tasks
│   │   └── config.py              # EVALS_DIR
│   ├── fs_logging/                # OpenAI turn-input HTML reports
│   │   ├── openai_turn_inputs.py  # OpenAITurnInputLogger
│   │   ├── openai_input_compare.py
│   │   └── openai_input_formatting.py
│   ├── image_generation/          # DALL-E 3 + Flux 2 Klein
│   │   ├── generation.py          # process_tasks, generate_image_dalle/replicate
│   │   ├── replicate.py           # call_replicate, remove_background
│   │   └── core.py                # Public re-exports
│   ├── prompts/                   # Stack-aware prompt assembly
│   │   ├── system_prompt.py       # The single SYSTEM_PROMPT string
│   │   ├── pipeline.py            # build_prompt_messages(...)
│   │   ├── plan.py                # derive_prompt_construction_plan
│   │   ├── policies.py            # build_selected_stack_policy / image_policy
│   │   ├── prompt_types.py        # Stack literal, UserTurnInput, etc.
│   │   ├── message_builder.py     # OpenAI ChatCompletion-shaped messages
│   │   ├── request_parsing.py     # parse_prompt_content / parse_prompt_history
│   │   ├── create/                # First-generation prompts (image/text/video)
│   │   └── update/                # Subsequent-edit prompts (history / file_snapshot)
│   ├── routes/                    # FastAPI routers
│   │   ├── generate_code.py       # /generate-code WebSocket pipeline
│   │   ├── screenshot.py          # POST /api/screenshot
│   │   ├── home.py                # GET /
│   │   ├── evals.py               # 9 eval routes
│   │   └── model_choice_sets.py   # 8 model-set tuples per key combo
│   ├── tests/                     # pytest suite (18 test files)
│   ├── video/                     # Video-mode helpers
│   │   ├── cost_estimation.py     # Token + USD estimates
│   │   └── utils.py               # bytes ↔ data-url + duration probing
│   ├── ws/constants.py            # APP_ERROR_WEB_SOCKET_CODE = 4332
│   ├── config.py                  # Env-var loading
│   ├── custom_types.py            # InputMode literal
│   ├── llm.py                     # Llm enum + MODEL_PROVIDER, OPENAI_MODEL_CONFIG
│   ├── main.py                    # FastAPI app
│   ├── start.py                   # uvicorn launcher with --port flag
│   ├── run_evals.py               # Image-mode batch eval CLI
│   ├── run_image_generation_evals.py  # Image-gen smoke test
│   ├── utils.py                   # Prompt pretty-printing helpers
│   ├── pyproject.toml
│   ├── poetry.lock
│   ├── pytest.ini
│   └── Dockerfile
├── frontend/                      # React + Vite app
│   ├── src/
│   │   ├── App.tsx                # Root component, state machine
│   │   ├── main.tsx               # Router setup (/, /evals/*)
│   │   ├── generateCode.ts        # WebSocket client
│   │   ├── config.ts              # VITE_* env reading
│   │   ├── constants.ts           # Close codes 4332 / 4333
│   │   ├── types.ts               # AppState, EditorTheme, Settings, ...
│   │   ├── urls.ts                # External link constants
│   │   ├── lib/
│   │   │   ├── stacks.ts          # Stack enum + STACK_DESCRIPTIONS
│   │   │   ├── models.ts          # CodeGenerationModel enum + descriptions
│   │   │   ├── prompt-history.ts  # Asset registration + history shaping
│   │   │   ├── takeScreenshot.ts  # html2canvas iframe screenshot
│   │   │   └── utils.ts           # cn(), capitalize()
│   │   ├── hooks/
│   │   │   ├── usePersistedState.ts  # localStorage-backed useState
│   │   │   ├── useBrowserTabIndicator.ts
│   │   │   └── useThrottle.ts
│   │   ├── store/
│   │   │   ├── app-store.ts       # Zustand: ephemeral UI state
│   │   │   └── project-store.ts   # Zustand: commits/variants/assets
│   │   └── components/            # ~70 components organized by feature
│   ├── public/                    # Favicons + Twitter card
│   ├── tests/                     # Jest + Puppeteer e2e
│   ├── package.json
│   ├── vite.config.ts
│   ├── tailwind.config.js
│   ├── components.json            # shadcn/ui config
│   └── Dockerfile
├── design-docs/                   # Architecture decision records (markdown)
├── blog/                          # Public-facing blog posts
├── docker-compose.yml
├── README.md, AGENTS.md, CLAUDE.md, Evaluation.md, TESTING.md, Troubleshooting.md
├── package.json (workspaces root)
└── plan.md
```

### 8.2 The pipeline middleware pattern

Every WebSocket request flows through a fresh `Pipeline` instance with 6 middlewares, defined in `backend/routes/generate_code.py:1316-1330`:

```python
@router.websocket("/generate-code")
async def stream_code(websocket: WebSocket):
    pipeline = Pipeline()
    pipeline.use(WebSocketSetupMiddleware())
    pipeline.use(ParameterExtractionMiddleware())
    pipeline.use(StatusBroadcastMiddleware())
    pipeline.use(PromptCreationMiddleware())
    pipeline.use(CodeGenerationMiddleware())
    pipeline.use(PostProcessingMiddleware())
    await pipeline.execute(websocket)
```

`Pipeline.execute` reverses the list and wraps each middleware to compose the chain. `PipelineContext` is the shared state object: it carries the `WebSocket`, an optional `WebSocketCommunicator`, raw `params`, validated `extracted_params`, `prompt_messages`, `variant_models`, `variant_completions`, etc.

### 8.3 Provider abstraction

```python
# backend/agent/providers/base.py
class ProviderSession(Protocol):
    async def stream_turn(self, on_event: EventSink) -> ProviderTurn: ...
    def append_tool_results(self, turn: ProviderTurn,
                            executed_tool_calls: list[ExecutedToolCall]) -> None: ...
    async def close(self) -> None: ...
```

Each implementation:

| Implementation | Streaming | Tool format | Notes |
|----------------|-----------|-------------|-------|
| `OpenAIProviderSession` | Responses API streaming events | OpenAI native `tools=[...]` | Uses `get_openai_api_name()` and `get_openai_reasoning_effort()` from `llm.py` for the per-model API name + reasoning effort. |
| `AnthropicProviderSession` | Messages API streaming | Anthropic `tools=[{name, description, input_schema}]` | Image inputs go through `agent/providers/anthropic/image.py` for base64 conversion. |
| `GeminiProviderSession` | `generate_content_stream` events | `genai.types.Tool(function_declarations=...)` | Video frames extracted at `DEFAULT_VIDEO_FPS=10`. |

`create_provider_session()` (the factory) inspects `MODEL_PROVIDER` from `llm.py`:

```python
if model in OPENAI_MODELS:
    if not openai_api_key: raise Exception("OpenAI API key is missing.")
    return OpenAIProviderSession(...)
if model in ANTHROPIC_MODELS: ...
if model in GEMINI_MODELS: ...
raise ValueError(f"Unsupported model: {model.value}")
```

### 8.4 Tool runtime

Tools defined in `backend/agent/tools/definitions.py::canonical_tool_definitions(image_generation_enabled)`:

| Tool | Required params | Optional params | What it does |
|------|-----------------|-----------------|--------------|
| `create_file` | `content` | `path` (default `index.html`) | Sets `file_state.content` to the extracted HTML and returns success. |
| `edit_file` | `old_text`, `new_text` (or `edits[]`) | `path`, `count` (default 1, `-1` = all) | Exact string replacement(s). Returns unified diff + first changed line. |
| `generate_images` | `prompts[]` | — | Calls `process_tasks(prompts, OPENAI_API_KEY, OPENAI_BASE_URL, model="dalle3")` or the Replicate equivalent. Only available if `image_generation_enabled`. |
| `remove_background` | `image_urls[]` | — | Calls `remove_background(url, REPLICATE_API_KEY)` for each. |
| `retrieve_option` | `option_number` (1-based) | — | Returns the HTML of variant N from `option_codes`. |

The runtime (`AgentToolRuntime.execute`) dispatches by `tool_call.name`. Failures yield `ToolExecutionResult(ok=False, ...)` with `summary={"error": "..."}` so the frontend can surface them.

### 8.5 Prompt construction

The system prompt (`backend/prompts/system_prompt.py:SYSTEM_PROMPT`) is a single multi-paragraph string that:

- Sets persona and tone ("expert front-end coder", concise, language-mirroring).
- Describes the 5 tools and how to use them.
- Includes per-stack "include this script" instructions:
  - Tailwind: `https://cdn.tailwindcss.com`
  - Bootstrap: `https://cdn.jsdelivr.net/npm/bootstrap@5.3.2/dist/css/bootstrap.min.css` (with SRI hash)
  - React: `https://cdn.jsdelivr.net/npm/react@18.0.0/umd/react.development.js`, react-dom@18, `https://unpkg.com/@babel/standalone/babel.min.js`
  - Ionic: `https://cdn.jsdelivr.net/npm/@ionic/core/dist/ionic/ionic.esm.js`
  - Vue: `https://registry.npmmirror.com/vue/3.3.11/files/dist/vue.global.js`
- Universal: Font Awesome 5.15.3 from cdnjs.

Per-request, `build_prompt_messages(stack, input_mode, generation_type, prompt, history, file_state, image_generation_enabled)` derives the `PromptConstructionPlan` and dispatches to one of:

| Construction strategy | When used | Where built |
|-----------------------|-----------|-------------|
| `create_from_input` (image) | First image-mode generation | `prompts/create/image.py::build_image_prompt_messages` |
| `create_from_input` (text) | First text-mode generation | `prompts/create/text.py::build_text_prompt_messages` |
| `create_from_input` (video) | First video-mode generation | `prompts/create/video.py::build_video_prompt_messages` |
| `update_from_history` | Edit when prior history exists | `prompts/update/from_history.py::build_update_prompt_from_history` |
| `update_from_file_snapshot` | Edit when only `fileState.content` exists (history was lost) | `prompts/update/from_file_snapshot.py::build_update_prompt_from_file_snapshot` |

Image policies (per `prompts/policies.py::build_user_image_policy`):

```text
Image generation enabled:
  "Image generation is enabled for this request. Use generate_images for missing assets when needed."

Image generation disabled:
  "Image generation is disabled for this request. Do not call generate_images. Use provided media,
   CSS effects, or placeholder URLs (https://placehold.co)."
```

### 8.6 The `Llm` enum and provider mapping

```python
# backend/llm.py — every supported model
class Llm(Enum):
    GPT_4_1_2025_04_14 = "gpt-4.1-2025-04-14"
    GPT_5_2_CODEX_LOW = "gpt-5.2-codex (low thinking)"
    # ... 5.2 codex {low, medium, high, xhigh}
    # ... 5.3 codex {low, medium, high, xhigh}
    # ... 5.4 2026-03-05 {none, low, medium, high, xhigh}
    CLAUDE_SONNET_4_6 = "claude-sonnet-4-6"
    CLAUDE_4_5_SONNET_2025_09_29 = "claude-sonnet-4-5-20250929"
    CLAUDE_4_5_OPUS_2025_11_01 = "claude-opus-4-5-20251101"
    CLAUDE_OPUS_4_6 = "claude-opus-4-6"
    GEMINI_3_FLASH_PREVIEW_HIGH = "gemini-3-flash-preview (high thinking)"
    GEMINI_3_FLASH_PREVIEW_MINIMAL = "gemini-3-flash-preview (minimal thinking)"
    GEMINI_3_1_PRO_PREVIEW_HIGH = "gemini-3.1-pro-preview (high thinking)"
    GEMINI_3_1_PRO_PREVIEW_MEDIUM = "gemini-3.1-pro-preview (medium thinking)"
    GEMINI_3_1_PRO_PREVIEW_LOW = "gemini-3.1-pro-preview (low thinking)"
```

`MODEL_PROVIDER: dict[Llm, str]` and the convenience sets `OPENAI_MODELS`, `ANTHROPIC_MODELS`, `GEMINI_MODELS` are all derived from the same mapping — single source of truth for provider routing.

`OPENAI_MODEL_CONFIG` (in `llm.py`) maps each Codex/GPT enum value to its **API name** (e.g. `"gpt-5.2-codex"`) and **reasoning effort** (`"low"`/`"medium"`/`"high"`/`"xhigh"`/`"none"`). The OpenAI provider calls `get_openai_api_name(model)` and `get_openai_reasoning_effort(model)` to assemble the request.

### 8.7 Variant model selection

In `backend/routes/model_choice_sets.py`:

```python
VIDEO_VARIANT_MODELS = (
    Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL,
    Llm.GEMINI_3_1_PRO_PREVIEW_HIGH,
)

ALL_KEYS_MODELS_DEFAULT = (
    Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL,
    Llm.GPT_5_2_CODEX_HIGH,
    Llm.GEMINI_3_FLASH_PREVIEW_HIGH,
    Llm.GEMINI_3_1_PRO_PREVIEW_HIGH,
)

ALL_KEYS_MODELS_TEXT_CREATE = (
    Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL,
    Llm.GPT_5_2_CODEX_HIGH,
    Llm.CLAUDE_OPUS_4_6,
    Llm.GEMINI_3_1_PRO_PREVIEW_LOW,
)

ALL_KEYS_MODELS_UPDATE = (
    Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL,
    Llm.GPT_5_4_2026_03_05_LOW,
)

GEMINI_ANTHROPIC_MODELS = (Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL, Llm.GEMINI_3_1_PRO_PREVIEW_LOW,
                           Llm.CLAUDE_OPUS_4_6, Llm.GEMINI_3_FLASH_PREVIEW_HIGH,
                           Llm.GEMINI_3_1_PRO_PREVIEW_HIGH)
GEMINI_OPENAI_MODELS    = (Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL, Llm.GEMINI_3_1_PRO_PREVIEW_LOW,
                           Llm.GPT_5_2_CODEX_HIGH, Llm.GPT_5_2_CODEX_MEDIUM)
OPENAI_ANTHROPIC_MODELS = (Llm.CLAUDE_OPUS_4_6, Llm.GPT_5_2_CODEX_HIGH, Llm.GPT_5_2_CODEX_MEDIUM)
GEMINI_ONLY_MODELS      = (Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL, Llm.GEMINI_3_1_PRO_PREVIEW_LOW,
                           Llm.GEMINI_3_FLASH_PREVIEW_HIGH, Llm.GEMINI_3_1_PRO_PREVIEW_HIGH)
ANTHROPIC_ONLY_MODELS   = (Llm.CLAUDE_OPUS_4_6, Llm.CLAUDE_SONNET_4_6)
OPENAI_ONLY_MODELS      = (Llm.GPT_5_2_CODEX_HIGH, Llm.GPT_5_2_CODEX_MEDIUM)
```

`ModelSelectionStage._get_variant_models()` picks the right tuple by inspecting which keys are present, then **cycles** through the tuple to fill `num_variants` slots: `[A, B]` × num=4 → `[A, B, A, B]`.

### 8.8 WebSocket message types

12 server→client messages (defined as a Literal in `generate_code.py:562-575`):

| Type | Payload | When fired |
|------|---------|------------|
| `chunk` | `{value}` | Reserved (not currently emitted). |
| `status` | `{value}` | Status updates per variant ("Generating code..."). |
| `setCode` | `{value}` | Updated full HTML; replaces the variant's preview. |
| `error` | `{value}` | Connection-fatal error before close. |
| `variantComplete` | `{value}` | Variant N finished (ws.close(1000) follows). |
| `variantError` | `{value}` | Variant N failed; other variants continue. |
| `variantCount` | `{value: "N"}` | Tells frontend how many variants this run. |
| `variantModels` | `{data: {models: [...]}}` | Debug-only; only when `IS_DEBUG_ENABLED`. |
| `thinking` | `{value, eventId}` | Streaming thinking-token deltas. |
| `assistant` | `{value, eventId}` | Streaming assistant-text deltas. |
| `toolStart` | `{data: {name, input}, eventId}` | Tool call started. |
| `toolResult` | `{data: {name, output, ok}, eventId}` | Tool call finished. |

WebSocket close codes:

| Code | Meaning | Source |
|------|---------|--------|
| `1000` | Success (normal close) | Standard |
| `4332` | App-side error (`APP_ERROR_WEB_SOCKET_CODE`) | `backend/ws/constants.py` |
| `4333` | User-cancelled (`USER_CLOSE_WEB_SOCKET_CODE`) | `frontend/src/constants.ts` |

### 8.9 Frontend state machines

```
AppState:   INITIAL → CODING → CODE_READY
               ↑                    │
               └────── reset() ─────┘

ScreenRecorderState: INITIAL → RECORDING → FINISHED
```

`AppState` lives in `app-store`. `INITIAL` shows the StartPane; `CODING` and `CODE_READY` show the PreviewPane and Sidebar. Browser-tab favicon flips to "coding" while in `CODING` (via `useBrowserTabIndicator`).

The project history is a **commit graph** — each generation is a `Commit` (`type: "ai_create" | "ai_edit" | "code_create"`) with a `parentHash` and N `variants`. `head` is the active commit hash. `nanoid` provides commit hashes and asset IDs.

### 8.10 Testing strategy

**Backend (pytest, `backend/tests/`)** — 18 test files covering:

| Test file | Covers |
|-----------|--------|
| `test_agent_tool_runtime.py` | `create_file`, `edit_file`, image-gen, retrieve_option execution + error paths. |
| `test_agent_tools.py` | Tool definition shapes. |
| `test_batching.py` | `process_tasks` Replicate batching (BATCH_SIZE=20). |
| `test_codegen_utils.py` | `extract_html_content` regex on doctype, `<file>`-wrapped, partial tags. |
| `test_evals_openai_input_compare.py` | `/openai-input-compare` route. |
| `test_image_generation_replicate.py` | Replicate polling, error cases. |
| `test_model_selection.py` | `ModelSelectionStage` branches per key combo. |
| `test_openai_input_compare.py` | Comparison logic. |
| `test_openai_provider_session.py` | OpenAI provider streaming + tool result feedback. |
| `test_openai_reasoning_parser.py` | Reasoning-effort header parsing. |
| `test_openai_turn_input_logging.py` | HTML report generation. |
| `test_parameter_extraction_stage.py` | All validation paths. |
| `test_prompt_summary.py` | `format_prompt_summary` / `print_prompt_summary` boxes. |
| `test_prompts.py` | `build_prompt_messages` for all combinations. |
| `test_request_parsing.py` | `parse_prompt_content` / `parse_prompt_history`. |
| `test_screenshot.py` | URL normalization + capture flow. |
| `test_status_broadcast.py` | StatusBroadcastMiddleware. |
| `test_token_usage.py` | `TokenUsage.cost`/`accumulate`. |

Run: `cd backend && poetry run pytest`.

**Frontend (Jest)** — `frontend/src/lib/prompt-history.test.ts`, `frontend/src/components/history/utils.test.ts`. Plus a Puppeteer e2e at `frontend/src/tests/qa.test.ts` (gated by `RUN_E2E=true`).

Run: `cd frontend && yarn test` (Jest only) or `yarn test:qa` (e2e against `http://localhost:5173/`).

### 8.11 Code patterns and conventions

- **Type strictness:** Pyright in `basic` mode for backend. TypeScript strict for frontend.
- **No async generators in pipeline middlewares** — they all conform to `(ctx, next_func) -> None`.
- **Tool calls feed back into the same session** via `session.append_tool_results(turn, executed_tool_calls)`. Each provider's session implementation knows how to thread the tool results into the next turn's prompt.
- **HTML extraction** is deliberately tolerant: regex tries `<file path="...">...</file>` first (the explicit format), then doctype + html, then bare `<html>...</html>`, finally returns the raw text.
- **Print-debug logging** is the convention; structured logging is not used. Wrap with your favorite logging shim if running at scale.
- **WebSocket safety:** every send is guarded by `is_closed` to avoid noisy `ConnectionClosedOK` exceptions when the client disconnects mid-stream.

### 8.12 Potential improvements (technical debt)

| Item | Where | Notes |
|------|-------|-------|
| Replace `print(...)` with structured logging | Throughout backend | High operational value, low risk. |
| Add `frontend/.env.example` to git | `frontend/` | Currently only the `.env.jest.example` is committed. |
| Authentication layer | `backend/main.py` | No built-in auth. Add an API-key header check or OAuth proxy if exposing. |
| CORS hardening | `backend/main.py` | `allow_origins=["*"]` should be configurable. |
| SSRF protection on screenshot route | `backend/routes/screenshot.py` | Add allow-list / block private IPs / metadata services. |
| Eval-route auth + path sanitization | `backend/routes/evals.py` | Folder query params accept arbitrary FS paths. Fine for trusted networks, dangerous otherwise. |
| Streaming `chunk` event is reserved but unused | `MessageType` Literal | Either remove it or wire up incremental token streaming. |
| OpenAPI / docs disabled at startup | `backend/main.py` | Re-enable behind an admin route or auth-gate when needed for debugging. |
| moviepy version pinned at `^1.0.3` | `pyproject.toml` | Modern moviepy 2.x has different API; community issue #522 reports build failures on Python 3.12+. Decide between freezing 3.10/3.11 or upgrading moviepy. |

### 8.13 Contributing guide

Pre-commit hooks (`backend/.pre-commit-config.yaml`):

```yaml
- check-yaml
- check-added-large-files
```

To add a new model:

1. Append to `Llm` enum in `backend/llm.py`.
2. Add to `MODEL_PROVIDER` dict so it routes correctly.
3. If OpenAI: add to `OPENAI_MODEL_CONFIG` (api_name + reasoning_effort).
4. Add pricing in `backend/agent/providers/pricing.py`.
5. Mirror to `frontend/src/lib/models.ts::CodeGenerationModel` and `CODE_GENERATION_MODEL_DESCRIPTIONS`.
6. Update `model_choice_sets.py` if the model should be a default variant.
7. Run `poetry run pytest` and `yarn test`.

To add a new stack:

1. Extend `Stack` literal in `backend/prompts/prompt_types.py`.
2. Mirror to `frontend/src/lib/stacks.ts::Stack` and `STACK_DESCRIPTIONS`.
3. Add the include directives to `prompts/system_prompt.py::SYSTEM_PROMPT`.
4. Update tests.

---

## 9. Integration Points

### 9.1 LLM providers

| Provider | API base | SDK | Auth header |
|----------|----------|-----|-------------|
| OpenAI | `https://api.openai.com/v1` (or `OPENAI_BASE_URL`) | `openai==2.16.0` | `Authorization: Bearer <key>` |
| Anthropic | `https://api.anthropic.com` | `anthropic^0.84.0` | `x-api-key: <key>` |
| Google Gemini | `generativelanguage.googleapis.com` | `google-genai^1.16.1` | API key in URL or via SDK |

### 9.2 Image generation

| Service | Endpoint | Used for |
|---------|----------|----------|
| OpenAI Images | `https://api.openai.com/v1/images/generations` | DALL-E 3, `1024x1024`, `quality=standard`, `style=natural`. |
| Replicate | `https://api.replicate.com/v1/models/{model_path}/predictions` and `/predictions` for versioned | Flux 2 Klein for primary image gen. Background removal via versioned endpoint with hash `a029dff3...`. |

### 9.3 Screenshot capture

| Service | Endpoint | Notes |
|---------|----------|-------|
| ScreenshotOne | `https://api.screenshotone.com/take` | Hardcoded params: `full_page=true`, `block_ads=true`, `block_cookie_banners=true`, `block_trackers=true`, `cache=false`. Mobile viewport `342x684`, desktop `1280x832`. 60s `httpx.AsyncClient` timeout. |

### 9.4 REST API

The backend exposes 11 routes (10 HTTP + 1 WebSocket). Authentication: **none**.

| Method | Path | Handler | Purpose |
|--------|------|---------|---------|
| `GET` | `/` | `home.py:get_status` | Health banner HTML. |
| `WS` | `/generate-code` | `generate_code.py:stream_code` | Main code-gen pipeline. |
| `POST` | `/api/screenshot` | `screenshot.py:app_screenshot` | URL → PNG via ScreenshotOne. |
| `GET` | `/eval_input_files` | `evals.py:get_eval_input_files` | List `evals/inputs/*.png`. |
| `GET` | `/evals?folder=PATH` | `evals.py:get_evals` | Pair input PNG + output HTML for a folder. |
| `GET` | `/pairwise-evals?folder1=...&folder2=...` | `evals.py:get_pairwise_evals` | Side-by-side compare two output folders. |
| `GET` | `/best-of-n-evals?folder1=...&folder2=...&...` | `evals.py:get_best_of_n_evals` | N-way compare. |
| `POST` | `/openai-input-compare` | `evals.py:compare_openai_inputs_for_evals` | Diff two OpenAI input JSON payloads. |
| `POST` | `/run_evals` | `evals.py:run_evals` | Run image-mode evals for given models + stack. |
| `POST` | `/run_evals_stream` | `evals.py:run_evals_stream` | Same as above, NDJSON streaming progress. |
| `GET` | `/models` | `evals.py:get_models` | List of all `Llm` values + `Stack` values. |
| `GET` | `/output_folders` | `evals.py:get_output_folders` | List `evals/results/*` directories. |

### 9.5 Frontend routes

```ts
// main.tsx
<Route path="/" element={<App />} />
<Route path="/evals" element={<AllEvalsPage />} />
<Route path="/evals/single" element={<EvalsPage />} />
<Route path="/evals/pairwise" element={<PairwiseEvalsPage />} />
<Route path="/evals/best-of-n" element={<BestOfNEvalsPage />} />
<Route path="/evals/run" element={<RunEvalsPage />} />
<Route path="/evals/openai-input-compare" element={<OpenAIInputComparePage />} />
```

The `/evals/*` routes are intended for offline model quality work; they share the no-auth posture of the backend.

### 9.6 Observability

`langfuse` is a declared dependency (`^3.0.2`) used to wrap agent turn inputs for trace export. To enable, set the standard Langfuse env vars (`LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_HOST`) — the SDK auto-instruments without code changes.

The `OpenAITurnInputLogger` (in `fs_logging/openai_turn_inputs.py`) writes per-turn HTML reports under `LOGS_PATH` (or a default subdir) when `OPENAI_TURN_INPUT_CONSOLE` is set. Useful for debugging tool-call argument parsing.

---

_Generated April 2026 from abi/screenshot-to-code at aaaa838. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._