Screenshot to Code — Administrator & Developer Guide

A source-based reference for operators, architects, and contributors maintaining abi/screenshot-to-code. Covers the FastAPI backend, the Vite/React frontend, the agentic LLM pipeline (OpenAI / Anthropic / Gemini), the prompt-construction layer, deployment topology, security model, and every configurable surface.

1. Architecture Overview

screenshot-to-code is a two-tier app: a FastAPI Python backend that hosts a single WebSocket route plus a handful of REST routes, and a React/Vite frontend that drives the UI and streams updates. The entire stack runs locally; the only outbound traffic is to LLM providers and (optionally) ScreenshotOne and Replicate.

┌────────────────────────────────────────────────────────────────────────┐
│                           Browser (frontend)                           │
│   React 18 + Vite + Tailwind + Radix UI + Zustand                      │
│   ┌─────────────────────────────────────────────────────────────────┐  │
│   │ App.tsx (state machine: INITIAL → CODING → CODE_READY)          │  │
│   │ generateCode.ts ──── WebSocket ───► ws://host:7001/generate-code│  │
│   │ Zustand stores: app-store, project-store                        │  │
│   │ usePersistedState (localStorage: 'setting', 'app-theme')        │  │
│   └─────────────────────────────────────────────────────────────────┘  │
└─────────────────┬──────────────────────────────────────────────────────┘
                  │  WebSocket JSON frames
                  ▼
┌────────────────────────────────────────────────────────────────────────┐
│                      FastAPI Backend (port 7001)                       │
│   main.py: CORS allow_origins=['*'], 4 routers                         │
│   ┌─────────────────────────────────────────────────────────────────┐  │
│   │ /generate-code  (WebSocket)  ◄── Pipeline of 6 middlewares      │  │
│   │ /api/screenshot (POST)       ◄── ScreenshotOne proxy            │  │
│   │ /                  (GET)     ◄── healthz HTML                   │  │
│   │ /eval_input_files, /evals, /run_evals, /run_evals_stream, ...   │  │
│   └─────────────────────────────────────────────────────────────────┘  │
│              │                                                         │
│              ▼ (CodeGenerationMiddleware → AgenticGenerationStage)     │
│   ┌─────────────────────────────────────────────────────────────────┐  │
│   │ ProviderSession (Protocol)                                       │  │
│   │   ├── OpenAIProviderSession        (openai 2.16.0)              │  │
│   │   ├── AnthropicProviderSession     (anthropic ^0.84.0)          │  │
│   │   └── GeminiProviderSession        (google-genai ^1.16.1)       │  │
│   │ Tools: create_file, edit_file, generate_images,                 │  │
│   │        remove_background, retrieve_option                       │  │
│   └─────────────────────────────────────────────────────────────────┘  │
└─────────────────┬──────────────────────────────────────────────────────┘
                  │
   ┌──────────────┼─────────────────────────────────────────────────┐
   │              │                                                 │
   ▼              ▼                                                 ▼
┌────────┐   ┌────────────┐                                ┌──────────────┐
│ OpenAI │   │ Anthropic  │                                │ Gemini       │
│  API   │   │   API      │                                │ google-genai │
└────────┘   └────────────┘                                └──────────────┘
                                                                    │
                                       ┌────────────────┐ ┌─────────┴────────┐
                                       │  Replicate     │ │ ScreenshotOne    │
                                       │ flux-2-klein   │ │ /take            │
                                       └────────────────┘ └──────────────────┘

Note

The backend exposes no OpenAPI / Swagger UI — FastAPI(openapi_url=None, docs_url=None, redoc_url=None) in backend/main.py. This is by design (reduces attack surface for the cloud build). All routes are documented in this guide instead.

Component inventory

Technology stack

Data flow — image-mode create generation

1. User drops PNG → ImageUpload converts to data URL.
2. App.tsx::doCreate() calls registerAssetIds() to store the image in projectStore.assetsById.
3. doGenerateCode() builds GenerationRequest, opens WebSocket to /generate-code.
4. Frontend sends JSON: { generationType, inputMode, prompt: {text, images, videos},
   generatedCodeConfig, openAiApiKey, anthropicApiKey, geminiApiKey, ... }
5. WebSocketSetupMiddleware → accept().
6. ParameterExtractionMiddleware → ExtractedParams (validated stack, input_mode, keys).
7. StatusBroadcastMiddleware → "variantCount" + N "Generating code..." status messages.
8. PromptCreationMiddleware → build_create_prompt_from_input(image, ...) → list[ChatMessage].
9. CodeGenerationMiddleware → ModelSelectionStage picks 4 models → AgenticGenerationStage
   spawns 4 asyncio tasks, each with its own AgentEngine + ProviderSession.
10. Each agent loop:
     a. session.stream_turn(on_event) → fires assistant_delta / thinking_delta / tool_call_delta.
     b. If tool calls: execute via AgentToolRuntime, send toolStart + toolResult frames.
     c. Loop up to max_steps=20 turns.
11. Variant completes → setCode + variantComplete frames sent.
12. PostProcessingMiddleware → cleanup. WebSocket closes (code 1000 = success, 4332 = app error,
    4333 = user cancel).

Note

The pipeline is intentionally a classic middleware chain, not a DAG. Each middleware has a single process(context, next_func) signature. Pipeline.execute() reverses the list and wraps each middleware to compose the chain.

2. System Requirements

Backend

Frontend

Container runtime

Important

The backend Dockerfile copies backend/.env at build time only if python-dotenv is invoked at runtime (load_dotenv() in main.py). You should bind-mount .env (or --env-file) into the container rather than baking secrets into the image.

3. Configuration Reference

3.1 Configuration files

3.2 Backend environment variables

All read in backend/config.py. Loaded via os.environ.get(...) at module import after load_dotenv().

Additional env vars used by helper scripts:

3.3 Frontend environment variables (Vite)

All prefixed with VITE_ and read via import.meta.env. See frontend/src/config.ts.

3.4 In-browser settings (localStorage)

Persisted via usePersistedState hook (see frontend/src/hooks/usePersistedState.ts).

Important

Keys in localStorage survive across tabs and reloads. Anyone with physical access to the browser profile can read them. For shared workstations, prefer backend .env.

3.5 Hard-coded constants

3.6 CLI flags

The only CLI surface is backend/start.py (a development convenience for uvicorn).

poetry run python start.py --port 7002

4. Deployment & Scaling

4.1 Docker Compose (single host)

# docker-compose.yml (excerpt; edit freely)
services:
  backend:
    build: ./backend
    ports:
      - "7001:7001"
    env_file:
      - ./backend/.env
  frontend:
    build: ./frontend
    ports:
      - "5173:5173"

Bring up:

docker compose up -d --build
docker compose logs -f backend

Tear down:

docker compose down

4.2 Reverse-proxy deployment (production-ish)

If you want a single TLS endpoint:

# /etc/nginx/sites-available/screenshot-to-code
server {
    listen 443 ssl http2;
    server_name s2c.example.com;
    # ... TLS config ...

    # Frontend (static or dev preview)
    location / {
        proxy_pass http://localhost:5173;
    }

    # Backend HTTP
    location /api/ {
        proxy_pass http://localhost:7001;
    }

    # Backend WebSocket — required upgrade headers
    location /generate-code {
        proxy_pass http://localhost:7001;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 600s;        # generation can be long
        proxy_send_timeout 600s;
    }
}

Then set in frontend/.env:

VITE_WS_BACKEND_URL=wss://s2c.example.com/generate-code
VITE_HTTP_BACKEND_URL=https://s2c.example.com/api

Warning

Default CORS in backend/main.py is allow_origins=["*"]. For production deployments behind a reverse proxy, harden it to your specific origin: allow_origins=["https://s2c.example.com"].

4.3 Scaling considerations

The backend is stateless between WebSocket sessions — no shared in-memory state across requests. You can horizontally scale uvicorn workers with uvicorn main:app --workers N or run multiple containers behind a sticky-session-aware load balancer (sticky sessions matter because the entire generation lifecycle lives on one socket).

Bottlenecks in practice:

1. Provider rate limits. Each variant = 1 LLM session; image-create runs 4 in parallel. Heavy concurrency hits provider TPM caps fast.
2. moviepy frame extraction. Video mode is CPU-bound during the moviepy pass. Single-process by default.
3. Replicate polling. Each generate_images call spins up a polling loop (100 polls × 100ms = 10s ceiling).

Tip

If you're running multiple uvicorn workers, beware the print(...) debug statements in the codebase — they interleave between workers. Pipe stdout through a structured logger if you need correlation.

5. Operations

5.1 Routine procedures

• [ ] Deploy: docker compose up -d --build
• [ ] Verify backend: curl http://localhost:7001/ returns the green-banner HTML
• [ ] Verify frontend: open http://localhost:5173, run a quick HTML+Tailwind generation
• [ ] Sync .env: ensure OPENAI_API_KEY (or peer keys) match the latest provider rotation
• [ ] Backend tests: cd backend && poetry run pytest (uses pytest.ini)
• [ ] Frontend tests: cd frontend && yarn test (Jest, jest.config.js)
• [ ] E2E QA test (Puppeteer): cd frontend && yarn test:qa

5.2 Logs

The backend writes to stdout. Notable lines:

Note

Set IS_DEBUG_ENABLED=true to also receive the variantModels WebSocket frame (the names of the 4 selected models) on the client side, useful for live A/B observation.

5.3 Monitoring runbooks

RB-1: User reports "Error generating code. Please contact support."

1. Inspect backend logs for the most recent block — look for [GENERATE_CODE] Unexpected error: ....
2. If repeated traces show provider auth failures, rotate the relevant API key.
3. If traces show Agent exceeded max tool turns, the model got stuck in a loop. Lower expectations, simplify input, or bump max_steps in agent/engine.py:152 (with caution — costs scale linearly).

RB-2: All variants timing out

1. Check provider status pages.
2. Try a single-provider mode (set only OPENAI_API_KEY) to bypass ModelSelectionStage fallback chains.
3. Check egress firewall — corporate proxies sometimes block streaming completions.

RB-3: Image generation failing with Inference timed out

1. Replicate polling exhausted at MAX_POLLS=100 × POLL_INTERVAL_SECONDS=0.1 = 10 seconds. Either Flux is slow today or the prompt is rejected.
2. Look for Inference errored out: <message> lines first.
3. Temporarily disable image generation in user Settings to unblock generations.

RB-4: Eval streaming endpoint stuck

1. /run_evals_stream emits NDJSON via StreamingResponse. If clients hang, the most likely cause is a model thread silently dying.
2. Restart the backend. Re-run with the --diff_mode flag to skip already-completed inputs.

5.4 Backup & restore

Nothing the backend writes is durable user state. The only on-disk artefact is backend/evals/results/<timestamp>/ for batched evaluation outputs. Back these up like any other artefact directory; they are safe to delete.

The frontend has zero server-side state. User project history lives in browser memory only and is lost on page refresh (except for settings, which persist in localStorage). This is intentional — it's a local sandbox tool.

5.5 Upgrades

git pull
cd backend && poetry install --no-interaction
cd ../frontend && yarn install --frozen-lockfile && yarn build
docker compose up -d --build

Tip

Pin openai==2.16.0 in any forks; the backend explicitly depends on this version (changes to the Responses API in newer versions break the OpenAI provider session).

6. Performance

6.1 Latency profile (typical, single user)

6.2 Cost model

Pricing tables ship in backend/agent/providers/pricing.py keyed by API model name. Excerpt:

Cost per generation is the sum of all 4 variants' TokenUsage.cost(pricing) calculations. The TokenUsage dataclass (in agent/providers/token_usage.py) carries input, output, cache_read, cache_write, total and exposes accumulate(other) and cost(pricing).

6.3 Tuning levers

6.4 Caching

There is no application-layer caching. Provider-side caching is implicit:

• Anthropic prompt caching — automatic on identical prompt prefixes; reflected in cache_read and cache_write token counts.
• OpenAI prompt caching — automatic via the Responses API.
• Gemini caching — automatic for repeated content.

The system prompt (in prompts/system_prompt.py) is identical across requests, so its tokens are typically cache-resident on Anthropic.

7. Security

7.1 Threat model

7.2 Authentication

The backend has no authentication. It is designed for self-hosting on localhost or behind a trusted network. If you expose it publicly:

Caution

Adding TLS does not equal adding auth. With CORS at * and no per-request authentication, a publicly-reachable backend lets anyone invoke generations against your provider keys. Always run behind a reverse-proxy authenticator (basic auth, OIDC, IP allow-list) or a VPN.

Warning

The /run_evals and /best-of-n-evals routes accept arbitrary filesystem paths via the folder query parameter and will read any HTML/PNG it finds. Do not expose these to untrusted users — see community issue #579 for the security audit reference (CORS wildcard + eval() mention).

7.3 Input validation

ParameterExtractionStage (in backend/routes/generate_code.py) validates:

• generatedCodeConfig ∈ Stack literal (html_css, html_tailwind, react_tailwind, bootstrap, vue_tailwind, ionic_tailwind); else ValueError.
• inputMode ∈ InputMode literal (image, video, text); else ValueError.
• generationType ∈ ["create", "update"]; else ValueError.

URLs sent to ScreenshotOne are normalized in backend/routes/screenshot.py::normalize_url:

• No scheme → prepended with https://.
• Scheme of http/https → kept as-is.
• Scheme is anything else (ftp, file, etc.) → ValueError("Unsupported protocol").
• domain:port form → treated as https://domain:port.

Important

The screenshot route does not sanitize against SSRF. A malicious user with backend access could pass https://169.254.169.254/... to read cloud metadata if the backend runs on AWS/GCP/Azure. Run with appropriate network restrictions or add an allow-list for outbound URLs.

7.4 Secret handling

7.5 Dependency provenance

The backend uses Poetry with poetry.lock for deterministic installs. The frontend uses yarn.lock and package-lock.json. Audit in CI:

poetry export -f requirements.txt | safety check
yarn audit --level high

7.6 PDF / Markdown safety in evals

/openai-input-compare accepts arbitrary JSON, parsed into structured payloads via OpenAIInputComparison types. Bad JSON returns 400; otherwise the payload is iterated structurally — there is no shell-out and no eval() in the implementation despite the surface name (community issue #579 refers to historical concern about eval() use; current code does not use eval()).

8. Developer Internals

8.1 Repository structure

screenshot-to-code/
├── backend/                       # FastAPI service
│   ├── agent/                     # Agent loop + provider adapters
│   │   ├── engine.py              # AgentEngine (max_steps=20 tool loop)
│   │   ├── runner.py              # Agent (alias for AgentEngine)
│   │   ├── state.py               # AgentFileState, message-derived seeding
│   │   ├── providers/             # Per-provider streaming sessions
│   │   │   ├── base.py            # ProviderSession Protocol, StreamEvent
│   │   │   ├── factory.py         # create_provider_session(model, ...)
│   │   │   ├── openai.py          # OpenAIProviderSession (Responses API)
│   │   │   ├── anthropic/         # AnthropicProviderSession + image helpers
│   │   │   ├── gemini.py          # GeminiProviderSession + tool serialization
│   │   │   ├── pricing.py         # MODEL_PRICING dict
│   │   │   ├── token_usage.py     # TokenUsage dataclass
│   │   │   └── types.py           # Backwards-compat aliases
│   │   └── tools/                 # Tool definitions + runtime
│   │       ├── definitions.py     # canonical_tool_definitions(...)
│   │       ├── runtime.py         # AgentToolRuntime (executes tools)
│   │       ├── parsing.py         # JSON-arg + partial-arg extraction
│   │       └── summaries.py       # Pretty-printed tool input/output
│   ├── codegen/utils.py           # extract_html_content() regex extractor
│   ├── debug/DebugFileWriter.py   # IS_DEBUG_ENABLED scratch writer
│   ├── evals/                     # Multi-model batch eval harness
│   │   ├── core.py                # generate_code_for_image(image, stack, model)
│   │   ├── runner.py              # run_image_evals(...) + count_pending_eval_tasks
│   │   └── config.py              # EVALS_DIR
│   ├── fs_logging/                # OpenAI turn-input HTML reports
│   │   ├── openai_turn_inputs.py  # OpenAITurnInputLogger
│   │   ├── openai_input_compare.py
│   │   └── openai_input_formatting.py
│   ├── image_generation/          # DALL-E 3 + Flux 2 Klein
│   │   ├── generation.py          # process_tasks, generate_image_dalle/replicate
│   │   ├── replicate.py           # call_replicate, remove_background
│   │   └── core.py                # Public re-exports
│   ├── prompts/                   # Stack-aware prompt assembly
│   │   ├── system_prompt.py       # The single SYSTEM_PROMPT string
│   │   ├── pipeline.py            # build_prompt_messages(...)
│   │   ├── plan.py                # derive_prompt_construction_plan
│   │   ├── policies.py            # build_selected_stack_policy / image_policy
│   │   ├── prompt_types.py        # Stack literal, UserTurnInput, etc.
│   │   ├── message_builder.py     # OpenAI ChatCompletion-shaped messages
│   │   ├── request_parsing.py     # parse_prompt_content / parse_prompt_history
│   │   ├── create/                # First-generation prompts (image/text/video)
│   │   └── update/                # Subsequent-edit prompts (history / file_snapshot)
│   ├── routes/                    # FastAPI routers
│   │   ├── generate_code.py       # /generate-code WebSocket pipeline
│   │   ├── screenshot.py          # POST /api/screenshot
│   │   ├── home.py                # GET /
│   │   ├── evals.py               # 9 eval routes
│   │   └── model_choice_sets.py   # 8 model-set tuples per key combo
│   ├── tests/                     # pytest suite (18 test files)
│   ├── video/                     # Video-mode helpers
│   │   ├── cost_estimation.py     # Token + USD estimates
│   │   └── utils.py               # bytes ↔ data-url + duration probing
│   ├── ws/constants.py            # APP_ERROR_WEB_SOCKET_CODE = 4332
│   ├── config.py                  # Env-var loading
│   ├── custom_types.py            # InputMode literal
│   ├── llm.py                     # Llm enum + MODEL_PROVIDER, OPENAI_MODEL_CONFIG
│   ├── main.py                    # FastAPI app
│   ├── start.py                   # uvicorn launcher with --port flag
│   ├── run_evals.py               # Image-mode batch eval CLI
│   ├── run_image_generation_evals.py  # Image-gen smoke test
│   ├── utils.py                   # Prompt pretty-printing helpers
│   ├── pyproject.toml
│   ├── poetry.lock
│   ├── pytest.ini
│   └── Dockerfile
├── frontend/                      # React + Vite app
│   ├── src/
│   │   ├── App.tsx                # Root component, state machine
│   │   ├── main.tsx               # Router setup (/, /evals/*)
│   │   ├── generateCode.ts        # WebSocket client
│   │   ├── config.ts              # VITE_* env reading
│   │   ├── constants.ts           # Close codes 4332 / 4333
│   │   ├── types.ts               # AppState, EditorTheme, Settings, ...
│   │   ├── urls.ts                # External link constants
│   │   ├── lib/
│   │   │   ├── stacks.ts          # Stack enum + STACK_DESCRIPTIONS
│   │   │   ├── models.ts          # CodeGenerationModel enum + descriptions
│   │   │   ├── prompt-history.ts  # Asset registration + history shaping
│   │   │   ├── takeScreenshot.ts  # html2canvas iframe screenshot
│   │   │   └── utils.ts           # cn(), capitalize()
│   │   ├── hooks/
│   │   │   ├── usePersistedState.ts  # localStorage-backed useState
│   │   │   ├── useBrowserTabIndicator.ts
│   │   │   └── useThrottle.ts
│   │   ├── store/
│   │   │   ├── app-store.ts       # Zustand: ephemeral UI state
│   │   │   └── project-store.ts   # Zustand: commits/variants/assets
│   │   └── components/            # ~70 components organized by feature
│   ├── public/                    # Favicons + Twitter card
│   ├── tests/                     # Jest + Puppeteer e2e
│   ├── package.json
│   ├── vite.config.ts
│   ├── tailwind.config.js
│   ├── components.json            # shadcn/ui config
│   └── Dockerfile
├── design-docs/                   # Architecture decision records (markdown)
├── blog/                          # Public-facing blog posts
├── docker-compose.yml
├── README.md, AGENTS.md, CLAUDE.md, Evaluation.md, TESTING.md, Troubleshooting.md
├── package.json (workspaces root)
└── plan.md

8.2 The pipeline middleware pattern

Every WebSocket request flows through a fresh Pipeline instance with 6 middlewares, defined in backend/routes/generate_code.py:1316-1330:

@router.websocket("/generate-code")
async def stream_code(websocket: WebSocket):
    pipeline = Pipeline()
    pipeline.use(WebSocketSetupMiddleware())
    pipeline.use(ParameterExtractionMiddleware())
    pipeline.use(StatusBroadcastMiddleware())
    pipeline.use(PromptCreationMiddleware())
    pipeline.use(CodeGenerationMiddleware())
    pipeline.use(PostProcessingMiddleware())
    await pipeline.execute(websocket)

Pipeline.execute reverses the list and wraps each middleware to compose the chain. PipelineContext is the shared state object: it carries the WebSocket, an optional WebSocketCommunicator, raw params, validated extracted_params, prompt_messages, variant_models, variant_completions, etc.

8.3 Provider abstraction

# backend/agent/providers/base.py
class ProviderSession(Protocol):
    async def stream_turn(self, on_event: EventSink) -> ProviderTurn: ...
    def append_tool_results(self, turn: ProviderTurn,
                            executed_tool_calls: list[ExecutedToolCall]) -> None: ...
    async def close(self) -> None: ...

Each implementation:

create_provider_session() (the factory) inspects MODEL_PROVIDER from llm.py:

if model in OPENAI_MODELS:
    if not openai_api_key: raise Exception("OpenAI API key is missing.")
    return OpenAIProviderSession(...)
if model in ANTHROPIC_MODELS: ...
if model in GEMINI_MODELS: ...
raise ValueError(f"Unsupported model: {model.value}")

8.4 Tool runtime

Tools defined in backend/agent/tools/definitions.py::canonical_tool_definitions(image_generation_enabled):

The runtime (AgentToolRuntime.execute) dispatches by tool_call.name. Failures yield ToolExecutionResult(ok=False, ...) with summary={"error": "..."} so the frontend can surface them.

8.5 Prompt construction

The system prompt (backend/prompts/system_prompt.py:SYSTEM_PROMPT) is a single multi-paragraph string that:

• Sets persona and tone ("expert front-end coder", concise, language-mirroring).
• Describes the 5 tools and how to use them.
• Includes per-stack "include this script" instructions:
  • Tailwind: https://cdn.tailwindcss.com
  • Bootstrap: https://cdn.jsdelivr.net/npm/bootstrap@5.3.2/dist/css/bootstrap.min.css (with SRI hash)
  • React: https://cdn.jsdelivr.net/npm/react@18.0.0/umd/react.development.js, react-dom@18, https://unpkg.com/@babel/standalone/babel.min.js
  • Ionic: https://cdn.jsdelivr.net/npm/@ionic/core/dist/ionic/ionic.esm.js
  • Vue: https://registry.npmmirror.com/vue/3.3.11/files/dist/vue.global.js
• Universal: Font Awesome 5.15.3 from cdnjs.

Per-request, build_prompt_messages(stack, input_mode, generation_type, prompt, history, file_state, image_generation_enabled) derives the PromptConstructionPlan and dispatches to one of:

Image policies (per prompts/policies.py::build_user_image_policy):

Image generation enabled:
  "Image generation is enabled for this request. Use generate_images for missing assets when needed."

Image generation disabled:
  "Image generation is disabled for this request. Do not call generate_images. Use provided media,
   CSS effects, or placeholder URLs (https://placehold.co)."

8.6 The Llm enum and provider mapping

# backend/llm.py — every supported model
class Llm(Enum):
    GPT_4_1_2025_04_14 = "gpt-4.1-2025-04-14"
    GPT_5_2_CODEX_LOW = "gpt-5.2-codex (low thinking)"
    # ... 5.2 codex {low, medium, high, xhigh}
    # ... 5.3 codex {low, medium, high, xhigh}
    # ... 5.4 2026-03-05 {none, low, medium, high, xhigh}
    CLAUDE_SONNET_4_6 = "claude-sonnet-4-6"
    CLAUDE_4_5_SONNET_2025_09_29 = "claude-sonnet-4-5-20250929"
    CLAUDE_4_5_OPUS_2025_11_01 = "claude-opus-4-5-20251101"
    CLAUDE_OPUS_4_6 = "claude-opus-4-6"
    GEMINI_3_FLASH_PREVIEW_HIGH = "gemini-3-flash-preview (high thinking)"
    GEMINI_3_FLASH_PREVIEW_MINIMAL = "gemini-3-flash-preview (minimal thinking)"
    GEMINI_3_1_PRO_PREVIEW_HIGH = "gemini-3.1-pro-preview (high thinking)"
    GEMINI_3_1_PRO_PREVIEW_MEDIUM = "gemini-3.1-pro-preview (medium thinking)"
    GEMINI_3_1_PRO_PREVIEW_LOW = "gemini-3.1-pro-preview (low thinking)"

MODEL_PROVIDER: dict[Llm, str] and the convenience sets OPENAI_MODELS, ANTHROPIC_MODELS, GEMINI_MODELS are all derived from the same mapping — single source of truth for provider routing.

OPENAI_MODEL_CONFIG (in llm.py) maps each Codex/GPT enum value to its API name (e.g. "gpt-5.2-codex") and reasoning effort ("low"/"medium"/"high"/"xhigh"/"none"). The OpenAI provider calls get_openai_api_name(model) and get_openai_reasoning_effort(model) to assemble the request.

8.7 Variant model selection

In backend/routes/model_choice_sets.py:

VIDEO_VARIANT_MODELS = (
    Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL,
    Llm.GEMINI_3_1_PRO_PREVIEW_HIGH,
)

ALL_KEYS_MODELS_DEFAULT = (
    Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL,
    Llm.GPT_5_2_CODEX_HIGH,
    Llm.GEMINI_3_FLASH_PREVIEW_HIGH,
    Llm.GEMINI_3_1_PRO_PREVIEW_HIGH,
)

ALL_KEYS_MODELS_TEXT_CREATE = (
    Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL,
    Llm.GPT_5_2_CODEX_HIGH,
    Llm.CLAUDE_OPUS_4_6,
    Llm.GEMINI_3_1_PRO_PREVIEW_LOW,
)

ALL_KEYS_MODELS_UPDATE = (
    Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL,
    Llm.GPT_5_4_2026_03_05_LOW,
)

GEMINI_ANTHROPIC_MODELS = (Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL, Llm.GEMINI_3_1_PRO_PREVIEW_LOW,
                           Llm.CLAUDE_OPUS_4_6, Llm.GEMINI_3_FLASH_PREVIEW_HIGH,
                           Llm.GEMINI_3_1_PRO_PREVIEW_HIGH)
GEMINI_OPENAI_MODELS    = (Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL, Llm.GEMINI_3_1_PRO_PREVIEW_LOW,
                           Llm.GPT_5_2_CODEX_HIGH, Llm.GPT_5_2_CODEX_MEDIUM)
OPENAI_ANTHROPIC_MODELS = (Llm.CLAUDE_OPUS_4_6, Llm.GPT_5_2_CODEX_HIGH, Llm.GPT_5_2_CODEX_MEDIUM)
GEMINI_ONLY_MODELS      = (Llm.GEMINI_3_FLASH_PREVIEW_MINIMAL, Llm.GEMINI_3_1_PRO_PREVIEW_LOW,
                           Llm.GEMINI_3_FLASH_PREVIEW_HIGH, Llm.GEMINI_3_1_PRO_PREVIEW_HIGH)
ANTHROPIC_ONLY_MODELS   = (Llm.CLAUDE_OPUS_4_6, Llm.CLAUDE_SONNET_4_6)
OPENAI_ONLY_MODELS      = (Llm.GPT_5_2_CODEX_HIGH, Llm.GPT_5_2_CODEX_MEDIUM)

ModelSelectionStage._get_variant_models() picks the right tuple by inspecting which keys are present, then cycles through the tuple to fill num_variants slots: [A, B] × num=4 → [A, B, A, B].

8.8 WebSocket message types

12 server→client messages (defined as a Literal in generate_code.py:562-575):

WebSocket close codes:

8.9 Frontend state machines

AppState:   INITIAL → CODING → CODE_READY
               ↑                    │
               └────── reset() ─────┘

ScreenRecorderState: INITIAL → RECORDING → FINISHED

AppState lives in app-store. INITIAL shows the StartPane; CODING and CODE_READY show the PreviewPane and Sidebar. Browser-tab favicon flips to "coding" while in CODING (via useBrowserTabIndicator).

The project history is a commit graph — each generation is a Commit (type: "ai_create" | "ai_edit" | "code_create") with a parentHash and N variants. head is the active commit hash. nanoid provides commit hashes and asset IDs.

8.10 Testing strategy

Backend (pytest, backend/tests/) — 18 test files covering:

Run: cd backend && poetry run pytest.

Frontend (Jest) — frontend/src/lib/prompt-history.test.ts, frontend/src/components/history/utils.test.ts. Plus a Puppeteer e2e at frontend/src/tests/qa.test.ts (gated by RUN_E2E=true).

Run: cd frontend && yarn test (Jest only) or yarn test:qa (e2e against http://localhost:5173/).

8.11 Code patterns and conventions

• Type strictness: Pyright in basic mode for backend. TypeScript strict for frontend.
• No async generators in pipeline middlewares — they all conform to (ctx, next_func) -> None.
• Tool calls feed back into the same session via session.append_tool_results(turn, executed_tool_calls). Each provider's session implementation knows how to thread the tool results into the next turn's prompt.
• HTML extraction is deliberately tolerant: regex tries <file path="...">...</file> first (the explicit format), then doctype + html, then bare <html>...</html>, finally returns the raw text.
• Print-debug logging is the convention; structured logging is not used. Wrap with your favorite logging shim if running at scale.
• WebSocket safety: every send is guarded by is_closed to avoid noisy ConnectionClosedOK exceptions when the client disconnects mid-stream.

8.12 Potential improvements (technical debt)

8.13 Contributing guide

Pre-commit hooks (backend/.pre-commit-config.yaml):

- check-yaml
- check-added-large-files

To add a new model:

1. Append to Llm enum in backend/llm.py.
2. Add to MODEL_PROVIDER dict so it routes correctly.
3. If OpenAI: add to OPENAI_MODEL_CONFIG (api_name + reasoning_effort).
4. Add pricing in backend/agent/providers/pricing.py.
5. Mirror to frontend/src/lib/models.ts::CodeGenerationModel and CODE_GENERATION_MODEL_DESCRIPTIONS.
6. Update model_choice_sets.py if the model should be a default variant.
7. Run poetry run pytest and yarn test.

To add a new stack:

1. Extend Stack literal in backend/prompts/prompt_types.py.
2. Mirror to frontend/src/lib/stacks.ts::Stack and STACK_DESCRIPTIONS.
3. Add the include directives to prompts/system_prompt.py::SYSTEM_PROMPT.
4. Update tests.

9. Integration Points

9.1 LLM providers

9.2 Image generation

9.3 Screenshot capture

9.4 REST API

The backend exposes 11 routes (10 HTTP + 1 WebSocket). Authentication: none.

9.5 Frontend routes

// main.tsx
<Route path="/" element={<App />} />
<Route path="/evals" element={<AllEvalsPage />} />
<Route path="/evals/single" element={<EvalsPage />} />
<Route path="/evals/pairwise" element={<PairwiseEvalsPage />} />
<Route path="/evals/best-of-n" element={<BestOfNEvalsPage />} />
<Route path="/evals/run" element={<RunEvalsPage />} />
<Route path="/evals/openai-input-compare" element={<OpenAIInputComparePage />} />

The /evals/* routes are intended for offline model quality work; they share the no-auth posture of the backend.

9.6 Observability

langfuse is a declared dependency (^3.0.2) used to wrap agent turn inputs for trace export. To enable, set the standard Langfuse env vars (LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST) — the SDK auto-instruments without code changes.

The OpenAITurnInputLogger (in fs_logging/openai_turn_inputs.py) writes per-turn HTML reports under LOGS_PATH (or a default subdir) when OPENAI_TURN_INPUT_CONSOLE is set. Useful for debugging tool-call argument parsing.