Screenshot to Code — User Guide

Convert any screenshot, mockup, design, or screen recording into clean, production-ready front-end code using a multi-model agentic LLM pipeline. Drop in an image, pick a stack, watch four variants generate side-by-side, then pick the best one and refine it iteratively in plain English.

1. Quick Start

Screenshot to Code is a self-hosted web app made of two services: a FastAPI backend that talks to LLM providers (OpenAI, Anthropic, Google) and a React + Vite frontend that handles uploads, previews, and version history. You bring the API keys; nothing leaves your machine except the requests to the model providers you authorize.

Key features

• Upload a screenshot, paste a URL, or drop in a screen recording — generate working HTML in under a minute.
• Six output stacks: HTML + Tailwind, HTML + CSS, React + Tailwind, Bootstrap, Vue + Tailwind (beta), Ionic + Tailwind (beta).
• Four parallel "variants" per generation, each from a different model — pick the best result.
• Iterative editing: type make the header sticky and switch to a green palette and the agent edits the existing file in place via edit_file tool calls.
• Optional Flux 2 / DALL-E 3 image generation for placeholder assets.
• Built-in screenshot capture via the ScreenshotOne integration.
• Video → app pipeline (Gemini-only) that re-creates an interactive web app from a screen recording.

Prerequisites: Python 3.10+, Node.js 22 (Yarn), and at least one of: OpenAI API key, Anthropic API key, or Gemini API key.

One-command install (using Docker Compose):

git clone https://github.com/abi/screenshot-to-code.git
cd screenshot-to-code
echo "OPENAI_API_KEY=sk-..." > backend/.env
docker compose up -d --build

Open http://localhost:5173 in your browser. Drag a screenshot onto the start pane.

Tip

You don't actually have to set keys in backend/.env — you can also paste them directly into the in-app Settings dialog (gear icon, top-left). Browser-only storage; the keys are sent on each WebSocket request and never persisted server-side.

2. Installation

2.1 System requirements

2.2 Install methods

Docker Compose (recommended)

git clone https://github.com/abi/screenshot-to-code.git
cd screenshot-to-code
cp backend/.env.example backend/.env  # if it exists; otherwise create it
# Add at least one provider key to backend/.env, then:
docker compose up -d --build

The docker-compose.yml at the repo root brings up two services:

• backend (FastAPI on 7001)
• frontend (Vite preview on 5173)

Open http://localhost:5173.

Local backend + frontend (development)

Backend (uses Poetry):

cd backend
poetry install
poetry shell
echo "OPENAI_API_KEY=sk-..." >> .env
poetry run python start.py            # --port 7001 by default

Frontend (uses Yarn):

cd frontend
yarn
yarn dev                              # http://localhost:5173

Hosted version

A managed instance lives at screenshottocode.com. It uses the same code path with VITE_IS_DEPLOYED=true and bills via prepaid credit packs (100 generations / $36) — full source still available locally.

2.3 Configure API keys

The backend reads four LLM-related environment variables and one optional image-generation key. All keys are optional individually, but at least one of OPENAI_API_KEY, ANTHROPIC_API_KEY, or GEMINI_API_KEY must be set or the model selector throws No OpenAI, Anthropic, or Gemini API key found.

Create backend/.env:

# At least one of these is required
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GEMINI_API_KEY=AI...

# Optional — point OpenAI calls at a proxy
OPENAI_BASE_URL=https://my-proxy.example.com/v1

# Optional — enables the generate_images tool
REPLICATE_API_KEY=r8_...

Important

If you only have an OpenAI key, GPT-4 Vision access is required (the image-input pipeline depends on it). If your account hasn't paid into the API at least once, image uploads will return OpenAI Model not found. See the official Troubleshooting.

Warning

Video mode requires Gemini. Without GEMINI_API_KEY, attempting a video-to-code generation throws Video mode requires a Gemini API key. Please add GEMINI_API_KEY to backend/.env or in the settings dialog.

2.4 Verify the install

Hit the backend health endpoint:

curl http://localhost:7001/

<h3>Your backend is running correctly. Please open the front-end URL (default is http://localhost:5173) to use screenshot-to-code.</h3>

Open the frontend; you should see the Start pane with an upload zone and a stack selector.

3. Configuration

User-facing settings live in two places: the Settings dialog (gear icon → cog) for API keys and editor preferences, and the stack selector in the start pane / sidebar for the output framework. Both persist to localStorage under the key setting.

3.1 Settings dialog

Note

Keys entered in Settings take precedence over the backend .env. Anything you paste into the dialog only ever lives in your browser's local storage; it's sent on each WebSocket payload and discarded after the request.

3.2 Stack selection

Use the dropdown labeled Stack: in the start pane. Six choices:

Tip

Video mode (inputMode === "video") hides the stack selector entirely — video output is locked to HTML + Tailwind. Switch to image or text mode if you need to change stacks.

3.3 Choosing a default model

The default model on first launch is Claude Opus 4.5 (claude-opus-4-5-20251101). Change it in Settings → Code Generation Model. Models in beta show a small badge.

The "thinking" suffix is Codex/Gemini reasoning effort. Higher = slower + more expensive but more accurate on tricky layouts.

4. Core Usage

4.1 Generate from a screenshot

1. Open http://localhost:5173. The Start pane shows a unified input pane with four tabs: Upload, URL, Text, Import.
2. Click Upload (or drag a PNG/JPEG onto the pane).
3. Pick your Stack: from the dropdown.
4. The first generation kicks off automatically. The app transitions from INITIAL → CODING → CODE_READY.
5. The right pane shows live preview as the agent streams setCode events. The chat sidebar shows thinking deltas, tool calls, and the working code.

What happens server-side: the React app opens a WebSocket to ws://127.0.0.1:7001/generate-code and sends a JSON payload (stack, input mode, image data URL, all four API keys, generation type). The backend pipeline:

1. WebSocketSetupMiddleware accepts the socket.
2. ParameterExtractionMiddleware validates generatedCodeConfig, inputMode, parses prompt + history.
3. StatusBroadcastMiddleware announces variant count (4 for create, 2 for update, 2 for video).
4. PromptCreationMiddleware assembles system prompt + stack policy + image policy.
5. CodeGenerationMiddleware picks variant models from model_choice_sets.py based on which keys are present, then runs all variants in parallel via asyncio.gather. Each variant uses a per-provider session (OpenAIProviderSession, AnthropicProviderSession, GeminiProviderSession) with five tools available: create_file, edit_file, generate_images, remove_background, retrieve_option.
6. PostProcessingMiddleware finalizes; the WebSocket closes.

Tip

Multiple screenshots in one go: drop several images at once and the agent organizes them into linked pages, tabs, or numbered scaffolds. The image-create prompt explicitly handles "Screenshot 1", "Screenshot 2"... when the images are unrelated.

4.2 Generate from a URL

1. In the Start pane, switch to the URL tab.
2. Paste any HTTP/HTTPS URL.
3. Make sure your ScreenshotOne API key is set in Settings (free tier: 100 screenshots/month).
4. Click generate.

Under the hood: POST /api/screenshot calls ScreenshotOne (https://api.screenshotone.com/take) with block_ads, block_cookie_banners, block_trackers, viewport=1280×832. The returned PNG becomes a base64 data URL and the same image-create flow runs.

Note

URLs without a protocol get https:// auto-prepended. Bare domains with ports (e.g. example.com:8080) are also normalized. Anything else throws Unsupported protocol.

4.3 Generate from a text description

1. Switch to the Text tab.
2. Type something like Login form with email, password, and a "Sign in with Google" button. Use a soft purple accent.
3. Pick a stack and submit.

The text-create prompt tells the model to "make it look modern and sleek; use modern, professional fonts and colors; follow UX best practices."

4.4 Generate from a screen recording (video mode)

1. Switch to the Upload tab and drop in a .mov, .mp4, or .webm file (or use the on-screen recorder).
2. Make sure GEMINI_API_KEY is set.
3. Submit.

Video mode runs two Gemini variants (gemini-3-flash-preview minimal and gemini-3.1-pro-preview high). The pipeline:

• moviepy extracts frames at 10 fps.
• Frames are re-packed into a single image-detail prompt instructing the model to "watch the entire video carefully... re-create the same app exactly such that the same user interactions will produce the same results."
• The model emits a fully interactive HTML/Tailwind app with mocked data instead of real backend calls.

Warning

Cost adds up quickly. A 30-second clip at 10 fps is 300 frames; even on Flash Preview, this can exceed $0.50/run. Use the cost estimator in backend/video/cost_estimation.py to project before you generate.

4.5 Iterative editing

Once you have generated code (CODE_READY state), the sidebar reveals an Update chat box.

1. Type natural-language instructions: make the call-to-action bigger, swap to a dark theme, add a footer with copyright.
2. Optionally drop additional reference images via the paperclip icon.
3. Press Enter.

The frontend sends a new payload with generationType: "update", the existing code as fileState.content, and the running variant history. The backend picks 2 update-mode models (gemini-3-flash-preview minimal and gpt-5.4 low) and the agent uses edit_file tool calls (exact string replacements + unified diff) instead of regenerating the full HTML.

Tip

Select-and-edit is available for html_tailwind and html_css stacks. Click any element in the preview, type your instruction, and the request body includes that element's HTML so the model edits exactly that span.

4.6 Working with variants

When generation starts, you see 4 variants (or 2 for edits) as thumbnails along the top of the preview pane.

The sidebar shows live status for each variant: gray (pending), green (complete), red (error or cancelled).

4.7 Download and import

5. Common Tasks

Recipe 1 — Replicate a SaaS landing page from a screenshot

1. Open the app and confirm OPENAI_API_KEY (or any other key) is set.
2. Pick Stack: HTML + Tailwind.
3. Drag landing-page.png onto the Upload tab.
4. Wait ~30s while the four variants run (you can watch them stream in real-time).
5. Pick the variant that nailed the navbar layout.
6. Type make the hero CTA stick to the bottom on mobile and press Enter.
7. Click Download.

Recipe 2 — Recreate an admin dashboard mockup using React

1. Stack: React + Tailwind.
2. Upload admin-dashboard.png.
3. Once code generates, click Code → Preview. The preview iframe runs React via the UMD CDN scripts injected by the system prompt (react@18, react-dom@18, @babel/standalone).
4. Edit refinements: add a sidebar with Dashboard, Users, Billing, Settings entries.

Recipe 3 — Build from text and iterate

1. Text tab. Prompt: A modern pricing table with Free / Pro / Team tiers. Include a "Most popular" badge on Pro.
2. Generate. The text prompt is wrapped with the stack-aware system prompt that includes Tailwind/React/Bootstrap include directives.
3. Update: make the Pro card use a violet accent and lift it 8 pixels above the others.

Recipe 4 — Capture a live website by URL

1. Drop your ScreenshotOne API key into Settings → Screenshot by URL.
2. Switch to URL tab, paste https://stripe.com.
3. The backend route POST /api/screenshot fetches a 1280×832 PNG with ads/trackers blocked.
4. The app generates from that screenshot like any other image.

Recipe 5 — Replace placeholder images with real ones

By default, Placeholder Images is on. The agent can call the generate_images tool with prompts; the backend routes them to:

• Flux 2 Klein via Replicate (black-forest-labs/flux-2-klein-4b) — this is the default flux mode.
• DALL-E 3 via OpenAI (dall-e-3, 1024x1024, quality=standard, style=natural) — used when the runner is configured to use dalle3.

Either way you need REPLICATE_API_KEY (for Flux) or your OPENAI_API_KEY to cover image generation.

Recipe 6 — Save tokens by disabling image generation

Toggle Image Generation → Placeholder Images OFF in Settings. The system prompt switches policy to: "Image generation is disabled for this request. Do not call generate_images. Use provided media, CSS effects, or placeholder URLs (https://placehold.co)."

Recipe 7 — Re-create an interactive prototype from a Loom recording

1. Record (or download) a screen capture demonstrating the interaction you want.
2. Switch to Video input. Upload the file.
3. Set GEMINI_API_KEY if not already configured.
4. Generate. The video frames are sent to Gemini Pro/Flash; the model wires up <button> handlers, modal toggles, etc. with vanilla JS.

Recipe 8 — Use a custom OpenAI proxy

For self-hosters routing through an OpenAI-compatible proxy (e.g. Helicone, an internal gateway):

1. Set OPENAI_BASE_URL=https://gateway.example.com/v1 in backend/.env, or paste it in Settings → OpenAI Base URL (only visible when VITE_IS_DEPLOYED is not true).
2. Restart the backend if you used .env.

Note

The Base URL field is intentionally hidden in the cloud-hosted version (IS_PROD=true) for security.

Recipe 9 — Compare model output side-by-side

Visit http://localhost:5173/evals/best-of-n (admin route, no auth). Point it at multiple result folders to see eval inputs vs. each model's HTML rendered as iframes. Useful when you're tuning model selection.

Recipe 10 — Cancel a running generation

Click the cancel button (X icon) in the sidebar while in CODING state. The frontend sends close code 4333 (USER_CLOSE_WEB_SOCKET_CODE); the backend tears down the session and the variant statuses roll back.

6. Troubleshooting

Error: No API key

Cause: ParameterExtractionStage ran but found neither an env-var nor a settings-dialog value for any of OpenAI / Anthropic / Gemini.

Solution:

1. Open Settings (gear icon).
2. Paste at least one provider key.
3. Re-generate.

Prevention: Set OPENAI_API_KEY in backend/.env and restart so it survives browser cache clears.

Error: Video mode requires a Gemini API key. Please add GEMINI_API_KEY to backend/.env or in the settings dialog

Cause: Video input mode is locked to VIDEO_VARIANT_MODELS (GEMINI_3_FLASH_PREVIEW_MINIMAL, GEMINI_3_1_PRO_PREVIEW_HIGH); both require a Gemini key.

Solution: Add GEMINI_API_KEY to backend/.env (then restart) or to Settings → Gemini API key.

Prevention: Get a Gemini key from Google AI Studio before trying video mode.

Error: Incorrect OpenAI key. Please make sure your OpenAI API key is correct…

Cause: OpenAI returned AuthenticationError.

Solution:

1. Verify the key on OpenAI's API keys page.
2. Confirm there is no leading/trailing whitespace.
3. Re-paste in Settings.

Error: OpenAI Model not found … Please make sure you have followed the instructions correctly to obtain an OpenAI key with GPT vision access

Cause: Account doesn't have GPT-4 Vision access, typically because it has never been billed.

Solution: Add at least $5 of credit to your OpenAI account, then retry. See community issue #499 and the official Troubleshooting.md.

Error: OpenAI error - 'You exceeded your current quota, please check your plan and billing details.'

Cause: OpenAI rate limit / quota.

Solution: Wait, top up, or switch to Anthropic / Gemini in Settings.

Error: Error generating code. Please contact support.

Cause: Every variant raised an exception inside AgenticGenerationStage._run_variant. Common community-reported causes (issues #503, #502, #484, #464):

1. Bad/expired API key.
2. Image too large or unsupported format.
3. Model temporarily unavailable.

Solution:

1. Open the browser DevTools console — the variantError payloads tell you which provider failed.
2. Check the backend log for the full traceback (the AgenticGenerationStage prints Error in variant N: with traceback.print_exception).
3. If it's a transient model error, retry. Otherwise rotate keys.

Error: Error generating code. Check the Developer Console AND the backend logs for details. Feel free to open a Github issue.

Cause: WebSocket closed with non-1000 code that isn't 4332 or 4333. Network blip, backend crash, or proxy interfering.

Solution:

1. Confirm the backend is alive: curl http://localhost:7001/.
2. Confirm VITE_WS_BACKEND_URL matches your backend host (default ws://127.0.0.1:7001).
3. Check for corporate firewalls blocking WebSockets.

Error: Error taking screenshot

Cause: ScreenshotOne returned non-200 or empty bytes (URL unreachable, key invalid, target blocking robots).

Solution:

1. Confirm the URL loads in a normal browser.
2. Verify your ScreenshotOne key.
3. Try a simpler URL to isolate the cause.

Error: Update requests require history or fileState.content

Cause: Sent an update request without prior generation (no head commit, no file state).

Solution: Generate a base version first; updates can only follow an existing ai_create or ai_edit commit.

Error: Update called with no head (frontend toast: No current version set)

Cause: UI tried to doUpdate while head === null. Usually a state-restoration glitch.

Solution: Refresh the page. If it persists, clear localStorage and start a new project.

Error: Regenerate called with no head (toast: No current version set)

Cause: Regenerate button pressed before a successful first generation.

Solution: Run a create first, then regenerate.

Backend won't start with Installing moviepy (1.0.3): Failed

Cause: Poetry can't build moviepy on the current Python — community issue #522.

Solution:

1. Use Python 3.10 or 3.11 (avoid 3.12+ if you hit moviepy build errors).
2. On macOS, install ffmpeg: brew install ffmpeg.
3. On Linux, install build essentials: apt install build-essential libsm6 libxext6.

Generation hangs, no preview update

Cause: WebSocket dropped silently or model's tool calls are stuck (community issue #464).

Solution:

1. Open DevTools → Network → WS — check the socket is still open and frames are flowing.
2. Cancel and retry; the agent has a max_steps = 20 budget that throws Agent exceeded max tool turns if reached.

"Select and edit seems to trigger an error" (issue #523)

Cause: Select-and-edit only supports HTML stacks (html_tailwind and html_css). Switching to react_tailwind or bootstrap mid-session can leave the feature half-enabled.

Solution: Switch back to an HTML stack or refresh the page.

Frontend builds but runtime says Connecting to backend @ ws://127.0.0.1:7001 and never connects

Cause: Backend not running, or you're on a different host than the default.

Solution: Set VITE_WS_BACKEND_URL=ws://YOUR_HOST:7001 and VITE_HTTP_BACKEND_URL=http://YOUR_HOST:7001 in frontend/.env and rebuild.

7. FAQ

Q: Can I use this with my own LLM provider that isn't OpenAI / Anthropic / Gemini?

Not out of the box. The supported providers are OpenAI, Anthropic, and Google Gemini, dispatched via backend/agent/providers/factory.py. There are open community feature requests for DeepSeek (issue #531) and Ollama (issue #354) but neither is wired up. You can sometimes shim through OPENAI_BASE_URL if your provider is OpenAI-API-compatible.

Q: Where are my API keys stored?

Browser localStorage (under the key setting) when entered via the Settings dialog. Or the backend/.env file when set there. Nothing is persisted server-side beyond the lifetime of one WebSocket request.

Q: Does the app track me?

The deployed screenshottocode.com build injects a Plausible script for traffic analytics (privacy-friendly, no cookies). Self-hosted builds with VITE_IS_DEPLOYED unset do not include any tracking.

Q: How do I switch which 4 models run for each generation?

The model sets are hard-coded in backend/routes/model_choice_sets.py. Edit ALL_KEYS_MODELS_DEFAULT, ALL_KEYS_MODELS_TEXT_CREATE, ALL_KEYS_MODELS_UPDATE, etc., then restart. See the Admin Guide for a full breakdown.

Q: Can I run more than 4 variants?

No. NUM_VARIANTS = 4 (image/text create) and NUM_VARIANTS_VIDEO = 2 are constants in backend/config.py. Edit them and restart, but be mindful of cost — each variant is a full LLM call.

Q: How big can my screenshots be?

Each provider sets its own image size cap. For OpenAI, image data URLs over a few MB tend to fail (community issue #557). Resize to ≤ 1280px wide for best results.

Q: Why are some models marked (beta)?

Beta models in the dropdown are GPT 5.2/5.3/5.4 Codex variants and Gemini 3.x previews. They tend to give the strongest visual fidelity but their pricing/availability is changing under the hood. The non-beta default is Claude Opus 4.5.

Q: How do I export or share a project's history?

The history sidebar lets you click any past commit (variant snapshots, edit chains) and download that exact HTML via the Download button. There is no native export of the full project graph in this version.

Q: Does the app support DeepSeek or open-source models like Ollama?

Not directly. Tracked in issues #531 and #354. For now, the only approved providers are OpenAI, Anthropic, and Gemini.

Q: Can I generate Android / iOS / Flutter UIs?

Not currently. Output is web-only (HTML / CSS / JS / React / Vue / Bootstrap / Ionic). Issue #504 tracks the request for Android-native generation; no implementation yet.

Q: How is the generated code licensed?

The repository is MIT-licensed. The generated code is yours to do with as you please.