Gatsby Administrator & Developer Guide

1. Architecture Overview

Gatsby is a React-based static site generator organized as a monorepo of ~90 packages. At build time, it sources data from arbitrary inputs (filesystems, CMSs, APIs), unifies them into a GraphQL schema, runs user-defined queries, and renders static HTML with embedded React for client-side hydration. The result is a public/ directory of static assets ready for CDN deployment.

High-Level Architecture

┌──────────────────────────────────────────────────────────────┐
│                       gatsby-cli                             │
│  CLI entry point: develop / build / serve / clean / info     │
└──────────────┬───────────────────────────────────────────────┘
               │
┌──────────────▼───────────────────────────────────────────────┐
│                    gatsby (core)                              │
│                                                              │
│  ┌─────────────┐  ┌──────────────┐  ┌─────────────────────┐ │
│  │ Bootstrap    │  │ Data Layer   │  │ Build Pipeline      │ │
│  │ (initialize, │  │ (sourceNodes,│  │ (createPages,       │ │
│  │  load        │  │  schema,     │  │  extract queries,   │ │
│  │  plugins)    │  │  GraphQL)    │  │  run queries,       │ │
│  └──────┬──────┘  └──────┬───────┘  │  build HTML,        │ │
│         │                │          │  webpack bundle)     │ │
│         │                │          └──────────┬──────────┘ │
│  ┌──────▼────────────────▼─────────────────────▼──────────┐ │
│  │              State Machine (xstate)                     │ │
│  │  Services: buildSchema, createPages, extractQueries,    │ │
│  │  runQueries, buildHTML, graphQLTypegen                   │ │
│  └─────────────────────────────────────────────────────────┘ │
│                                                              │
│  ┌───────────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │ Redux Store       │  │ LMDB Cache   │  │ Worker Pool  │  │
│  │ (pages, nodes,    │  │ (persistent  │  │ (parallel    │  │
│  │  schema, config)  │  │  datastore)  │  │  processing) │  │
│  └───────────────────┘  └──────────────┘  └──────────────┘  │
└──────────────────────────────────────────────────────────────┘
               │
┌──────────────▼───────────────────────────────────────────────┐
│                     Plugin Ecosystem                          │
│  Source: filesystem, WordPress, Contentful, Drupal, Shopify  │
│  Transform: remark, sharp, MDX, JSON, YAML, CSV, XML        │
│  General: image, offline, manifest, sitemap, TypeScript      │
└──────────────────────────────────────────────────────────────┘
               │
┌──────────────▼───────────────────────────────────────────────┐
│                     Output: public/                           │
│  Static HTML, JS bundles, CSS, optimized images, page-data   │
└──────────────────────────────────────────────────────────────┘

Component Inventory

Technology Stack

Build Lifecycle (Data Flow)

The Gatsby build process follows a deterministic state machine:

1. Initialize — Load gatsby-config.js, resolve plugins, set up Redux store, initialize LMDB cache
2. onPreInit — First plugin hook; runs immediately after plugin load
3. onPreBootstrap — Run before bootstrap sequence
4. Source Nodes (sourceNodes) — Plugins create data nodes in the Redux store
5. onCreateNode — Transformer plugins react to new nodes (e.g., Markdown → HTML)
6. Create Schema (createSchemaCustomization) — Plugins define custom GraphQL types
7. Build Schema — Gatsby infers types from nodes and builds the GraphQL schema
8. Create Resolvers (createResolvers) — Plugins add custom field resolvers
9. Create Pages (createPages) — Plugins create pages from data
10. Extract Queries — Babel plugin scans components for graphql tagged template literals
11. Run Queries — Execute all page queries and static queries against the schema
12. Build HTML — Render each page to static HTML via server-side rendering
13. Bundle JavaScript — Webpack bundles client-side JavaScript
14. onPostBuild — Final plugin hook after build completes

Note

In develop mode, steps 10-13 run incrementally. Gatsby uses query-on-demand to only execute queries for pages currently viewed in the browser, significantly reducing dev server startup time.

2. System Requirements

Node.js Compatibility

The engines field in the root package.json enforces:

{
  "engines": {
    "yarn": "^1.17.3",
    "node": ">=18.0.0 <26",
    "npm": ">=8.0.0"
  }
}

Warning

Node.js 23 and 24 have a known buffer bounds issue on macOS (#39223). Use Node.js 18 or 20 for production builds.

3. Configuration Reference

Configuration Files

gatsby-config.js Options

Environment Variable Reference

The following environment variables are recognized by Gatsby core and official plugins. Variables prefixed with GATSBY_ are embedded in the client-side bundle.

Core Build Variables

Experimental Feature Flags

Cache and Data Variables

Development Server Variables

Important

ENABLE_GATSBY_REFRESH_ENDPOINT is required for CMS preview integrations (e.g., Drupal, Contentful). When set to true, a POST to /__refresh triggers a full data refresh in development.

Image Processing Variables

Plugin-Specific Variables

Benchmark and Internal Variables

CLI Flags Reference

gatsby develop

gatsby build

gatsby serve

4. Deployment & Scaling

Build Output Structure

After gatsby build, the public/ directory contains:

public/
├── index.html                    # Homepage
├── 404.html                      # 404 page
├── page-data/
│   ├── index/page-data.json     # Page data for homepage
│   ├── 404.html/page-data.json  # Page data for 404
│   └── sq/d/                    # Static query results
├── static/                       # Static assets (copied from src/static/)
├── chunk-map.json               # Webpack chunk mapping
├── webpack.stats.json           # Webpack build statistics
└── [page-paths]/                # Static HTML for each page

Deployment Targets

Gatsby produces static files deployable to any static hosting platform:

Adapter System

Gatsby 5 introduced adapters for platform-specific optimizations:

// gatsby-config.js
const adapter = require("gatsby-adapter-netlify").default

module.exports = {
  adapter: adapter({
    excludeDatastoreFromEngineFunction: false,
  }),
}

The adapter hooks into the build pipeline to:

• Configure server-side rendering routing
• Optimize function bundling for the target platform
• Set up CDN cache headers
• Configure redirects and rewrites

Note

The GATSBY_CONTINUE_BUILD_ON_ADAPTER_MISMATCH env var allows builds to proceed when the detected deployment platform doesn't match the configured adapter.

Scaling Large Builds

For sites with thousands of pages:

1. Increase Node.js heap:

   NODE_OPTIONS="--max-old-space-size=8192" gatsby build
   

2. Maximize CPU utilization:

   GATSBY_CPU_COUNT=logical_cores gatsby build
   

3. Enable parallel sourcing:

   GATSBY_EXPERIMENTAL_PARALLEL_SOURCING=true gatsby build
   

4. Preserve download cache:

   GATSBY_EXPERIMENTAL_PRESERVE_FILE_DOWNLOAD_CACHE=true gatsby build
   

Tip

The gatsby-worker package provides a custom worker pool (not Node.js worker_threads directly) that serializes tasks via IPC. The WorkerPool class in packages/gatsby-worker/src/index.ts manages lifecycle, task queuing, and error propagation. Configuring GATSBY_CPU_COUNT directly controls pool size.

5. Operations

Cache Management

Gatsby uses multiple cache layers:

# Full cache clear
gatsby clean

# Selective cache management
rm -rf .cache/webpack  # Clear only webpack cache

Tip

In CI environments, persist .cache/ and public/ between builds for incremental builds. This can reduce build times by 50-80% for sites that primarily add/update content.

Monitoring Build Performance

Enable verbose output to diagnose slow builds:

VERBOSE=true gatsby build

This reports per-lifecycle timing:

success open and validate gatsby-configs - 0.058s
success load plugins - 0.284s
success source and transform nodes - 12.345s  ← bottleneck
success building schema - 0.265s
success createPages - 1.234s
success run page queries - 45.678s  ← bottleneck
success Building production JavaScript and CSS bundles - 23.456s
success Building static HTML for pages - 8.901s

For distributed tracing, use OpenTracing:

GATSBY_OPEN_TRACING_CONFIG_FILE=./jaeger-config.js gatsby build

Note

The gatsby-plugin-benchmark-reporting package in the monorepo provides automated build performance reporting. It captures onPostBuild data and sends it to a configurable endpoint via BENCHMARK_REPORTING_URL.

Health and Diagnostics

The gatsby info command outputs environment diagnostics:

gatsby info --clipboard

System:
  OS: macOS 14.0
  CPU: (8) arm64 Apple M1
  Shell: /bin/zsh
Binaries:
  Node: 20.10.0
  npm: 10.2.3
  Yarn: 1.22.19
npmPackages:
  gatsby: ^5.13.0 => 5.13.3
  gatsby-plugin-image: ^3.13.0 => 3.13.1
  gatsby-plugin-sharp: ^5.13.0 => 5.13.1

6. Performance

Webpack Configuration

Gatsby's webpack configuration (packages/gatsby/src/utils/webpack.config.js) supports four stages:

Key webpack behaviors:

• Code splitting — automatic chunk splitting per route
• Asset hashing — content-based filenames for long-term caching
• Polyfills — core-js included by default (disable with polyfill: false)
• Public path — configurable via GATSBY_WEBPACK_PUBLICPATH or pathPrefix

// Custom webpack config via gatsby-node.js
exports.onCreateWebpackConfig = ({ stage, getConfig, rules, loaders, plugins, actions }) => {
  // Access current config
  const config = getConfig()

  // Modify via setWebpackConfig (merged with existing)
  actions.setWebpackConfig({
    resolve: {
      alias: {
        '@components': path.resolve(__dirname, 'src/components'),
      },
    },
  })

  // Or replace entire config (use with caution)
  actions.replaceWebpackConfig(config)
}

Image Processing Pipeline

gatsby-plugin-sharp wraps the Sharp library (libvips) for image transformations:

• Formats: JPEG, PNG, WebP, AVIF, TIFF
• Operations: resize, crop, rotate, duotone, blur, grayscale
• Lazy processing: In development, images are processed on-demand when requested by the browser
• Worker pool: Image operations run in parallel worker processes
• Error handling: SharpError class wraps failures with file context

The image CDN subsystem (packages/gatsby-plugin-utils/src/polyfill-remote-file/) supports encrypted URL generation for CDN-optimized image delivery:

Query-on-Demand

In development, Gatsby defers query execution until a page is viewed:

• Only 404 pages, currently active browser tabs, and pending websocket requests run immediately
• Static queries (sq-- prefix) and slice queries (slice-- prefix) always run
• Controlled via GATSBY_QUERY_ON_DEMAND (default: true in develop)
• GATSBY_ENABLE_QUERY_ON_DEMAND_IN_CI overrides for CI environments

7. Security

Client-Side Data Exposure

Caution

All GATSBY_* environment variables are embedded in the webpack bundle and visible in client-side JavaScript. Never use this prefix for secrets, API keys, or credentials.

• .env.development and .env.production files should be in .gitignore
• Server-only variables (without GATSBY_ prefix) are safe for secrets during build
• GraphQL data is serialized to page-data.json files in public/ — any data queried is publicly accessible

Authentication Patterns

Gatsby supports client-side authentication via client-only routes:

// gatsby-node.js — mark routes as client-only
exports.onCreatePage = async ({ page, actions }) => {
  if (page.path.match(/^\/app/)) {
    page.matchPath = "/app/*"
    actions.createPage(page)
  }
}

The SSR build will render a shell for these routes; client-side React handles authentication checks.

Serverless Function Security

Functions in src/api/ receive standard Node.js req/res objects. Platform-specific middleware (e.g., Netlify Identity, Vercel auth) provides authentication for serverless endpoints.

Refresh Endpoint

When ENABLE_GATSBY_REFRESH_ENDPOINT=true:

• The /__refresh endpoint accepts POST requests to trigger data refresh
• GATSBY_REFRESH_TOKEN provides optional bearer token authentication
• The initial webhook body can be passed via GATSBY_INITIAL_WEBHOOK_BODY

Warning

The refresh endpoint is intended for development and CMS preview integrations only. Do not expose it in production without authentication.

Supply Chain

The monorepo uses:

• Yarn 1.22.19 with workspaces for deterministic resolution
• lint-staged + husky for pre-commit checks
• ESLint + Prettier for code quality enforcement
• TypeScript strict mode for type safety in core packages

8. Developer Internals

Repository Structure

gatsbyjs/gatsby/
├── packages/                    # ~90 npm packages
│   ├── gatsby/                  # Core framework (5.8M chars)
│   │   ├── src/
│   │   │   ├── commands/        # CLI command implementations
│   │   │   ├── services/        # Build lifecycle services (21 files)
│   │   │   ├── utils/           # Core utilities (89 files, 350K chars)
│   │   │   ├── schema/          # GraphQL schema builder
│   │   │   ├── redux/           # Redux store, actions, reducers
│   │   │   ├── query/           # Query extraction, planning, execution
│   │   │   ├── datastore/       # LMDB-backed data storage
│   │   │   ├── state-machines/  # XState state machines
│   │   │   └── internal-plugins/ # Built-in plugins (dev-404, partytown)
│   │   └── cache-dir/           # Client-side runtime (loader, navigation)
│   ├── gatsby-cli/              # CLI entry point
│   ├── create-gatsby/           # Project scaffolding
│   ├── gatsby-plugin-sharp/     # Image processing (75K chars)
│   ├── gatsby-source-filesystem/ # File sourcing (23K chars)
│   ├── gatsby-source-wordpress/ # WordPress sourcing (50K chars)
│   ├── gatsby-transformer-remark/ # Markdown transformation (31K chars)
│   ├── gatsby-core-utils/       # Shared utilities (35K chars)
│   ├── gatsby-worker/           # Worker pool (22K chars)
│   └── [60+ more packages]
├── starters/                    # Starter templates
│   ├── blog/
│   ├── default/
│   ├── gatsby-starter-minimal-ts/
│   ├── gatsby-starter-plugin/
│   └── gatsby-starter-wordpress-blog/
├── examples/                    # Example projects (~50)
├── e2e-tests/                   # Cypress E2E test suites
├── integration-tests/           # Integration test suites
├── benchmarks/                  # Performance benchmarks
├── deprecated-packages/         # Deprecated packages (recipes, telemetry, admin)
├── scripts/                     # Monorepo tooling (changelog, publish)
└── peril/                       # GitHub bot (Danger.js)

State Machine Architecture

Gatsby uses XState to orchestrate the build pipeline. The state machines in packages/gatsby/src/state-machines/ define:

• Data Layer Machine — Sources nodes, builds schema, creates pages
• Query Running Machine — Extracts queries, calculates dirty queries, executes them
• Waiting Machine — Handles the develop-mode idle loop (listens for file changes, webhooks)

Services (in packages/gatsby/src/services/) are invoked by the state machines:

Redux Store Shape

The internal Redux store (in packages/gatsby/src/redux/) tracks:

• pages — Map of all pages (path → page object)
• nodes — All data nodes (internal, backed by LMDB)
• schema — GraphQL schema object
• config — Resolved gatsby-config.js
• components — Map of page component paths
• staticQueryComponents — Static query definitions
• status — Build status flags
• queries — Query tracking and dirty state
• slices — Gatsby Slice components

Node API Reference

These hooks are exported from gatsby-node.js:

Browser API Reference

These hooks are exported from gatsby-browser.js:

Node API Helpers

Every Node API hook receives a helpers object with:

Client-Side Runtime

The packages/gatsby/cache-dir/ directory contains the client-side runtime:

Key Utility Modules

The packages/gatsby/src/utils/ directory (89 files, 350K chars) contains the core utility layer:

Worker Pool Architecture

packages/gatsby-worker/ provides a custom worker pool:

• WorkerPool class manages child processes via child_process.fork()
• TaskQueue implements FIFO task scheduling with priority support
• Workers are identified by GATSBY_WORKER_ID env var
• In-flight task state can be dumped to disk via GATSBY_WORKER_IN_FLIGHT_DUMP_LOCATION
• Error: "There is no worker with ..." indicates a worker process crashed

Plugin Architecture

Each Gatsby plugin is an npm package that can implement:

1. gatsby-node.js — Node.js build-time hooks
2. gatsby-browser.js — Client-side browser hooks
3. gatsby-ssr.js — Server-side rendering hooks
4. package.json — with "keywords": ["gatsby-plugin"]

Plugin options are validated via pluginOptionsSchema using Joi:

exports.pluginOptionsSchema = ({ Joi }) => {
  return Joi.object({
    path: Joi.string().required().description(`Path to source directory`),
    name: Joi.string().default(`default`).description(`Source instance name`),
    ignore: Joi.array().items(Joi.string()).description(`File patterns to ignore`),
  })
}

Theme System

Gatsby themes are plugins that can:

• Define their own gatsby-config.js (merged with site config)
• Provide pages, components, and data
• Support component shadowing — the site can override any theme component by placing a file at src/{theme-package-name}/{original-path}

Theme resolution follows the webpack theme component shadowing system in packages/gatsby/src/internal-plugins/webpack-theme-component-shadowing/.

9. Integration Points

Source Plugin Contract

Source plugins implement sourceNodes to create nodes:

exports.sourceNodes = async ({ actions, createNodeId, createContentDigest }) => {
  const { createNode } = actions

  // Fetch data from external source
  const data = await fetchDataFromAPI()

  data.forEach(item => {
    createNode({
      ...item,
      id: createNodeId(`my-source-${item.id}`),
      parent: null,
      children: [],
      internal: {
        type: `MySourceType`,
        contentDigest: createContentDigest(item),
      },
    })
  })
}

Transformer Plugin Contract

Transformer plugins implement onCreateNode to transform source nodes:

exports.onCreateNode = async ({ node, actions, loadNodeContent, createNodeId, createContentDigest }) => {
  if (node.internal.mediaType !== `text/markdown`) return

  const { createNode, createParentChildLink } = actions
  const content = await loadNodeContent(node)

  const markdownNode = {
    id: createNodeId(`${node.id} >>> MarkdownRemark`),
    children: [],
    parent: node.id,
    internal: {
      type: `MarkdownRemark`,
      content,
      contentDigest: createContentDigest(content),
    },
  }

  createNode(markdownNode)
  createParentChildLink({ parent: node, child: markdownNode })
}

gatsby-source-filesystem

Creates File nodes from local filesystem paths:

gatsby-source-wordpress

Sources WordPress content via WPGraphQL:

• Hooks: onPluginInit, createSchemaCustomization, sourceNodes, onCreatePage, onCreateDevServer
• State management: Rematch (Redux-like) with async local storage
• Browser hook: onRouteUpdate hydrates inline WordPress images
• Auth: HTTP Basic Auth via HTTPBASICAUTH_USERNAME/HTTPBASICAUTH_PASSWORD
• Preview: Integrates with GATSBY_IS_PREVIEW and ENABLE_GATSBY_REFRESH_ENDPOINT
• Error handling: Validates GraphQL responses, detects empty strings, redirects, and connection failures

gatsby-transformer-remark

Transforms Markdown files to MarkdownRemark nodes:

• Processor: Unified/remark with GFM and footnotes support
• Output types: MarkdownRemark (HTML, excerpt, headings, table of contents, word count, time to read)
• Caching: AST, HTML, headings, and TOC are cached by content digest
• Sub-plugins: Supports remark sub-plugins via createSchemaCustomization
• Heading extraction: Generates IDs and extracts {id, value, depth} tuples

Deprecated Packages

The following packages are in deprecated-packages/:

Testing Strategy

Running tests:

# All tests
yarn jest

# Specific package
yarn jest packages/gatsby-plugin-sharp

# Integration tests
yarn test:integration

# Type checking
yarn typecheck

Contributing

The monorepo uses:

• Yarn workspaces (packages/*) for dependency management
• Lerna for versioning and publishing
• ESLint with Google style base + custom rules (backtick quotes, camelCase, file naming)
• Prettier for formatting
• Husky for pre-commit hooks (lint-staged)
• TypeScript for core packages (.ts/.tsx)

Development workflow:

# Bootstrap all packages
yarn bootstrap

# Watch for changes
yarn watch

# Run tests
yarn jest

# Lint
yarn lint

# Format
yarn format

The changelog is automatically generated by scripts/gatsby-changelog-generator/ during Lerna version bumps.

Error Classes