Apache Airflow Administrator & Architecture Guide

This guide is the operational and architectural reference for Apache Airflow 3.3.0. It is written for the people who deploy, configure, secure, scale, and maintain Airflow — platform engineers, SREs, data-platform architects, and developers extending the system. Where the User Guide covers using Airflow, this guide covers running it: how the components fit together, every configuration surface that matters, deployment topologies, executor selection, the metadata database, security, observability, and the internals you need to operate (or replicate) the platform with confidence.

Note

Airflow is a large, multi-product monorepo. This guide centers on the flagship deployable — the apache-airflow core platform — and treats the Task SDK, airflowctl, the Helm chart, the 87 provider packages, and the experimental Go/Java edge SDKs as the extensions they are. Each gets focused coverage; the core gets depth.

1. Executive Summary

Apache Airflow is a Python-based workflow orchestration platform. Workflows ("DAGs") are authored as code; a set of long-running services parse them, schedule task instances, execute those tasks via a pluggable executor, and record all state in a relational metadata database. A FastAPI-based API server exposes both a REST API and a React 19 web UI.

What you are operating:

• A scheduler (the brain — creates DAG runs and schedules tasks).
• An API server (FastAPI + uvicorn — serves the REST API and UI).
• A DAG processor (parses DAG files from bundles into the database).
• A triggerer (runs asynchronous deferrable tasks efficiently).
• An executor (where tasks actually run — Local, Celery, Kubernetes, Edge, or cloud-native).
• A metadata database (PostgreSQL or MySQL in production; SQLite for dev only).
• Optional workers (for Celery) and edge workers (for the Edge executor).

Key 3.x architectural facts an operator must internalize:

Maturity & support: Airflow 3 is the current major line (3.x first released Apr 2025, in active maintenance). Airflow 2.x is in limited maintenance with EOL Apr 22, 2026. Run the latest 3.x patch.

2. Architecture

2.1 Component topology

                          ┌──────────────────────────────────────────────┐
                          │              Metadata Database                │
                          │      (PostgreSQL / MySQL; SQLite = dev)        │
                          │  DAG runs · task instances · connections ·     │
                          │  variables · pools · serialized DAGs · XComs   │
                          └───────▲───────────▲───────────▲───────────▲────┘
                                  │           │           │           │
        ┌─────────────────────────┘           │           │           └────────────────────┐
        │                                      │           │                                │
 ┌──────┴───────┐                      ┌───────┴──────┐ ┌──┴────────────┐            ┌───────┴────────┐
 │  Scheduler   │                      │  DAG         │ │  API Server   │            │   Triggerer    │
 │  - create    │                      │  Processor   │ │  (FastAPI +   │            │  - async       │
 │    DAG runs  │                      │  - parse DAG │ │   uvicorn)    │            │    deferrable  │
 │  - schedule  │                      │    bundles   │ │  - REST API   │            │    operators   │
 │    task inst │                      │  - serialize │ │  - React UI   │            │                │
 │  - heartbeat │                      │    to DB     │ │  - Execution  │            └────────────────┘
 └──────┬───────┘                      └──────────────┘ │    API        │
        │ hand off workloads                            └───────▲───────┘
        ▼                                                       │ Execution API (versioned)
 ┌──────────────────────────────────────────────┐              │
 │                 Executor                       │     ┌────────┴─────────────────────────┐
 │  Local (in-proc) │ Celery │ Kubernetes │ Edge  │     │  Task runtime (Task SDK)          │
 │  │ AWS ECS/Batch/Lambda                        │◄────┤  workers / pods / edge nodes run  │
 └──────────────────────────────────────────────┘     │  task code, call back via API     │
                                                        └───────────────────────────────────┘
        ▲                                                        ▲
        │ reads DAG files from                                   │
 ┌──────┴───────────────┐                                ┌───────┴────────┐
 │   DAG Bundles        │                                │  Providers     │
 │  (versioned sources  │                                │  (87 packages: │
 │   of dags folder)    │                                │  operators,    │
 └──────────────────────┘                                │  hooks, etc.)  │
                                                          └────────────────┘

2.2 Data and control flow

1. The DAG processor reads DAG files from configured bundles, parses them, and writes serialized DAGs to the metadata DB. (Parsing is isolated from scheduling so a bad DAG can't take down the scheduler.)
2. The scheduler reads serialized DAGs, creates DAG runs when schedules fire (or assets update), and creates task instances, scheduling those whose dependencies are met. It uses row-level locking to allow multiple schedulers (HA).
3. The scheduler hands runnable workloads to the executor, which places task execution onto workers/pods/processes.
4. Task code runs against the Task SDK and communicates with Airflow only through the versioned Execution API exposed by the API server — never touching the database directly. This decoupling is what enables remote and edge workers and forward-compatible authoring.
5. Deferrable operators yield control to the triggerer, which polls external conditions asynchronously and resumes the task when ready — freeing worker slots while waiting.
6. The API server serves the REST API and React UI for human and programmatic interaction.

2.3 Why the split matters operationally

• Scheduler HA: Multiple schedulers can run concurrently (scheduler.use_row_level_locking = True). Watch for timetable-specific race conditions (e.g. partitioned/asset timetables, community issue #68045) and patch promptly.
• DAG processor isolation: Parse errors and slow imports affect only parsing throughput, not scheduling decisions already in the DB.
• Execution API boundary: Because tasks never hit the DB directly, you can run workers outside your trusted network (edge), and the Task SDK stays stable across core upgrades.

3. Component Inventory

4. System Requirements

Caution

SQLite + LocalExecutor are single-writer and cannot support concurrent task execution safely. They exist for standalone and local development only.

5. Installation & Deployment

5.1 Installation methods

Important

The dual library/application nature of Airflow makes unconstrained installs unreliable. Always pin with the constraint file for your exact Airflow and Python version: https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.10.txt.

5.2 Reference single-node (non-Kubernetes) deployment

1. Provision PostgreSQL; create a database and user.
2. Install Airflow + needed providers into a virtualenv with constraints.
3. Set configuration (env vars or airflow.cfg):

   export AIRFLOW__DATABASE__SQL_ALCHEMY_CONN='postgresql+psycopg2://airflow:***@db:5432/airflow'
   export AIRFLOW__CORE__EXECUTOR='LocalExecutor'   # or CeleryExecutor for multi-node
   export AIRFLOW__CORE__FERNET_KEY='<generated-fernet-key>'
   export AIRFLOW__API__SECRET_KEY='<random-secret>'
   

4. Initialize the schema: airflow db migrate.
5. Run each service under a process supervisor (systemd, supervisord):
   • airflow scheduler
   • airflow dag-processor
   • airflow triggerer
   • airflow api-server
6. Put a TLS-terminating reverse proxy in front of the API server.

Tip

Generate a Fernet key once and store it in your secret manager:

from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())

Reuse the same key across all services. Rotating it later requires airflow rotate-fernet-key.

5.3 Kubernetes (Helm chart)

The official apache-airflow Helm chart deploys scheduler, API server, DAG processor, triggerer, optional Celery or Kubernetes executors, PgBouncer (connection pooling), and supporting resources. It versions independently of core; pin both. Detailed values are in the chart's own documentation — operationally, the chart is the supported way to run Airflow on Kubernetes.

Note

Community issue #67951 tracks adding template (tpl) support to .Values.volumes, .Values.volumeMounts, and pgbouncer.configSecretName — relevant if you template chart values dynamically.

5.4 Deployment checklist

• [ ] PostgreSQL/MySQL provisioned and reachable from all services
• [ ] sql_alchemy_conn set via env/secret (not committed to airflow.cfg)
• [ ] Fernet key generated and shared across all components
• [ ] api.secret_key set and consistent across API server replicas
• [ ] Executor chosen and its provider installed
• [ ] DAG bundles configured (dag_processor.dag_bundle_config_list)
• [ ] airflow db migrate run before first start and after every upgrade
• [ ] Auth manager selected and admin access provisioned
• [ ] Remote logging configured (so logs survive worker churn)
• [ ] Reverse proxy / TLS in front of the API server
• [ ] Health checks wired to scheduler (8974) and API server (8080)
• [ ] Backups of the metadata DB scheduled

6. Configuration Mastery

Airflow's configuration lives in airflow.cfg (under $AIRFLOW_HOME), but every option can be set via an environment variable named AIRFLOW__<SECTION>__<KEY> (uppercase, double underscores). Environment variables win over the file, which is why containerized deployments configure Airflow almost entirely through env. There is also a _CMD / _SECRET suffix convention to source a value from a shell command or a secrets backend instead of inlining it.

The authoritative option set is defined in airflow-core/src/airflow/config_templates/config.yml, which contains 28 sections and roughly 600 options, each carrying a default and a version_added. The sections:

core, database, logging, metrics, traces, secrets, api, workers, sdk, api_auth, execution_api, lineage, operators, email, smtp, sentry, scheduler, connection_test, triggerer, kerberos, sensors, dag_processor, state_store, profiling, callbacks.

The following subsections document the options operators touch most. Defaults are taken directly from config.yml.

6.1 [core] — fundamentals

Warning

Set load_examples = False and change the simple_auth_manager_users default away from admin:admin before any non-local deployment.

6.2 [database] — metadata store

Tip

With many scheduler/worker processes, the sum of pool_size + max_overflow across all processes can exhaust DB connections. On Kubernetes, front PostgreSQL with PgBouncer (the Helm chart includes it).

6.3 [api] — API server & UI

Important

When running multiple API server replicas, set a fixed api.secret_key (and core.fernet_key) so sessions and encrypted values are consistent across all of them.

6.4 [scheduler] — scheduling behavior

Tip

Enable scheduler.enable_health_check and point your orchestrator's liveness probe at port 8974. Tune max_tis_per_query and max_dagruns_* together when scaling to thousands of tasks.

6.5 [dag_processor] — DAG parsing

6.6 [logging] — task and system logs

Important

In any deployment where workers are ephemeral (Kubernetes, Celery autoscaling, Edge), enable remote logging to durable storage. Otherwise task logs vanish when the worker/pod is recycled.

6.7 [secrets] — secrets backends

A secrets backend lets connections and variables be resolved from an external store instead of the metadata DB. The lookup order is: secrets backend → environment variables → metadata DB.

6.8 Other notable sections

• [metrics] — StatsD/OpenTelemetry metrics emission.
• [traces] — OpenTelemetry distributed tracing.
• [workers] / [sdk] / [execution_api] — Task-runtime and Execution-API tuning (worker API retries, execution-API behavior).
• [api_auth] — API authentication backends/tokens.
• [smtp] / [email] — Email/alert delivery.
• [sentry] — Sentry error reporting.
• [kerberos] — Kerberos integration for secured Hadoop/enterprise environments.
• [triggerer] — Deferrable-task capacity and behavior.
• [state_store] — Backend for task/asset state storage.

Tip

Audit your effective configuration with airflow config list, and catch deprecated options after an upgrade with airflow config lint.

7. Executors

The executor is the single most consequential operational choice. Core ships only LocalExecutor; everything else is a provider you install and select via core.executor.

Note

Airflow supports multiple executors simultaneously — assign tasks to an executor via the task's queue/executor configuration. Start with LocalExecutor (single node) or CeleryExecutor/KubernetesExecutor (scaled), and add specialized executors only where they earn their complexity.

Caution

CeleryExecutor adds a broker (Redis/RabbitMQ) and result backend as additional operational dependencies. KubernetesExecutor trades per-task pod startup latency for isolation and elasticity. Choose deliberately based on task volume, duration, and isolation needs.

8. The Metadata Database

The metadata DB is the source of truth for all Airflow state: DAG runs, task instances, serialized DAGs, XComs, connections, variables, pools, and job heartbeats.

8.1 Schema management

airflow db migrate      # apply pending migrations (run after install & every upgrade)
airflow db check        # verify connectivity and that migrations are current
airflow db check-migrations  # block until migrations are applied (useful in init containers)
airflow db downgrade    # revert to an earlier revision (dangerous)
airflow db reset        # DROP everything and recreate (destructive — dev only)
airflow db shell        # open an interactive DB shell

Migrations are Alembic-based (alembic.ini, database.alembic_ini_file_path).

8.2 Maintenance

Airflow accumulates history. Periodically archive/purge old rows:

# Archive rows older than a cutoff into archive tables, then optionally drop them
airflow db clean --clean-before-timestamp 2026-01-01
airflow db export-archived --output-path ./archive
airflow db drop-archived

Warning

db clean, db drop-archived, db downgrade, and db reset mutate or destroy data. Always back up the database first and test on a copy. Run cleanup during low-traffic windows.

8.3 HA and connections

• Multiple schedulers rely on row-level locking (scheduler.use_row_level_locking = True) — supported on PostgreSQL and MySQL, not SQLite.
• Use connection pooling (PgBouncer) to keep connection counts sane as you scale processes.
• Set sql_alchemy_pool_recycle below your DB/proxy idle-timeout to avoid stale connections.

9. Security

9.1 Authentication & authorization (auth managers)

Airflow 3 makes auth pluggable via core.auth_manager:

Caution

The default SimpleAuthManager with admin:admin is for local use. For production, either configure SimpleAuthManager with a real passwords file (core.simple_auth_manager_passwords_file) and strong credentials, or install the FAB auth manager for RBAC, or integrate an external IdP (Keycloak/OIDC). Note community issue #67476: older builds ignored the next deep-link parameter during SimpleAuthManager auto-login — keep core patched.

9.2 Secrets at rest — the Fernet key

Connection passwords and variable values are encrypted with the Fernet key (core.fernet_key).

• Generate one key, store it in your secret manager, and use it across all components.
• To rotate: add the new key, then run airflow rotate-fernet-key to re-encrypt existing secrets.
• Losing the key means losing the ability to decrypt all stored secrets.

9.3 Secrets backends

Move secrets out of the metadata DB entirely with a [secrets] backend (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager). Connections and variables are then resolved from the external store. This is the recommended posture for regulated environments.

9.4 Network & transport

• Terminate TLS at a reverse proxy in front of the API server (or use api.ssl_cert/ssl_key directly).
• Keep api.expose_config = False and api.expose_stacktrace = False in production.
• Restrict CORS via api.access_control_allow_origins.
• Use [api_auth] tokens for programmatic/REST access; airflowctl authenticates with a token (AIRFLOW_CLI_TOKEN).

9.5 Execution isolation

Because task code reaches Airflow only through the versioned Execution API, workers do not need database credentials. This lets you run workers (especially edge workers) in lower-trust networks while keeping the metadata DB private. Combine with core.default_impersonation / run-as-user controls where supported.

Important

DAG files are executed Python during parsing. Treat the dags folder/bundles as a code-trust boundary: only deploy DAGs from sources you trust, since a malicious DAG file runs with the DAG processor's privileges.

10. Observability & Operations

10.1 Health checks

10.2 Metrics & traces

• [metrics] emits StatsD/OpenTelemetry metrics (scheduler loop timing, pool usage, task duration). Wire to your metrics stack.
• [traces] emits OpenTelemetry spans for distributed tracing across components.
• [sentry] integrates error reporting.

10.3 Logging operations

• Default logs under base_log_folder; the UI fetches task logs over worker_log_server_port (8793) and triggerer logs over 8794.
• Enable remote_logging to S3/GCS/Azure/Elasticsearch (via the relevant provider) for ephemeral workers.

10.4 Scaling playbook

10.5 Upgrades

1. Read the release notes for breaking changes.
2. Back up the metadata DB.
3. Install the new core + providers with the matching constraint file.
4. Run airflow db migrate.
5. Run airflow config lint to catch deprecated options.
6. Roll services (DAG processor, scheduler, triggerer, API server).

11. Provider & Plugin Internals

11.1 Providers

Providers are independently versioned Python packages (apache-airflow-providers-*) discovered at runtime via a provider-discovery mechanism. Each declares, through a provider.yaml/metadata, the operators, hooks, sensors, transfers, connection types, secrets backends, executors, logging handlers, and auth managers it contributes. Inspect what's installed:

airflow providers list
airflow providers hooks
airflow providers secrets
airflow providers executors
airflow providers auth-managers

87 providers ship in this repository, covering AWS (amazon), Google (google), Azure (microsoft.azure), Kubernetes (cncf.kubernetes), the common databases, messaging systems, and many SaaS integrations.

11.2 Plugins

Plugins (in core.plugins_folder) extend Airflow with custom operator links, UI elements (React plugins), macros, and more. List loaded plugins with airflow plugins. The repository includes React plugin tooling (dev/react-plugin-tools) for building UI plugins against the new React UI.

12. Repository & Build Internals (for contributors and replicators)

This section documents the codebase well enough that a developer — or a future AI — could navigate, build, and extend it.

12.1 Monorepo layout

12.2 The two CLIs — don't confuse them

• The airflow CLI (airflow-core/src/airflow/cli/cli_config.py) is the user/operator surface, built on argparse with a GroupCommand/ActionCommand structure. This is what the User Guide documents.
• Breeze (dev/breeze, a large Click application) is the contributor tool for building images, running tests, managing releases, and CI. Its ~500 Click commands dominate naive CLI extraction but are not end-user commands. Operators of a production Airflow do not run Breeze.

Note

Frontend/UI work is React 19 + Vite + Chakra UI + TanStack Query, with an OpenAPI-generated client (openapi-gen) bound to the FastAPI API. The FAB provider still ships a legacy Flask/webpack UI for its auth-manager screens.

12.3 Building & testing

Development uses Breeze (Dockerized): breeze start-airflow boots a full stack with hot reload; breeze testing runs the suite; breeze ci-image/prod-image build images; breeze release-management handles the elaborate ASF release process. Dependency resolution is constraint-driven (generate-constraints), and provider releases are independently versioned and tagged.

13. Improvements & Technical Debt (observed)

Grounded in the source and the project's own issue tracker:

• Multi-scheduler timetable races: Duplicate DAG runs can occur for partitioned/asset timetables under HA (issue #68045). If you run HA with such timetables, validate carefully and track the fix.
• Kubernetes client/proxy regression: Older cncf.kubernetes client versions broke KubernetesPodOperator behind corporate proxies (issue #67845, resolved by supporting client >=36.0.1). Keep providers current.
• UI polish items: DAG code syntax highlighting with f-strings (#67986) and DAG code readability improvements (#67359) are cosmetic UI items.
• SimpleAuthManager deep-link redirect (#67476): auto-login dropped the next parameter in older builds.
• Chart templating gaps (#67951): request to add tpl support to chart volumes/secret names.
• UI dependency consolidation (#61834): redundant UI dependencies to be consolidated.

General operational debt to plan around: the experimental status of the Go/Java edge SDKs and EdgeExecutor; the per-task pod startup latency of KubernetesExecutor; and the broker operational burden of CeleryExecutor.

14. Operational Troubleshooting

Symptom: Scheduler appears alive but nothing schedules

Diagnose: airflow jobs check --job-type SchedulerJob; confirm the DAG processor is running and DAGs are serialized (airflow dags reserialize); check parallelism and pool exhaustion. Resolve: Restart the stalled service; raise parallelism/pool slots; verify DB connectivity (airflow db check).

Symptom: "Too many connections" on the database

Cause: (pool_size + max_overflow) × processes exceeds DB limits. Resolve: Introduce PgBouncer; reduce per-process pool sizes; raise the DB connection limit.

Symptom: Task logs missing in the UI

Cause: Ephemeral workers without remote logging, or the log server port blocked. Resolve: Enable logging.remote_logging to durable storage; ensure ports 8793/8794 are reachable from the API server.

Symptom: Migrations fail or schema mismatch on startup

Cause: Upgraded binaries without running migrations. Resolve: airflow db migrate; use airflow db check-migrations as an init-container gate before starting services.

Symptom: Deferrable sensors never resume

Cause: No triggerer running, or triggerer at capacity. Resolve: Start/scale airflow triggerer; raise --capacity.

Symptom: Secrets fail to decrypt after redeploy

Cause: Fernet key mismatch across components or after rotation without rotate-fernet-key. Resolve: Ensure a single shared core.fernet_key; run airflow rotate-fernet-key only with both keys configured.

15. Quick Reference

Production configuration minimums (env vars)

Service ports

Database lifecycle commands

Config introspection