Apache Airflow User Guide

Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. You define your pipelines as Python code; Airflow's scheduler then runs your tasks on a pool of workers while honoring the dependencies you declared. A rich web UI lets you visualize pipelines in production, watch progress, inspect logs, and retry or backfill work when something goes wrong.

This guide is written for the people who use Airflow day to day: data engineers and analysts who author DAGs, and operators who trigger, monitor, and troubleshoot runs from the command line or the web interface. It covers installation, core concepts, the complete airflow command-line interface, the everyday tasks you perform against a running environment, and a deep troubleshooting and FAQ reference.

Note

Architecture deep dives, the full airflow.cfg configuration reference, deployment topologies, executor selection, database tuning, and security hardening live in the companion Admin Guide. This guide links there whenever a topic crosses into operations.

1. What Airflow Is (and Is Not)

Airflow orchestrates batch workflows. You express a pipeline as a DAG (Directed Acyclic Graph) — a collection of tasks with directional dependencies — written in Python. The scheduler reads your DAG files, decides which task instances are ready to run, and hands them to an executor, which runs them on workers. Everything that happens is recorded in a metadata database and surfaced in the UI and REST API.

Airflow's design philosophy, taken directly from the project:

• Dynamic — Pipelines are defined in Python, so DAGs can be generated and parameterized in code.
• Extensible — A large library of built-in operators ships in provider packages, and you can write your own.
• Flexible — Airflow leans on the Jinja templating engine for rich, runtime-parameterized commands.

Important

Airflow works best with workflows that are mostly static and slowly changing — where the DAG structure is similar from one run to the next. Tasks should be idempotent (re-running produces the same result without duplicating data) and should not pass large volumes of data between each other. For passing small bits of metadata between tasks, use Airflow's XCom feature; for high-volume data, delegate to an external system (a warehouse, Spark, etc.) and let Airflow orchestrate it.

Caution

Airflow is not a streaming solution. It is frequently used to process near-real-time data by pulling from streams in batches, but it is not an event-by-event stream processor.

Versioning at a glance

Airflow follows strict SemVer. Crucially, core Airflow and each provider version independently: an amazon or google provider release has its own MAJOR/MINOR cycle and can be installed against multiple Airflow versions. The Helm chart and the language API clients also version independently. This guide documents Airflow core 3.3.0.

2. The Airflow Family at a Glance

Airflow is developed as a single repository that ships several separately installed, separately consumed products. As a user you will almost always start with the core platform; the rest are extensions you add when you need them.

Tip

Most readers only need three of these: core (run the platform), providers (connect to your systems), and the Task SDK (author your DAGs). Start there. The remainder of this guide is organized that way.

3. Quick Start

The fastest way to see Airflow run end to end is the all-in-one standalone command. It initializes a local SQLite metadata database, creates an admin user, and starts the scheduler, the API server (with the UI), the DAG processor, and the triggerer together in one process group.

pip (virtualenv)

# 1. Create and activate a virtual environment (Python 3.10-3.14)
python -m venv airflow-venv
source airflow-venv/bin/activate

# 2. Install Airflow with a known-good constraint set
pip install 'apache-airflow==3.3.0' \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.10.txt"

# 3. Choose where Airflow keeps its home directory (dags, logs, config, db)
export AIRFLOW_HOME=~/airflow

# 4. Run everything at once
airflow standalone

Docker image

# Pull the official image and run the all-in-one quick start
docker run -it --rm -p 8080:8080 \
  apache/airflow:3.3.0 standalone

When standalone finishes booting it prints the generated admin credentials:

standalone | Airflow is ready
standalone | Login with username: admin  password: <generated-password>

Open http://localhost:8080 in your browser, log in with those credentials, and you will see the Dags view populated with the bundled example DAGs.

Warning

airflow standalone is for local exploration and development only. It uses SQLite and a single-process LocalExecutor, neither of which is safe for production. For real deployments, see the Admin Guide's deployment chapter.

Tip

The admin password is written to $AIRFLOW_HOME/simple_auth_manager_passwords.json.generated. If you lose the printed value, read it from there.

4. Installation

4.1 Requirements

Airflow 3.3.0 is tested against the following. POSIX-compliant operating systems only — on Windows, run Airflow inside WSL2 or a Linux container.

Caution

SQLite and the default LocalExecutor are fine for trying Airflow but must not be used in production. MariaDB is explicitly not tested or recommended. Use PostgreSQL or MySQL for any shared or production environment.

4.2 Installing from PyPI with constraints

Airflow is simultaneously a library (you import it in your DAGs) and an application (you run its services). Because of that dual nature, its dependencies are kept deliberately open in pyproject.toml, which means a bare pip install apache-airflow can occasionally resolve to an unusable combination. The project therefore publishes constraint files — pinned, known-to-work dependency sets — per Airflow version and per Python minor version.

Important

Always install Airflow with a --constraint URL. The URL encodes both the Airflow version and your Python version. Only pip is officially supported; Poetry and pip-tools are not.

# Plain core install, Python 3.10
pip install 'apache-airflow==3.3.0' \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.10.txt"

Successfully installed apache-airflow-3.3.0 ...

4.3 Installing with extras (providers)

"Extras" pull in the provider packages and their dependencies in one step. Specify them in brackets:

# Core + PostgreSQL + Google providers
pip install 'apache-airflow[postgres,google]==3.3.0' \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.10.txt"

Common extras include postgres, mysql, celery, cncf.kubernetes, amazon, google, microsoft.azure, slack, http, and ssh. You can also install any provider directly:

pip install apache-airflow-providers-amazon

Note

Providers version independently from core. Installing a provider does not change your Airflow version, and you can upgrade a provider without upgrading Airflow.

4.4 First-run setup (manual)

If you are not using standalone, initialize the environment explicitly:

export AIRFLOW_HOME=~/airflow

# Create/upgrade the metadata database schema
airflow db migrate

DB: sqlite:////home/you/airflow/airflow.db
Performing upgrade to the metadata database ...
Database migrating done!

Then start the services you need in separate terminals or as managed processes (see §6, The Components You Run).

4.5 Verifying the install

airflow version

3.3.0

airflow info

airflow info prints a full environment report — Airflow version, Python version, executor, configured paths, installed providers, and system details. Run it first whenever you file a bug report.

5. Core Concepts You Need Before Authoring

You can use most of Airflow knowing just a handful of terms. (Architecture-level internals are in the Admin Guide.)

6. The Components You Run

A production Airflow is several long-running services. As a user you mostly interact with the CLI and UI, but it helps to know what each process does because you start them with the airflow CLI.

Note

In Airflow 3 the old webserver command is gone — the UI is served by api-server. DAG parsing has also been split into its own dag-processor service. If you are migrating from Airflow 2, update your service definitions accordingly.

Minimal manual startup for development:

airflow scheduler &
airflow dag-processor &
airflow triggerer &
airflow api-server --port 8080

7. Authoring Your First DAG

DAGs are Python files placed in your dags folder (default $AIRFLOW_HOME/dags). Author them against the Task SDK (airflow.sdk), the stable public authoring interface.

7.1 A complete example

from __future__ import annotations

import pendulum

from airflow.sdk import DAG, task
from airflow.providers.standard.operators.bash import BashOperator


with DAG(
    dag_id="example_etl",
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    tags=["example", "etl"],
) as dag:

    extract = BashOperator(
        task_id="extract",
        bash_command="echo 'pulling data' && sleep 1",
    )

    @task
    def transform() -> dict:
        return {"rows": 42}

    @task
    def load(payload: dict) -> None:
        print(f"loaded {payload['rows']} rows")

    # Dependency wiring: extract -> transform -> load
    extract >> load(transform())

# Save as $AIRFLOW_HOME/dags/example_etl.py
# Within ~30s the DAG processor parses it and it appears in the UI.

7.2 What the pieces mean

• dag_id — Unique name for the DAG. It is how you reference the DAG everywhere in the CLI and UI.
• schedule — When the DAG runs. Accepts cron strings ("0 6 * * *"), shorthand presets ("@daily", "@hourly"), timedelta objects, or an asset expression for data-aware scheduling.
• start_date — The first logical date the scheduler considers.
• catchup — If False (recommended for most DAGs), Airflow does not run all missed intervals between start_date and now when you first enable the DAG.
• tags — Labels for filtering DAGs in the UI.
• >> — The dependency operator. a >> b means "run a, then b".
• @task — The TaskFlow decorator: turns a plain Python function into a task, and its return value becomes an XCom automatically passed to downstream tasks.

Tip

Prefer importing from airflow.sdk (the Task SDK) rather than deep internal modules. The SDK is the supported, forward-compatible surface; internal imports can break between minor releases.

7.3 Validating a DAG before you ship it

# Parse the file and report import errors without scheduling anything
airflow dags list-import-errors

# Run the whole DAG once locally, in-process, for a date — great for development
airflow dags test example_etl 2026-01-01

[2026-01-01T00:00:00+0000] INFO - Marking run <DagRun example_etl @ 2026-01-01> successful

airflow dags test executes the DAG end to end without needing a running scheduler — the single most useful authoring-loop command.

8. The airflow Command-Line Interface

The airflow CLI is your primary control surface. It is organized into command groups (each managing one kind of object) plus a handful of top-level action commands (mostly to start services). This section documents the full user-facing command tree of Airflow 3.3.0.

Note

Every command accepts -h/--help. Many object commands accept --output (table, json, yaml, plain) so you can pipe results into scripts. Service commands accept --pid, --daemon, --stdout, --stderr, and --log-file for process management.

8.1 Top-level map

8.2 airflow dags — manage DAGs

Common examples:

# List DAGs as JSON
airflow dags list --output json

# Trigger a run now, passing a JSON configuration to the run
airflow dags trigger example_etl --conf '{"region": "us-east-1"}'

# Pause / resume
airflow dags pause example_etl
airflow dags unpause example_etl

# Inspect recent runs that failed
airflow dags list-runs --dag-id example_etl --state failed

# When was this DAG going to run next?
airflow dags next-execution example_etl

# airflow dags trigger output
Created <DagRun example_etl @ 2026-06-04T12:00:00+00:00: manual__2026-06-04T12:00:00+00:00, state:queued>

Tip

airflow dags trigger --conf '{...}' passes a run-time configuration dictionary that your tasks read via the runtime context ({{ dag_run.conf }} in templates). This is the standard way to parameterize manual runs.

8.3 airflow tasks — manage and test tasks

# List tasks in a DAG
airflow tasks list example_etl

# Dry-run a single task for a date (prints logs to your terminal)
airflow tasks test example_etl transform 2026-01-01

# Why is this task stuck?
airflow tasks failed-deps example_etl load 2026-01-01

# See the exact templated command/SQL that would execute
airflow tasks render example_etl extract 2026-01-01

Important

airflow tasks test runs the task immediately and in-process, ignoring dependencies, pools, and the scheduler. It is a development tool — it does not record a normal task instance state the way a scheduled run does.

8.4 airflow backfill — run historical intervals

Backfills run a DAG across a range of past dates. In Airflow 3, backfill is a first-class object you create and manage rather than a flag on dags.

airflow backfill create \
  --dag-id example_etl \
  --from-date 2026-01-01 \
  --to-date 2026-01-31

Note

Use backfills deliberately. They can create many DAG runs at once; combine with pools (§8.6) to cap how much they hit downstream systems.

8.5 airflow connections — manage external credentials

Connections store the endpoints and credentials your operators use, keyed by conn_id.

# Add a Postgres connection via a connection URI
airflow connections add my_pg \
  --conn-uri 'postgresql://user:[email removed]:5432/analytics'

# List and test
airflow connections list --output table
airflow connections test my_pg

# Back up all connections to a file
airflow connections export connections.json

# airflow connections test my_pg
Connection success!

Caution

Connection secrets are encrypted at rest using the Fernet key. Treat exported files (which may contain plaintext credentials) as sensitive and delete them after use. See the Admin Guide's security chapter for secrets backends (Vault, AWS Secrets Manager, etc.).

8.6 airflow pools — limit concurrency

Pools cap how many task instances can run against a shared resource at once.

# Create a pool of 5 slots for a rate-limited API
airflow pools set api_pool 5 "Limit concurrent calls to the vendor API"

# List and inspect
airflow pools list --output table
airflow pools get api_pool

# Import/export pools as JSON
airflow pools export pools.json
airflow pools import pools.json

In a task, assign it to the pool with pool="api_pool".

8.7 airflow variables — runtime values

# Set a JSON variable
airflow variables set feature_flags '{"new_pipeline": true}'

# Get it back (optionally a nested key)
airflow variables get feature_flags

# Bulk import/export
airflow variables export variables.json
airflow variables import variables.json

In a DAG, read it with Variable.get("feature_flags", deserialize_json=True) from the Task SDK.

Tip

Reading a Variable at the top level of a DAG file runs on every parse and hammers the database. Read Variables inside tasks (or use Jinja templating {{ var.json.feature_flags }}) instead of at module scope.

8.8 airflow assets — data-aware scheduling

Assets let a DAG be scheduled to run when another DAG updates a dataset, rather than on a clock. Schedule a consumer DAG with schedule=[my_asset].

8.9 airflow config — view configuration

# Print the full effective configuration
airflow config list

# Read a single value
airflow config get-value core executor

# Lint config for deprecated/removed options
airflow config lint

# airflow config get-value core executor
LocalExecutor

Note

By default the API server does not expose configuration in the UI (expose_config = False) for security. The CLI always reads the local config. The full option reference is in the Admin Guide.

8.10 airflow providers — inspect installed providers

The providers group introspects what each installed provider contributes. Useful subcommands:

airflow providers list --output table
airflow providers hooks

8.11 airflow db — database operations

Mostly an operator concern, but two are commonly run by users on dev environments:

# Create/upgrade the schema (run after install and after upgrades)
airflow db migrate

# Verify connectivity and schema health
airflow db check

# Open an interactive DB shell
airflow db shell

# Purge old metadata (use with care — see Admin Guide)
airflow db clean --clean-before-timestamp 2026-01-01

Warning

airflow db clean, db reset, and db downgrade are destructive. db reset drops all metadata. Read the Admin Guide before running any of them against a shared environment.

8.12 Other top-level commands

airflow version          # Print version (3.3.0)
airflow info             # Full environment report (attach to bug reports)
airflow cheat-sheet      # Print a quick command cheat sheet
airflow plugins          # List loaded plugins
airflow rotate-fernet-key  # Re-encrypt secrets after rotating the Fernet key

9. The Web UI

The web interface is served by airflow api-server (default http://localhost:8080). It is the fastest way to observe and intervene in running pipelines.

Typical UI workflow when a task fails:

1. Open Grid, find the red (failed) task instance.
2. Click it and open Logs to read the traceback.
3. Fix the cause (data, connection, code).
4. Clear the task instance (or the downstream subtree) to re-run it — equivalent to airflow tasks clear.

Tip

Clearing a task in the Grid view re-runs it and, by default, its downstream tasks. This is the most common recovery action and mirrors the airflow tasks clear CLI command.

10. Using Providers

Providers are how Airflow talks to the outside world. Each provider ships operators, hooks, sensors, and sometimes transfers, secrets backends, or executors. Airflow 3.3.0 has 87 providers spanning cloud platforms, databases, messaging, and more.

Pattern for using a provider operator:

from airflow.sdk import DAG
from airflow.providers.amazon.aws.operators.s3 import S3CreateBucketOperator

with DAG(dag_id="s3_example", schedule=None, catchup=False) as dag:
    create_bucket = S3CreateBucketOperator(
        task_id="create_bucket",
        bucket_name="my-airflow-bucket",
        aws_conn_id="aws_default",   # references a stored Connection
    )

The *_conn_id argument points at a Connection you created with airflow connections add. The operator's hook reads that connection to authenticate.

Note

Discover what an installed provider offers with airflow providers hooks, airflow providers list, and the provider's own documentation. Operators are imported from airflow.providers.<provider>.....

A small sample of widely used providers:

11. Driving a Remote Airflow with airflowctl

airflowctl (the apache-airflow-ctl package) is a standalone command-line client that operates a remote Airflow instance over its REST API. Unlike the airflow CLI — which must run on a host with the metadata database — airflowctl only needs network access to an API server and an auth token.

pip install apache-airflow-ctl

# Point it at your environment and authenticate
export AIRFLOW_CLI_ENVIRONMENT=prod
airflowctl auth login --api-url https://airflow.example.com --token "$AIRFLOW_CLI_TOKEN"

# Now drive the remote instance
airflowctl dags list
airflowctl dags trigger example_etl

Tip

Use airflowctl from CI pipelines and operator laptops to manage managed/remote Airflow deployments where you have no shell access to the scheduler host.

12. Common Recipes

12.1 Re-run a failed task and everything after it

airflow tasks clear example_etl \
  --task-regex 'transform' \
  --downstream \
  --start-date 2026-06-01 --end-date 2026-06-04 --yes

12.2 Trigger a parameterized one-off run

airflow dags trigger example_etl --conf '{"full_refresh": true}'

12.3 Promote connections/variables between environments

# On source
airflow connections export conns.json
airflow variables export vars.json

# On target
airflow connections import conns.json
airflow variables import vars.json

12.4 Back up before an upgrade

airflow db export-archived --output-path ./archive   # archive old rows first if desired
airflow connections export conns.json
airflow variables export vars.json
airflow pools export pools.json

12.5 Throttle a busy DAG

Assign its expensive tasks to a small pool, then cap the DAG itself with max_active_runs and max_active_tasks in the DAG definition.

13. Troubleshooting

Format: the exact symptom, the cause, the fix, and how to prevent recurrence. These are grounded in real, community-reported issues and Airflow's own error surface.

Error: DAG does not appear in the UI

Cause: The DAG processor hasn't parsed it, or the file has an import error. Solution:

airflow dags list-import-errors
airflow dags reserialize

If the file is valid, confirm it is inside the configured dags folder and that the dag-processor service is running. Prevention: Run airflow dags test <dag_id> <date> before deploying; keep DAG files importable (no heavy top-level work).

Error: airflow command error: the following arguments are required: ...

Cause: A required positional/option argument was omitted. Solution: Run the command with -h to see required args, e.g. airflow dags trigger -h. Prevention: Use --output json and scripted wrappers for repeated commands.

Error: Task stuck in queued or scheduled and never runs

Cause: No worker capacity, an exhausted pool, or a stopped scheduler. Solution:

airflow jobs check --job-type SchedulerJob   # is a scheduler alive?
airflow tasks failed-deps <dag_id> <task_id> <date>
airflow pools list                            # are slots available?

Prevention: Monitor scheduler health (scheduler.enable_health_check, see Admin Guide) and size pools to real capacity.

Error: DAG code syntax highlighting is wrong with f-strings (UI display)

Cause: Known UI rendering quirk in the Code view for some f-string patterns (community issue #67986). It is a display artifact only — execution is unaffected. Solution: None required; the code runs correctly. Verify with airflow tasks render. Prevention: N/A — cosmetic.

Error: KubernetesPodOperator fails behind a corporate proxy

Cause: A no_proxy regression in older Kubernetes client versions used by the cncf.kubernetes provider (community issue #67845). Solution: Upgrade the cncf.kubernetes provider to a build that supports Kubernetes client >=36.0.1. Prevention: Keep providers current; pin provider versions via constraints.

Error: Deep links redirect to the homepage after login (SimpleAuthManager)

Cause: The next parameter was ignored during auto-login on first visit (community issue #67476). Solution: Upgrade to a release where the fix is applied; re-navigate to the deep link after authenticating. Prevention: Keep core current.

Error: Duplicate DAG runs under multiple schedulers (HA)

Cause: A race condition for certain partitioned/asset timetables when running multiple schedulers in high-availability (community issue #68045). Solution: If you observe duplicates, reduce to a single scheduler for the affected timetable or apply the latest patch release. Prevention: Track the fix in your upgrade cadence; test HA timetables before relying on them.

Error: sqlite / "database is locked" or poor concurrency

Cause: Running SQLite (the default) with concurrent tasks. Solution: Switch database.sql_alchemy_conn to PostgreSQL or MySQL (Admin Guide). Prevention: Never use SQLite beyond local experimentation.

Error: pip install apache-airflow produces a broken environment

Cause: Installed without constraints, so dependencies resolved to an incompatible set. Solution: Reinstall with the matching --constraint URL for your Airflow and Python version. Prevention: Always pass --constraint (see §4.2).

14. Frequently Asked Questions

Q: Which Python versions are supported? Python 3.10 through 3.14 for Airflow 3.3.0.

Q: Can I run Airflow on Windows? Not natively. Use WSL2 or a Linux container. Only Linux is a supported production environment.

Q: What's the difference between airflow standalone and running the services separately? standalone runs the scheduler, API server, DAG processor, and triggerer together with SQLite and a generated admin user — for local use only. Production runs each service separately against PostgreSQL/MySQL.

Q: Where did airflow webserver go? In Airflow 3 the UI and REST API are served by airflow api-server. There is no webserver command.

Q: Where did airflow users and airflow roles go? User/role management is no longer in core. It is provided by the auth manager you configure. The default SimpleAuthManager uses a simple username/password scheme; the FAB provider supplies full RBAC. See the Admin Guide.

Q: How do I pass data between tasks? Small metadata via XCom (TaskFlow return values do this automatically). Large data should be written to external storage and referenced, not pushed through XCom.

Q: How do I parameterize a manual run? airflow dags trigger <dag_id> --conf '{"key": "value"}', then read {{ dag_run.conf }} (or context["dag_run"].conf) in your tasks.

Q: What is catchup and should I enable it? catchup=True makes Airflow run every missed interval back to start_date when a DAG is first enabled. Most users set catchup=False and use explicit backfills when they need historical runs.

Q: How do I install operators for AWS / Google / Kubernetes? Install the relevant provider, e.g. pip install apache-airflow-providers-amazon, or use an extra like apache-airflow[amazon].

Q: What is a "bundle"? A DAG bundle is a versioned source of DAG files that the DAG processor reads. It lets Airflow track which version of your DAG code produced a given run.

Q: How do I test a single task quickly? airflow tasks test <dag_id> <task_id> <date> runs it in-process and prints logs, without touching the scheduler.

Q: How do I find out what's installed and where? airflow info prints the full environment report (version, Python, executor, paths, providers).

Q: Why shouldn't I read a Variable at the top of my DAG file? The DAG file is parsed frequently; top-level reads hit the database on every parse. Read Variables inside tasks or via Jinja templating.

Q: How do I drive a remote/managed Airflow without shell access? Use airflowctl (the apache-airflow-ctl package) against the REST API with a token.

15. Quick Reference

Service commands

Most-used object commands

Key environment variables

Tip

Any airflow.cfg setting can be overridden with an environment variable named AIRFLOW__SECTION__KEY (double underscores, uppercased). This is the most common way to configure Airflow in containers. The full option list is in the Admin Guide.