Data Engineering Zoomcamp — Admin & Architecture Guide

This guide is the technical companion to the Data Engineering Zoomcamp repository. Where the User Guide explains how to take the course, this guide explains how the course's reference implementations are built: the architecture of each module's stack, the exact configuration shipped in the repo, the security posture of the bundled environments, and the operational commands to run and tear them down.

Audience: instructors maintaining the materials, engineers studying the reference stacks, and anyone adapting these pipelines toward production.

Important

Every container stack, credential, and Terraform plan in this repository is a teaching artifact tuned for local development. The defaults prioritize ease of learning over security. This guide flags exactly where those defaults must change before any non-local use. See Section 6 — Security.

1. Executive Summary

The repository is a curriculum, not a deployable product, but it contains a complete, coherent reference data platform assembled from open-source and cloud-managed components. Across seven modules it stands up, in order:

The unifying workload is the NYC TLC taxi trip dataset (yellow/green, CSV), which flows through every module so learners see the same data ingested, warehoused, transformed, batch-processed, and streamed.

Repository facts (commit c59a5ea):

• 406 files, ~134 KB of analyzed source across 44 high-signal folders.
• Polyglot: Python (Click/argparse scripts, PySpark, PyFlink), Java (Kafka examples), SQL (BigQuery, dbt), HCL (Terraform), YAML (Docker Compose, Kestra flows, dbt config).
• Primary languages by content: Jupyter Notebook + Python, with Java and YAML configuration.

2. Repository Architecture

2.1 Top-level layout

data-engineering-zoomcamp/
├── 01-docker-terraform/      # Docker, Postgres, ingestion pipeline, Terraform
│   ├── docker-sql/           #   11-lesson workshop + pipeline/
│   └── terraform/            #   GCP + AWS Terraform variants
├── 02-workflow-orchestration/# Kestra compose stack + 11 example flows
│   └── flows/
├── 03-data-warehouse/        # BigQuery SQL + GCS upload helpers (extras/)
├── 04-analytics-engineering/ # dbt project (taxi_rides_ny/) + class notes
├── 05-data-platforms/        # Bruin notes + tutorial pointers
├── 06-batch/                 # PySpark code + per-OS Spark setup
├── 07-streaming/             # PyFlink workshop, Kafka theory (Java), extras
├── cohorts/                  # Archived 2022–2026 materials + current homework
└── projects/                 # Capstone instructions

The current curriculum lives in the numbered top-level folders. The cohorts/ tree is an append-only archive: each year captures that cohort's homework, deadlines, and tooling (e.g., Airflow/Mage in 2022–2024, since superseded by Kestra). When reading the repo, treat cohorts/ as historical unless you are looking up a specific year's homework.

Note

The single declared "entry point" (07-streaming/workshop/live/main.py) is a placeholder stub (print("Hello from streaming-workshop!")). There is no application entry point for the repository as a whole — each module is run independently via its own README, Compose file, or script. This is expected for a curriculum repo.

2.2 The recurring data flow

Every module is a variation on the same pipeline shape, which is the course's central architectural lesson:

        ┌─────────────┐     ┌──────────────┐     ┌───────────────┐     ┌────────────┐
Source  │  Extract    │ --> │  Load/Stage  │ --> │  Transform    │ --> │  Serve     │
(CSV)   │  (HTTP/wget)│     │ (Postgres/   │     │ (SQL/dbt/     │     │ (BigQuery/ │
        │             │     │  GCS/Kafka)  │     │  Spark/Flink) │     │  reports)  │
        └─────────────┘     └──────────────┘     └───────────────┘     └────────────┘

• Module 1 runs this locally: CSV → Postgres via a Python script.
• Module 2 orchestrates it: Kestra flows do ETL (local Postgres) then ELT (GCS → BigQuery).
• Modules 3–4 focus on the warehouse + transform stages (BigQuery, dbt).
• Module 6 swaps the engine for distributed batch (Spark).
• Module 7 makes it real-time (Kafka → Flink → Postgres).

A defining teaching point is the ETL vs. ELT distinction (Module 2.4): load before transform when the warehouse can do the heavy lifting at scale.

3. Module Reference Stacks

This section documents each module's deployable stack as configured in the repo.

3.1 Module 1 — Docker + Postgres + Terraform

Local data stack (docker-sql/pipeline/docker-compose.yaml):

Ingestion pipeline (pipeline/ingest_data.py): a Click CLI that streams a month of taxi CSV from github.com/DataTalksClub/nyc-tlc-data and chunk-loads it via SQLAlchemy (postgresql+psycopg://…) with a tqdm progress bar. It declares an explicit dtype map (all 16 taxi columns) and parses tpep_pickup_datetime / tpep_dropoff_datetime as dates. Default chunk size is 100,000 rows; the first chunk writes the header with if_exists='replace', subsequent chunks append.

Container image (pipeline/Dockerfile):

FROM python:3.13.11-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/
WORKDIR /code
ENV PATH="/code/.venv/bin:$PATH"
COPY pyproject.toml .python-version uv.lock ./
RUN uv sync --locked
COPY ingest_data.py .
ENTRYPOINT ["python", "ingest_data.py"]

Note

Dependency management uses uv with a committed uv.lock and uv sync --locked for reproducible builds. The slim Python 3.13 base plus copy-lock-first layering keeps the image small and cache-friendly. (The module's 08 lesson shows an ENTRYPOINT variant using uv run python; the committed Dockerfile invokes python directly since uv sync populates the venv on PATH.)

Terraform (terraform/terraform/terraform_with_variables/):

• Provider: hashicorp/google (pinned 5.6.0; the basic example pins 4.51.0). Authenticates via a service-account JSON file (var.credentials) or GOOGLE_APPLICATION_CREDENTIALS.
• Resources: google_storage_bucket (force-destroy, 1-day abort-incomplete- upload lifecycle) and google_bigquery_dataset.
• Variables: project, region (default us-central1), location (default US), bq_dataset_name (demo_dataset), gcs_bucket_name, gcs_storage_class (STANDARD).

An AWS equivalent (terraform_with_variable_AWS/) mirrors the GCP design for learners on AWS:

It also enables S3 versioning and defaults to region eu-north-1.

3.2 Module 2 — Kestra orchestration

Stack (02-workflow-orchestration/docker-compose.yml):

Kestra's KESTRA_CONFIGURATION (embedded YAML) wires a Postgres-backed repository and queue, local storage at /app/storage, basic auth ([email removed] / Admin1234!), and a tmp working dir under /tmp/kestra-wd. It bind-mounts /var/run/docker.sock so task runners can spawn containers.

Warning

The Kestra service runs as root and mounts the host Docker socket. The compose comment notes this is for development only; production should use a rootless setup (e.g. Podman Compose). A mounted Docker socket is effectively host-root access.

Flow catalog (flows/) — eleven YAML flows demonstrating the full feature surface:

Module 2.5 adds an AI layer: Kestra's AI Copilot configured for Google Gemini (gemini-2.5-flash, api-key: ${GEMINI_API_KEY}) plus a RAG demo. This is the architectural lesson that context engineering (Copilot/RAG) beats an un-grounded LLM for generating correct, current flow code.

3.3 Module 3 — BigQuery warehouse

No long-running services; the module is SQL + GCS loaders. Helper scripts in extras/ (web_to_gcs.py, web_to_gcs_with_progress_bar.py) download taxi CSVs and upload to GCS using the Google Cloud Storage client. The destination bucket is read from the GCP_GCS_BUCKET environment variable (placeholder default dtc-data-lake-bucketname). SQL files (big_query.sql, big_query_ml.sql) hold the partitioning/clustering and BigQuery ML demonstrations.

Architecturally the module teaches: external tables over GCS, partitioning (prune by date) vs clustering (co-locate by column), cost control, and in-warehouse ML via CREATE MODEL.

3.4 Module 4 — dbt transformation

Project (taxi_rides_ny/dbt_project.yml):

• name: taxi_rides_ny, profile: taxi_rides_ny, require-dbt-version: [">=1.7.0", "<3.0.0"].
• Layered materializations: staging → view, intermediate → table, marts → table.
• Dev sampling vars: dev_start_date = 2019-01-01, dev_end_date = 2019-02-01.
• flags: require_generic_test_arguments_property: true (modern dbt test syntax).

Two execution targets share one project: local (DuckDB + dbt Core, zero cost) and cloud (BigQuery + dbt Cloud). The model tree under models/ (staging/, intermediate/, marts/, marts/reporting/) plus seeds/, macros/, snapshots/, and tests/ is a canonical dbt project structure.

Note

Profiles are environment-specific and not committed in a usable form — learners create ~/.dbt/profiles.yml (or use dbt Cloud's connection). The setup guides (setup/local_setup.md, setup/cloud_setup.md) provide the exact profile blocks.

3.5 Module 5 — Bruin data platform

Bruin is introduced via its hosted template (bruin init zoomcamp my-taxi-pipeline) rather than committed code; the repo carries reference notes (notes/) for completed implementations. Architecture concepts: a project (.bruin.yml with environments and connections) contains pipelines (one schedule each) made of assets (SQL, Python, or YAML files that each produce a table/view). A three-layer pipeline (ingestion → staging → reports) over DuckDB mirrors the dbt project's shape but within a single integrated tool that also covers ingestion, quality, lineage, and orchestration. The Bruin MCP server exposes the platform to AI agents.

3.6 Module 6 — Spark batch

No bundled cluster; learners install Spark locally (per-OS guides in setup/) or use Dataproc. The reference job code/06_spark_sql.py is an argparse CLI (--input_green, --input_yellow, --output) that reads Parquet, renames the green/yellow timestamp columns to a common schema, registers temp views, and runs a revenue aggregation. 06_spark_sql_big_query.py is the same job writing to BigQuery via the Spark-BigQuery connector. code/download_data.sh bulk-downloads a year of a taxi type into data/raw/<type>/<year>/<month>/.

Architecture coverage: cluster anatomy (driver/executors), GroupBy and Join internals (shuffles), RDDs, a local standalone cluster, Dataproc submission, and GCS/BigQuery connectivity.

3.7 Module 7 — Streaming (Redpanda + Flink)

Workshop stack (workshop/docker-compose.yml): a redpanda broker (redpandadata/redpanda:v25.3.9) plus Flink (JobManager/ TaskManager via Dockerfile.flink, with an ARM64 variant) and Postgres as the sink. Target topology:

Producer (Python, kafka-python) → Redpanda (Kafka) → Flink (PyFlink) → PostgreSQL

Redpanda exposes dual listeners — internal redpanda:29092 (other containers, e.g. Flink) and external localhost:9092 (host producer/consumer) — because Kafka clients bootstrap then reconnect to advertised addresses. Resource flags (--smp 1, --reserve-memory 0M, --overprovisioned) tune it for a laptop.

Two reference tracks accompany the workshop:

• theory/java/kafka_examples/ — a Gradle Java project (kafka-clients 3.3.1, kafka-streams, Confluent Avro serializers 7.3.1, avro 1.11.0, JUnit 5) demonstrating producers, consumers, Streams topologies, custom SerDes, and Avro schema management with Schema Registry.
• extras/ — prior-year Python/PyFlink examples (kafka-python, confluent-kafka, faust, pyspark) and a PyFlink job (apache-flink==1.16.0).

4. System Requirements

5. Configuration Mastery

5.1 Credentials and connection settings (as shipped)

5.2 Terraform variables

5.3 Version pins worth noting

Tip

The README explicitly warns against using kestra/kestra:develop — it is a bleeding-edge build that may break flows. Pinning is a deliberate reproducibility practice the course models throughout.

6. Security Considerations

This repository's stacks are intentionally insecure for learning convenience. The following are the concrete issues an admin must address before any exposure beyond localhost.

6.1 Hard-coded development credentials

Every bundled service ships with trivial, public credentials (root/root, [email removed]/root, [email removed]/Admin1234!, kestra/k3str4). These are fine on an isolated laptop and must never be reused on any network-reachable host.

Caution

Do not deploy any of these docker-compose files to a public VM as-is. Bound services (Postgres 5432, pgAdmin 8085, Kestra 8080, Redpanda 9092) would be exploitable immediately with the published passwords.

6.2 Docker socket exposure (Kestra)

The Kestra service runs as root and mounts /var/run/docker.sock. This grants the container full control of the host's Docker daemon — equivalent to host root. The compose file documents this and points to a rootless production setup.

6.3 Service-account key handling

Multiple modules use a GCP service-account JSON key. The materials repeatedly warn:

• Never commit the key to Git (use a proper .gitignore; Module 1 references the GitHub Terraform .gitignore).
• In Kestra, store credentials in the KV Store or as Secrets, not inline in flow YAML.
• For Terraform, a documented fallback uses short-lived impersonation tokens (google_service_account_access_token, 3600s lifetime) instead of a downloaded key — the more secure pattern when org policy forbids key creation.

6.4 Secrets in AI workflows

Module 2.5 uses a Gemini API key via ${GEMINI_API_KEY} (environment variable), and the README warns against committing API keys — use env vars or the KV Store.

6.5 Cloud cost as a security/ops concern

force_destroy = true on buckets and terraform destroy discipline are emphasized to prevent orphaned, billable cloud resources. Backfilling the full taxi dataset is explicitly deferred to the cloud (Module 2.4) precisely because it would exhaust local resources.

7. Operations

7.1 Running each stack

# Module 1 — local Postgres + pgAdmin
cd 01-docker-terraform/docker-sql/pipeline && docker compose up -d

# Module 2 — Kestra + Postgres
cd 02-workflow-orchestration && docker compose up -d   # UI: http://localhost:8080

# Module 7 — Redpanda broker (then Flink services)
cd 07-streaming/workshop && docker compose up redpanda -d

7.2 Importing Kestra flows programmatically

for f in flows/*.yaml; do
  curl -X POST -u '[email removed]:Admin1234!' \
    http://localhost:8080/api/v1/flows/import -F "fileUpload=@$f"
done

7.3 Cloud provisioning lifecycle

gcloud auth application-default login
terraform init
terraform plan  -var="project=$GCP_PROJECT_ID"
terraform apply -var="project=$GCP_PROJECT_ID"
# ... do the module work ...
terraform destroy

7.4 Teardown checklist

• [ ] docker compose down for every started stack (add --volumes to wipe data).
• [ ] terraform destroy for any provisioned cloud infra.
• [ ] Delete GCS buckets / BigQuery datasets created outside Terraform (e.g. via Kestra 07_gcp_setup).
• [ ] Remove pg-network if created manually: docker network rm pg-network.
• [ ] Revoke or delete any temporary service-account keys.

7.5 Health and verification

• Kestra metadata Postgres has a pg_isready healthcheck with depends_on conditions sequencing startup.
• Verify Redpanda with docker compose ps (expect STATUS: Up).
• For pgAdmin↔Postgres, connectivity depends on both being on the same Docker network and using the container name as host.

8. Code Patterns & Quality

The repository models several good engineering practices that are themselves teaching content:

• Locked, reproducible builds — uv sync --locked with committed uv.lock; copy-lockfiles-first Docker layering.
• Explicit schemas — ingest_data.py declares an exact dtype map and date columns rather than relying on inference, preventing silent type drift.
• Chunked, idempotent loading — header-first replace then append, so re-runs are safe.
• Pinned dependencies everywhere — images, providers, plugins, and the Kestra version are all pinned for reproducibility.
• Layered transformation — dbt staging(view)→intermediate(table)→ marts(table) and Bruin ingestion→staging→reports both teach the medallion pattern.
• Dev sampling — dbt date-window vars and partial backfills keep local iteration cheap.
• Dual-cloud parity — the GCP and AWS Terraform variants document a service-by-service mapping, reinforcing cloud-agnostic concepts.

Known data/code issues (from GitHub)

The project's issue tracker surfaces real, learner-reported items an admin or contributor should be aware of:

Note

Most open issues are support questions and homework clarifications rather than code defects — consistent with a curriculum repo whose "users" are learners.

9. Potential Improvements

For maintainers and contributors adapting these materials:

• Consolidate duplicated compose definitions. Postgres/pgAdmin service blocks are repeated across Modules 1, 2, and cohort archives with slight drift (named volume vs. bind mount, version: key present/absent). A shared base or documented canonical block would reduce divergence.
• Externalize credentials even for local dev. Shipping a .env.example with the root/root/Admin1234! values referenced via ${VAR} would let learners change them in one place and build the habit early.
• Fix the streaming entry-point stub. 07-streaming/workshop/live/main.py is a placeholder; the live workshop builds the real producer/consumer elsewhere. Either wire it up or rename it to avoid confusion.
• Address the duplicate delivery_report() (#737) in the Avro example.
• Normalize README video-embed markup (#591) so thumbnails render reliably.
• Default Terraform to the impersonation-token pattern rather than a downloaded key, making the more secure path the default rather than the fallback.

10. Quick Reference

Ports

Key files

Commands

docker compose up -d / down [--volumes]   # per-module stacks
terraform init / plan / apply / destroy   # cloud infra
dbt deps / seed / run / test / build      # transformations
uv sync --locked ; uv run python <s>.py   # Python envs
./06-batch/code/download_data.sh <type> <year>   # bulk data download