# Data Engineering Zoomcamp — User Guide The **Data Engineering Zoomcamp** is a free, 9-week, hands-on course from [DataTalks.Club](https://datatalks.club/) that teaches you to build an end-to-end data pipeline from scratch using industry-standard tools. This repository (`DataTalksClub/data-engineering-zoomcamp`) holds every lesson, code sample, configuration file, and homework for the course. This guide shows you how to **use the repository**: how to enroll or study at your own pace, how to set up your environment, and how to work through each module's hands-on materials. It is written for learners — whether you joined a live cohort or found the repo on your own and want to teach yourself. > [!NOTE] > This is a **learning repository**, not an installable software package. > "Using" it means cloning it, following the module instructions, and running > the example pipelines on your own machine and cloud account. There is no > `pip install`; the deliverable is your own skill and a portfolio project. --- ## 1. Quick Start If you want to start learning in the next ten minutes, here is the shortest path. ```bash # 1. Clone the course repository git clone https://github.com/DataTalksClub/data-engineering-zoomcamp.git cd data-engineering-zoomcamp # 2. Open Module 1 and start the Docker + Postgres workshop cd 01-docker-terraform/docker-sql ``` Then open [`01-docker-terraform/docker-sql/README.md`](01-docker-terraform/docker-sql/README.md) and follow the eleven numbered lessons in order. Each module folder is self-contained and pairs a `README.md` with short video lectures. > [!TIP] > The course is **video-first**. Every module README embeds YouTube links > next to each topic. Watch the video, then run the commands from the README. > The full playlist is the > [Course Playlist on YouTube](https://www.youtube.com/playlist?list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb). ### What you will build By the end of the course you will have built a complete data platform that: - Ingests the public **NYC taxi trip dataset** from raw CSV files - Containers everything with **Docker** - Provisions cloud infrastructure with **Terraform** - Orchestrates workflows with **Kestra** - Loads data into a **BigQuery** data warehouse - Transforms it with **dbt** - Processes it in batch with **Apache Spark** - Streams it in real time with **Kafka / Flink** - Wraps up with a **capstone project** of your own design --- ## 2. How to Enroll You can take the course in two ways: as part of a scheduled **cohort** (with deadlines, a leaderboard, and a certificate) or **self-paced** at any time. ### Join the next cohort | Item | Detail | |------|--------| | **2026 cohort start** | 12 January 2026 | | **Register** | [Sign-up form (Airtable)](https://airtable.com/shr6oVXeQvSI5HuWD) | | **Cost** | Free | After you sign up, read [`after-sign-up.md`](after-sign-up.md) for next steps. ### Study self-paced All materials are public and remain available after each cohort ends. To learn independently: 1. Watch the course videos in module order. 2. Join the [DataTalks.Club Slack](https://datatalks.club/slack.html) and the `#course-data-engineering` channel. 3. Use the [FAQ document](https://datatalks.club/faq/data-engineering-zoomcamp.html) when you get stuck. > [!NOTE] > Self-paced learners can do every exercise and homework, but **certificates > are only issued to learners who complete a live cohort** within its > deadlines. See [Section 11](#11-certificates). ### Prerequisites To get the most out of the course you should have: - Basic coding experience - Familiarity with SQL - Experience with Python (helpful but not required) No prior data engineering experience is necessary. --- ## 3. Course Map The repository is organized as one folder per module, numbered in the order you should take them. Workshops and a final project are interleaved. | # | Module | Folder | Core tools | |---|--------|--------|------------| | 1 | Containerization & Infrastructure as Code | [`01-docker-terraform/`](01-docker-terraform/) | Docker, Docker Compose, PostgreSQL, Terraform, GCP | | 2 | Workflow Orchestration | [`02-workflow-orchestration/`](02-workflow-orchestration/) | Kestra | | — | Workshop 1: Data Ingestion | `cohorts/2026/workshops/dlt.md` | dlt | | 3 | Data Warehousing | [`03-data-warehouse/`](03-data-warehouse/) | BigQuery | | 4 | Analytics Engineering | [`04-analytics-engineering/`](04-analytics-engineering/) | dbt, DuckDB, BigQuery | | 5 | Data Platforms | [`05-data-platforms/`](05-data-platforms/) | Bruin | | 6 | Batch Processing | [`06-batch/`](06-batch/) | Apache Spark / PySpark | | 7 | Streaming | [`07-streaming/`](07-streaming/) | Kafka, Redpanda, Flink | | — | Final Project | [`projects/`](projects/) | Everything above | > [!NOTE] > The dataset used throughout is the **NYC Taxi & Limousine Commission (TLC)** > trip data. The course uses the **CSV** copies hosted at > [github.com/DataTalksClub/nyc-tlc-data](https://github.com/DataTalksClub/nyc-tlc-data/releases), > *not* the Parquet files on the nyc.gov site. CSV is easier for newcomers to > inspect with a spreadsheet or text editor. ### Past cohorts The [`cohorts/`](cohorts/) directory archives previous years (2022–2026), including homework, deadlines, and tools that have since been replaced (for example, Airflow and Mage in older cohorts). Use the current top-level module folders for the latest version; browse `cohorts/` only if you want historical material or a tool a past cohort used. --- ## 4. Setting Up Your Environment You do not need to install everything up front. Each module tells you what it needs. Install these as you reach the modules that use them. | Tool | Needed from | Why | |------|-------------|-----| | **Git** | Module 1 | Clone the repo | | **Docker + Docker Compose** | Module 1 | Run Postgres, pgAdmin, Kestra, Redpanda | | **Python 3.x** + [`uv`](https://docs.astral.sh/uv/) | Module 1 | Run ingestion scripts and manage virtual environments | | **Terraform** | Module 1 | Provision cloud infrastructure | | **A Google Cloud (GCP) account** | Module 1–3 | Free-tier cloud storage and BigQuery | | **dbt** (Core or Cloud) | Module 4 | Transform warehouse data | | **Apache Spark / PySpark** | Module 6 | Batch processing | > [!TIP] > The course standardizes on **`uv`** for Python environment and dependency > management. It is fast and replaces `pip` + `venv`. Where older notes mention > `pip install`, you can usually substitute `uv add` / `uv run`. ### Clone the repository ```bash git clone https://github.com/DataTalksClub/data-engineering-zoomcamp.git cd data-engineering-zoomcamp ``` ```text data-engineering-zoomcamp/ ├── 01-docker-terraform/ ├── 02-workflow-orchestration/ ├── 03-data-warehouse/ ├── 04-analytics-engineering/ ├── 05-data-platforms/ ├── 06-batch/ ├── 07-streaming/ ├── cohorts/ # past-year archives + current homework └── projects/ # capstone instructions ``` > [!IMPORTANT] > Several modules require a **Google Cloud account**. GCP offers a free tier > and trial credits that are sufficient for the course. Always run > `terraform destroy` and delete cloud resources when you finish a module to > avoid charges. See [Section 5.3](#53-provision-cloud-infrastructure-with-terraform). --- ## 5. Module 1 — Docker, Postgres & Terraform Module 1 is the foundation and the most hands-on. You will containerize a Postgres database, write a Python ingestion pipeline, and provision cloud infrastructure with Terraform. Work through the eleven lessons in [`01-docker-terraform/docker-sql/`](01-docker-terraform/docker-sql/) in order. ### 5.1 Run PostgreSQL in a container From [`docker-sql/04-postgres-docker.md`](01-docker-terraform/docker-sql/04-postgres-docker.md): ```bash docker run -it --rm \ -e POSTGRES_USER="root" \ -e POSTGRES_PASSWORD="root" \ -e POSTGRES_DB="ny_taxi" \ -v ny_taxi_postgres_data:/var/lib/postgresql \ -p 5432:5432 \ postgres:18 ``` Connect to it with `pgcli`: ```bash uv add --dev pgcli uv run pgcli -h localhost -p 5432 -u root -d ny_taxi ``` When prompted, the password is `root`. ```sql -- List tables, create one, query it \dt CREATE TABLE test (id INTEGER, name VARCHAR(50)); INSERT INTO test VALUES (1, 'Hello Docker'); SELECT * FROM test; \q ``` > [!NOTE] > The default credentials are `root` / `root` and the database is `ny_taxi`. > These values are used consistently across Modules 1 and 2, so memorize them. ### 5.2 Ingest the taxi data The repository ships a ready-made ingestion script, [`pipeline/ingest_data.py`](01-docker-terraform/docker-sql/pipeline/ingest_data.py), built with [Click](https://click.palletsprojects.com/). It downloads a month of yellow-taxi CSV data and loads it into Postgres in chunks. ```bash cd 01-docker-terraform/docker-sql/pipeline uv run python ingest_data.py \ --pg-user=root \ --pg-pass=root \ --pg-host=localhost \ --pg-port=5432 \ --pg-db=ny_taxi \ --year=2021 \ --month=1 \ --target-table=yellow_taxi_data ``` All ingestion options: | Option | Default | Description | |--------|---------|-------------| | `--pg-user` | `root` | PostgreSQL user | | `--pg-pass` | `root` | PostgreSQL password | | `--pg-host` | `localhost` | PostgreSQL host | | `--pg-port` | `5432` | PostgreSQL port | | `--pg-db` | `ny_taxi` | Database name | | `--year` | `2021` | Year of taxi data to download | | `--month` | `1` | Month of taxi data to download | | `--target-table` | `yellow_taxi_data` | Destination table | | `--chunksize` | `100000` | Rows read per chunk | > [!TIP] > The script reads the CSV with a progress bar and writes the table header > first, then appends data chunk by chunk. If you re-run it against the same > `--target-table`, it **replaces** the existing table — safe to re-run. ### 5.3 Add pgAdmin (web database UI) `pgcli` is great for quick queries; **pgAdmin** is a web UI for richer browsing. Both containers must share a Docker network so they can find each other. ```bash # Create a shared network docker network create pg-network # Postgres on the network, named so others can reach it docker run -it \ -e POSTGRES_USER="root" -e POSTGRES_PASSWORD="root" -e POSTGRES_DB="ny_taxi" \ -v ny_taxi_postgres_data:/var/lib/postgresql \ -p 5432:5432 --network=pg-network --name pgdatabase \ postgres:18 # pgAdmin on the same network (in another terminal) docker run -it \ -e PGADMIN_DEFAULT_EMAIL="[email removed]" \ -e PGADMIN_DEFAULT_PASSWORD="root" \ -v pgadmin_data:/var/lib/pgadmin \ -p 8085:80 --network=pg-network --name pgadmin \ dpage/pgadmin4 ``` Open [http://localhost:8085](http://localhost:8085), log in with `[email removed]` / `root`, then **Register → Server** with host `pgdatabase`, port `5432`, user `root`, password `root`. ### 5.4 Dockerize the pipeline Build the ingestion script into an image and run it on the network so it can reach the `pgdatabase` container ([`08-dockerizing-ingestion.md`](01-docker-terraform/docker-sql/08-dockerizing-ingestion.md)): ```bash cd pipeline docker build -t taxi_ingest:v001 . docker run -it \ --network=pg-network \ taxi_ingest:v001 \ --pg-user=root --pg-pass=root \ --pg-host=pgdatabase --pg-port=5432 --pg-db=ny_taxi \ --target-table=yellow_taxi_trips ``` > [!IMPORTANT] > Inside Docker, `--pg-host` must be the **container name** (`pgdatabase`), not > `localhost`. `localhost` inside a container refers to the container itself. ### 5.5 One command with Docker Compose Instead of starting Postgres and pgAdmin by hand, the [`pipeline/docker-compose.yaml`](01-docker-terraform/docker-sql/pipeline/docker-compose.yaml) starts both at once: ```bash cd 01-docker-terraform/docker-sql/pipeline docker compose up -d # start in background docker compose down # stop ``` This brings up `pgdatabase` (Postgres 18, port 5432) and `pgadmin` (port 8085) with named volumes so your data survives restarts. ### 5.6 Provision cloud infrastructure with Terraform Module 1 closes by creating a Google Cloud Storage bucket and a BigQuery dataset with Terraform. From [`terraform/README.md`](01-docker-terraform/terraform/README.md): ```bash # Authenticate for this session gcloud auth application-default login terraform init # initialize terraform plan -var="project=" # preview terraform apply -var="project=" # create terraform destroy # tear down when done ``` > [!CAUTION] > Always run `terraform destroy` when you finish a module. Leaving cloud > resources running can incur charges. Also **never commit your service-account > JSON key** to Git — treat it like a password. Windows users should follow the extra steps in [`terraform/windows.md`](01-docker-terraform/terraform/windows.md). --- ## 6. Module 2 — Workflow Orchestration with Kestra Module 2 replaces the manual ingestion script with orchestrated **flows** in [Kestra](https://kestra.io/), an open-source, YAML-based orchestrator. The eleven example flows in [`flows/`](02-workflow-orchestration/flows/) build up from "Hello World" to a scheduled, cloud-backed ELT pipeline with an AI assistant. ### Start Kestra ```bash cd 02-workflow-orchestration docker compose up -d ``` Open the Kestra UI at [http://localhost:8080](http://localhost:8080). Shut down with `docker compose down`. > [!NOTE] > The bundled [`docker-compose.yml`](02-workflow-orchestration/docker-compose.yml) > runs Kestra `v1.1`, a dedicated Kestra Postgres, plus the same `ny_taxi` > Postgres and pgAdmin from Module 1. Default Kestra login is > `[email removed]` / `Admin1234!`. ### Add the example flows Paste each YAML into the Kestra editor, or import them all via the API: ```bash curl -X POST -u '[email removed]:Admin1234!' \ http://localhost:8080/api/v1/flows/import \ -F fileUpload=@flows/04_postgres_taxi.yaml ``` The flows progress through the core concepts: | Flow | What it teaches | |------|-----------------| | `01_hello_world.yaml` | Tasks, inputs, outputs, variables, triggers, concurrency | | `02_python.yaml` | Running Python inside a flow | | `03_getting_started_data_pipeline.yaml` | Extract via HTTP → transform in Python → query with DuckDB | | `04_postgres_taxi.yaml` | Load yellow/green taxi CSVs into local Postgres | | `05_postgres_taxi_scheduled.yaml` | Schedule the pipeline + backfill history | | `06_gcp_kv.yaml` | Store GCP settings in Kestra's KV Store | | `07_gcp_setup.yaml` | Create the GCS bucket and BigQuery dataset | | `08_gcp_taxi.yaml` | ELT: load taxi data to GCS → BigQuery | | `09_gcp_taxi_scheduled.yaml` | Schedule + backfill the full cloud dataset | | `10_chat_without_rag.yaml` | Ask an LLM without context (baseline) | | `11_chat_with_rag.yaml` | Retrieval-Augmented Generation grounded in real docs | > [!TIP] > Section 2.5 of the module shows how to use **Kestra's AI Copilot** (powered > by Google Gemini) to generate flows from a natural-language prompt, and > compares it to plain ChatGPT to illustrate why *context matters*. You will > need a free [Gemini API key](https://aistudio.google.com/app/apikey). > [!WARNING] > Your `GCP_CREDS` service account contains sensitive data. Keep it in Kestra's > KV Store or as a [Secret](https://kestra.io/docs/concepts/secret) — never > hard-code it into a flow or commit it to Git. --- ## 7. Module 3 — Data Warehousing with BigQuery Module 3 ([`03-data-warehouse/`](03-data-warehouse/)) moves from a local database to a cloud **data warehouse**. It is mostly video + SQL, with these topics: - Introduction to BigQuery and external vs. native tables - **Partitioning** and **clustering** (and when to use each) - BigQuery cost and performance best practices - BigQuery internals - **BigQuery ML** — training and deploying models with SQL Helper scripts in [`extras/`](03-data-warehouse/extras/) download the taxi data and upload it to GCS — for example [`web_to_gcs.py`](03-data-warehouse/extras/web_to_gcs.py) and a progress-bar variant. ```bash # The upload target bucket is read from an environment variable export GCP_GCS_BUCKET="your-bucket-name" python 03-data-warehouse/extras/web_to_gcs.py ``` > [!NOTE] > `web_to_gcs.py` reads the bucket name from the `GCP_GCS_BUCKET` environment > variable (default placeholder: `dtc-data-lake-bucketname`). Set it to your own > bucket before running. The accompanying SQL files (`big_query.sql`, `big_query_ml.sql`) hold every query demonstrated in the videos. --- ## 8. Module 4 — Analytics Engineering with dbt Module 4 ([`04-analytics-engineering/`](04-analytics-engineering/)) transforms the raw warehouse data into clean analytical models with [dbt](https://www.getdbt.com/). The complete dbt project lives in [`taxi_rides_ny/`](04-analytics-engineering/taxi_rides_ny/). ### Choose your setup path | Path | Stack | Cost | Prerequisite | |------|-------|------|--------------| | 🏠 **Local** | DuckDB + dbt Core | Free | None — the guide loads the data for you | | ☁️ **Cloud** | BigQuery + dbt Cloud | Free tier (varies) | Completed Module 3 with taxi data in BigQuery | Follow [`setup/local_setup.md`](04-analytics-engineering/setup/local_setup.md) or [`setup/cloud_setup.md`](04-analytics-engineering/setup/cloud_setup.md). ### Run the dbt project ```bash cd 04-analytics-engineering/taxi_rides_ny dbt deps # install packages dbt seed # load lookup/seed data dbt run # build all models dbt test # run data tests dbt build # seed + run + test in dependency order ``` The project is configured in [`dbt_project.yml`](04-analytics-engineering/taxi_rides_ny/dbt_project.yml): - Models are layered **staging** (views) → **intermediate** (tables) → **marts** (tables). - A `dev` environment samples a date window (`dev_start_date = 2019-01-01`, `dev_end_date = 2019-02-01`) so you don't process the whole dataset while developing. > [!NOTE] > This module focuses on **yellow and green** taxi data for **2019–2020**. FHV > data, even if you loaded it in Module 3, is not used here. > [!TIP] > If you hit out-of-memory errors during `dbt build` with DuckDB, see > [`setup/duckdb_troubleshooting.md`](04-analytics-engineering/setup/duckdb_troubleshooting.md). > New to dbt? The class notes in > [`class_notes/`](04-analytics-engineering/class_notes/) walk through every > concept, and the > [dbt Fundamentals](https://learn.getdbt.com/courses/dbt-fundamentals) course > is a recommended primer. --- ## 9. Module 5 — Data Platforms with Bruin Module 5 ([`05-data-platforms/`](05-data-platforms/)) introduces [Bruin](https://getbruin.com/), a single platform that combines ingestion, transformation, orchestration, data quality, and lineage. The hands-on tutorial is a fill-in-the-blanks template: ```bash bruin init zoomcamp my-taxi-pipeline cd my-taxi-pipeline ``` You then complete the TODO-marked configuration and asset files, guided by inline comments, to build a three-layer pipeline (ingestion → staging → reports) over the NYC taxi data with DuckDB. Common Bruin commands: | Command | Purpose | |---------|---------| | `bruin run` | Execute the pipeline | | `bruin validate` | Check pipeline and asset definitions | | `bruin lineage` | Show data lineage | The reference notes in [`notes/`](05-data-platforms/notes/) contain completed implementations and cover projects, pipelines, **assets** (SQL / Python / YAML), variables, the Bruin **MCP** for AI agents, and deploying to **Bruin Cloud**. > [!TIP] > Bruin has a VS Code / Cursor extension and an MCP server, so you can build and > query the pipeline conversationally with an AI agent (Section 5.4). --- ## 10. Modules 6 & 7 — Batch and Streaming ### Module 6 — Batch Processing with Spark Module 6 ([`06-batch/`](06-batch/)) teaches **Apache Spark / PySpark**. Install Spark for your OS first: - [Windows](06-batch/setup/windows.md) - [Linux](06-batch/setup/linux.md) - [macOS](06-batch/setup/macos.md) Download the data, then run the example Spark SQL job ([`code/06_spark_sql.py`](06-batch/code/06_spark_sql.py)): ```bash # Download a year of a taxi type into data/raw/... ./06-batch/code/download_data.sh yellow 2020 # Run the Spark SQL job python 06-batch/code/06_spark_sql.py \ --input_green=data/raw/green/2020/ \ --input_yellow=data/raw/yellow/2020/ \ --output=data/report/revenue/ ``` | Spark job option | Description | |------------------|-------------| | `--input_green` | Path to green-taxi Parquet input | | `--input_yellow` | Path to yellow-taxi Parquet input | | `--output` | Destination path for results | The module covers Spark DataFrames, Spark SQL, the anatomy of a cluster, internals of **GroupBy** and **Joins**, RDDs, running a local standalone cluster, **Google Dataproc**, and connecting Spark to BigQuery. The `06_spark_sql_big_query.py` variant writes results straight to BigQuery. > [!NOTE] > If local Spark setup is difficult, the README offers a **Google Colab** > fallback — but try the local setup first; it is closer to real work. ### Module 7 — Stream Processing Module 7 ([`07-streaming/`](07-streaming/)) is built around the **PyFlink workshop** in [`workshop/`](07-streaming/workshop/), which builds a real-time pipeline: ```text Producer (Python) → Kafka (Redpanda) → Flink → PostgreSQL ``` Prerequisites: Docker + Docker Compose, [`uv`](https://docs.astral.sh/uv/), and a SQL client (`pgcli`, DBeaver, pgAdmin, or DataGrip). ```bash cd 07-streaming/workshop docker compose up redpanda -d # start the Kafka-compatible broker docker compose ps # confirm it's running ``` > [!NOTE] > The workshop uses **Redpanda** — a Kafka-compatible broker that needs no JVM > and no ZooKeeper, so it starts in seconds with one container. Every Kafka > client library works against it unchanged. Producers/consumers connect on > port `9092` (external) and Flink connects on `29092` (internal). Two optional study tracks accompany the workshop: - [`theory/`](07-streaming/theory/) — Kafka concepts with **Java** code examples (`kafka_examples/`), covering producers, consumers, Kafka Streams, KSQL, and Avro schema management. - [`extras/`](07-streaming/extras/) — supplementary Python and PyFlink examples from previous years (`kafka-python`, `confluent-kafka`, `faust`, `pyspark`). --- ## 11. Certificates Certificates are awarded to learners who complete a **live cohort** within its deadlines. From [`certificates.md`](certificates.md), eligibility requires: - Submitting the required **homework** by the deadlines, and - Completing and peer-reviewing the **capstone project**. After the cohort ends and results are finalized, a certificate link is published that you can add to your LinkedIn profile. Self-paced learners can do all the work but are not issued certificates. --- ## 12. The Capstone Project The [`projects/`](projects/) folder holds the final-project instructions. You apply everything from the course to a real-world dataset of your choice: 1. Pick a dataset and a problem. 2. Build an end-to-end pipeline (ingestion → warehouse → transformation → dashboard). 3. Submit your project. 4. **Peer-review** other learners' projects (required for your own to count). See [`projects/datasets.md`](projects/datasets.md) for dataset suggestions if you don't have one in mind. > [!TIP] > Reuse the course's reference stack for your capstone — Docker + Terraform + > Kestra + BigQuery + dbt is a complete, employer-recognizable pipeline. --- ## 13. Getting Help & Community The course runs on community. When you get stuck: - **Slack** — Join [DataTalks.Club Slack](https://datatalks.club/slack.html) and the [`#course-data-engineering`](https://app.slack.com/client/T01ATQK62F8/C01FABYF2RG) channel. - **Ask good questions** — Follow [`asking-questions.md`](asking-questions.md) so people can help you quickly (search first, share errors and what you tried). - **FAQ** — The [course FAQ](https://datatalks.club/faq/data-engineering-zoomcamp.html) answers most common setup and homework questions. - **Telegram** — [Announcements channel](https://t.me/dezoomcamp). > [!TIP] > Learn in public. [`learning-in-public.md`](learning-in-public.md) explains how > sharing your progress (blog posts, notes, LinkedIn) earns bonus points in a > cohort and builds your portfolio. The community has contributed dozens of > note sets, linked at the bottom of each module README. For a curated list of further reading, see [`awesome-data-engineering.md`](awesome-data-engineering.md). --- ## 14. Troubleshooting Common issues drawn from the module READMEs and the project's GitHub issues. ### Error: Kestra UI won't load — port `8080` already in use **Cause:** pgAdmin or another service is already bound to port 8080. **Solution:** Change Kestra's port mapping in `docker-compose.yml` from `8080:8080` to `18080:8080`, then open [http://localhost:18080](http://localhost:18080). **Prevention:** Check `docker ps` for port conflicts before starting a new stack. ### Error: `BigQueryError ... CSV table references column position 17, but line contains only 14 columns` **Cause:** The CSV in GCS was not fully downloaded or uploaded (a network/ transfer glitch), so its column count is short — it is *not* a real schema mismatch. **Solution:** Re-run the entire execution, including re-downloading the CSV from GitHub and re-uploading it to GCS. **Prevention:** Verify file sizes after upload; let downloads finish before loading. ### Error: pgAdmin can't connect to Postgres **Cause:** The two containers are on different Docker networks, or you used `localhost` as the host. **Solution:** Put both containers on the same network (`--network=pg-network`) and use the **container name** (`pgdatabase`) as the host in pgAdmin. **Prevention:** Prefer Docker Compose, which puts all services on one network automatically. ### Error: DuckDB out-of-memory during `dbt build` (Module 4) **Cause:** Building all models over the full dataset exceeds available memory. **Solution:** Follow [`04-analytics-engineering/setup/duckdb_troubleshooting.md`](04-analytics-engineering/setup/duckdb_troubleshooting.md); limit the date range with the `dev_start_date` / `dev_end_date` vars. **Prevention:** Develop against the sampled `dev` window, not the full history. ### Error: `pip install cheat.sh`-style "package not found" for course tools **Cause:** Course tooling is run via Docker or `uv`, not always installed from PyPI directly. **Solution:** Use the exact commands in each module README (`uv add`, `docker run`, `docker compose up`). **Prevention:** Don't substitute your own install method until you've completed a module once as written. ### Homework answer seems wrong **Cause:** Homework and datasets occasionally have reported discrepancies (see open GitHub issues such as the Module 1 Question 1 thread). **Solution:** Check the cohort's homework file under `cohorts/2026//homework.md` and ask in Slack — instructors and peers clarify quickly. **Prevention:** Always use the **current cohort's** homework file, not an older year's. --- ## 15. FAQ **Q: Is the course really free?** Yes. All videos, code, and materials are free and remain public after each cohort. **Q: Do I have to wait for a cohort to start?** No. You can study self-paced anytime. Cohorts add deadlines, a leaderboard, and a certificate. **Q: How much does the cloud cost?** GCP's free tier and trial credits cover the course if you tear down resources (`terraform destroy`, delete buckets/datasets) when done. Module 4's local path (DuckDB) and several workshops are entirely free. **Q: Do I need to know data engineering already?** No. You need basic coding and some SQL. Python helps but isn't required. **Q: Which Python tool does the course use — pip or uv?** The current materials use **`uv`** for environments and dependencies. Older cohort notes may show `pip`/`venv`; the commands are equivalent. **Q: What dataset will I work with?** The NYC TLC yellow/green taxi trip data, in CSV form, from [DataTalksClub/nyc-tlc-data](https://github.com/DataTalksClub/nyc-tlc-data/releases). **Q: Can I use AWS instead of GCP?** The course is GCP-focused, but Module 1 includes an AWS Terraform equivalent (`terraform_with_variable_AWS/`) that mirrors the GCS + BigQuery setup with S3 + Glue for learners who prefer AWS. **Q: Where are the old Airflow / Mage / Prefect materials?** In `cohorts//`. The current course uses Kestra for orchestration; past tools are archived for reference. **Q: How do I get a certificate?** Complete a live cohort's homework and capstone within the deadlines. Self-paced study does not include a certificate. **Q: Can I contribute notes or fixes?** Yes. Each module README has a "Community notes" section — open a pull request to add your notes above the marker line. **Q: Who runs the course?** DataTalks.Club, led by Alexey Grigorev with a team of instructors (Michael Shoemaker, Will Russell, Anna Geller, Juan Manuel Perafan, Arsalan Noorafkan, and others). It is sponsored by Kestra, Bruin, and dltHub. --- ## 16. Quick Reference ### Default credentials & ports | Service | Where | Credentials | Port | |---------|-------|-------------|------| | Postgres (`ny_taxi`) | Modules 1–2 | `root` / `root` | `5432` | | pgAdmin | Modules 1–2 | `[email removed]` / `root` | `8085` | | Kestra UI | Module 2 | `[email removed]` / `Admin1234!` | `8080` | | Redpanda (Kafka) | Module 7 | — | `9092` (ext) / `29092` (int) | > [!CAUTION] > These are **development-only** credentials shipped for learning. Never reuse > them, or the bundled `docker-compose` files, for anything exposed to the > internet. See the Admin Guide for hardening guidance. ### Everyday commands ```bash # Start / stop a module's stack docker compose up -d docker compose down # Run a Python script in its environment uv run python