Dagster Essentials

Learn the basics of Dagster, a Python-based platform that enables you to build robust, production-ready data pipelines. In this course, you’ll learn how to represent a data pipeline as the data assets it produces and orchestrate a pipeline you’ll make with Dagster.

Get Started

Course prerequisites

This course is geared towards users who are unfamiliar with Dagster and looking to get started.

Beginner level

Python knowledge

While you don’t need to be a Python expert to get started, you do need some Python familiarity to complete this course and use Dagster. In Lesson 2, we’ll cover Dagster’s specific installation requirements. Here are some Pythonic concepts used, along with resources to learn about them:
SQL knowledge

You won’t be writing complex SQL, but you will need to understand the concept of select statements, what tables are, and how to make them. If you’d like a 5-minute crash course, here’s a short article and cheatsheet on using SQL.
Command line familiarity

Like the Python requirement, you don’t need to be an expert in the command line - just familiar with it. This course provides copy-and-pasteable commands so you won’t have to write your own, but you should know how to open and use the terminal. If you need some help getting started, check out Code Academy’s course.

Course curriculum

1. What's data engineering?
2. What's an orchestrator?
3. Orchestration approaches
4. Why is asset-centric orchestration good for data engineering?
5. Project preview
6. Knowledge check
1. Prerequisites and installation
2. Set up local
3. Set up Codespaces
4. Project files
5. Knowledge check
1. Overview
2. What's an asset?
3. Knowledge check
4. Defining your first asset
5. Knowledge check
6. Asset materialization
7. Viewing run details
8. Troubleshooting failed runs
9. Practice: Create a taxi_zones_file asset
10. Lesson recap
1. Overview
2. What's a dependency?
3. Assets and database execution
4. Loading data into a database
5. Practice: Create a taxi_zones asset
6. Assets with in-memory computations
7. Practice: Create a trips_by_week asset
1. Overview
2. What's the Definitions object?
3. What's a code location?
4. Code locations in the Dagster UI
5. Knowledge check
1. Overview
2. What's a resource?
3. Knowledge check
4. Setting up a database resource
5. Using resources in assets
6. Practice: Refactoring assets to use resources
7. Analyzing resource usage using the Dagster UI
8. Recap
9. Knowledge check
1. Overview
2. What are schedules?
3. Practice: Create a weekly_update_job
4. Creating a schedule
5. Practice: Create a weekly_update_schedule
6. Updating the Definitions object
7. Jobs and schedules in the Dagster UI
8. Knowledge check
1. Overview
2. What are partitions and backfills?
3. Creating a partition
4. Practice: Create a weekly partition
5. Adding partitions to assets
6. Practice: Partition the taxi_trips asset
7. Creating a schedule with a date-based partition
8. Practice: Partition the trips_by_week asset
9. Partitions and backfills in the Dagster UI
10. Lesson recap
11. Knowledge check
1. Overview
2. What's a sensor?
3. Configuring asset creation
4. Creating an asset triggered by a sensor
5. Creating a job
6. Building the sensor
7. Updating the Definitions object
8. Sensors in the Dagster UI
9. Enabling the sensor
10. Knowledge check
1. Capstone project
1. Overview
2. What's metadata?
3. Definition metadata: Asset descriptions
4. Definition metadata: Asset groups
5. Practice: Grouping assets
6. Materialization metadata
7. Practice: Add metadata to taxi_zones_file
8. Asset metadata as Markdown
9. Knowledge check

Skills you'll gain in this course

Confidence in writing a data pipeline in Dagster
Viewing your data pipelines as the data assets that they produce
Understanding of essential Dagster concepts and benefits

Get started with Dagster Essentials

This course will take 6-10 hours to complete.
You can pause at anytime and pick up right where you left off.

Start Learning

Reviews from our Fans!

Filling the Gap: ‘Dagster Essentials’ Delivers a Comprehensive Masterclass

Juan Domenech

The dedication and effort the team put into creating comprehensive content are evident. Every module resonates with quality, making it an enriching learning experience. Highly recommended for anyone diving into Dagster.

Read Less