Post

I've been leading the data infrastructure efforts at my job (I used to work as a Data Engineer in big tech) and the stack we've landed on is so enjoyable to work with. @dagster.io - Orchestration @duckdb.org - Database @motherduck.com - Data Warehouse/storage DBT - Data modeling

Dagsters "software defined assets" model is such a good heuristic for managing data orchestration pipelines. It's architecture is also incredibly simple and easy to maintain. Controlling dependencies, visualizing lineage, and integrations with tons of other tech make it a joy to work with.

I was always wondering how Dagster and DBT work together. Doesn't DBT already have dependencies, lineage and so on? Where does Dagster come in then?

There’s a dagster-dbt integration library that exposes the DBT models and metadata to be translated into Dagster assets + metadata. It’s a pretty seamless integration. You even get the DBT tests as asset checks 😎. Dagster then just runs the DBT binary during materialization so it just works.

Sounds like a fun stack to work with! How does DuckLake fit into the stack? My understanding is that it would be like an alternative to MotherDuck here.

DuckLake is an open table format similar to Apache Iceberg. So your data is stored as files and both Iceberg and DuckLake store additional data file metadata which act as a mapping to the to the actual data files. The difference is, iceberg is based on files and DuckLake is based on a DB.

For large datasets the metadata is so useful because you get things like schema evolution, time travel (querying the db as it was in the past), and a few other features. DuckLakes format is based on a database storing the metadata and iceberg is based on parsing actual files at query time.