Post

I've been leading the data infrastructure efforts at my job (I used to work as a Data Engineer in big tech) and the stack we've landed on is so enjoyable to work with. @dagster.io - Orchestration @duckdb.org - Database @motherduck.com - Data Warehouse/storage DBT - Data modeling

It's a very simple stack whose core philosophy is _just write SQL_. DuckDB makes managing the data _easy_ and allows us to ingest sources natively from just about anywhere we'd need to. Using DuckLake also means we have almost 1:1 parity with an Iceberg stack w/o the overhead of metadata files.

Dagsters "software defined assets" model is such a good heuristic for managing data orchestration pipelines. It's architecture is also incredibly simple and easy to maintain. Controlling dependencies, visualizing lineage, and integrations with tons of other tech make it a joy to work with.

I was always wondering how Dagster and DBT work together. Doesn't DBT already have dependencies, lineage and so on? Where does Dagster come in then?

And do you use Dagster to run anything else or is it only for running DBT?

Dagster is orchestrating all our pipelines end to end. We're pulling data from APIs, working on app db ingestion, (soon) orchestrating LLM training and (soon) isolated federated db shares. DBT is just the modeling part of our pipeline atm.