Post

jd
@jd.codes

I've been leading the data infrastructure efforts at my job (I used to work as a Data Engineer in big tech) and the stack we've landed on is so enjoyable to work with. @dagster.io - Orchestration @duckdb.org - Database @motherduck.com - Data Warehouse/storage DBT - Data modeling

jd
@jd.codes

It's a very simple stack whose core philosophy is _just write SQL_. DuckDB makes managing the data _easy_ and allows us to ingest sources natively from just about anywhere we'd need to. Using DuckLake also means we have almost 1:1 parity with an Iceberg stack w/o the overhead of metadata files.

jd
@jd.codes

Dagsters "software defined assets" model is such a good heuristic for managing data orchestration pipelines. It's architecture is also incredibly simple and easy to maintain. Controlling dependencies, visualizing lineage, and integrations with tons of other tech make it a joy to work with.

0 reposts1 like

jd
@jd.codes

There's a bit of a learning curve with this stack, especially if you come from tools Airflow, Spark, or Trino. But once you're able to understand it, it's surprisingly simple. This is one of the things I'm most proud to have been working on in my career.

Jorin
@jorin.me

I was always wondering how Dagster and DBT work together. Doesn't DBT already have dependencies, lineage and so on? Where does Dagster come in then?

jd
@jd.codes

There’s a dagster-dbt integration library that exposes the DBT models and metadata to be translated into Dagster assets + metadata. It’s a pretty seamless integration. You even get the DBT tests as asset checks 😎. Dagster then just runs the DBT binary during materialization so it just works.

jd
@jd.codes

Dagster is orchestrating all our pipelines end to end. We're pulling data from APIs, working on app db ingestion, (soon) orchestrating LLM training and (soon) isolated federated db shares. DBT is just the modeling part of our pipeline atm.