Post

I've been leading the data infrastructure efforts at my job (I used to work as a Data Engineer in big tech) and the stack we've landed on is so enjoyable to work with. @dagster.io - Orchestration @duckdb.org - Database @motherduck.com - Data Warehouse/storage DBT - Data modeling

It's a very simple stack whose core philosophy is _just write SQL_. DuckDB makes managing the data _easy_ and allows us to ingest sources natively from just about anywhere we'd need to. Using DuckLake also means we have almost 1:1 parity with an Iceberg stack w/o the overhead of metadata files.

Dagsters "software defined assets" model is such a good heuristic for managing data orchestration pipelines. It's architecture is also incredibly simple and easy to maintain. Controlling dependencies, visualizing lineage, and integrations with tons of other tech make it a joy to work with.