Post

It helps our company is a year old and the data platform is of course very green. There maybe a world where this stack has unforeseen limitations that’ll impact us, but that’s hard to imagine with what we know now.

It's a very simple stack whose core philosophy is _just write SQL_. DuckDB makes managing the data _easy_ and allows us to ingest sources natively from just about anywhere we'd need to. Using DuckLake also means we have almost 1:1 parity with an Iceberg stack w/o the overhead of metadata files.

Dagsters "software defined assets" model is such a good heuristic for managing data orchestration pipelines. It's architecture is also incredibly simple and easy to maintain. Controlling dependencies, visualizing lineage, and integrations with tons of other tech make it a joy to work with.

There's a bit of a learning curve with this stack, especially if you come from tools Airflow, Spark, or Trino. But once you're able to understand it, it's surprisingly simple. This is one of the things I'm most proud to have been working on in my career.

StretchArmstrong

@stretcharm1.bsky.social

7 Sept 2025

i have a similar stack @dagster.io @getdbt.com dlt and databricks in azure. Dagtster and dbt work together like a dream. love @duckdb.org for local dev and poc wrangling. might try motherduck at some point especially when ducklake has matured. using k8s and grafana for alerts

Jorin

@jorin.me

I was always wondering how Dagster and DBT work together. Doesn't DBT already have dependencies, lineage and so on? Where does Dagster come in then?

6 Sept 2025

There’s a dagster-dbt integration library that exposes the DBT models and metadata to be translated into Dagster assets + metadata. It’s a pretty seamless integration. You even get the DBT tests as asset checks 😎. Dagster then just runs the DBT binary during materialization so it just works.

Jorin

@jorin.me

Sounds like a fun stack to work with! How does DuckLake fit into the stack? My understanding is that it would be like an alternative to MotherDuck here.

6 Sept 2025

DuckLake is an open table format similar to Apache Iceberg. So your data is stored as files and both Iceberg and DuckLake store additional data file metadata which act as a mapping to the to the actual data files. The difference is, iceberg is based on files and DuckLake is based on a DB.

6 Sept 2025

For large datasets the metadata is so useful because you get things like schema evolution, time travel (querying the db as it was in the past), and a few other features. DuckLakes format is based on a database storing the metadata and iceberg is based on parsing actual files at query time.

Willi

@willigo09.bsky.social

What were the main reasons to go with dagster instead of e.g. airflow or prefect?

We felt that Dagsters focus on materialized assets instead of the DAG itself fit our model better. The baked in metadata management, lineage, and scheduling capabilities were also a big plus. Airflow is incredible software but the overhead for managing these things is too high for our small team.

Willi

@willigo09.bsky.social

Thanks! Any experience with prefect?