Post

I've been leading the data infrastructure efforts at my job (I used to work as a Data Engineer in big tech) and the stack we've landed on is so enjoyable to work with. @dagster.io - Orchestration @duckdb.org - Database @motherduck.com - Data Warehouse/storage DBT - Data modeling

It's a very simple stack whose core philosophy is _just write SQL_. DuckDB makes managing the data _easy_ and allows us to ingest sources natively from just about anywhere we'd need to. Using DuckLake also means we have almost 1:1 parity with an Iceberg stack w/o the overhead of metadata files.

Sounds like a fun stack to work with! How does DuckLake fit into the stack? My understanding is that it would be like an alternative to MotherDuck here.

DuckLake is an open table format similar to Apache Iceberg. So your data is stored as files and both Iceberg and DuckLake store additional data file metadata which act as a mapping to the to the actual data files. The difference is, iceberg is based on files and DuckLake is based on a DB.

For large datasets the metadata is so useful because you get things like schema evolution, time travel (querying the db as it was in the past), and a few other features. DuckLakes format is based on a database storing the metadata and iceberg is based on parsing actual files at query time.

And how does DuckLake play together with MotherDuck in your stack? They are both for storing data, right? Are you using them to store different data?

MotherDuck is providing the storage for DuckLake parquet files (in their s3 buckets currently although you can BYO) and providing compute instances. They have a feature called read scaling that allows you to federate read only duckdb compute instances to specific db share which is useful for us.

Thanks for explaining! I didn't know that you can use DuckLake from within MotherDuck like that. That's nice! Do you run a separate Postgres instance as metadata store for DuckLake?