Anders

this is the clearest case for Arrow that I've ever seen. I love that it's high-level but also doesn't shy away from details when it's important. this should be a #databs canon text imho. thanks @ianmcook.bsky.social !

1) my top-level takeaways on DeepSeek's smallpond: a distributed data processing framework used for training LLMs #databs

10) So sick that a smallpond pipeline returns a LogicalPlan representing a DAG where each node is a distinct data processing task. Imagine if a dbt DAG resulted in a single logical plan that operates across multiple engines. and then they can optimize the plan before execution as well! 🤯🤯🤯

1) my top-level takeaways on DeepSeek's smallpond: a distributed data processing framework used for training LLMs #databs

8) this HN called out that smallpond abstracts supports using different query engines for different jobs (shuffling vs sorting). Very bullish on this future of right tool for right job and making it as simple as a config news.ycombinator.com/item?id=4323...

1) my top-level takeaways on DeepSeek's smallpond: a distributed data processing framework used for training LLMs #databs

6) Making smallpond 5 years ago would have been very difficult! but the emergence of lower-level, off-the-shelf components greatly accelerate the development time. Within the year, we'll to see this new paradigm catch on. Future examples will probably be using DataFusion not DuckDB.

1) my top-level takeaways on DeepSeek's smallpond: a distributed data processing framework used for training LLMs #databs

4) smallpond is a bespoke data processing framework using off-the-shelf, OSS components (ray, arrow, duckdb, polars). Why didn't they use {TOOL}? My guesses ❌ dbt: they wanted Python Dataframe API ❌ Airflow: not as close to metal as Ray ❌ pytorch or ray[data]: idk tbh

1) my top-level takeaways on DeepSeek's smallpond: a distributed data processing framework used for training LLMs #databs

2) so cool to begin to see the data infra that supports training LLMs. Open weights is cool, and RAG makes sense, but as a former XGBooster turned "data engineer", seeing the data cleaning pipelines is what I've most wanted to see.

post one of a new series kicking off today whose larger thrust is effectively: understanding SQL, not just a job for the database! Post 1 lays out what the levels of understanding docs.getdbt.com/blog/the-lev... #databs

in AWS Glue Data Catalog, you can now read and write Iceberg tables using the Iceberg REST API. aws.amazon.com/blogs/big-da... previously platforms had to write a custom integration to read/write Glue Iceberg tables, not you can use the REST API! #databs #apacheiceberg #aws

4/ what about S3 Tables? aren't they then "bad" by not using the OSS API spec? 🤷 data teams are predominately on a single cloud, so what they need is a standard within a single provider. and, like @benesch.bsky.social said, always better to not have to think about catalogs and have it 'just work'

in AWS Glue Data Catalog, you can now read and write Iceberg tables using the Iceberg REST API. aws.amazon.com/blogs/big-da... previously platforms had to write a custom integration to read/write Glue Iceberg tables, not you can use the REST API! #databs #apacheiceberg #aws

2/ lately, I've been stoked more on the Iceberg REST catalog than the table format itself bc it radically simplifies things both for A. end users: in terms of what they need to know B: data platforms: in terms of the number standalone integrations needed

Hey #databs, I'm submitting to speak at Data Council in April, vote with a like to the reply with the talk that you think is the most interesting! www.datacouncil.ai/cfp-2025

3. Disruptive or Sustaining? over the next six years, what impact will recent innovations have on practitioners lives and the vendor ecosystem? Going deep specifically on OSS projects like: #iceberg, #substrait and Single-node compute (e.g. #DuckDB, #Datafusion) #databs

Hey #databs, I'm submitting to speak at Data Council in April, vote with a like to the reply with the talk that you think is the most interesting! www.datacouncil.ai/cfp-2025

1. Surfing Disruption: lessons learned from making dbt Mesh work across platforms yn top of a still very nascent Iceberg #databs