colton

Join me next week at @motherduck.com's Data Outpost conference where I'll be running a workshop on how to build a data analytics agent on the eve.dev framework!

It was great to join @christianminich.com in this webinar. He shared a lot of great insights in how he's using AI for data engineering.

I wrote an Almanack like Charlie Munger and Naval Ravikant did, but about life wisdom and my time of using Dagster for data orchestration over the years. Since using it in 2018, the story has gone from complexity to composability and from orchestrating to a full data platform.

Learn how we built a light weight evaluation framework for the official Dagster skills. With short feedback loops and quantitative measurements, we were able to make quick improvements that improving decision routing and context management. dagster.io/blog/evaluat...

If you're a Python developer using Claude Code, or something similar, we just published a blog that covers 10 rules we add to our prompt for drastically improved code gen. And we include the full prompt at the end of the post! dagster.io/blog/dignifi...

🚀 NEW on We ❤️ Open Source 🚀 How can a small team answer 16,000+ community Qs/month? Colton Padden (@colton.boo) reveals Dagster’s smart AI-powered doc strategy—built for humans and LLMs. allthingsopen.org/articles/sca... #WeLoveOpenSource #OpenSource #AI #TechDocs #Dagster #CommunitySupport

“Flexibility without fragmentation” In addition to some solid alteration, it’s also a great explanation of how to build a data team and how your system can and should scale. @colton.boo and Dennis Hume have an awesome new ebook on this topic, check it out! #dataBS dagster.io/how-to-scale...

Read this post you work in data analytics. A similar mindset problem pervades sales engineering as well, especially in the demo / early stages of discovery. Understand your prospect’s context. Who are they? What do they care about? What value can you provide them? That’s what you show.

MotherDuck is live now! DuckLake & The Future of Open Table Formats ft. Jordan Tigani & Hannes Mühleisen www.youtube.com/live/yn07s-_...

New Oxidise Your Command Line just dropped. Some that stand out to me are: xh, fselect, and presenterm! www.youtube.com/watch?v=rWMQ...

mpv might just be the best way to watch twitch.tv. mpv.io

@pycon.us might just be my favorite conference ever. everyone is so nice!

I'm writing about #opentableformats. With the newest entries of AWS S3 tables and Cloudflare R2, it's interesting to reassess the market and see where the future with managed Iceberg Tables leads us. What's your take? And anything missing on the evolution below?

I sat down to chat with Chris about "Project Airlift", a way to easily observe Airflow environments directly from Dagster. With a couple of lines of code, you can get a holistic picture into all of your environments, and then optionally migrate them!

🚨 New Course Alert 🚨 We are pleased to announce that the new "Testing with Dagster" course is now live on Dagster University. dagster.io/blog/dagster...

🚀 Want to preview the brand new `dg` cli, and a framework for building and working with YAML DSLs built on top of Dagster called "Components"? You can find a link to demo videos and a GitHub discussion in the thread!

Joe Naso and I just published an e-book covering the essential topics needed to build a data platform! Check it out if you're looking to learn more about data modeling, ingestion patterns, and common data architectures. dagster.io/how-to-build...

Ok, Claude Code, consider me impressed!

I think this is one of the most important books data people could be reading right now, especially if you need to work language models which need all the semantics they can get.

Come join us if you’d like to learn more about LLM routing from experts in the field!

We’re building a new static type checker for Python, from scratch, in Rust. From a technical perspective, it’s probably our most ambitious project yet. We’re about 800 PRs deep!

As someone who works at an open core company similar to Preset / Superset, this post really resonates with my experience. Buying a “hosted version of an open source library” made by that libraries creators has more benefits than “just” the hosting. preset.io/blog/running...

docs.dagster.io

I'm really proud to share that the new Dagster docs are live; what an effort!

This is a great resource covering the tools you need as a data engineer!

Loved this discussion between @barrald.bsky.social from Hex and @pedramnavid.com from Dagster on the data ecosystem, and the impact of AI on the work of data engineers and analysts! www.youtube.com/watch?v=8JxD...

I'm always really enjoying these presentations, sharing all the code and showcasing what's possible today. Amazing integrating BI tools such as Power BI, Looker, Tableau, or Sigma. I believe that's the first time an open-source orchestrator is fully end-to-end. 📺 youtu.be/z3trqkKPbsI?...

In case you missed it, Shifting Left and Moving Forward with @matsonj.com, @colton.boo, and me is now live on Youtube! youtu.be/z3trqkKPbsI?...

Consolation in the data transformation startup space! Congrats to the SDF team 👏 #dataBS blog.sdf.com/p/dbt-labs-h...

Join us tomorrow where @alexnoonan.bsky.social, @matsonj.com, and @colton.boo will discuss: 🦆 Building an end-to-end data platform ingesting Bluesky data into MotherDuck, and PowerBI 👈 Shifting Left in Data Engineering 🧠 And AI-powered Data Engineering best practices lp.dagster.io/deep-dive-sh...

Did you know Dagster asset checks can directly access data from an asset using the built-in I/O manager. Here's a quick snippet showing how it's done!

This time of year is full of reflection and self-improvement. That's why our upcoming deep dive (Jan 14 at 9 a.m. PT) with @matsonj.com at MotherDuck will focus on improving data engineering workflows and leveraging AI tools at their best. Register here: lp.dagster.io/deep-dive-sh...

It's really exciting to see much the dlt ecosystem has grown!

Kurt Vonnegut’s short story, “EPICAC”, is about a man who depends on a sentient supercomputer to charm the woman he loves, and it feels more relevant than ever—you can find it in the collection of stories, “Welcome to the Monkey House”. www.goodreads.com/book/show/49...

Buckle up because we're banging into the new year with my annual retrospective of the last year in databases! Highlights include license change blowback, Databricks vs. Snowflake gangwar, @duckdb.org's shotgun weddings, and buying a quarterback to impress your lover: www.cs.cmu.edu/~pavlo/blog/...

Turns out it's pretty easy to render 3D models with p5js! Here's how I'm embedding models in blog post markdown with Nuxt, Nuxt Content, and p5. cmpadden.github.io/articles/nux...

I might need to go touch some grass… git-wrapped.com/profiles/cmp...

This video from Hank Green on not having an internal monologue is wild. I have aphantasia, and distinctly remember "gaining" an internal monologue as a child. But his description of thinking, and self, are unlike any I've heard before. www.youtube.com/watch?v=XmTM...