Dmitriy Ryaboy

Dmitriy Ryaboy

@squarecog.bsky.social

Works with data, runs with swords.

2.8K followers234 following244 posts

Working on a bit of Rust code and it's so confusing how this faster and more efficient language is so much slower to iterate on and sends my laptop fans spinning so much harder than the Python & TypeScript stack I've been spoiled by.

That awkward moment when your tool doesn't find 25 planted bugs it was supposed to find in a benchmark, but does find 5 that were there unintentionally.

The founding of the East India Company, in 1600: "a barrel of beer a day was authorised ‘for the better holding together of the workemen from running from ther worke to drinke’." 400 years later, tech offices pretty much running the same playbook.

The corollary here is, if the thing is really throw-away and there is no upkeep to worry about... build might be right now, where it was not justified before, if you have the tokens.

A cool thing about the dblect tooling is that you can `noqa` it inside macros. Something sqlfluff doesn't support right now, but i found it necessary for significant dbt usage.

Rough order of planned `dblect` releases: 1. Structural hazards: static analysis across dbt models 2. Logical types + contracts: eg no summing amounts without grouping by currencies 3. Var/flag worlds: awareness of flag values that change generated SQL 4. PBT: generate examples and test execution

I'm working on a tool, dblect, to help test dbt project changes for bugs in meaning, not just sql validity or data diffs. Last weekend it got the ability to call out bugs like summing up money amounts without grouping by currency. This is good stuff, man.

It's genuinely hard and takes more work to do a close read of Claude docs and designs than those of a mid-level SE, because Claude makes similar mistakes but it _sounds_ a lot more authoritative. The normal tells of someone not really thinking through it aren't there, they're... different.

I'm writing something that lets you say "the columns amount:int and currency:varchar make up the type Money". The right name for that is probably SemanticType, but I'm concerned folks will get confused and think this is connected to Semantic Layers. Does DomainType work?

Two days ago, I joked "Codeim, I mean Windsurf, I mean whatever it is called now - probably Devin." Yesterday they actually renamed it to Devin. What should I do with my newly discovered powers?

What if you could have nice types on top of relations, with type inference through your sql workflows. With type refinements. And you could write correctness contracts on them. And it was statically checked. And maybe you could also generate PBT style tests on top. Would that be nice?

It's been like 15 years since the Twitter API started putting ids as both numbers and strings cause js sucks, and yet there are Real Serious Companies still tossing 64 bit int ids around in json, like loaded footguns.

Old sigmod and vldb papers are gold. I'm working on a thing at the intersection of program analysis and db theory (aka pbt for dbt) and it's hella fun. What a gem the Olston, Chopra, Srivastava 2009 sigmod paper was. Undervalued.

This is a pretty intriguing idea for future proofing file formats. It does assume wasm is future proof, of course, but that feels like a safer bet than "assume readers are updated"

Andy Pavlo@andypavlo.bsky.social

Our F3 files embed small WASM programs to decode data. If somebody creates a new encoding and the DBMS does not have native impl, it can still read data using WASM passing Arrow buffers. Our experiments show WASM is 15-20% slower than native. We use @spiraldb.com's Vortex encoding impls.

If you love this sort of thing, read up on C-store, which introduced this idea in 2005 and commercialized it in Vertica. Stonebraker, Sam Madden, Daniel Abadi. Parquet was also partially inspired by Vertica (and Google's Dremel, and PaX by Natassa Ailamaki et al) :-).

DuckDB@duckdb.org

Are you streaming into your Lakehouse? Traditional formats suffered with the “many small files” problem — OLAP engines merge them reactively with long jobs. ⏳ DuckLake takes a proactive path: Data Inlining + async flush to parquet while always keeping data queryable ⚡