Yaroslav Tkachenko

Working with data streaming systems? You have to read "The Dataflow Model Revisited" paper! www.vldb.org/pvldb/vol19/...

The biggest takeaway for me: windowing is just another form of GROUP BY, not the foundation of streaming analytics! Users should define what they want in SQL declaratively and let an incremental view maintenance engine handle the streaming complexity.

I'll be speaking about optimizing Kafka client serialization next week. Register here: factorhouse.io/events/kafka...

I've been working on a feature for Streamling, and I needed to filter out data based on statistics (with min/max/null counts). DataFusion (which Streamling uses) already has built-in support for this - PruningPredicate! It can rewrite expressions if they're compatible with available statistics.

For example: ts >= 400 AND city = 'Vancouver' can be rewritten as: ts_max >= 400 AND city_min <= 'Vancouver' AND 'Vancouver' <= city_max

It's used for reading Parquet files! The implementation is based on the Snowflake paper.

independentengineer.co“Hourly Billing Is Nuts” by Jonathan StarkThe biggest mistake is charging hourly.

One of my biggest learnings of the past 12 months as a consultant was ditching hourly pricing. I no longer have an hourly rate. You shouldn't either: www.independentengineer.co/p/hourly-bil...

github.comGitHub - datafusion-contrib/StreamFusion: World's First OSS Flink Accelerator built on Apache DataFusionWorld's First OSS Flink Accelerator built on Apache DataFusion - datafusion-contrib/StreamFusion

Cool project that's worth checking out if you're running Flink at scale: github.com/datafusion-c...

So far, when data streaming and AI are mentioned together, it typically describes the problem of maintaining a fresh, relevant context: e.g. use stream-processing to join/aggregate/enrich data streams, and pass them to an agent as context for inference. And honestly, I'm sold.

But is there anything else? E.g. using Kafka for agent-to-agent communication? Streaming long agent sessions? Workflow orchestration?

s2.devStream per agent sessionWhy AI agent sessions need their own durable streams for memory, auditability, isolation, branching, and reliable resume.

I really like what S2 is doing: s2.dev/blog/agent-s... , looking for more examples like this!

streamacademy.ioAdvanced Apache Flink CourseAdvanced Apache Flink: Deep dive into modern Flink 2 internals, DataStream & Table APIs, custom operators, and production best practices. Intensive course with hands-on workshops

I'm considering adding a few updates to my Advanced Apache Flink course. What would you like to see? streamacademy.io/course/advan...

datadoghq.comHow we extended Apache DataFusion to execute one query across many machines | DatadogLearn how Datadog built Distributed DataFusion to scale interactive Apache DataFusion queries across multiple machines.

If you enjoy working with large-scale data processing systems and you want to stay relevant, you have to learn Apache DataFusion. Slowly but surely, it's replacing many other tools. The RAD stack (Rust, Arrow, DataFusion) is the future. One example: www.datadoghq.com/blog/enginee...

streamingdata.techData Streaming Summit 2026Conference highlights.

I attended the Data Streaming Summit last week. Just published a post with a few highlights from the event: www.streamingdata.tech/p/data-strea...

Watermarks can be a really scary concept when you start working with stateful streaming systems. And all popular frameworks don't offer anything better than "bounded out-of-orderness": pick a number of seconds and hope that all data arrives within that interval.

But when you do control the producers (e.g. application events), the producers themselves can be the source of watermarks! The idea is not new, and it's covered in several famous papers like Google's MillWheel (see the screenshot).

It can actually be translated to modern Kafka producers emitting watermark signals. Anyone interested in a POC?

But in reality, there is almost always late data, so you end up with data loss (and incorrect results). This is a really hard problem, especially when you don't control your producers (e.g. web/mobile click stream data, IoT telemetry, etc.)

Earlier today at the Data Streaming Summit. Thanks Tun Shwe for the picture! #DSS26

Does anyone use IVM systems like Materialize or Feldera in production? What's your experience? Feel free to DM.

cwiki.apache.orgFLIP-611: Running Flink jobs in MiniCluster using the Kubernetes Operator - Apache Flink - Apache Software FoundationDUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.

A very cool FLIP (Flink Improvement Proposal) has recently been accepted: Running Flink jobs in MiniCluster using the Kubernetes Operator. cwiki.apache.org/confluence/s...

The FLIP addresses the problem by running both components (JobManager and TaskManager) in the same JVM process on a single pod. It's a big step in making Flink simpler and more efficient to operate.

I especially like that you can easily migrate from this setup to a "classic" deployment by simply using a savepoint. Start small, then scale out.

A very cool FLIP (Flink Improvement Proposal) has recently been accepted: Running Flink jobs in MiniCluster using the Kubernetes Operator. cwiki.apache.org/confluence/s...

Historically, every Flink pipeline required: - At least two pods (JobManager and TaskManager). - Each pod needs at least 0.5 vCPU and 1GB RAM (it's possible but really tricky to go lower).

So that's 1 vCPU and 2GB RAM for any pipeline to start, even if it processes one message a minute. Imagine running that at scale (I don't need to imagine...)? At the same time, other tools (e.g., Rust- or Go-based) can have a much lower footprint.

It's fascinating how many projects now offer a native CDC (Change Data Capture) experience. A few years ago, your choice was to use Debezium or ... *checks notes* that's it. Postgres logical replication for CDC has become almost... boring?

Is this because of LLMs? Or have more engineers become database replication experts? 🤔

independentengineer.coThe Rise of a Specialist in the Age of AIWhy LLM coding tools may increase the value of specialist software engineers—and why expertise, judgment, and critical thinking still matter.

The value of specialists will only increase as LLM adoption grows. Sure, anyone can generate code in any language and in any framework, but how do you know it's optimal? Or that you need it in the first place? Some thoughts: www.independentengineer.co/p/the-rise-o...

Flamegraph CPU profiling is likely the most efficient way to investigate performance issues in complex distributed systems like Apache Flink. Instead of guessing why something is slow, measure it.

My Advanced Apache Flink course includes a chapter on profiling, where I show various performance issues and their associated flamegraphs. #DataStreaming #ApacheFlink

streamacademy.io/course/advan...

Flamegraph CPU profiling is likely the most efficient way to investigate performance issues in complex distributed systems like Apache Flink. Instead of guessing why something is slow, measure it.

Did you know that Flink ships with a built-in flamegraph profiler that you can trigger with a single click? Set "rest.profiling.enabled" to true and open any TaskManager tab. Over the years, I've helped many companies make Flink profiling an easy-to-use tool.

I will not stop reading code anytime soon. LLMs can generate code that's correct and even performant, but I frequently find it too complex. And I hate accidental complexity - it compounds very quickly and leads to cruft, bugs, and, ultimately, slow and unreliable software.

I spent several extra days, but, as a result, I managed to remove a lot of unnecessary complexity, and I'm pretty satisfied with the result. But if you are new to the field, you don't have the same level of expertise to question LLM's output, and it's a bit scary, especially long-term.

Don't give up on your judgement!

And I kept finding new abstractions that seemed too complex. After 20 years of building software, I've developed a pretty sensitive complexity detector. You look at a new struct with 7 fields, and it raises all kinds of alarms.

So I pushed back: can we simplify this specific abstraction? The answer: "Yes, this can be simplified. Here's why the gate ended up complex, and two ways to simplify it."

Then, implementation. Rust project, Opus 5.5, Claude Code, subagents, a bunch of useful skills, a well-crafted AGENTS file and, most importantly, a fully self-contained end-to-end testing suite with hundreds of tests that the agent can run.

The implementation was done, and I spent several days reviewing it. Just to be clear, my goal wasn't to understand every single line, but to build mental models about the abstractions that the LLM generated.

Here's an example from last week. I worked on a fairly complex feature. First, iterating on the spec. Lots of research, going back and forth with the agent, finally coming up with a decent high-level design doc.

I'll be speaking at the Data Streaming Summit next week. Covering one of my favourite topics: columnar data for streaming workloads. Interested in attending? Use the DSSBUDDY26 discount code to get a free ticket!

sap1ens.comConsulting | sap1ens.comConsulting

Want to make your streaming pipelines more efficient? I help companies make data streaming systems faster, cheaper, and more reliable - whether that means finding bottlenecks, reducing infrastructure spend, or rethinking part of the architecture. Work with me: sap1ens.com/consulting/

I currently have some additional consulting capacity and am looking to work with a few new teams.

streamingdata.techImproving Cost Efficiency of Data Streaming PipelinesLearn practical ways to cut data streaming costs in Kafka and Flink by optimizing batching, serialization, state, network traffic, and capacity.

Most teams that operate large-scale data streaming pipelines start to care about cost in one way or another. Streaming is perceived as "expensive", and it's frequently questioned. Today I'm attempting to address some concerns: www.streamingdata.tech/p/improving-...

streamnative.ioIntroducing the Stream Materialization Framework: From Streams to Governed Data AssetsLakestream's Stream Materialization Framework turns a stream into governed Iceberg, ClickHouse, or OpenSearch tables without a separate connector.

Streamhouse is old news, StreamNative has just released Open Lakestream. Lakestream is their take on materializing streaming data. streamnative.io/blog/introdu...

The idea is to use object storage: "Because compaction works directly against object storage, materialization adds no load to the brokers serving producers and consumers. It requires no consumer groups and no separate connector cluster, and its compute scales independently from the brokers."

However, instead of just focusing on lakehouses, they also support databases like ClickHouse and OpenSearch! I like this new take on connectors.