Post

Watermarks can be a really scary concept when you start working with stateful streaming systems. And all popular frameworks don't offer anything better than "bounded out-of-orderness": pick a number of seconds and hope that all data arrives within that interval.

Yaroslav Tkachenko

@sap1ens.com

But in reality, there is almost always late data, so you end up with data loss (and incorrect results). This is a really hard problem, especially when you don't control your producers (e.g. web/mobile click stream data, IoT telemetry, etc.)

9/10/26, 4:06 pm0 reposts1 like

Yaroslav Tkachenko

@sap1ens.com

But when you do control the producers (e.g. application events), the producers themselves can be the source of watermarks! The idea is not new, and it's covered in several famous papers like Google's MillWheel (see the screenshot).

Yaroslav Tkachenko

@sap1ens.com

It can actually be translated to modern Kafka producers emitting watermark signals. Anyone interested in a POC?