Post

I've read a couple papers on property based testing and the examples are usually very DS&A oriented because it's where testing invariants and defining properties is easy. For business rules it's much more difficult to pin down exact behavior. Requirements might not be precise enough.

Yeah the invariant-based stuff has somewhat low ceilings in what you can do, but stateful properties have an incredible amount of expressiveness for larger scale integration tests around whole workflows that would in theory be very powerful as a backstop for LLM-based development.

Do you mean something like this? antithesis.com/resources/de... by "stateful properties"?

Can you link to some sources on what you mean by stateful properties?

I’m still interested to see how it’ll unfold, but despite shipping code faster and using more general forms of specs when engineering with LLMs, the testing methods used never made use of the gained time nor of the more general spec. Instead the trend is to just generate more examples and review.

I’m not sure what may lie behind these observations, which goals and pressure gradients shape them, but it feels funny/interesting that the gains appear to be “reinvested” in specific ways that reinforce existing tradeoffs, rather than seizing opportunities for a new balance.

Part of it just makes me feel the stuff I wished I could see grow never really mattered to the “group” (see how nobody even looks that way even when there’d be more room now than ever!) and it’s a very funny feeling to contend with. A bit disconcerting to see the distance grow, but better to know?

Part of this is that property testing feels magical - the examples are always for things like "assert that is sorted(sort(x))" Which is nice _in theory_, but the docs don't really have much in the way of more complicated examples.

Stateful properties would be so good for the LLM integration flows because they’d even help minimizing context usage. Stateful docs exist (and Anthropic owes me a check for having ingested them), but a problem is that unless the volume is high, it’s hard to amplify these practices via training set.

At some point the property tests I can come up with just feel like some extra ceremony around example tests, but then I lose nice properties like being able to easily debug!

I’d need to show some examples at work, like everything we have that generated extensive scenarios to walk through a release integration scenario could be given as a recipe for it in stateful property tests and have it shrink the failing steps for you.