Post

When writing about Property-Based testing, I found they were much more solid and comprehensive than usual tests, due to their more general (properties vs examples). They weren’t used because they felt more complex and time-intensive to write, despite being shorter and having to change them less.

I've read a couple papers on property based testing and the examples are usually very DS&A oriented because it's where testing invariants and defining properties is easy. For business rules it's much more difficult to pin down exact behavior. Requirements might not be precise enough.

Yeah the invariant-based stuff has somewhat low ceilings in what you can do, but stateful properties have an incredible amount of expressiveness for larger scale integration tests around whole workflows that would in theory be very powerful as a backstop for LLM-based development.

Can you link to some sources on what you mean by stateful properties?

Here’s a book chapter I had written about it: propertesting.com/book_case_st... The model for execution is described at a higher level in propertesting.com/book_statefu...

If I’m understanding quickly, testing by generating arbitrary reified operations and applying them to both a target system and a dual model of the system, then asserting some observations, is that close?

An interesting bit is that people often write out a partial spec in their prompt, and finding ways to translate that to a more specific set of properties would likely improve accuracy.

Cool. The model doesn’t have to be strictly in memory though. Maybe you have a legacy system and a new target for migration, you could use this technique to verify matching behavior

Yep, that specific approach is often called “oracle testing” because you have a complete perfect implementation you can compare with.