Post

I think you’re in a local minimum here; tools get better and people get better at using them. Eventually these tools will be as trustworthy as a compiler, and it would make about as much sense to manually program as it does to manually compile.

I can see how deterministic tools like compilers and table saws can be evaluated and adopted with confidence that they are a net positive. But blackboxed probabilistic tools like LLMs seem to defy rational evaluation and require "vibes" instead. So I'm not sure those are apt comparisons.

0 reposts20 likes

Of course we can run survey studies to evaluate LLM-based tools. But these will often be stuck in declarative bias hell, rely on questionable proxy metrics, or suffer from confounding factors when measuring long term impacts.

It seems to me that evaluating the productivity impacts of LLMs is closer to evaluating some management technique — "is Agile more productive?" — than to evaluating the productivity impacts of a compiler or table saw.