Post

this is interesting, but feels at best incomplete. i think there are at least 3 separate reasons we haven't yet seen consistently good writing from AI, at least in long form (microfiction is arguably mostly solved): 🧵

1. the problem is just way harder. good writing is mostly an aesthetic judgment, unlike code or science, where correctness is often verifiable. people are much less forgiving! if AI writes 120K words of code that *works*, devs are impressed even if they think it's insecure spaghetti code.

whereas if AI writes a broadly coherent 120K-word novel, nobody is impressed unless they also like the quality of individual paragraphs. so an AI writer has to be able to write like almost *every* human writer to be "good", and it has to produce high quality writing fractally, *at every scale*.

2. for reasons related to the above, learning to produce coherent and high-quality long form writing via pretraining or RL is probably much harder than learning to generate code or even do math. lack of verifiability, and the inherent subjectivity of writing, really hurts you here.

one way to think about it is that the manifold you need to learn in the latent space is much larger, because there are many more ways to be good. you probably need an incredible amount of feedback from expert judges, and you have the standard chaining problem, but with very few intermediate labels.

3. it's arguably more of a harness problem than a base model problem. i think @moultano.bsky.social's point that information needs to come from somewhere is correct, buuut... it doesn't have to be in the prompt! it could be in the weights (hard for reason (2)), or it could be in the harness.

which is just to say that, even if base model development halted in its current state, we probably *will* still get a good AI novelist in the next couple of years. it's just that it's not trivial to build the right harness, and not a lot of resources are being thrown at that particular problem.

buuuut even then, i'm skeptical that any particular AI, short of genuine ASI, would ever be *widely* received as a good writer, for the same reason that even the most popular human writers are usually appreciated by only a small subset of people. it's just inherently subjective.

Isn't another option 4. A large fraction of the writing on the internet is not great, and there's not much of a training signal that tells the training algo when it's digesting good vs not so good text. Therefore it will steer towards the middle of the pack?