Post

Figuring out a workflow to make Claude better at creating diagrams and illustrated explanations. By default it is pretty crap - see this example. Labels overlap. Arrows in the wrong place. Text too small. But it's also decent at identifying what's wrong once it takes a screenshot.

Claude's second attempt at fixing some of the problems, with a lot of additional steering from me on improving the visual details. It's still unable to make an arrow go where it needs to, or make two elements connect with a line. But it's a decent starting point - enough for me to cleanup by hand.

The best diagram outputs an LLM can get to are still crap by my standards. It helps speed me up a bit and give me starting points, but it's still disappointing that their visual reasoning and graphic design skills are really weak. The layouts and visual language are always predictably boring too

My theory here is the major labs have done very little or no RLHF on visual explanations and diagrams? Seems like a blind spot. Image generation alone or text generation alone are very well optimised but the combination seems uniquely hard for current models.

We’ve reached identical conclusions. If you figure this out I’d be very keen to hear how you did it!

Something that helped when I was writing my prompt caching post was constraining it to a smaller grid. I created a little SVG library in React where I could create nested grids for easier positioning and the LLM seemed to appreciate that. It also made it easier to clean up when it got it wrong.