Post

Figuring out a workflow to make Claude better at creating diagrams and illustrated explanations. By default it is pretty crap - see this example. Labels overlap. Arrows in the wrong place. Text too small. But it's also decent at identifying what's wrong once it takes a screenshot.

Claude's second attempt at fixing some of the problems, with a lot of additional steering from me on improving the visual details. It's still unable to make an arrow go where it needs to, or make two elements connect with a line. But it's a decent starting point - enough for me to cleanup by hand.

The best diagram outputs an LLM can get to are still crap by my standards. It helps speed me up a bit and give me starting points, but it's still disappointing that their visual reasoning and graphic design skills are really weak. The layouts and visual language are always predictably boring too

My theory here is the major labs have done very little or no RLHF on visual explanations and diagrams? Seems like a blind spot. Image generation alone or text generation alone are very well optimised but the combination seems uniquely hard for current models.

We’ve reached identical conclusions. If you figure this out I’d be very keen to hear how you did it!

Something that helped when I was writing my prompt caching post was constraining it to a smaller grid. I created a little SVG library in React where I could create nested grids for easier positioning and the LLM seemed to appreciate that. It also made it easier to clean up when it got it wrong.

I'm thinking like: gestalt principles translated into clear rules (put related items <8px apart, put other items >32px apart), grid system, when connecting two items verify your lines are touching the edges, etc. Mermaid was suggested a bunch in this thread but for me its too constrained and ugly.

Here’s what I went with. You create grids with a width and height, under the hood it uses the parent container size (either a parent grid or a parent <svg> size) and divides it into chunks. Then everything takes an x and y to position within this constrained grid. If unspecified, center.

What you’re describing is something I’d love to have. Like Android’s “constraint layout.” Mermaid is ugly as sin and also a nightmare to customise style wise.