Post

Figuring out a workflow to make Claude better at creating diagrams and illustrated explanations. By default it is pretty crap - see this example. Labels overlap. Arrows in the wrong place. Text too small. But it's also decent at identifying what's wrong once it takes a screenshot.

1 repost37 likes

Claude's second attempt at fixing some of the problems, with a lot of additional steering from me on improving the visual details. It's still unable to make an arrow go where it needs to, or make two elements connect with a line. But it's a decent starting point - enough for me to cleanup by hand.

The best diagram outputs an LLM can get to are still crap by my standards. It helps speed me up a bit and give me starting points, but it's still disappointing that their visual reasoning and graphic design skills are really weak. The layouts and visual language are always predictably boring too

My theory here is the major labs have done very little or no RLHF on visual explanations and diagrams? Seems like a blind spot. Image generation alone or text generation alone are very well optimised but the combination seems uniquely hard for current models.

It’s been a while But there are markup languages for diagrams like mermaid. Not sure if there are any styling mechanisms for them similar to css. Perhaps a mermaid > structured HTML then style that with css?

One way I got to some interessting results was think in a more multi modal way. A prompt is translated into a blueprint diagram, then translate into node based logic gates structure (they are good at that), and then render the outcome. This one was prompt to new game.

We’ve reached identical conclusions. If you figure this out I’d be very keen to hear how you did it!

Gemini / Nano Banano became the best at it with the recent large update, any context I'm missing on why to use Claude here?

Using Claude because I need these to be in working code – SVG, CSS, HTML, JS. It allows me to have have fine-grained control over details. And I usually add in animations and interactions, plus I need them to be responsive for mobile. Nano banan still has a ways to go 😄 bsky.app/profile/magg...

Maggie Appleton@maggieappleton.com

I hadn't tried out Nano Banana for diagrams yet so I had a little play. Wonderfully promising and hilariously crap in the details. So close and yet so far.

What until it can work in layers, or generate native photoshop, illustrator files. Then we're all c̶o̶o̶k̶e̶d̶ unlocked for utopian "understand through diagrams" future (working on my ai optimism).

I've kind of enjoyed what it Opus 4.5 produces out to excalidraw diagrams. However, it does end up blocky pages in structure instead of flows similar to wireframes. It also struggles to leave enough room for text when wrapped in boxes

The dream is to use AI to get started with a set theme .. something you can point the AI at (url, screenshot, etc) .. prompt it and get a first pass. But the iteration step is tricky. More prompting? At that point I want to take over in a tool (PS, Illustrator) and get what I want. Or code.

For my use cases, Claude worked best at generating Mermaid diagrams ( which you can then render however you want). Downside was that they’re pretty ugly by default.

Yeah I've used mermaid diagrams before, but my understand is they're very simple and standardised? The final output I'm going for with is pretty refined. These are SVG diagrams for one of my essays so I need a lot of control over the visual aesthetics. And I usually add in animation and interaction

Yeah I hate the default themes but it is customizable, though not to the standard I expect from your essays. I wonder if it could work as a first step and then grab svg.

Oh I didn't know it was customisable at all! I'll take a look. Maybe models will be better at generating initial ideas with a more structured language like that, and I can work off it

SVG and CSS. I gave it some existing examples from my blog and style files (global CSS vars, etc.). Plus some guidelines: be subtle, use soft shadows and round borders, use these colours Agree it's not bad as a starting point. And seeing what I dislike helps me take it in a better direction.

Thanks! That actually raises a deeper question. If we take something “slightly off” from AI and fix it into “good,” do we lose the path where something “slightly off” started by us becoming “great” only through our hands? I wonder if starting with AI sets us on a lower quality curve from the start.

Yeah I thought this too – that starting with mediocre, predictable stuff narrows the range of exploration substantially. We can ward this off a bit by having high standards and/or a specific vision for the final result pre-prompting. There's a tradeoff calc here: easier/faster but slightly worse