Post

Lukas Wirth
@lukaswirth.dev

Turns out the TokenStream implementation of rust-analyzer's proc-macro server is super slow when it comes to one of the most common proc-macro operations: concatenation. I noticed that in Zed's codebase, there were ~4 derives that just straight up together took 8 seconds to expand in rust-analyzer.

Lukas Wirth
@lukaswirth.dev

All that proc-macro was doing though is `quote!`ing a bunch, not much more. So I spent the day rewriting the implementation of TokenStream used by the server to make use of an immutable shared tree instead to get super cheap concatenations.

0 reposts36 likes

Lukas Wirth
@lukaswirth.dev

And sure enough, this changed the 8 second expansion time into a 100ms! A bit more info can be found in the PR description here github.com/rust-lang/ru.... This should improve general speed of rust-analyzer in the majority of projects given how pervasive these kinds of macros are in Rust.

github.comproc-macro-srv: Reimplement token trees via immutable trees by Veykril · Pull Request #21097 · rust-lang/rust-analyzerFor macros, rust has the concept of TokenTrees and TokenStreams which are basically a tree-like form of the underlying tokenized input with (){}[] delimiters forming Groups (internal nodes) and all...
Lukas Wirth
@lukaswirth.dev

Oh almost forgot! For the folks using RustRover, you will benefit from this too as RustRover uses the proc-macro server for its proc-macro expansion as well!

Nate Butler
@nate.rip

Err, I’m sorry, and thank you? I might be responsible for some of those 🤣 That is absolutely incredible though, great work.

Lukas Wirth
@lukaswirth.dev

Thanks! Funnily enough the macros in Zed themselves aren't really that problematic. They don't do a lot of quote!ing so they are mostly fine. Except for the `derive_inspector_reflection` which I straight up disabled for r-a's analysis because it took like 16 seconds alone github.com/zed-industri...

github.comReduce macro burden for rust-analyzer by Veykril · Pull Request #42871 · zed-industries/zedThis enables optimizations for our own proc-macros as well as some heavy hitters. Additionally this gates the derive_inspector_reflection to be skipped for rust-analyzer as it currently slows down...
Nate Butler
@nate.rip

Honestly, ever since the brutal recompile experience after working in the gpui crate I’ve more or less stopped using proc macros. Super excited to try these again if you’re seeing this type of performance improvement with your changes!