Post

Will Nelson
@wnelson.dev

Today I shipped a huge change to syntax highlighting in Discord: code blocks and file previews on web are now being highlighted by Tree-sitter, instead of highlight.js.

Will Nelson
@wnelson.dev

Why, you might ask? highlight.js is a genuinely great project, but it suffers from one great weakness: regular expressions. This comes with 2 main downsides: RegEx is often vulnerable to denial-of-service, and it has no semantic understanding of the code.

Will Nelson
@wnelson.dev

highlight.js is fairly resilient to ReDoS, but even so we'd occasionally have to field some security reports about it. Tree-sitter is not immune to DoS either, but we can tune it to protect against DoS in ways that aren't possible with RegEx.

0 reposts8 likes

Will Nelson
@wnelson.dev

More importantly, however, tree-sitter is semantically aware, meaning that it can highlight code in context rather than based on keywords and pattern matching. This means that your code blocks in Discord are now as accurate as the best code editors. For example, None is not a keyword anymore.

Will Nelson
@wnelson.dev

Shout-out to the tree-sitter contributors and the hundreds of independent grammar authors for creating such a strong ecosystem, and shout-out @fasterthanli.me for building Arborium (the basis for our new syntax highlighter).

Noa 🪻
@noahbogart.com

i'm super interested to add some grammars. would you be willing to put up some documentation on how to add one via PR?

leaf
@iamleaf.bsky.social

this broke the css classes going from `{alias}` to something like `language-{main}`. example being `hbs` => `language-html`. they have different use cases. please, at the very least revert that change. the `language-` prefix doesn't matter here. keep the aliases separate.

Crimson Ghostpaw ♡
@gh0stp4wz.xyz

So we're just gonna use formatting for similar languages now instead of actually using the correct highlighter? that sure isn't gonna cause confusion. I can imagine the CSS is gonna be hell to deal with for this…