@alexdima are these before/after measurements in milliseconds or something else?

Do you happen to have millisecond measurements, or info on how you measured? Curious how this approach compares to https://github.com/trishume/syntect :)

Also curious if you have any ideas why bootstrap.min.css became 4.1x slower on VS Code Desktop?

Came here from link in release notes.

I have massive respect for you @alexdima + vscode team's ever prioritization of performance. Really cool that this uses webassembly under the hood. Wow!

This is insane! Keep it up!!

@slimsag Yes, the measurements are in millis. I got them by running F1 > Developer: Force Retokenize which flushes any cached tokens and synchronously (without yielding) tokenizes the entire file top-bottom. It then prints the time taken to the developer tools... [Normally tokenization runs for some time, then yields to allow the event loop to process events, then tokenizes again, etc].

https://github.com/trishume/syntect looks like a great project. I'm not sure how times can be compared objectively. The current JS TM grammar that VS Code ships with is not written with speed in mind (it is basically the entire TS TM grammar with some minor things removed). Philosophically, it is also written more like a parser, which leads sometimes to some extraordinarily complex regular expressions -- e.g.). Matching such regular expressions is no breeze for any regex engine:

Regarding bootstrap.min.css, I honestly didn't have a lot of time to dig deep into it, since I was so happy with the wins for the regular files. Minified code is something very specific to the web ecosystem (CSS/JS/HTML), and VS Code often cheats by ignoring such long lines. There is a setting to forcefully enable parsing of those, but that is used rarely in practice, since most folks end up opening a minified file due to a breakpoint or to check something out, and rarely have the intention of really editing the minified code.