EDIT: Apologies for the somewhat messy PR; perhaps conceptually some of these should go into different PRs. However, the changes are so small that I figured it was perhaps best to include them here as well.

Commit summary

Fix BQN lexer treating special characters as word chars

\w matches all alphanumeric Unicode characters, including ones (e.g., 𝕊) that BQN treats special. This is especially troublesome for variables; previously, something like

𝕊i

would have returned

(Token.Operator, '𝕊'),
(Token.Error, 'i'),

instead of

(Token.Operator, '𝕊'),
(Token.Name.Variable, 'i').

This extends to special sequences like \b, which care about the difference between \w and \W.

There are still some \b's in the code, but from what I can tell these don't produce any undesirable effects. E.g., for 2-modifiers, the BQN grammar already disallows writing something like F_𝕣_G instead of F _𝕣_ G. No guarantees that I caught everything, of course.

This is, of course, nonsense, there are plenty of situations where this could arise that are accepted by BQN; e.g., _F ← {𝔽_𝕣}, _D ← {𝔽_F𝕩}, etc. See the fixup commit.

BQN lexer: Parse _𝕣 and 𝕣 correctly

BQN Lexer: Replace function * with ⋆

BQN does not actually use * (ASTERISK) anywhere, but there is a primitive function ⋆ (STAR OPERATOR).

BQN Lexer: Allow underscores in numbers

As per the spec.