You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 1b79558
Browse filesBrowse the repository at this point in the historyBrowse files
fix(grammar): accept a lone '&' in JSX strings and text (#1736)
A plain '&' inside JSX -- a URL query in an attribute
(href="https://fonts.googleapis.com/css2?family=Inter&display=swap"),
or text such as <p>Tom & Jerry</p> -- produced a tree-sitter ERROR node
in .js, .jsx and .tsx files. The file was flagged parse_partial in the
coverage view, and the error recovery turned words from the URL
(create-next-app -> create/next/app) into identifiers that resolved to
unrelated symbols as false USAGE edges.
Root cause is upstream tree-sitter-javascript, which the tsx dialect
builds on: unescaped_{double,single}_jsx_string_fragment is
/([^"&]|&[^#A-Za-z])+/, so '&' is only accepted before a space/digit
or as a complete html_character_reference (& / &). jsx_text is
scanned by the external scanner, which stops at every '&', and nothing
in _jsx_child accepted a '&' that is not a reference. So '&' + letter in
a string, and any lone '&' in JSX text, had no valid token.
Fix: javascript and tsx become self-maintained forks of the pins we
already vendored (javascript @ 58404d8cf191; typescript tsx @
75b3874edb2d on its javascript base @ 3a837b6f3658), following the
arkts model. One patch (tools/tree-sitter-javascript/patches/
jsx-lone-ampersand.patch) is applied to both javascript bases:
- a one-character '&' token at lexical precedence -1, aliased to the
existing string_fragment (in strings) and jsx_text (in text); the
lower precedence keeps & / & lexing as html_character_reference
- the string-fragment regexes exclude a '&' directly before the closing
quote, so "a&" no longer runs past the end of the string
Node types are unchanged (same type/named set). The external scanners
stay upstream bytes. Parsers regenerated with tree-sitter-cli 0.25.10
(ABI 15; tsx moves from ABI 14, and its tree_sitter/ headers now match
arkts). Without the patch both fork dirs regenerate upstream exactly.
The upstream corpus passes unchanged: javascript 116/116; tsx gives the
same results as unpatched upstream.
Proof on vercel/next.js examples/ (canary 725705bd, 1,722 js/jsx/tsx
files): parse_partial 92 -> 47 (tsx 33 -> 0, jsx 2 -> 0, js 12 -> 2; the
two left are Babel do-expressions and Flow types), no newly flagged
file, code-graph nodes 14,739 -> 14,739, edges 24,773 -> 24,657: the
115 USAGE edges removed were all false cross-example edges from URL
fragments, plus a 4 SIMILAR_TO / +3 SEMANTICALLY_RELATED reshuffle
from the corrected ASTs.
Registered like arkts: MANIFEST.md fork rows + notes (ABI tally
recounted), audit FORKS map, THIRD_PARTY.md, vendored checksums.
LICENSEs stay upstream's MIT byte-identical.
Tests: parse_coverage gains a .js/.jsx/.tsx matrix (double/single-quoted
attribute, '&' before the quote, '&' + letter/space/digit in text, plus
&/& controls) and a guard that broken JSX is still flagged.
Signed-off-by: Martin Vogel <[email protected]>
- ABI distribution: **9×** ABI-13 **78×** ABI-14 **75×** ABI-15 (runtime ceiling is ABI 15; never vendor ABI 16 without a runtime upgrade)
14
+
— recounted from the tree 2026-09-25 after the tsx fork was regenerated with tree-sitter-cli 0.25.10, moving `tsx` from ABI 14 to ABI 15 (was `9×/79×/74×`).
14
15
— recounted from the tree 2026-08-30 after the perl v1.2.1 refresh moved `perl` from ABI 14 to ABI 15 (was `9×/80×/73×`). Neither side of the rebase had this right: main's line was correct for main, and this branch still carried the pre-2026-08-28 `7×/84×/65×`. Regenerate, never increment.
15
16
— recounted from the tree 2026-08-28. This line had drifted: it read `7×/86×/65×`, which sums to 158 against 161 vendored grammars, so it was wrong before Chialisp was added and incrementing it would have carried the error forward. Regenerate with:
@@ -32,6 +33,7 @@ The grammars were originally vendored as bare `parser.c`+`scanner.c` with **no r
32
33
- **objectscript_udl / objectscript_routine** (added 2026-06-24): vendored from [intersystems/tree-sitter-objectscript](https://github.com/intersystems/tree-sitter-objectscript) @ `a7ffcdf` — MIT, the InterSystems-official grammars (a niche vendor language, hence `vendor-maintained`, not in nvim-treesitter/Helix). **Re-vendor note:** each `scanner.c`'s upstream `#include "../../common/scanner.h"` is repointed to a per-directory `objectscript_common.h` (a verbatim copy of upstream `common/scanner.h`), because this repo's shared `vendored/common/scanner.h` belongs to the cfml/fsharp grammars and differs. The generated `parser.c`/`scanner.c` are otherwise byte-for-byte upstream — on re-vendor, re-apply only that single include rename. **Local modification (2026-07-16):** in `objectscript_common.h`, two loop counters `uint8_t i` were widened to `int i` (the `reverse_marker` scan and the `html_marker_buffer` reversal) to clear CodeQL `cpp/comparison-with-wider-type` — a false positive in practice (both lengths are hard-bounded by `MARKER_BUFFER_MAX_LEN = 30`, so `uint8_t` could never wrap), fixed for cleanliness. On re-vendor, re-apply this widening too (or upstream it at intersystems/tree-sitter-objectscript).
33
34
-**mojo** (added 2026-07-01): vendored from [lsh/tree-sitter-mojo](https://github.com/lsh/tree-sitter-mojo) @ `33193a99afe6` — MIT, ABI 15. Helix tracks `lsh/tree-sitter-mojo` as its Mojo grammar source, but the Helix-pinned commit (`3d7c53b8038f`) no longer resolves in the upstream repository after a force-push, so this vendor uses current upstream `main` rather than the stale registry SHA. Security review covered only the vendored C surface (`parser.c`, `scanner.c`, `tree_sitter/*.h`) plus upstream license/provenance metadata; no package manager hooks, workflow files, prompt/agent instruction files, or generated lockfiles were vendored.
- **javascript + tsx** (forked 2026-09-25, #1736): **self-maintained forks** of the pins we already vendored — [tree-sitter/tree-sitter-javascript](https://github.com/tree-sitter/tree-sitter-javascript) @ `58404d8cf191` and [tree-sitter/tree-sitter-typescript](https://github.com/tree-sitter/tree-sitter-typescript)'s `tsx` dialect @ `75b3874edb2d` (javascript base @ `3a837b6f3658`, the version upstream's package-lock pins, as for arkts). **Why:** upstream accepts a `&` inside a JSX string only before a space/digit or as a complete character reference, and never inside JSX text, so a plain URL query (`href="…?a=1&b=2"`, Google Fonts `&display=swap`) or `<p>Tom & Jerry</p>` produced an ERROR node — a `parse_partial` flag, plus false USAGE edges from identifiers the error recovery pulled out of the URL (on vercel/next.js `examples/`: 45 of 92 flagged files, 115 false cross-example USAGE edges). **The change** is one patch, `tools/tree-sitter-javascript/patches/jsx-lone-ampersand.patch`, applied to both javascript bases: a lone `&` that does not begin a complete `html_character_reference` is a one-character token (lexical precedence -1, so `&`/`&` still lex as references) aliased to the EXISTING `string_fragment` / `jsx_text` nodes, and the string-fragment regexes stop before a `&` that directly precedes the closing quote. Node types are unchanged (same type/named set; `node-types.json` only gains an empty `fields` map on those two entries). Without the patch both directories regenerate upstream exactly (javascript: byte-identical `parser.c` apart from the generator-version comment; tsx: identical state/symbol counts, ABI 14 → 15 from the newer CLI). **External scanners** are unchanged: `javascript/scanner.c` and `tsx/scanner.c` + `_common_scanner.h` stay the reviewed upstream bytes. Regenerate with `npx [email protected] generate` in `tools/tree-sitter-{javascript,tsx}/` (ABI 15) and copy `src/parser.c` (tsx: also `src/tree_sitter/*.h`) here. **On an upstream bump:** re-fetch the new pins, re-apply the patch, regenerate, re-run `test/corpus/jsx_ampersand.txt` — and drop the fork if upstream fixes it. **LICENSE:** upstream's MIT text byte-identical (registered in the audit's `FORKS` map); our copyright for the patch lives in the fork `grammar.js` headers and `THIRD_PARTY.md`.
35
37
- **plsql** (added 2026-08-27): vendored from [AndreasMaierDe/tree-sitter-plsql](https://github.com/AndreasMaierDe/tree-sitter-plsql) @ `28aebef209be` (full: `28aebef209be57169600e1aa41ca1431cc6c916f`, upstream tip; repo dormant since 2023-02) — MIT, ABI 14, **no external scanner** (`EXTERNAL_TOKEN_COUNT 0`). Not listed in nvim-treesitter/Helix (`community-niche`, an individual-maintained grammar); provenance verified directly against upstream (pin = `git ls-remote` HEAD; vendored `parser.c` byte-identical to the pinned clone apart from the include-quote local patch below; 0 non-ASCII bytes; 0 dangerous calls). Security review covered only the vendored C surface (`parser.c`, `tree_sitter/parser.h`) plus upstream license/provenance metadata; no package manager hooks, workflow files, prompt/agent instruction files, or generated lockfiles were vendored. Known upstream limitation: `CREATE TYPE ... AS OBJECT` currently yields ERROR nodes (pinned by `plsql_create_type_as_object_limitation` in `tests/test_extraction.c` + `tests/fixtures/plsql/create_type_as_object_limitation.tps`). Originally contributed as PR #1033 by Oğuz (@ouzsrcm); re-vendored from upstream per vendoring policy with the PR's language wiring distilled on top.
36
38
37
39
- **chialisp** (added 2026-08-28): **first-party** — authored in this repository, not vendored from anywhere. Grammar source + corpus tests live in `tools/tree-sitter-chialisp/`; regenerate with `npx [email protected] generate --abi 14` and copy `src/parser.c` + `src/tree_sitter/*.h` here. ABI 14, **no external scanner** (`EXTERNAL_TOKEN_COUNT 0`), 0 non-ASCII bytes. It is a deliberately GENERIC s-expression grammar (`source_file`/`list`/`symbol`/`string`/`number`/`hex`/`dot`/`comment`) modelling the clvm_tools reader rather than the Chialisp form vocabulary: `mod`/`defun`/`defconstant`/`include` are ordinary head symbols, and which lists are definitions is decided in `internal/cbm/extract_defs.c`, so a dialect that adds a form does not need a regenerated parser. Written because the only public grammar (`Quexington/tree-sitter-chialisp`) cannot parse the language: it required CRLF to terminate a comment (`/;.*\r\n/`) while real files are LF, rejected the `.` in `(include foo.clib)`, and accepted only a primitive after `(defconstant NAME …)` — each of which desynchronised the rest of the file. Acceptance gate: all five `chia-blockchain@main` reference files parse with **zero ERROR and zero MISSING nodes**. **LICENSE:** the project's own LICENSE, byte-identical to the repository root — no third-party copyright is carried, because there is no third party.
@@ -265,6 +265,8 @@ upstream author (correctly retained). The table now records the true origin.
265
265
| grammar | cur ABI | original upstream | license |
266
266
|---|:---:|---|---|
267
267
| arkts | 15 |**self-maintained fork** of tree-sitter/tree-sitter-typescript @ `75b3874edb2d` (v0.23.2; javascript base @ `3a837b6f3658` v0.23.1); grammar source in `tools/tree-sitter-arkts/`| MIT, (c) 2017 Max Brunsfeld — byte-identical to upstream; ArkTS additions (c) 2026 DeusData, see grammar.js + THIRD_PARTY.md |
268
+
| javascript | 15 |**self-maintained fork** of tree-sitter/tree-sitter-javascript @ `58404d8cf191` (the previously vendored pin, VERIFIED-BOTH) + the JSX lone-ampersand patch (#1736); grammar source in `tools/tree-sitter-javascript/`| MIT, (c) 2014 Max Brunsfeld — byte-identical to upstream; patch (c) 2026 DeusData, see grammar.js + THIRD_PARTY.md |
269
+
| tsx | 15 |**self-maintained fork** of tree-sitter/tree-sitter-typescript's `tsx` dialect @ `75b3874edb2d` (v0.23.2, the previously vendored pin, VERIFIED-BOTH; javascript base @ `3a837b6f3658` v0.23.1) + the same JSX lone-ampersand patch (#1736); grammar source in `tools/tree-sitter-tsx/`| MIT, (c) 2017 Max Brunsfeld — byte-identical to upstream; patch (c) 2026 DeusData, see grammar.js + THIRD_PARTY.md |
268
270
| assembly | 14 | RubixDev/tree-sitter-assembly (**repo deleted from GitHub** — our retained MIT copy, (c) 2023 RubixDev, is the surviving grant) | MIT |
0 commit comments