Skip to content

fix(shiftjis): decode the complete EUDC block - #445

Merged
bjohansebas merged 2 commits into
pillarjs:masterfrom
b84330808:fix-shiftjis-eudc-row-f9
Nov 7, 2026
Merged

bjohansebas merged 2 commits into
pillarjs:masterfrom
b84330808:fix-shiftjis-eudc-row-f9

Conversation

@b84330808

@b84330808 b84330808 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

The generated Shift_JIS table stops the EUDC range at pointer 10528 (bytes F9 40), leaving 187 valid byte pairs in the last row unmapped. Extend it through pointer 10715 (F9 FC) so the full block decodes to U+E000–U+E757, as required by the Encoding Standard.

For example, decoding F9 FC 41 now produces U+E757 followed by ASCII A, instead of U+FFFD followed by A. EUDC characters remain decode-only: encoding uses the configured replacement byte or invokes invalidCharHandler.

Changes

  • Update the generator's upper bound and regenerate encodings/tables/shiftjis.js. Extend encodeSkipVals through 0xF9FC to preserve encoding behavior.
  • Check all 1,880 EUDC mappings and their default encoding fallback. Cover handler replacements, skipped characters, reported positions and configurable replacement bytes.
  • Add streaming cases for the block boundaries, the last row, invalid trailing bytes and incomplete input. Verify following ASCII bytes and every chunk boundary with both backends and webpack/Chrome.
  • Include the 1,880 algorithmic EUDC mappings in the WPT decode comparator, which previously checked only assigned index entries and half-width katakana. Update its documentation and the changelog.

Validation

Local validation with Node.js 26.10.0 and Chrome Headless 156:

  • npm run test:ci: 803 passing, 8 pending per backend. Combined line coverage: 98.52%; all configured coverage thresholds pass.
  • npm run test:webpack: 730 passing, 8 skipped in Chrome.
  • WPT decode comparator: 9,667 Shift_JIS cases pass per backend. Restoring the old table reproduces exactly 187 failures.
  • npm run lint and git diff --check: pass.
  • Offline regeneration from the vendored JIS0208 index reproduces both the original and updated table payloads exactly.
  • Compared all 65,536 two-byte inputs and 1,112,064 Unicode scalar values per backend: decoding changes only for the 187 expected pairs; encoding output is unchanged.

The eight pending cases are the existing SBCS tests without reference data.

The original implementation was written with Claude Code.

b84330808 and others added 2 commits November 7, 2026 13:30
The Shift_JIS decoder in the Encoding Standard maps pointers 8836 to 10715
(0xF040 to 0xF9FC) to U+E000 to U+E757. The table generator stopped at
pointer 10528, so in the last EUDC row only F940 was mapped and F941 to F9FC
decoded to U+FFFD followed by the trail byte read as a new character.

Use the spec's upper bound and regenerate the table; 187 byte pairs change
and nothing else does. Extend encodeSkipVals to the end of the block so the
new pairs stay decode-only, like the rest of the EUDC block.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@bjohansebas
bjohansebas force-pushed the fix-shiftjis-eudc-row-f9 branch from d72e650 to 4be5c4e Compare November 7, 2026 18:36
@bjohansebas bjohansebas changed the title fix: decode the whole Shift_JIS EUDC block fix(shiftjis): decode the complete EUDC block Nov 7, 2026
@bjohansebas

Copy link
Copy Markdown
Member

Thanks so much for the PRs, they’re really useful!

@bjohansebas
bjohansebas merged commit b1e99b6 into pillarjs:master Nov 7, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants