feat: add lookup_token_approx for lossless sourcemap composition#392
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an “approximating” lookup API to SourceMap to support lossless sourcemap composition when a remapping query lands before the first token on a generated line (e.g., due to indentation). This preserves mappings in common composer workflows while keeping the existing lookup_token contract unchanged.
Changes:
- Add
SourceMap::lookup_token_approxto clamp “before-first-token” queries to the first token on that line. - Add
SourceMap::lookup_source_view_token_approx(andOwnedSourceMapforwarding methods) for theSourceViewTokenwrapper use-case. - Add targeted tests covering clamping behavior, non-clamping exact matches, empty/out-of-range handling, and source/name/content propagation.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/sourcemap.rs | Implements lookup_token_approx / lookup_source_view_token_approx and adds comprehensive tests for composition-oriented clamping behavior. |
| src/owned_sourcemap.rs | Exposes the new approx lookup APIs on OwnedSourceMap by delegating to the inner SourceMap. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Merging this PR will improve performance by 3.75%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | lookup_token[real_small] |
698.9 ns | 669.7 ns | +4.36% |
| ⚡ | lookup_token[real_medium] |
953.3 ns | 924.2 ns | +3.16% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing hyf0:fix/lookup-token-approx (f0e21c5) with main (6a54fed)
Footnotes
-
7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
…es::Boundary workaround Switch `collapse_sourcemaps` to `lookup_source_view_token_approx` (oxc-project/oxc-sourcemap#392): when a token's position falls before the first token on its line in the previous map (an indented line), clamp to that first token instead of dropping the mapping — the same approximation Rollup's `collapseSourcemaps` does. This fixes the dropped-mapping class behind #10070 across all collapse paths, so the module-level `Hires::Boundary` workaround from #10074 is removed. Snapshot changes: - 5 fixtures recover previously-dropped mappings (topics/deconflict/*, module_types/copy/*, misc/wrapped_esm). - The two mutation-path fixtures (issue_10070, mix-cjs-esm) move from column-level to line-level source precision, since the mutation map is no longer hi-res. Depends on oxc-project/oxc-sourcemap#392, pulled via [patch.crates-io] git dependency; not for merge until that lands in a published oxc_sourcemap.
lookup_token returns None when the queried column is before the first token on a line. That's correct for a plain position lookup but drops mappings when composing sourcemaps: an indented line's first token sits after the indent, so a column-0 query misses and the line loses its mapping. Add lookup_token_approx / lookup_source_view_token_approx, which clamp to the line's first token in that case, matching Rollup's collapseSourcemaps traceSegment. lookup_token is unchanged.
…sition test Model the full rolldown/rolldown#10070 pipeline in one integration test: codegen indents a line so its first anchor sits at column 1, the bundler samples the moved line at column 0, and collapsing the chain resolves the bundle token through the codegen map. The strict lookup drops the mapping (asserted); the approximating lookup keeps it. The composed map then round-trips through JSON and resolves the bundle position back to the original source via OwnedSourceMap, covering the delegation as well. Also align the new method's binary search with lookup_token (dst_col-only key, oxc-project#385).
34c65a7 to
f0e21c5
Compare
What
Adds
SourceMap::lookup_token_approx(+lookup_source_view_token_approx): likelookup_token, but when the queried column is before the first token on a line, it clamps to that first token instead of returningNone.lookup_tokenis unchanged.Why — with vs. without
lookup_tokenreturns the greatest token<= (line, col), so a query before a line's first token returnsNone. Correct for a plain position lookup — but it silently drops mappings when composing two sourcemaps, and it only surfaces on indented lines.Downstream, this is exactly what bit rolldown (rolldown/rolldown#10070): every indented statement that went through map composition lost its mapping, so a debugger could no longer jump from it back to source.
Prior art — Rollup's
collapseSourcemapsdoes exactly thisRollup's map composer approximates in this same situation instead of dropping the segment —
src/utils/collapseSourcemaps.tsL128-L136:When the column precedes the line's first segment (
searchStart === 0andsegments[0][0] > column),Math.max(0, searchStart - 1)clamps to index0— the line's first segment.lookup_token_approxreturns that same first token; for in-range columns it matcheslookup_token(Rollup's exact-hit path). An empty/out-of-range line returnsNonein both.Notes
lookup_token's "Nonebefore the first token" contract (and its test) is untouched.lookup_tokenand this method assume tokens are sorted by(dst_line, dst_col)— an existing invariant fromgenerate_lookup_table; the approx variant adds no new assumption.None(nothing to clamp to).Tests
A single end-to-end test,
tests/lookup_token_approx.rs::compose_sourcemaps_with_approx_lookup_keeps_indented_lines, models the whole #10070 pipeline: codegen indents a line (first anchor at column 1), the bundler samples the moved line at column 0, and collapsing the chain resolves the bundle token through the codegen map — asserting the strict lookup drops the mapping while the approximating lookup keeps it. The composed map then round-trips through JSON and resolves the bundle position back to the original source viaOwnedSourceMap(covering the delegation too).(Maintainer edit: rebased on main, replaced the original unit tests with the e2e test above, and aligned the new method's binary search with #385's
dst_col-only key.)