Skip to content

feat: add lookup_token_approx for lossless sourcemap composition#392

Merged
Boshen merged 2 commits into
oxc-project:mainfrom
hyfdev:fix/lookup-token-approx
Jul 3, 2026
Merged

feat: add lookup_token_approx for lossless sourcemap composition#392
Boshen merged 2 commits into
oxc-project:mainfrom
hyfdev:fix/lookup-token-approx

Conversation

@hyfdev

@hyfdev hyfdev commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

What

Adds SourceMap::lookup_token_approx (+ lookup_source_view_token_approx): like lookup_token, but when the queried column is before the first token on a line, it clamps to that first token instead of returning None. lookup_token is unchanged.

Why — with vs. without

lookup_token returns the greatest token <= (line, col), so a query before a line's first token returns None. Correct for a plain position lookup — but it silently drops mappings when composing two sourcemaps, and it only surfaces on indented lines.

Indented generated line (→ = leading tab):

    →globalThis.side = 1;
    ↑↑
    ↑└ col 1 — codegen's first token ("globalThis")
    └ col 0 — the tab, no token here

A composer remaps the line by querying column 0:

    lookup_token(line, 0)         →  None           ❌  the whole line drops out of the map
    lookup_token_approx(line, 0)  →  token @ col 1  ✅  the line stays mapped

Downstream, this is exactly what bit rolldown (rolldown/rolldown#10070): every indented statement that went through map composition lost its mapping, so a debugger could no longer jump from it back to source.

Prior art — Rollup's collapseSourcemaps does exactly this

Rollup's map composer approximates in this same situation instead of dropping the segment — src/utils/collapseSourcemaps.ts L128-L136:

// If a sourcemap does not have sufficient resolution to contain a
// necessary mapping, e.g. because it only contains line information or
// the column is not precise (e.g. the sourcemap is generated by esbuild, segment[0] may be shorter than the location of the first letter),
// we approximate by finding the closest segment whose segment[0] is less than the given column
if (segment[0] !== column && searchStart === searchEnd) {
    const approximatedSegmentIndex =
        segments[searchStart][0] > column ? Math.max(0, searchStart - 1) : searchStart;
    segment = segments[approximatedSegmentIndex];
}

When the column precedes the line's first segment (searchStart === 0 and segments[0][0] > column), Math.max(0, searchStart - 1) clamps to index 0 — the line's first segment. lookup_token_approx returns that same first token; for in-range columns it matches lookup_token (Rollup's exact-hit path). An empty/out-of-range line returns None in both.

Notes

  • Additive only; lookup_token's "None before the first token" contract (and its test) is untouched.
  • Both lookup_token and this method assume tokens are sorted by (dst_line, dst_col) — an existing invariant from generate_lookup_table; the approx variant adds no new assumption.
  • Empty line / line past the table still returns None (nothing to clamp to).

Tests

A single end-to-end test, tests/lookup_token_approx.rs::compose_sourcemaps_with_approx_lookup_keeps_indented_lines, models the whole #10070 pipeline: codegen indents a line (first anchor at column 1), the bundler samples the moved line at column 0, and collapsing the chain resolves the bundle token through the codegen map — asserting the strict lookup drops the mapping while the approximating lookup keeps it. The composed map then round-trips through JSON and resolves the bundle position back to the original source via OwnedSourceMap (covering the delegation too).

(Maintainer edit: rebased on main, replaced the original unit tests with the e2e test above, and aligned the new method's binary search with #385's dst_col-only key.)

@hyfdev
hyfdev marked this pull request as ready for review July 2, 2026 02:16
Copilot AI review requested due to automatic review settings July 2, 2026 02:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an “approximating” lookup API to SourceMap to support lossless sourcemap composition when a remapping query lands before the first token on a generated line (e.g., due to indentation). This preserves mappings in common composer workflows while keeping the existing lookup_token contract unchanged.

Changes:

  • Add SourceMap::lookup_token_approx to clamp “before-first-token” queries to the first token on that line.
  • Add SourceMap::lookup_source_view_token_approx (and OwnedSourceMap forwarding methods) for the SourceViewToken wrapper use-case.
  • Add targeted tests covering clamping behavior, non-clamping exact matches, empty/out-of-range handling, and source/name/content propagation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/sourcemap.rs Implements lookup_token_approx / lookup_source_view_token_approx and adds comprehensive tests for composition-oriented clamping behavior.
src/owned_sourcemap.rs Exposes the new approx lookup APIs on OwnedSourceMap by delegating to the inner SourceMap.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@codspeed-hq

codspeed-hq Bot commented Jul 2, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 3.75%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
✅ 18 untouched benchmarks
⏩ 7 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
lookup_token[real_small] 698.9 ns 669.7 ns +4.36%
lookup_token[real_medium] 953.3 ns 924.2 ns +3.16%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing hyf0:fix/lookup-token-approx (f0e21c5) with main (6a54fed)

Open in CodSpeed

Footnotes

  1. 7 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

hyfdev added a commit to rolldown/rolldown that referenced this pull request Jul 2, 2026
…es::Boundary workaround

Switch `collapse_sourcemaps` to `lookup_source_view_token_approx`
(oxc-project/oxc-sourcemap#392): when a token's position falls before the
first token on its line in the previous map (an indented line), clamp to that
first token instead of dropping the mapping — the same approximation Rollup's
`collapseSourcemaps` does. This fixes the dropped-mapping class behind #10070
across all collapse paths, so the module-level `Hires::Boundary` workaround
from #10074 is removed.

Snapshot changes:
- 5 fixtures recover previously-dropped mappings (topics/deconflict/*,
  module_types/copy/*, misc/wrapped_esm).
- The two mutation-path fixtures (issue_10070, mix-cjs-esm) move from
  column-level to line-level source precision, since the mutation map is no
  longer hi-res.

Depends on oxc-project/oxc-sourcemap#392, pulled via [patch.crates-io] git
dependency; not for merge until that lands in a published oxc_sourcemap.
@Boshen Boshen self-assigned this Jul 2, 2026
hyfdev and others added 2 commits July 2, 2026 22:48
lookup_token returns None when the queried column is before the first
token on a line. That's correct for a plain position lookup but drops
mappings when composing sourcemaps: an indented line's first token sits
after the indent, so a column-0 query misses and the line loses its
mapping.

Add lookup_token_approx / lookup_source_view_token_approx, which clamp to
the line's first token in that case, matching Rollup's collapseSourcemaps
traceSegment. lookup_token is unchanged.
…sition test

Model the full rolldown/rolldown#10070 pipeline in one integration test:
codegen indents a line so its first anchor sits at column 1, the bundler
samples the moved line at column 0, and collapsing the chain resolves the
bundle token through the codegen map. The strict lookup drops the mapping
(asserted); the approximating lookup keeps it. The composed map then
round-trips through JSON and resolves the bundle position back to the
original source via OwnedSourceMap, covering the delegation as well.

Also align the new method's binary search with lookup_token (dst_col-only
key, oxc-project#385).
@Boshen
Boshen force-pushed the fix/lookup-token-approx branch from 34c65a7 to f0e21c5 Compare July 2, 2026 14:59
@Boshen
Boshen merged commit 3d6948a into oxc-project:main Jul 3, 2026
8 checks passed
@oxc-guard oxc-guard Bot mentioned this pull request Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants