You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2aab973
Browse filesBrowse the repository at this point in the historyBrowse files
refactor: replace MapOffset tuple with ProbeOffset enum (apache#26203)
## Which issue does this PR close?
- Closesapache#25332 .
## Rationale for this change
`MapOffset` was a `(usize, Option<u64>)` tuple that encoded the resume
point of chunked hash map lookups by convention. One position had two
encodings (`(row, Some(0))` and `(row + 1, None)`), the unique-key fast
paths read only `offset.0`, and `HashJoinStream` relied on an
undocumented invariant (`offset == (0, None)` only before the first
chunk).
## What changes are included in this PR?
- Replace the tuple with `pub enum ProbeOffset { AtRow { row }, MidChain
{ row, next } }` in `join_hash_map.rs`, re-exported from `joins::utils`
next to `JoinHashMapType`.
- Normalize at the producer: `traverse_chain` returns `AtRow { row: row
+ 1 }` once a row's chain is finished, so every position has one
encoding and readers no longer handle `Some(0)`.
| Old | New |
|---|---|
| `(row, None)` | `AtRow { row }` |
| `(row, Some(next))`, `next != 0` | `MidChain { row, next }` |
| `(row, Some(0))` | `AtRow { row: row + 1 }` |
- Unique-key fast paths (`JoinHashMap`, `ArrayMap`) `debug_assert!` that
they only receive `AtRow`. The invariant holds by construction (`unique`
is fixed when the build side is built, and every chunk of a probe batch
uses the same map), so a check in debug builds and CI is enough to catch
a regression without adding a panic path to query execution.
- Add `ProcessProbeBatchState::is_first_chunk()` and use it at the three
first-chunk checks. The invariant it relies on (a returned offset is
never `AtRow { row: 0 }`) is documented on `ProbeOffset`.
## What is the testing strategy for this PR?
Covered by existing tests, with expected offsets updated to the new
encoding.
`test_unique_fast_path_with_uninserted_row` (apache#26018) told the fast path
apart from the chain walk by their different offset encodings, which
this PR removes. This PR added an unmatched probe row so the two paths
now emit different results (the fast path limits by probe rows, the
chain walk by matches), and checked that the test fails when the fast
path is disabled. cc @jayzhan211
## Are there any user-facing changes?
No behavior change.
API change: `JoinHashMapType::get_matched_indices_with_limit_offset` and
the public `get_matched_indices_with_limit_offset` function now take and
return `ProbeOffset` instead of `(usize, Option<u64>)`.
Co-authored-by: Jay Zhan <[email protected]>
0 commit comments