Core, Spark 4.2: Add PositionDeleteIndex.forEachInRange and use it in vectorized delete checks by JeonDaehong · Pull Request #18027 · apache/iceberg

JeonDaehong · 2026-09-09T12:05:45Z

Summary:

Vectorized reads probe the position delete index once per row, but the positions in a batch are a contiguous ascending range. The index can be traversed once instead.
Significant gain when few columns are projected. Not visible end to end on wide projections, where the delete check is a small share of scan CPU.
Needs a new API: PositionDeleteIndex.forEachInRange

In details:

When this helps. Scans with a narrow projection of integer columns, where the query is scan-dominated rather than aggregation-dominated, and the table is written as fewer, larger files. Under those conditions, the end-to-end query was 13-19% faster in our measurements (up to 29% for a single projected column on a dedicated instance). The gain is largest just below the Roaring array/bitmap boundary at ~6.25% deletes, not at high delete rates. It does not help when a string column is projected (one md5 column costs 8-10 integer columns of decode), when heavy aggregation dominates, or when the table is split into many small files. As a single rule: it is worth it when the delete check is more than ~21% of scan CPU.

ColumnarBatchUtil.buildRowIdMapping and buildIsDeleted call PositionDeleteIndex.isDeleted(pos) once for every row in a batch. Positions within a batch are a contiguous ascending range, so every call repeats the same work: extracting the high and low keys, bounds-checking the bitmap array, and binary searching the container array. None of it is amortized across the batch even though every call lands in the same one or two containers.

This adds PositionDeleteIndex.forEachInRange(posStart, posEnd, consumer), which traverses the deleted positions in a range in ascending order. The start is inclusive and the end exclusive, matching delete(posStart, posEnd) on the same interface. The default implementation falls back to the per-position loop, and BitmapPositionDeleteIndex overrides it to resolve the range against the 32-bit bitmaps it covers and walk each once with RoaringBitmap.forAllInRange. ColumnarBatchUtil takes that path when the scan has no equality deletes; tables with equality deletes keep the existing loop unchanged.

The measurements below were performed locally on 8M rows across 4 files, with one column projected and 3 repetitions per configuration in separate JVMs (delete-check CPU samples, async-profiler ctimer). The results are environment-dependent and are intended as comparative measurements under the same test conditions:

delete density	container	current	patched	reduction
0.5%	array	674	79	8.6x
6.1%	array	873	94	9.3x
7.0%	bitmap	420	113	3.7x
8.0%	bitmap	499	142	3.5x
50%	bitmap	584	225	2.6x

The gain is largest just below the array/bitmap boundary, which is where the removed work — a binary search per row — was most expensive. buildIsDeleted improves 14.9x–18.9x since it does not need to fill the gaps between deleted positions.

Correctness was checked before performance: count(*), sum(id) and min/max(id) over 7 tables are identical for both jars, and tables with equality deletes show no regression.

This touches spark/v4.2 only, to keep the diff reviewable. ColumnarBatchUtil is byte-identical in v3.5, v4.0 and v4.1, so I will backfill those in a follow-up if this direction looks right.

Closes #18026, which has the rest of the measurements — portability across three CPU microarchitectures, local disk and S3, warm and dropped page cache, task parallelism 1 through 16, a distributed cluster, and hardware counters — along with the conditions under which the numbers above do not hold.

… vectorized delete checks

JeonDaehong · 2026-09-09T14:42:33Z

Hello @aokolnychyi @huaxingao

Tagging you as the people who have worked most on these files. No rush at all, a look whenever you have time would be much appreciated.

I tried to make this easy to evaluate rather than something you have to take on trust. The change was measured across delete densities from 0.5% to 50%, on three CPU microarchitectures, and on both local disk and S3. I also used hardware counters to verify the underlying mechanism instead of inferring it from timings, which is actually how I realized my initial explanation for the cost was incorrect. The linked issue also lists the conditions under which these numbers do not hold.

I am happy to adjust the approach if you see a better way forward. If this direction looks right, I will backport it to v3.5, v4.0, and v4.1 in a follow-up PR.

JeonDaehong · 2026-09-11T13:03:03Z

@pvary

Hi, it’s been a while! I hope you’ve been doing well.

I reached out to the Core maintainers to ask for discussion and feedback on this PR, but I imagine everyone has been quite busy. So I wanted to ask if you might be able to take a look at it when you have a chance.

Eduard Tudenhoefner suggested on Slack that we discuss this through the mailing list, so I moved the discussion to the dev@iceberg mailing list. However, I haven’t received much response there yet, so I wasn’t quite sure how best to proceed and thought I’d reach out to you directly.

Sorry for the unexpected mention, and I’d really appreciate it if you could take a look when you have some time. Thank you!

pvary · 2026-09-11T13:50:05Z

@JeonDaehong: Could you please link the letter sent to the dev list?

JeonDaehong · 2026-09-11T13:56:27Z

@pvary

https://lists.apache.org/thread/5v8j1zj2l8ntygph47r0k61xyskxc5o1

Thank you

pvary · 2026-09-11T14:12:39Z

Ahh.. this went into the Spam folder for me 😢

I suggest way sorter letters and descriptions. Or at least a TL;DR; in the beginning otherwise maintainers pressed for time will skip your messages. I often make this mistake myself, but learning....

I ended up putting your description to an LLM, and I got this answer:

So if your query is:
- Narrow projection
- Scan-heavy
- DV-heavy
- Little aggregation/filtering

then you might actually see 13-19% faster queries.

Is this accurate?
You might want to highlight in which use-case your PR helps

JeonDaehong · 2026-09-11T14:32:45Z

@pvary

Ah, thank you for the feedback!
I'll revise it and post it to the mailing list again.
I'd really appreciate it if you could take a look and give me some feedback once I post it there. 😄

JeonDaehong · 2026-09-11T14:51:19Z

@pvary

I’ve resent the email as well! Here is a quick summary for you here too :D

TL;DR — yes, 13-19% is accurate and if anything conservative. But the
summary dropped three conditions, and one of them is backwards: more deletes
is not more benefit. I've rewritten the PR description to lead with when
this helps. Thanks for both points.

Is 13-19% accurate?

It's measured rather than extrapolated, on projections of 5 columns or fewer,
all integer. A scan-only query with a single projected column came out at 29%
on a dedicated instance. At 10 columns or more it falls inside my noise floor
(9.7% median run-to-run spread) and I don't claim anything there.

When it helps


Helps	Narrow projection and the columns are integers
	Scan-dominated query, little aggregation
	Delete rate near the array/bitmap boundary (~6.25%)
	Fewer, larger files
Doesn't help	One md5 string column costs 8-10 integer columns of decode, which sinks it on its own
	Heavy aggregation — the CPU saved is constant, so its share falls to ~3% as the aggregation grows
	Many small files — per-file DV load cost dilutes it (60.7% of scan CPU at 4 files, 49.7% at 488)

The one that's backwards

"DV-heavy" reads as "more deletes, more benefit", and it's the other way
round. The worst case — and so the largest gain — is just below 6.25%,
where the Roaring container is still a sorted array at its deepest. Past that
it flips to a bitmap and gets cheaper on its own. Deleting more rows can make
the scan faster, which is the least intuitive thing I found here.

One rule instead of six

The patch is worth it when the delete check is more than ~21% of scan CPU.
That single threshold predicted the crossover better than column count or
column type did.

Point taken on length — I've added a "when this helps" paragraph at the top of the description, and I'll lead with a TL;DR on the mailing list. Sorry it landed in your spam folder.

pvary · 2026-09-14T07:38:56Z

+      DeleteFilter<InternalRow> deletes,
+      long rowStartPosInBatch,
+      int batchSize) {
+    RowIdMappingBuilder builder = new RowIdMappingBuilder(rowStartPosInBatch, batchSize);


Could we shortcut if there are no deletes?

Done. The fast path now returns null right away when the index is empty, before touching the range.

We tried to optimize performance by checking if the bitmap index is empty before processing each batch, but it actually made things 53% slower because isEmpty() was doing an expensive full traversal under the hood. We’ve reverted the change.There is a cheap $O(1)$ alternative (highLowContainer.size() == 0), but since it touches shared methods outside this specific path, we left it out of this PR. Let me know if you'd like to pursue that in a follow-up!

pvary · 2026-09-14T07:39:50Z

+    }
+
+    /** Appends the live rows after the last deleted position and returns the live row count. */
+    int build() {


this is not a build method.

Yep. Split into appendRemainingLiveRows() and liveRowCount(), so the side effect and the accessor are separate.

pvary · 2026-09-14T07:42:49Z

+   * @param length the number of positions in the range
+   * @param consumer a consumer for the deleted positions in the range
+   */
+  default void forEachInRange(long posStart, int length, LongConsumer consumer) {


Other methods have long posStart, long posEnd. Shall we use that?

Done, and thanks for catching it... this turned out to be more than just a rename.

The old implementation relied on length being an int to guarantee a range spans at most two 32-bit keys, and the comment said so explicitly. With a long end that no longer holds, so the per-key loop is now general, and the inner traversal chunks any sub-range wider than Integer.MAX_VALUE (RoaringPositionBitmap.forEachInRange takes an int length, so a fully covered middle bitmap cannot be expressed in one call).

RoaringPositionBitmap.forEachInRange now follows setRange for parameter names, for the precondition, and for treating an empty range as a no-op, so the two read the same way. Added a test for a range spanning three keys, which is the path that did not exist before.

pvary · 2026-09-14T07:43:27Z

+    if (length <= 0) {
+      return;
+    }


Shall we throw instead?

Done. It now uses the same precondition and message as setRange ("Start position must not exceed end position: [%s, %s)"), and an empty range where start equals end stays a no-op, also matching setRange.

pvary · 2026-09-14T07:45:00Z

+   * {@link #isDeleted(long)} once per position. Implementations backed by a bitmap can locate the
+   * containers covering the range once and walk them, instead of resolving the container for every
+   * position.


No impl details in the javadoc

Done. dropped the paragraph about locating containers, left only the contract.

pvary · 2026-09-14T07:45:42Z

+  @Test
+  void testPositionOnlyPathMatchesPerRowPath() {
+    // the range traversal must agree with a straightforward per-row probe on every batch
+    java.util.Random random = new java.util.Random(20260818L);


nit: no fully qualified name

pvary · 2026-09-14T07:46:26Z

+ * PositionDeleteIndex#isDeleted(long)} reports as deleted in the same range, in ascending order,
+ * for every implementation.
+ */
+public class TestPositionDeleteIndexForEachInRange {


package private

Removed the file and moved the tests into the existing two classes to match their style.

pvary · 2026-09-14T07:46:42Z

+  private static final long BITMAP_SIZE = 0xFFFFFFFFL + 1L;
+
+  @Test
+  public void testEmptyIndex() {


package private - no test prefix for new tests

also moved these into TestRoaringPositionBitmap and TestBitmapPositionDeleteIndex to follow the existing public void testX() convention.

pvary · 2026-09-14T07:47:09Z

+ * PositionDeleteIndex#isDeleted(long)} reports as deleted in the same range, in ascending order,
+ * for every implementation.
+ */
+public class TestPositionDeleteIndexForEachInRange {


Why new test class?
Shall we put these into TestRoaringPositionBitmap, or TestBitmapPositionDeleteIndex?

Thought about creating a new test class, but on second thought it didn't seem right, so I dropped it and split the tests by what they actually exercise

TestRoaringPositionBitmap : container and key boundaries, a range spanning three keys, ranges past the allocated bitmaps, run containers after runLengthEncode, the new precondition, and a randomized comparison against contains.

TestBitmapPositionDeleteIndex : the range contract (inclusive start, exclusive end), ascending order, empty index and empty range, and a randomized comparison against isDeleted for both the bitmap-backed index and the interface default.

org.apache.iceberg.deletes went from 55 to 63 tests.

pvary · 2026-09-14T07:49:39Z

+      IsDeletedBuilder builder = new IsDeletedBuilder(isDeleted, rowStartPosInBatch);
+      deletedPositions.forEachInRange(rowStartPosInBatch, batchSize, builder);
+      for (int index = 0; index < builder.deletedRowCount(); index++) {
+        deletes.incrementDeleteCount();


Could we just call it inside the IsDeletedBuilder.accept?

Done. IsDeletedBuilder now takes the DeleteFilter and increments in accept, so the counter field and its accessor are gone along with the loop.

pvary · 2026-09-14T07:55:33Z

+    for (int index = liveRowId; index < batchSize; index++) {
+      deletes.incrementDeleteCount();
+    }


This is a bit tricky to read. Is this more clear? Optimizer will fix it anyways.

Suggested change

for (int index = liveRowId; index < batchSize; index++) {

deletes.incrementDeleteCount();

}

int deletedRowCount = batchSize - liveRowId;

for (int i = 0; i < deletedRowCount; i++) {

deletes.incrementDeleteCount();

}

I applied the same idea you suggested for IsDeletedBuilder here as well
RowIdMappingBuilder.accept is called once per deleted position, so incrementing there removes this loop entirely rather than making it easier to read. That means I didn't commit the suggestion literally — let me know if you'd rather keep an explicit loop.

pvary · 2026-09-14T07:56:04Z

Could you please add JMH test to show the gains?

JeonDaehong · 2026-09-14T08:38:57Z

@pvary
Thanks a lot for the detailed feedback! I'll make sure to address all of your comments, including the JMH test, in about 5 hours after I wrap up work. :D

JeonDaehong · 2026-09-14T13:42:01Z

@pvary

Could you please add JMH test to show the gains?

Added PositionDeleteIndexBenchmark in core/src/jmh. It reads 5M positions in 5,000-position batches, the way the vectorized reader does, at densities on either side of the array/bitmap boundary.

JMH 1.37, JDK 21.0.10:

Benchmark	(density)	Mode	Cnt	Score	Error	Units
`PositionDeleteIndexBenchmark.probePerPosition`	0.5	ss	5	82.893	± 13.257	ms/op
`PositionDeleteIndexBenchmark.probePerPosition`	6.1	ss	5	111.101	± 43.291	ms/op
`PositionDeleteIndexBenchmark.probePerPosition`	12.0	ss	5	68.924	± 18.444	ms/op
`PositionDeleteIndexBenchmark.traverseRange`	0.5	ss	5	0.369	± 0.378	ms/op
`PositionDeleteIndexBenchmark.traverseRange`	6.1	ss	5	1.304	± 0.891	ms/op
`PositionDeleteIndexBenchmark.traverseRange`	12.0	ss	5	2.456	± 1.660	ms/op

The spread is wide at five single-shot iterations, but the two arms are two orders of magnitude apart, so the comparison holds either way.

Two things are worth noting here:

First, the per-position cost is not monotonic in density: 6.1% is the worst case because the container is still a sorted array near its deepest, and 12% is cheaper because it has flipped to a bitmap.
Second, this ratio is much larger than the 2.6–9.3x quoted for the Spark path: This is because this benchmark measures only the index lookup performance, whereas the delete-check subtree in a real scan also contains DV deserialization, the mapping array write, and the index wrapper, none of which this change touches.

pvary · 2026-09-15T12:31:22Z

+      long lowEnd =
+          key == endKey
+              ? Integer.toUnsignedLong(pos32Bits(posEndExclusive - 1)) + 1
+              : MAX_POS_32_BITS + 1;


Is this the same as 1L << 32 used elsewhere?

Switched to 1L << 32 to keep it consistent, and removed the now-unused MAX_POS_32_BITS

pvary · 2026-09-15T15:51:14Z

@JeonDaehong: I suggest adding this as a topic for tomorrow’s Iceberg Community Sync so you can present the proposal to the broader community and gather feedback. You can add it to the agenda here: https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit?tab=t.0#heading=h.z3dncl7gr8m1

pvary · 2026-09-15T16:15:12Z


    PositionDeleteIndex deletedPositions = deletes.deletedRowPositions();
+
+    if (deletedPositions != null && !deletes.hasEqDeletes()) {


Is this intentionally different from the similar check in buildRowIdMapping?

Not intentional.
I folded the branches in buildRowIdMapping
and left buildIsDeleted behind, so the two read differently for the same
condition.

They now have the same shape, and buildIsDeleted also skips the per-row
loop when there is nothing deleted at all

if (!deletes.hasEqDeletes()) { // positions in a batch form a contiguous ascending range, so the index // can be traversed once for the whole range instead of being probed // once per row if (deletedPositions != null) { IsDeletedCollector collector = new IsDeletedCollector(deletes, isDeleted, rowStartPosInBatch); deletedPositions.forEachInRange( rowStartPosInBatch, rowStartPosInBatch + batchSize, collector); } return isDeleted; }

JeonDaehong · 2026-09-15T23:19:03Z

@pvary

The kafka-connect job failed while resolving dependencies... Maven Central returned 403s for zstd-jni, lz4-java, and javax.ws.rs-api, all of which are pulled in by kafka-clients.
It failed after 18s before any tests ran, and this PR doesn't touch that module.
Looks like a transient issue.
I don't have permission to re-run it
could you kick it off when you get a chance?

JeonDaehong · 2026-09-15T23:24:43Z

@pvary

Thanks for suggesting that I present the proposal !
I’m giving some thought to the timing and the English presentation, and I’ll get back to you shortly.

JeonDaehong · 2026-09-16T07:43:47Z

@pvary

Thanks for the suggestion! I’d be happy to add it to the agenda and sync with the community, but I’m a little concerned about presenting in English given the timing in Korea.

Would it be okay to add it as a discussion item instead, so I can simply sync with the community without giving a formal presentation?

If so, where would I be able to follow up on the discussion, including any Q&A or feedback?

pvary · 2026-09-16T08:47:32Z

Sure thing!
No need to have a formal presentation.

Simply a few sentences:

Identified a way to improve applying the delete filter for vectorized reads
Improves performance significantly when there are only a few columns projected
Needs a new API: PositionDeleteIndex.forEachInRange, RoaringPositionBitmap.forEachInRange - Are you ok with this?
Link to the PR

The community members are clever, don't need too much detail. They will ask details if they need it.

JeonDaehong · 2026-09-16T13:25:50Z

@pvary

Thank you for guiding me!

[Daehong] Reducing per-row position delete index probes in the vectorized read path
* Vectorized reads probe the position delete index once per row, but the positions in a batch are a contiguous ascending range. The index can be traversed once instead.
* Significant gain when few columns are projected. Not visible end to end on wide projections, where the delete check is a small share of scan CPU.
* Needs a new API: PositionDeleteIndex.forEachInRange. Is the community OK with that?
* PR: #18027

I wrote it this way.

I hope it leads to a great discussion!

As an engineer who loves Iceberg, I'm thrilled to be a part of this journey.

pvary · 2026-09-16T14:44:18Z

As an engineer who loves Iceberg, I'm thrilled to be a part of this journey.

See you there! 😄

JeonDaehong · 2026-09-17T00:55:08Z

@pvary
I was wondering if there were any updates or outcomes from the discussion. Would it be possible to know what was discussed?

pvary · 2026-09-17T09:42:04Z

I was wondering if there were any updates or outcomes from the discussion. Would it be possible to know what was discussed?

You can find the recordings on the Apache Iceberg youtube channel.

There were no objections against adding the new API.

@nastra promised a second opinion on the PR.

nastra

I did a high-level pass of the APIs but haven't checked the tests yet

nastra · 2026-09-17T10:33:29Z

+   * Traverses the deleted positions within the given range in ascending order, applying the
+   * provided consumer.
+   *
+   * <p>Callers that test a contiguous range of positions should prefer this method over calling


not sure if that's helpful, maybe just remove this

Thanks for feedback.
I've removed it.

nastra · 2026-09-17T10:35:35Z

+   * @param consumer a consumer for the deleted positions in the range
+   * @throws IllegalArgumentException if posStart &gt; posEnd
+   */
+  default void forEachInRange(long posStart, long posEnd, LongConsumer consumer) {


Suggested change

default void forEachInRange(long posStart, long posEnd, LongConsumer consumer) {

default void forEachInRange(long posStartInclusive, long posEndExclusive, LongConsumer consumer) {

we use similar naming in RoaringPositionBitmap.setRange(..), so I think being explicit about the ranges is helpful. Please also update the implementing methods

also I think it's not entirely clear from the method naming that this will only look at deleted positions in the range. forEach itself iterates over all positions, so one would assume that forEachInRange would do the same without checked for deleted positions

maybe we should make the predicate part of the method parameters

PositionDeleteIndex.forEach(consumer) only calls the consumer for the position set in the index. So for only the deleted positions. This is the same for the forEachInRange.

Good catch on the naming!

PositionDeleteIndex.forEach(consumer) only calls the consumer for the position set in the index. So for only the deleted positions. This is the same for the forEachInRange.

sorry I missed that, in that case I'm ok with naming this forEachInRange()

Renamed in the interface and both implementations.
Thanks to Peter for the clarification -- keeping forEachInRange().

nastra · 2026-09-17T10:43:33Z


+  @Override
+  public void forEachInRange(long posStart, long posEnd, LongConsumer consumer) {
+    Preconditions.checkArgument(


what about just throwing an UOE stating that this isn't supported? Not sure we're adding any value by checking the start/end but essentially not doing anything

I requested that we should have the same behaviour between the implementations. That is why the check.
I'm not sure that throwing an UOE is a good idea. BaseDeleteLoader uses PositionDeleteIndex.empty() for data files where no position is deleted. We might throw if we start using forEachInRange there, even if the correct response is not to call the consumer at all.

Agreed
Thanks for the BaseDeleteLoader pointer, that is the case I had in mind too. Keeping it as is: the override also avoids the default implementation walking the whole range calling isDeleted, which is always false here.

nastra · 2026-09-17T11:13:45Z

+  @Test
+  public void testForEachInRangeEmptyRange() {
+    PositionDeleteIndex index = indexOf(0L, 1L, 2L);
+    assertThat(collect(index, 0L, 0L)).isEmpty();


we also need tests where end < start

Added testForEachInRangeInvalidRange. The range is validated in three separate places -- the default method, BitmapPositionDeleteIndex (via RoaringPositionBitmap) and EmptyPositionDeleteIndex -- so the test covers all three. I verified it by disabling each precondition in turn and confirming the test fails for each.

JeonDaehong · 2026-09-17T22:48:08Z

@nastra
Thanks for taking a look! I appreciate it. Please take your time with the tests, and let me know if you have any feedback.

nastra · 2026-09-21T09:17:08Z

    return positions;
  }
+
+  private static List<Long> collect(PositionDeleteIndex index, long posStart, long posEnd) {


maybe name this forEachInRange or positionsInRange?

Renamed to positionsInRange. Parameters now match the other helper too.

nastra · 2026-09-21T09:19:58Z

+    RoaringPositionBitmap bitmap = new RoaringPositionBitmap();
+    bitmap.setRange(10L, 20L);
+
+    // the beginning is inclusive and the end is exclusive


can be omitted

nastra · 2026-09-21T09:20:24Z

    }
  }

+  private static List<Long> collectInRange(


same naming as in the other test

Renamed to positionsInRange as well.

nastra

left a few minor comments but generally LGTM

JeonDaehong · 2026-09-21T13:21:02Z

Thanks @nastra for the approval, and thanks @pvary for carrying the review this far.

Once this PR is merged, I'll open the backport to Spark 3.5, 4.0, and 4.1.

pvary · 2026-09-21T16:04:03Z

Merged to main.
Thanks @JeonDaehong for the performance improvement PR and @nastra for the review!

@JeonDaehong: Please don't forget to create a PR to backport the changes to the other Spark branches.

JeonDaehong · 2026-09-22T03:44:28Z

@pvary

Thanks for merging! I’ll create the backport PRs for the other Spark branches. 👍

…nd 4.1 (#18207) Backport of #18027

Core, Spark 4.2: Add PositionDeleteIndex.forEachInRange and use it in…

1fe358d

… vectorized delete checks

github-actions Bot added spark core labels Sep 9, 2026

JeonDaehong changed the title ~~Core, Spark 4.2: Avoid per-row path rehashing in toPositionIndexes~~ Core, Spark 4.2: Add PositionDeleteIndex.forEachInRange and use it in vectorized delete checks Sep 9, 2026

JeonDaehong mentioned this pull request Sep 11, 2026

Spark: vectorized reads probe the position delete index once per row - 43–58% of scan CPU on narrow projections #18026

Closed

pvary reviewed Sep 14, 2026

View reviewed changes

Core, Spark 4.2: Address review comments on forEachInRange

6d57d5b

Spark 4.2: Revert the empty-index shortcut, it costs more than it saves

c4c976a

pvary reviewed Sep 15, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/deletes/RoaringPositionBitmap.java Resolved

pvary reviewed Sep 15, 2026

View reviewed changes

Core, Spark 4.2: Fix range traversal across the signed int boundary

2c3d95a

pvary reviewed Sep 15, 2026

View reviewed changes

Spark 4.2: Make the no-equality-delete gate consistent in both methods

dfca55f

pvary requested a review from nastra September 17, 2026 09:42

nastra reviewed Sep 17, 2026

View reviewed changes

Core: Address review comments on forEachInRange

760b2dc

nastra reviewed Sep 21, 2026

View reviewed changes

nastra approved these changes Sep 21, 2026

View reviewed changes

Core: Rename range test helpers and drop a redundant comment

14741e2

pvary approved these changes Sep 21, 2026

View reviewed changes

pvary merged commit 8a6830a into apache:main Sep 21, 2026
42 checks passed

JeonDaehong mentioned this pull request Sep 22, 2026

Spark: Backport forEachInRange vectorized delete checks to 3.5, 4.0 and 4.1 #18207

Merged

pvary pushed a commit that referenced this pull request Sep 23, 2026

Spark: Backport forEachInRange vectorized delete checks to 3.5, 4.0 a…

89a6a87

…nd 4.1 (#18207) Backport of #18027

JeonDaehong mentioned this pull request Oct 5, 2026

Partition paths format float and double values differently from iceberg-java apache/iceberg-rust#3326

Closed

JeonDaehong mentioned this pull request Oct 14, 2026

perf(arrow): build the delete row selection from deleted ranges apache/iceberg-rust#3454

Merged


		PositionDeleteIndex deletedPositions = deletes.deletedRowPositions();

		if (deletedPositions != null && !deletes.hasEqDeletes()) {

	default void forEachInRange(long posStart, long posEnd, LongConsumer consumer) {
	default void forEachInRange(long posStartInclusive, long posEndExclusive, LongConsumer consumer) {

Conversation

JeonDaehong commented Sep 9, 2026 • edited by pvary Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JeonDaehong commented Sep 9, 2026

Uh oh!

JeonDaehong commented Sep 11, 2026

Uh oh!

pvary commented Sep 11, 2026

Uh oh!

JeonDaehong commented Sep 11, 2026

Uh oh!

pvary commented Sep 11, 2026

Uh oh!

JeonDaehong commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JeonDaehong commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Is 13-19% accurate?

When it helps

The one that's backwards

One rule instead of six

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JeonDaehong Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pvary Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pvary commented Sep 14, 2026

Uh oh!

JeonDaehong commented Sep 14, 2026

Uh oh!

JeonDaehong commented Sep 14, 2026

Uh oh!

JeonDaehong commented Sep 9, 2026 •

edited by pvary

Loading

JeonDaehong commented Sep 11, 2026 •

edited

Loading

JeonDaehong commented Sep 11, 2026 •

edited

Loading

JeonDaehong Sep 14, 2026 •

edited

Loading

pvary Sep 14, 2026 •

edited

Loading