Skip to content

Commit 482ce27

Browse files
committed
Merge remote-tracking branch 'apache/main' into iceberg-issue-6234
# Conflicts: # docs/source/contributor-guide/ffi.md # spark/src/main/scala/org/apache/spark/sql/comet/execution/arrow/CometNativeArrowSource.scala # spark/src/test/scala/org/apache/comet/CometIcebergWriteActionSuite.scala # spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala # spark/src/test/scala/org/apache/spark/CometExecIteratorLifecycleSuite.scala
2 parents 90a3a1e + 9f68a41 commit 482ce27

534 files changed

Lines changed: 28098 additions & 3901 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/review-comet-ffi-pr/SKILL.md‎

Lines changed: 32 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ that in mind.
3636

3737
| Doc | What you need from it |
3838
| ---------------------------------------------------- | --------------------------------------------------------------------------- |
39-
| `docs/source/contributor-guide/ffi.md` | Both data-flow directions, ownership rules, lifecycle, alignment workaround |
39+
| `docs/source/contributor-guide/ffi.md` | Both data-flow directions, ownership rules, lifecycle, offset normalization |
4040
| `docs/source/contributor-guide/memory_management.md` | The "Crossing the FFI boundary" section: who is charged for a batch's bytes |
4141

4242
Read `ffi.md` in full before the diff. The two directions have different ownership semantics and
@@ -53,10 +53,10 @@ reviewing one with the other's mental model is the most common way to miss a bug
5353

5454
- [ ] **JVM to native: no defensive deep copies.** The C Stream transfers ownership by reference
5555
count, so native can buffer imported batches in `SortExec` or the shuffle writer without
56-
copying. A new `.clone()` of the data, a `copy_array`, or a "to be safe" deep copy on this
57-
path is a real throughput cost. Ask what it is protecting against.
56+
copying. A new `.clone()` of the data, a `MutableArrayData` copy, or a "to be safe" deep copy
57+
on this path is a real throughput cost. Ask what it is protecting against.
5858
- [ ] **JVM to native: dropping the reader is the release.** When `ScanExec` drops its
59-
`AlignedArrowStreamReader`, the stream's release callback fires synchronously back into the
59+
`ArrowArrayStreamReader`, the stream's release callback fires synchronously back into the
6060
JVM and closes the `ArrowReader` and its `VectorSchemaRoot`. Anything that extends the
6161
reader's lifetime, stores it somewhere longer-lived, or drops it early changes when JVM
6262
off-heap buffers are freed.
@@ -107,30 +107,37 @@ a new subclass needs no new case as long as `getValueVector` returns an Arrow ve
107107
- [ ] A new `CometVector` subclass whose `getValueVector` is synthesized rather than owned has a
108108
defined lifetime relative to the export.
109109
- [ ] Import is symmetric. `ScanExec::pull_next` runs every imported column through
110-
`import_column`, which decodes invalid UTF-8 to Spark's rendering before
111-
`copy_or_unpack_array` with `CopyMode::UnpackOrClone`. A PR that adds a column type has to say
112-
what that pair does to it.
110+
`import_column`, which decodes invalid UTF-8 to Spark's rendering and otherwise keeps the
111+
imported buffers. A PR that adds a column type has to say what that step does to it.
113112
- [ ] The row-count check still holds. `exportBatch` requires every column to report the same value
114113
count and throws otherwise, which is the guard that catches a vector exported at the wrong
115114
length.
116115

117-
## 4. Alignment and Schema
118-
119-
- [ ] **`AlignedArrowStreamReader` is not dead code.** It calls `align_buffers` on every batch
120-
because Java's allocator hands back `Decimal128` buffers at 8-byte rather than 16-byte
121-
alignment, which the stock `ArrowArrayStreamReader` rejects
122-
([arrow-rs#10028](https://github.com/apache/arrow-rs/issues/10028)). It can only be replaced
123-
with the stock reader once Comet is on arrow 59 or newer, where
124-
[arrow-rs#10030](https://github.com/apache/arrow-rs/pull/10030) aligns on import. If the PR
125-
removes it, check the arrow version in `native/Cargo.toml` actually supports that.
116+
## 4. Alignment, Offsets, and Schema
117+
118+
- [ ] **JVM to native: arrow-rs does the alignment.** Java's allocator hands back `Decimal128`
119+
buffers at 8-byte rather than 16-byte alignment. Since arrow 59, `from_ffi` and
120+
`from_ffi_and_data_type` realign them on import
121+
([arrow-rs#10030](https://github.com/apache/arrow-rs/pull/10030)), so `ScanExec` reads the
122+
stream with the stock `ArrowArrayStreamReader`. The `realigns_under_aligned_decimal128` test
123+
in `scan.rs` guards this. A PR that downgrades arrow, or that imports through anything other
124+
than those two functions, must keep that test passing.
125+
- [ ] **Native to JVM: exported offsets must be zero.** Arrow Java ignores `ArrowArray.offset` at
126+
every level on import, so every array native exports to the JVM first goes through
127+
`zero_offsets` (in `native/common/src/ffi_offsets.rs`). `move_to_spark` applies it to
128+
executed batches and decoded shuffle blocks, and `JvmScalarUdfExpr` to the inputs of the
129+
JVM UDF bridge. A new export path has to call it too, or a sliced boolean, top-level or
130+
nested, reaches the JVM misaligned
131+
([#6288](https://github.com/apache/datafusion-comet/issues/6288)).
126132
- [ ] **Schema reconciliation stays truthful.** `CometArrowStream.reconcileStreamSchema` advertises
127133
the stream's schema from the actual `CometVector` types in the first batch rather than the
128134
consumer's Spark-declared types, so that the cast in native `build_record_batch` fires. A PR
129135
that changes either side needs to change both, and a new "just declare the Spark type" path
130136
quietly reintroduces the drift this exists to handle.
131-
- [ ] **Dictionary handling.** Import uses `CopyMode::UnpackOrClone`: dictionary columns are
132-
unpacked into new native arrays, everything else is an `Arc` clone. A change here affects both
133-
correctness and allocation volume.
137+
- [ ] **Dictionaries are decoded on the JVM.** No input stream carries a dictionary:
138+
`ColumnarBatchArrowReader` decodes dictionary-encoded columns before export, and
139+
`reconcileStreamSchema` advertises the value type. A new reader that exports a dictionary
140+
would reach `ScanExec` as is, and only the cast in `build_record_batch` would unpack it.
134141

135142
## 5. Memory Accounting
136143

@@ -148,7 +155,10 @@ FFI bugs rarely reproduce in a small unit test. Ask what evidence the PR offers:
148155
- Does an existing suite exercise the new path with a non-trivial number of batches, rather than
149156
one batch that happens to work?
150157
- Are nested and dictionary-encoded types covered, since they take different import paths?
151-
- Are Decimal128 columns covered, since that is what the alignment workaround exists for?
158+
- Are Decimal128 columns covered, since they need an alignment that Java's allocator does not
159+
guarantee?
160+
- On the native to JVM side, are sliced inputs covered, since Arrow Java ignores a non-zero
161+
offset?
152162
- For a lifecycle change, is there a test that closes or cancels early?
153163
- Rust tests in `native/core` need `DYLD_LIBRARY_PATH=$JAVA_HOME/lib/server` on macOS and
154164
`LD_LIBRARY_PATH` on Linux. If the PR adds one that needs the JVM, check CI actually runs it.
@@ -162,9 +172,8 @@ FFI bugs rarely reproduce in a small unit test. Ask what evidence the PR offers:
162172
- The **JVM to Native** and **Native to JVM** architecture diagrams, if a stage is added, removed,
163173
or renamed.
164174
- The list of `ArrowReader` implementations, if the PR adds an input shape.
165-
- The **Buffer Alignment** section, which describes `AlignedArrowStreamReader` as a temporary
166-
workaround with a stated exit condition. If the PR removes the reader or bumps arrow past 59,
167-
that section has to go or change.
175+
- The **Array Offsets** section, if the PR adds a native to JVM export path or changes what
176+
`zero_offsets` does.
168177
- The **Schema Reconciliation** and **Ownership and Lifecycle** paragraphs, which state invariants
169178
rather than describing code. These are the easiest to leave quietly wrong.
170179
- Class and file paths named in the prose, if the PR moves anything.

‎.ai/skills/review-comet-pr/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ change.
3939
| The PR touches | Also use |
4040
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------- |
4141
| `spark/src/main/scala/org/apache/comet/serde/`, `QueryPlanSerde.scala`, `native/spark-expr/`, `expr.proto`, `comet_scalar_funcs.rs` | `review-comet-expression-pr` |
42-
| `CometExecIterator`, `CometNativeArrowSource`, `NativeUtil`, `CometVector` subclasses, `jni_api.rs`, `scan.rs`, `aligned_stream_reader.rs`, anything using `FFI_Arrow*` | `review-comet-ffi-pr` |
42+
| `CometExecIterator`, `CometNativeArrowSource`, `NativeUtil`, `CometVector` subclasses, `jni_api.rs`, `scan.rs`, anything using `FFI_Arrow*` | `review-comet-ffi-pr` |
4343
| `native/core/src/execution/memory_pools/`, `CometTaskMemoryManager`, `CometShuffleMemoryAllocator`, `MemoryConsumer` / `try_grow` / `reserve` call sites, pool configs | `review-comet-memory-pr` |
4444
| `native/shuffle/`, `spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/`, `spark/src/main/java/org/apache/spark/shuffle/`, `CometShuffleExchangeExec` | `review-comet-shuffle-pr` |
4545
| `IcebergWriteStrategy`, `IcebergWriteExec`, `IcebergCommitExec`, `CometIcebergWriteExec`, `CometIcebergNativeWrite`, `iceberg_write.rs`, `iceberg_partition_path.rs`, the iceberg-rust pin | `review-comet-iceberg-write-pr` |

‎.asf.yaml‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,12 @@ github:
110110
branch-0.17:
111111
required_pull_request_reviews:
112112
required_approving_review_count: 1
113+
branch-1.0:
114+
required_pull_request_reviews:
115+
required_approving_review_count: 1
116+
branch-1.1:
117+
required_pull_request_reviews:
118+
required_approving_review_count: 1
113119
# GitHub's merge queue. Classic `protected_branches` above has no way to
114120
# express this, but .asf.yaml rulesets accept a raw Rulesets API payload, so
115121
# no INFRA ticket is needed. Same approach as apache/iceberg-python and
@@ -142,16 +148,22 @@ github:
142148
# ALLGREEN: every entry's own build has to be green, not just the
143149
# last one's. HEADGREEN would let an entry whose build failed ride
144150
# in on a green head, which papers over a flake but equally papers
145-
# over a real failure that a later entry happens to mask.
151+
# over a real failure that a later entry happens to mask. It would
152+
# also let untested code in: `merge_group.base_sha` is the queue
153+
# commit of the entry ahead, so each entry's build runs only the
154+
# suites its own changed files select, and a green head says
155+
# nothing about the suites an earlier entry needed.
146156
grouping_strategy: ALLGREEN
147157
# Speculation depth: how many merge_group builds run at once. Each
148158
# queue entry gets its own build of itself plus everything ahead of
149159
# it, and merge limits do not combine those builds, so this is the
150-
# throughput dial. At 1 every merge serializes behind a ~2.5h
151-
# pipeline; 2 sustains roughly 19 merges a day, against a median of
152-
# 6 and a busiest-day 22 over the last month. Raise it if the queue
153-
# backs up, at the price of that many concurrent pipelines.
154-
max_entries_to_build: 2
160+
# throughput dial. In September 2026 a full pipeline took a median
161+
# of 92 minutes, so 2 capped a queue that never ran dry at roughly
162+
# 30 merges a day while approvals arrived at about 20 a day, and
163+
# the backlog grew. 4 roughly doubles the cap. The price is that
164+
# many concurrent pipelines, and more rebuilds when an entry fails,
165+
# since every entry behind it that was already building restarts.
166+
max_entries_to_build: 4
155167
# These only cap how many already-green entries land in one merge;
156168
# they do not batch the builds. No minimum and no wait, so a lone
157169
# PR merges the moment its build is green.

‎.github/actions/rust-test/action.yaml‎

Lines changed: 6 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -41,36 +41,26 @@ runs:
4141
cd native
4242
cargo install cargo-machete --version 0.7.0 && cargo machete
4343
44+
# Borrows the Maven repository the TPC-H/TPC-DS jobs keep warm on main, a
45+
# superset of what `common` needs. This job no longer runs on push to main
46+
# (see linux-test-rust in ../../workflows/pr_build_linux.yml), so an entry
47+
# of its own would never be refreshed.
4448
- name: Restore Maven dependencies
45-
id: maven-cache
4649
uses: actions/cache/restore@v4
4750
with:
4851
path: |
4952
~/.m2/repository
5053
/root/.m2/repository
51-
key: ${{ runner.os }}-rust-maven-${{ hashFiles('**/pom.xml') }}
54+
key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}
5255
restore-keys: |
53-
${{ runner.os }}-rust-maven-
56+
${{ runner.os }}-java-maven-
5457
5558
- name: Build common module (pre-requisite for Rust tests)
5659
shell: bash
5760
run: |
5861
cd common
5962
../mvnw -B clean compile -DskipTests
6063
61-
# Saved only on main: an entry written from a pull request or a
62-
# `gh-readonly-queue/*` branch cannot be restored by any later run, and it
63-
# evicts main's from the shared budget. See "Large caches are written on
64-
# main only" in ../../workflows/README.md.
65-
- name: Save Maven dependencies
66-
if: ${{ github.ref == 'refs/heads/main' && steps.maven-cache.outputs.cache-hit != 'true' }}
67-
uses: actions/cache/save@v4
68-
with:
69-
path: |
70-
~/.m2/repository
71-
/root/.m2/repository
72-
key: ${{ runner.os }}-rust-maven-${{ hashFiles('**/pom.xml') }}
73-
7464
- name: Install nextest
7565
shell: bash
7666
run: |

‎.github/actions/setup-iceberg-builder/action.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,3 +37,25 @@ runs:
3737
run: |
3838
cd apache-iceberg
3939
git apply ../dev/diffs/iceberg/${{inputs.iceberg-version}}.diff
40+
41+
# The Gradle wrapper downloads its distribution from services.gradle.org
42+
# the first time it runs, with no retry of its own, so a connection reset
43+
# or TLS failure there used to fail the test step before any test ran.
44+
# Install it here instead, where a retry costs seconds; the test step then
45+
# finds Gradle already in place.
46+
- name: Install the Gradle distribution
47+
shell: bash
48+
run: |
49+
cd apache-iceberg
50+
for attempt in 1 2 3; do
51+
if ./gradlew --version; then
52+
exit 0
53+
fi
54+
if [ "$attempt" -eq 3 ]; then
55+
echo "::error::Could not install the Gradle distribution after $attempt attempts."
56+
exit 1
57+
fi
58+
delay=$((15 * attempt))
59+
echo "::warning::Gradle distribution download failed (attempt $attempt of 3); retrying in ${delay}s."
60+
sleep "$delay"
61+
done

0 commit comments

Comments
 (0)