Skip to content

Commit 62a41cd

Browse files
committed
Merge remote-tracking branch 'apache/main' into pr6940-parth-test
2 parents 5dd2fb9 + 4e54f1c commit 62a41cd

370 files changed

Lines changed: 32327 additions & 5935 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/review-comet-memory-pr/SKILL.md‎

Lines changed: 27 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -100,19 +100,27 @@ configuration, from the inside out:
100100

101101
- [ ] A new decorator forwards **every** `MemoryPool` method to its inner pool. A partial
102102
implementation makes `reserved()` disagree between levels.
103-
- [ ] **`fair_unified` checks the requesting consumer against its share, and the total against
104-
the pool.** It divides `pool_size` by the number of registered consumers and rejects if what
105-
the consumer holds plus the request exceeds that quotient, or if the pool's total reserved
106-
plus the request exceeds `pool_size`. With two consumers and an 8 GiB pool, once one holds
107-
3 GiB the other can still reserve up to 4 GiB. The pool keeps a running total for each
108-
consumer id, so the sibling reservations that `new_empty()`, `split()` and `take()` create
109-
count against one share. A PR that checks `reservation.size()` instead lets an operator with
110-
several reservations, such as a sort's streaming merge, take other consumers' shares. It
111-
also depends on when DataFusion updates the size, which is after it calls `try_grow` but
112-
before it calls `shrink`. If the PR changes either comparison it changes when every query
113-
spills, so it needs benchmark evidence, not reasoning.
114-
- [ ] **`num_consumers` counts every consumer in the task**, across every native plan, because the
115-
pool is task-shared.
103+
- [ ] **`fair_unified` holds the consumers that can spill to a share of the task's share.** The
104+
task's share is `spark.memory.offHeap.size` divided by the tasks on the executor running
105+
native plans, which `RunningTasks` counts while each task holds its task-shared pool, or
106+
`pool_size` if that is smaller. A consumer that can spill is
107+
refused if what it holds plus the request exceeds what the consumers that cannot spill leave
108+
of the task's share, divided by the registered consumers that can spill, or if the task's
109+
total plus the request exceeds the task's share. A consumer that cannot spill is refused only
110+
if the task's total plus the request exceeds `pool_size`, which binds before Spark's own
111+
limit only while the deprecated fraction is below `1 / N`. With eight running tasks, a 16 GiB
112+
pool and two consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge holds
113+
512 MiB. The pool keeps a running total for each consumer id, so the sibling reservations
114+
that `new_empty()`, `split()` and `take()` create count against one share. A PR that checks
115+
`reservation.size()` instead lets an operator with several reservations, such as a sort's
116+
streaming merge, take other consumers' shares. It also depends on when DataFusion updates the
117+
size, which is after it calls `try_grow` but before it calls `shrink`. A PR that limits a
118+
consumer that cannot spill to a share, or sizes the shares from the whole pool, brings back
119+
#7088: one operator takes the task's share from Spark and a merge or hash join build fails
120+
the task. If the PR changes any of these comparisons it changes when every query spills, so
121+
it needs benchmark evidence, not reasoning.
122+
- [ ] **The consumers that divide the task's share are every consumer that can spill in the
123+
task**, across every native plan, because the pool is task-shared.
116124
- [ ] **Task-shared pool lifetime.** `acquire_task_shared_pool` keeps a process-wide
117125
`HashMap<task_attempt_id, Weak<TaskSharedMemoryPool>>`. The returned `Arc` is the only
118126
lifetime handle, so there is no explicit release to forget and a `createPlan` that fails
@@ -143,14 +151,16 @@ reclamation, that is a significant change and needs to say what it does to in-fl
143151
## 5. Budget and Configuration
144152

145153
`CometExecIterator.getMemoryConfig` computes the budget once per executor and passes it across JNI
146-
as `memory_limit`:
154+
as `memory_limit`, along with `spark.memory.offHeap.size` itself as `off_heap_size`, which
155+
`fair_unified` divides among the tasks running native plans:
147156

148157
```text
149158
memory_limit = spark.memory.offHeap.size * spark.comet.exec.memoryPool.fraction
159+
off_heap_size = spark.memory.offHeap.size
150160
```
151161

152162
- [ ] A new memory config goes through `getMemoryConfig` rather than being read separately in
153-
native code.
163+
native code, and a new value crossing JNI is added to `createPlan` on both sides.
154164
- [ ] Changing `memoryPool.fraction` semantics affects every deployment that tuned it as a haircut
155165
for the accounting gap.
156166
- [ ] On Kubernetes, `spark.memory.offHeap.size` is **part of** the pod limit, not headroom on top
@@ -204,8 +214,8 @@ will not catch. Check:
204214
- The **pool stack** diagram, if a decorator is added, removed, or reordered
205215
- The **pool type** table, if a pool type is added, removed, or changes how it is sized
206216
- The `memory_limit` formula, if the budget calculation changes
207-
- The **fair pool** description, which spells out the share and pool-total checks and their
208-
consequences
217+
- The **fair pool** description, which spells out the task share, the share and task-total
218+
checks, and their consequences
209219
- The **Crossing the FFI boundary** section, which names the operators that reserve for imported
210220
batches
211221
- The **Open problems** list. If the PR closes one of these gaps, the entry must be removed or

‎.asf.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,13 @@ github:
179179
# Lets a committer queue a PR before its checks have finished, which is
180180
# most of the point of having a queue.
181181
allow_auto_merge: true
182+
# Limit open pull requests per user without write access.
183+
#
184+
# With coding agents, the number of open PRs now far exceeds our review capacity.
185+
# Setting a limit helps reviewers focus on fewer PRs at a time.
186+
creation_cap:
187+
enabled: true
188+
max_open_pull_requests: 3
182189
# publishes the content of the `asf-site` branch to
183190
# https://datafusion.apache.org/comet/
184191
publish:

‎.github/actions/rust-test/action.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,28 @@ runs:
2323
steps:
2424
# Note: cargo fmt check is now handled by the lint job that gates this workflow
2525

26+
- name: Set up embedded Python for native UDF tests
27+
shell: bash
28+
run: |
29+
apt-get update
30+
apt-get install -y --no-install-recommends python3 python3-dev python3-venv
31+
python3 -m venv /tmp/comet-rust-python
32+
/tmp/comet-rust-python/bin/pip install "pyarrow>=14" cloudpickle
33+
echo "PYO3_PYTHON=/tmp/comet-rust-python/bin/python" >> "$GITHUB_ENV"
34+
echo "PYTHONPATH=$(/tmp/comet-rust-python/bin/python -c 'import sysconfig; print(sysconfig.get_path("purelib"))')" >> "$GITHUB_ENV"
35+
2636
- name: Check Cargo clippy
2737
shell: bash
2838
run: |
2939
cd native
3040
cargo clippy --color=never --all-targets --workspace -- -D warnings
3141
42+
- name: Check native Python UDF feature
43+
shell: bash
44+
run: |
45+
cd native
46+
cargo clippy --color=never -p datafusion-comet --all-targets --features python-udf -- -D warnings
47+
3248
- name: Check compilation
3349
shell: bash
3450
run: |
@@ -83,6 +99,13 @@ runs:
8399
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
84100
RUST_BACKTRACE=1 cargo nextest run
85101
102+
- name: Run native Python UDF tests
103+
shell: bash
104+
run: |
105+
cd native
106+
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
107+
RUST_BACKTRACE=1 cargo nextest run -p datafusion-comet --lib --features python-udf python_udf
108+
86109
# The steps above lint and test the accounting allocator over the system allocator. Lint and
87110
# test it over jemalloc too, since that is a different allocator backend and nothing else in CI
88111
# builds it.

‎.github/workflows/delta_contrib_test.yml‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -68,8 +68,12 @@ jobs:
6868
path: |
6969
~/.m2/repository
7070
/root/.m2/repository
71-
key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-${{ inputs.spark }}
71+
key: ${{ runner.os }}-delta-maven-${{ hashFiles('**/pom.xml') }}-${{ inputs.spark }}
72+
# Its own prefix: the Spark SQL and Comet test jobs fall back to the
73+
# newest `java-maven-` entry, and this repository (built with
74+
# `-Pdelta`) broke Spark's Arrow tests when they restored it.
7275
restore-keys: |
76+
${{ runner.os }}-delta-maven-
7377
${{ runner.os }}-java-maven-
7478
7579
- name: Restore Cargo cache
@@ -126,7 +130,7 @@ jobs:
126130
path: |
127131
~/.m2/repository
128132
/root/.m2/repository
129-
key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-${{ inputs.spark }}
133+
key: ${{ runner.os }}-delta-maven-${{ hashFiles('**/pom.xml') }}-${{ inputs.spark }}
130134

131135
# The container job above has no Docker socket, so CometDeltaS3Suite cancels itself
132136
# there. This job runs it on the plain runner, where Testcontainers can start MinIO.
@@ -163,8 +167,12 @@ jobs:
163167
with:
164168
path: |
165169
~/.m2/repository
166-
key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-3.5
170+
key: ${{ runner.os }}-delta-maven-${{ hashFiles('**/pom.xml') }}-3.5
171+
# Its own prefix: the Spark SQL and Comet test jobs fall back to the
172+
# newest `java-maven-` entry, and this repository (built with
173+
# `-Pdelta`) broke Spark's Arrow tests when they restored it.
167174
restore-keys: |
175+
${{ runner.os }}-delta-maven-
168176
${{ runner.os }}-java-maven-
169177
170178
- name: Restore Cargo cache
@@ -210,7 +218,7 @@ jobs:
210218
with:
211219
path: |
212220
~/.m2/repository
213-
key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-3.5
221+
key: ${{ runner.os }}-delta-maven-${{ hashFiles('**/pom.xml') }}-3.5
214222

215223
# `delta` is in the default feature set, so no regular job builds without it;
216224
# this keeps the feature-off build and its "built without the delta feature"
@@ -261,7 +269,7 @@ jobs:
261269
- uses: actions/checkout@v7
262270

263271
- name: Setup Python
264-
uses: actions/setup-python@v6
272+
uses: actions/setup-python@v7
265273
with:
266274
python-version: "3.12"
267275

‎.github/workflows/iceberg_spark_test_reusable.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,7 @@ jobs:
155155
rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups
156156
# COMET_ICEBERG_WRITE_REPORT_DIR records which writer ran each Iceberg
157157
# write; see dev/ci/summarize-iceberg-writes.py.
158-
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true COMET_ICEBERG_WRITE_REPORT_DIR="$PWD/build/comet-iceberg-writes" \
158+
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true COMET_ICEBERG_DATA_FILE_CONCURRENCY_LIMIT=1 COMET_ICEBERG_WRITE_REPORT_DIR="$PWD/build/comet-iceberg-writes" \
159159
./gradlew -DsparkVersions=${{ inputs.spark-short }} -DscalaVersion=${{ inputs.scala }} -DflinkVersions= -DkafkaVersions= \
160160
:iceberg-spark:iceberg-spark-${{ inputs.spark-short }}_${{ inputs.scala }}:test \
161161
--init-script ../dev/ci/iceberg-test-shards.gradle \
@@ -235,7 +235,7 @@ jobs:
235235
run: |
236236
cd apache-iceberg
237237
rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups
238-
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true COMET_ICEBERG_WRITE_REPORT_DIR="$PWD/build/comet-iceberg-writes" \
238+
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true COMET_ICEBERG_DATA_FILE_CONCURRENCY_LIMIT=1 COMET_ICEBERG_WRITE_REPORT_DIR="$PWD/build/comet-iceberg-writes" \
239239
./gradlew -DsparkVersions=${{ inputs.spark-short }} -DscalaVersion=${{ inputs.scala }} -DflinkVersions= -DkafkaVersions= \
240240
:iceberg-spark:iceberg-spark-extensions-${{ inputs.spark-short }}_${{ inputs.scala }}:test \
241241
-Pquick=true -x javadoc
@@ -286,6 +286,6 @@ jobs:
286286
run: |
287287
cd apache-iceberg
288288
rm -rf /root/.m2/repository/org/apache/parquet # somehow parquet cache requires cleanups
289-
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true ./gradlew -DsparkVersions=${{ inputs.spark-short }} -DscalaVersion=${{ inputs.scala }} -DflinkVersions= -DkafkaVersions= \
289+
ENABLE_COMET=true ENABLE_COMET_ONHEAP=true COMET_ICEBERG_DATA_FILE_CONCURRENCY_LIMIT=1 ./gradlew -DsparkVersions=${{ inputs.spark-short }} -DscalaVersion=${{ inputs.scala }} -DflinkVersions= -DkafkaVersions= \
290290
:iceberg-spark:iceberg-spark-runtime-${{ inputs.spark-short }}_${{ inputs.scala }}:integrationTest \
291291
-Pquick=true -x javadoc

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -542,6 +542,7 @@ jobs:
542542
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleWriterSuite
543543
org.apache.spark.sql.comet.execution.shuffle.CometNativePositionalRoundRobinSuite
544544
org.apache.spark.sql.comet.execution.shuffle.CometDiskBlockWriterSuite
545+
org.apache.spark.sql.comet.execution.shuffle.CometBlockStoreShuffleReaderSuite
545546
org.apache.comet.exec.CometShuffleEncryptionSuite
546547
org.apache.comet.exec.CometShuffleManagerSuite
547548
org.apache.comet.exec.DisableAQECometShuffleSuite
@@ -568,6 +569,7 @@ jobs:
568569
org.apache.comet.exec.CometJoinSuite
569570
org.apache.comet.exec.CometTypedDatasetSuite
570571
org.apache.spark.sql.comet.CometMapInBatchSuite
572+
org.apache.spark.sql.comet.CometArrowPythonUdfSuite
571573
org.apache.spark.sql.execution.python.CometArrowPythonRunnerSuite
572574
org.apache.comet.CometNativeSuite
573575
org.apache.comet.DirectColumnarToRowConverterSuite
@@ -613,6 +615,7 @@ jobs:
613615
org.apache.spark.sql.comet.DecimalPrecisionSuite
614616
org.apache.spark.sql.comet.CometScanWithPlanDataSuite
615617
org.apache.spark.sql.comet.util.UtilsSuite
618+
org.apache.spark.sql.comet.CometBroadcastValueSuite
616619
org.apache.comet.vector.NativeUtilSuite
617620
org.apache.comet.vector.CometVectorUtilsSuite
618621
org.apache.comet.udf.CometUdfBridgeSuite
@@ -675,6 +678,7 @@ jobs:
675678
org.apache.comet.serde.CometScalarFunctionSuite
676679
org.apache.comet.serde.CometLiteralSuite
677680
org.apache.comet.serde.QueryPlanSerdeSuite
681+
org.apache.comet.serde.SparkFixesSuite
678682
org.apache.comet.CometFallbackInvarianceSuite
679683
fail-fast: false
680684
name: ${{ matrix.profile.name }} [${{ matrix.suite.name }}]

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -208,6 +208,7 @@ jobs:
208208
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleWriterSuite
209209
org.apache.spark.sql.comet.execution.shuffle.CometNativePositionalRoundRobinSuite
210210
org.apache.spark.sql.comet.execution.shuffle.CometDiskBlockWriterSuite
211+
org.apache.spark.sql.comet.execution.shuffle.CometBlockStoreShuffleReaderSuite
211212
org.apache.comet.exec.CometShuffleEncryptionSuite
212213
org.apache.comet.exec.CometShuffleManagerSuite
213214
org.apache.comet.exec.DisableAQECometShuffleSuite
@@ -234,6 +235,7 @@ jobs:
234235
org.apache.comet.exec.CometJoinSuite
235236
org.apache.comet.exec.CometTypedDatasetSuite
236237
org.apache.spark.sql.comet.CometMapInBatchSuite
238+
org.apache.spark.sql.comet.CometArrowPythonUdfSuite
237239
org.apache.spark.sql.execution.python.CometArrowPythonRunnerSuite
238240
org.apache.comet.CometNativeSuite
239241
org.apache.comet.DirectColumnarToRowConverterSuite
@@ -279,6 +281,7 @@ jobs:
279281
org.apache.spark.sql.comet.DecimalPrecisionSuite
280282
org.apache.spark.sql.comet.CometScanWithPlanDataSuite
281283
org.apache.spark.sql.comet.util.UtilsSuite
284+
org.apache.spark.sql.comet.CometBroadcastValueSuite
282285
org.apache.comet.vector.NativeUtilSuite
283286
org.apache.comet.vector.CometVectorUtilsSuite
284287
org.apache.comet.udf.CometUdfBridgeSuite
@@ -341,6 +344,7 @@ jobs:
341344
org.apache.comet.serde.CometScalarFunctionSuite
342345
org.apache.comet.serde.CometLiteralSuite
343346
org.apache.comet.serde.QueryPlanSerdeSuite
347+
org.apache.comet.serde.SparkFixesSuite
344348
org.apache.comet.CometFallbackInvarianceSuite
345349
346350
fail-fast: false

‎.github/workflows/pyarrow_udf_test.yml‎

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -44,12 +44,15 @@ jobs:
4444
- name: Spark 4.0
4545
maven_profiles: "-Pspark-4.0 -Pscala-2.13"
4646
pyspark: "4.0.4"
47+
native_python_udf: false
4748
- name: Spark 4.1
4849
maven_profiles: "-Pspark-4.1"
4950
pyspark: "4.1.3"
51+
native_python_udf: true
5052
- name: Spark 4.2
5153
maven_profiles: "-Pspark-4.2"
5254
pyspark: "4.2.0"
55+
native_python_udf: true
5356
container:
5457
# Pinned to the Debian 12 (bookworm) base so the system `python3` is 3.11. The default
5558
# `amd64/rust` image is Debian 13 (trixie) which ships Python 3.13 and no python3.11 apt
@@ -77,19 +80,34 @@ jobs:
7780
restore-keys: |
7881
${{ runner.os }}-java-maven-
7982
80-
- name: Build Comet (debug, ${{ matrix.name }} / Scala 2.13)
81-
run: |
82-
cd native && cargo build
83-
cd .. && ./mvnw -B install -DskipTests ${{ matrix.maven_profiles }}
84-
8583
- name: Install Python 3.11 and pip
8684
run: |
8785
apt-get update
88-
apt-get install -y --no-install-recommends python3 python3-venv python3-pip
86+
apt-get install -y --no-install-recommends python3 python3-dev python3-venv python3-pip
8987
python3 -m venv /tmp/venv
9088
/tmp/venv/bin/pip install --upgrade pip
9189
/tmp/venv/bin/pip install "pyspark==${{ matrix.pyspark }}" "pyarrow>=14" pandas pytest
9290
91+
- name: Build Comet (debug, ${{ matrix.name }} / Scala 2.13)
92+
env:
93+
PYO3_PYTHON: /tmp/venv/bin/python
94+
run: |
95+
if [ "${{ matrix.native_python_udf }}" = "true" ]; then
96+
(cd native && cargo build --features python-udf)
97+
else
98+
(cd native && cargo build)
99+
fi
100+
./mvnw -B install -DskipTests ${{ matrix.maven_profiles }}
101+
102+
- name: Run native Arrow UDF Scala suite
103+
if: matrix.native_python_udf
104+
env:
105+
PYSPARK_PYTHON: /tmp/venv/bin/python
106+
PYTHONPATH: /tmp/venv/lib/python3.11/site-packages
107+
run: |
108+
./mvnw -B test ${{ matrix.maven_profiles }} \
109+
-Dsuites=org.apache.spark.sql.comet.CometArrowPythonUdfSuite
110+
93111
- name: Run PyArrow UDF pytest
94112
env:
95113
# Spark launches Python workers in a fresh subprocess and looks up `python3`
@@ -98,11 +116,17 @@ jobs:
98116
# ModuleNotFoundError.
99117
PYSPARK_PYTHON: /tmp/venv/bin/python
100118
PYSPARK_DRIVER_PYTHON: /tmp/venv/bin/python
119+
# The embedded interpreter does not inherit PySpark worker sys.path setup.
120+
PYTHONPATH: /tmp/venv/lib/python3.11/site-packages
101121
run: |
102122
/tmp/venv/bin/python -m pytest -v \
103123
spark/src/test/resources/pyspark/test_pyarrow_udf.py
104124
/tmp/venv/bin/python -m pytest -v \
105125
spark/src/test/resources/pyspark/test_pyarrow_udf_dictionary_shuffle.py
126+
if [ "${{ matrix.native_python_udf }}" = "true" ]; then
127+
/tmp/venv/bin/python -m pytest -v \
128+
spark/src/test/resources/pyspark/test_native_arrow_udf_files.py
129+
fi
106130
/tmp/venv/bin/python -m pytest -v \
107131
spark/src/test/resources/pyspark/test_pyarrow_udf_fuzz.py
108132

0 commit comments

Comments
 (0)