Skip to content

Commit 450d37d

Browse files
Merge branch 'main' into docs/4420-docs-review-skill
2 parents 6ef1f41 + b137ca5 commit 450d37d

268 files changed

Lines changed: 24440 additions & 4004 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/review-comet-memory-pr/SKILL.md‎

Lines changed: 27 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -100,19 +100,27 @@ configuration, from the inside out:
100100

101101
- [ ] A new decorator forwards **every** `MemoryPool` method to its inner pool. A partial
102102
implementation makes `reserved()` disagree between levels.
103-
- [ ] **`fair_unified` checks the requesting consumer against its share, and the total against
104-
the pool.** It divides `pool_size` by the number of registered consumers and rejects if what
105-
the consumer holds plus the request exceeds that quotient, or if the pool's total reserved
106-
plus the request exceeds `pool_size`. With two consumers and an 8 GiB pool, once one holds
107-
3 GiB the other can still reserve up to 4 GiB. The pool keeps a running total for each
108-
consumer id, so the sibling reservations that `new_empty()`, `split()` and `take()` create
109-
count against one share. A PR that checks `reservation.size()` instead lets an operator with
110-
several reservations, such as a sort's streaming merge, take other consumers' shares. It
111-
also depends on when DataFusion updates the size, which is after it calls `try_grow` but
112-
before it calls `shrink`. If the PR changes either comparison it changes when every query
113-
spills, so it needs benchmark evidence, not reasoning.
114-
- [ ] **`num_consumers` counts every consumer in the task**, across every native plan, because the
115-
pool is task-shared.
103+
- [ ] **`fair_unified` holds the consumers that can spill to a share of the task's share.** The
104+
task's share is `spark.memory.offHeap.size` divided by the tasks on the executor running
105+
native plans, which `RunningTasks` counts while each task holds its task-shared pool, or
106+
`pool_size` if that is smaller. A consumer that can spill is
107+
refused if what it holds plus the request exceeds what the consumers that cannot spill leave
108+
of the task's share, divided by the registered consumers that can spill, or if the task's
109+
total plus the request exceeds the task's share. A consumer that cannot spill is refused only
110+
if the task's total plus the request exceeds `pool_size`, which binds before Spark's own
111+
limit only while the deprecated fraction is below `1 / N`. With eight running tasks, a 16 GiB
112+
pool and two consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge holds
113+
512 MiB. The pool keeps a running total for each consumer id, so the sibling reservations
114+
that `new_empty()`, `split()` and `take()` create count against one share. A PR that checks
115+
`reservation.size()` instead lets an operator with several reservations, such as a sort's
116+
streaming merge, take other consumers' shares. It also depends on when DataFusion updates the
117+
size, which is after it calls `try_grow` but before it calls `shrink`. A PR that limits a
118+
consumer that cannot spill to a share, or sizes the shares from the whole pool, brings back
119+
#7088: one operator takes the task's share from Spark and a merge or hash join build fails
120+
the task. If the PR changes any of these comparisons it changes when every query spills, so
121+
it needs benchmark evidence, not reasoning.
122+
- [ ] **The consumers that divide the task's share are every consumer that can spill in the
123+
task**, across every native plan, because the pool is task-shared.
116124
- [ ] **Task-shared pool lifetime.** `acquire_task_shared_pool` keeps a process-wide
117125
`HashMap<task_attempt_id, Weak<TaskSharedMemoryPool>>`. The returned `Arc` is the only
118126
lifetime handle, so there is no explicit release to forget and a `createPlan` that fails
@@ -143,14 +151,16 @@ reclamation, that is a significant change and needs to say what it does to in-fl
143151
## 5. Budget and Configuration
144152

145153
`CometExecIterator.getMemoryConfig` computes the budget once per executor and passes it across JNI
146-
as `memory_limit`:
154+
as `memory_limit`, along with `spark.memory.offHeap.size` itself as `off_heap_size`, which
155+
`fair_unified` divides among the tasks running native plans:
147156

148157
```text
149158
memory_limit = spark.memory.offHeap.size * spark.comet.exec.memoryPool.fraction
159+
off_heap_size = spark.memory.offHeap.size
150160
```
151161

152162
- [ ] A new memory config goes through `getMemoryConfig` rather than being read separately in
153-
native code.
163+
native code, and a new value crossing JNI is added to `createPlan` on both sides.
154164
- [ ] Changing `memoryPool.fraction` semantics affects every deployment that tuned it as a haircut
155165
for the accounting gap.
156166
- [ ] On Kubernetes, `spark.memory.offHeap.size` is **part of** the pod limit, not headroom on top
@@ -204,8 +214,8 @@ will not catch. Check:
204214
- The **pool stack** diagram, if a decorator is added, removed, or reordered
205215
- The **pool type** table, if a pool type is added, removed, or changes how it is sized
206216
- The `memory_limit` formula, if the budget calculation changes
207-
- The **fair pool** description, which spells out the share and pool-total checks and their
208-
consequences
217+
- The **fair pool** description, which spells out the task share, the share and task-total
218+
checks, and their consequences
209219
- The **Crossing the FFI boundary** section, which names the operators that reserve for imported
210220
batches
211221
- The **Open problems** list. If the PR closes one of these gaps, the entry must be removed or

‎.asf.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,9 @@ github:
116116
branch-1.1:
117117
required_pull_request_reviews:
118118
required_approving_review_count: 1
119+
branch-1.2:
120+
required_pull_request_reviews:
121+
required_approving_review_count: 1
119122
# GitHub's merge queue. Classic `protected_branches` above has no way to
120123
# express this, but .asf.yaml rulesets accept a raw Rulesets API payload, so
121124
# no INFRA ticket is needed. Same approach as apache/iceberg-python and

‎.github/actions/rust-test/action.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,28 @@ runs:
2323
steps:
2424
# Note: cargo fmt check is now handled by the lint job that gates this workflow
2525

26+
- name: Set up embedded Python for native UDF tests
27+
shell: bash
28+
run: |
29+
apt-get update
30+
apt-get install -y --no-install-recommends python3 python3-dev python3-venv
31+
python3 -m venv /tmp/comet-rust-python
32+
/tmp/comet-rust-python/bin/pip install "pyarrow>=14" cloudpickle
33+
echo "PYO3_PYTHON=/tmp/comet-rust-python/bin/python" >> "$GITHUB_ENV"
34+
echo "PYTHONPATH=$(/tmp/comet-rust-python/bin/python -c 'import sysconfig; print(sysconfig.get_path("purelib"))')" >> "$GITHUB_ENV"
35+
2636
- name: Check Cargo clippy
2737
shell: bash
2838
run: |
2939
cd native
3040
cargo clippy --color=never --all-targets --workspace -- -D warnings
3141
42+
- name: Check native Python UDF feature
43+
shell: bash
44+
run: |
45+
cd native
46+
cargo clippy --color=never -p datafusion-comet --all-targets --features python-udf -- -D warnings
47+
3248
- name: Check compilation
3349
shell: bash
3450
run: |
@@ -83,6 +99,13 @@ runs:
8399
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
84100
RUST_BACKTRACE=1 cargo nextest run
85101
102+
- name: Run native Python UDF tests
103+
shell: bash
104+
run: |
105+
cd native
106+
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
107+
RUST_BACKTRACE=1 cargo nextest run -p datafusion-comet --lib --features python-udf python_udf
108+
86109
# The steps above lint and test the accounting allocator over the system allocator. Lint and
87110
# test it over jemalloc too, since that is a different allocator backend and nothing else in CI
88111
# builds it.

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -497,6 +497,7 @@ jobs:
497497
- name: "scans"
498498
value: |
499499
org.apache.comet.parquet.CometParquetWriterSuite
500+
org.apache.comet.parquet.CometParquetWriterRetrySuite
500501
org.apache.comet.parquet.CometEmptyRelationParquetWriterSuite
501502
org.apache.spark.sql.comet.CometNativeWriteSuite
502503
org.apache.comet.parquet.ParquetReadV1Suite
@@ -541,6 +542,7 @@ jobs:
541542
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleWriterSuite
542543
org.apache.spark.sql.comet.execution.shuffle.CometNativePositionalRoundRobinSuite
543544
org.apache.spark.sql.comet.execution.shuffle.CometDiskBlockWriterSuite
545+
org.apache.spark.sql.comet.execution.shuffle.CometBlockStoreShuffleReaderSuite
544546
org.apache.comet.exec.CometShuffleEncryptionSuite
545547
org.apache.comet.exec.CometShuffleManagerSuite
546548
org.apache.comet.exec.DisableAQECometShuffleSuite
@@ -567,6 +569,7 @@ jobs:
567569
org.apache.comet.exec.CometJoinSuite
568570
org.apache.comet.exec.CometTypedDatasetSuite
569571
org.apache.spark.sql.comet.CometMapInBatchSuite
572+
org.apache.spark.sql.comet.CometArrowPythonUdfSuite
570573
org.apache.spark.sql.execution.python.CometArrowPythonRunnerSuite
571574
org.apache.comet.CometNativeSuite
572575
org.apache.comet.DirectColumnarToRowConverterSuite
@@ -612,6 +615,7 @@ jobs:
612615
org.apache.spark.sql.comet.DecimalPrecisionSuite
613616
org.apache.spark.sql.comet.CometScanWithPlanDataSuite
614617
org.apache.spark.sql.comet.util.UtilsSuite
618+
org.apache.spark.sql.comet.CometBroadcastValueSuite
615619
org.apache.comet.vector.NativeUtilSuite
616620
org.apache.comet.vector.CometVectorUtilsSuite
617621
org.apache.comet.udf.CometUdfBridgeSuite
@@ -674,6 +678,7 @@ jobs:
674678
org.apache.comet.serde.CometScalarFunctionSuite
675679
org.apache.comet.serde.CometLiteralSuite
676680
org.apache.comet.serde.QueryPlanSerdeSuite
681+
org.apache.comet.serde.SparkFixesSuite
677682
org.apache.comet.CometFallbackInvarianceSuite
678683
fail-fast: false
679684
name: ${{ matrix.profile.name }} [${{ matrix.suite.name }}]

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,7 @@ jobs:
163163
- name: "scans"
164164
value: |
165165
org.apache.comet.parquet.CometParquetWriterSuite
166+
org.apache.comet.parquet.CometParquetWriterRetrySuite
166167
org.apache.comet.parquet.CometEmptyRelationParquetWriterSuite
167168
org.apache.spark.sql.comet.CometNativeWriteSuite
168169
org.apache.comet.parquet.ParquetReadV1Suite
@@ -207,6 +208,7 @@ jobs:
207208
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleWriterSuite
208209
org.apache.spark.sql.comet.execution.shuffle.CometNativePositionalRoundRobinSuite
209210
org.apache.spark.sql.comet.execution.shuffle.CometDiskBlockWriterSuite
211+
org.apache.spark.sql.comet.execution.shuffle.CometBlockStoreShuffleReaderSuite
210212
org.apache.comet.exec.CometShuffleEncryptionSuite
211213
org.apache.comet.exec.CometShuffleManagerSuite
212214
org.apache.comet.exec.DisableAQECometShuffleSuite
@@ -233,6 +235,7 @@ jobs:
233235
org.apache.comet.exec.CometJoinSuite
234236
org.apache.comet.exec.CometTypedDatasetSuite
235237
org.apache.spark.sql.comet.CometMapInBatchSuite
238+
org.apache.spark.sql.comet.CometArrowPythonUdfSuite
236239
org.apache.spark.sql.execution.python.CometArrowPythonRunnerSuite
237240
org.apache.comet.CometNativeSuite
238241
org.apache.comet.DirectColumnarToRowConverterSuite
@@ -278,6 +281,7 @@ jobs:
278281
org.apache.spark.sql.comet.DecimalPrecisionSuite
279282
org.apache.spark.sql.comet.CometScanWithPlanDataSuite
280283
org.apache.spark.sql.comet.util.UtilsSuite
284+
org.apache.spark.sql.comet.CometBroadcastValueSuite
281285
org.apache.comet.vector.NativeUtilSuite
282286
org.apache.comet.vector.CometVectorUtilsSuite
283287
org.apache.comet.udf.CometUdfBridgeSuite
@@ -340,6 +344,7 @@ jobs:
340344
org.apache.comet.serde.CometScalarFunctionSuite
341345
org.apache.comet.serde.CometLiteralSuite
342346
org.apache.comet.serde.QueryPlanSerdeSuite
347+
org.apache.comet.serde.SparkFixesSuite
343348
org.apache.comet.CometFallbackInvarianceSuite
344349
345350
fail-fast: false

‎.github/workflows/pyarrow_udf_test.yml‎

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -44,12 +44,15 @@ jobs:
4444
- name: Spark 4.0
4545
maven_profiles: "-Pspark-4.0 -Pscala-2.13"
4646
pyspark: "4.0.4"
47+
native_python_udf: false
4748
- name: Spark 4.1
4849
maven_profiles: "-Pspark-4.1"
4950
pyspark: "4.1.3"
51+
native_python_udf: true
5052
- name: Spark 4.2
5153
maven_profiles: "-Pspark-4.2"
5254
pyspark: "4.2.0"
55+
native_python_udf: true
5356
container:
5457
# Pinned to the Debian 12 (bookworm) base so the system `python3` is 3.11. The default
5558
# `amd64/rust` image is Debian 13 (trixie) which ships Python 3.13 and no python3.11 apt
@@ -77,19 +80,34 @@ jobs:
7780
restore-keys: |
7881
${{ runner.os }}-java-maven-
7982
80-
- name: Build Comet (debug, ${{ matrix.name }} / Scala 2.13)
81-
run: |
82-
cd native && cargo build
83-
cd .. && ./mvnw -B install -DskipTests ${{ matrix.maven_profiles }}
84-
8583
- name: Install Python 3.11 and pip
8684
run: |
8785
apt-get update
88-
apt-get install -y --no-install-recommends python3 python3-venv python3-pip
86+
apt-get install -y --no-install-recommends python3 python3-dev python3-venv python3-pip
8987
python3 -m venv /tmp/venv
9088
/tmp/venv/bin/pip install --upgrade pip
9189
/tmp/venv/bin/pip install "pyspark==${{ matrix.pyspark }}" "pyarrow>=14" pandas pytest
9290
91+
- name: Build Comet (debug, ${{ matrix.name }} / Scala 2.13)
92+
env:
93+
PYO3_PYTHON: /tmp/venv/bin/python
94+
run: |
95+
if [ "${{ matrix.native_python_udf }}" = "true" ]; then
96+
(cd native && cargo build --features python-udf)
97+
else
98+
(cd native && cargo build)
99+
fi
100+
./mvnw -B install -DskipTests ${{ matrix.maven_profiles }}
101+
102+
- name: Run native Arrow UDF Scala suite
103+
if: matrix.native_python_udf
104+
env:
105+
PYSPARK_PYTHON: /tmp/venv/bin/python
106+
PYTHONPATH: /tmp/venv/lib/python3.11/site-packages
107+
run: |
108+
./mvnw -B test ${{ matrix.maven_profiles }} \
109+
-Dsuites=org.apache.spark.sql.comet.CometArrowPythonUdfSuite
110+
93111
- name: Run PyArrow UDF pytest
94112
env:
95113
# Spark launches Python workers in a fresh subprocess and looks up `python3`
@@ -98,11 +116,17 @@ jobs:
98116
# ModuleNotFoundError.
99117
PYSPARK_PYTHON: /tmp/venv/bin/python
100118
PYSPARK_DRIVER_PYTHON: /tmp/venv/bin/python
119+
# The embedded interpreter does not inherit PySpark worker sys.path setup.
120+
PYTHONPATH: /tmp/venv/lib/python3.11/site-packages
101121
run: |
102122
/tmp/venv/bin/python -m pytest -v \
103123
spark/src/test/resources/pyspark/test_pyarrow_udf.py
104124
/tmp/venv/bin/python -m pytest -v \
105125
spark/src/test/resources/pyspark/test_pyarrow_udf_dictionary_shuffle.py
126+
if [ "${{ matrix.native_python_udf }}" = "true" ]; then
127+
/tmp/venv/bin/python -m pytest -v \
128+
spark/src/test/resources/pyspark/test_native_arrow_udf_files.py
129+
fi
106130
/tmp/venv/bin/python -m pytest -v \
107131
spark/src/test/resources/pyspark/test_pyarrow_udf_fuzz.py
108132

‎benchmarks/tpc/README.md‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ python3 run.py --engine <engine> --benchmark <tpch|tpcds> [options]
4545
| `--iterations` | Number of iterations (default: 1) |
4646
| `--output` | Output directory (default: `.`) |
4747
| `--query` | Run a single query number |
48+
| `--plan-dir` | Optional directory to write formatted physical plans per query |
4849
| `--no-restart` | Skip Spark master/worker restart |
4950
| `--dry-run` | Print the spark-submit command without executing |
5051
| `--jfr` | Enable Java Flight Recorder profiling |
@@ -54,7 +55,7 @@ python3 run.py --engine <engine> --benchmark <tpch|tpcds> [options]
5455
| `--async-profiler-event` | Event type: `cpu`, `wall`, `alloc`, `lock`, etc. (default: `cpu`) |
5556
| `--async-profiler-format` | Output format: `flamegraph`, `jfr`, `collapsed`, `text` (default: `flamegraph`) |
5657

57-
Available engines: `spark`, `comet`, `comet-iceberg`, `gluten`
58+
Available engines: `spark`, `spark-iceberg`, `comet`, `comet-iceberg`, `gluten`
5859

5960
## Example usage
6061

@@ -168,21 +169,35 @@ $SPARK_HOME/bin/spark-submit \
168169
--warehouse $ICEBERG_WAREHOUSE
169170
```
170171

171-
### Run Iceberg benchmark
172+
### Run Iceberg benchmarks
172173

173174
```shell
174175
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
175176
export COMET_JAR=/opt/comet/comet-spark-spark3.5_2.12-0.10.0.jar
176177
export ICEBERG_JAR=/path/to/iceberg-spark-runtime-3.5_2.12-1.8.1.jar
177178
export ICEBERG_WAREHOUSE=/mnt/bigdata/iceberg-warehouse
178179
sudo ./drop-caches.sh
180+
181+
python3 run.py --engine spark-iceberg --benchmark tpch
179182
python3 run.py --engine comet-iceberg --benchmark tpch
180183
```
181184

185+
Use `spark-iceberg` as the Spark JVM Iceberg scan baseline and `comet-iceberg`
186+
as the Comet native Iceberg scan comparison. Both engines use `ICEBERG_JAR`,
187+
`ICEBERG_WAREHOUSE`, optional `ICEBERG_CATALOG`, and optional `ICEBERG_DATABASE`.
188+
182189
The benchmark uses Comet's native iceberg-rust integration, which is enabled by default.
183190
Verify native scanning is active by checking for `CometIcebergNativeScanExec` in the
184191
physical plan output.
185192

193+
### Plan capture
194+
195+
Pass `--plan-dir <dir>` to write each query's formatted physical plan to
196+
`<dir>/<engine-name>-q<query-label>.plan.txt`. Plan capture is opt-in and
197+
repeated iterations overwrite the same plan files. TPC-H writes up to 22 files per engine.
198+
TPC-DS writes up to 103 files per engine because q14, q23, q24, and q39 each contain two
199+
SELECT statements.
200+
186201
### create-iceberg-tables.py options
187202

188203
| Option | Required | Default | Description |

‎benchmarks/tpc/create-iceberg-tables.py‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,26 @@ def main(benchmark: str, parquet_path: str, warehouse: str, catalog: str, databa
118118
pass
119119

120120
for table in table_names:
121-
parquet_table_path = f"{parquet_path}/{table}.parquet"
121+
parquet_file_path = f"{parquet_path}/{table}.parquet"
122+
parquet_dir_path = f"{parquet_path}/{table}"
123+
parquet_file_exists = os.path.exists(parquet_file_path)
124+
parquet_dir_exists = os.path.exists(parquet_dir_path)
125+
126+
if parquet_file_exists:
127+
parquet_table_path = parquet_file_path
128+
if parquet_dir_exists:
129+
print(
130+
f"Warning: both '{parquet_file_path}' and '{parquet_dir_path}' exist; "
131+
f"using '{parquet_file_path}'",
132+
file=sys.stderr,
133+
)
134+
elif parquet_dir_exists:
135+
parquet_table_path = parquet_dir_path
136+
else:
137+
raise FileNotFoundError(
138+
f"Could not find Parquet data for table '{table}'. "
139+
f"Expected '{parquet_file_path}' or '{parquet_dir_path}'."
140+
)
122141
iceberg_table = f"{catalog}.{database}.{table}"
123142

124143
print(f"Converting {parquet_table_path} -> {iceberg_table}")

0 commit comments

Comments
 (0)