Skip to content

Commit 1a5b53d

Browse files
committed
Merge remote-tracking branch 'origin/main' into perf/fair-pool-lock-free
# Conflicts: # native/core/src/execution/memory_pools/fair_pool.rs
2 parents 48e2209 + 330b29c commit 1a5b53d

69 files changed

Lines changed: 2925 additions & 1083 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.ai/skills/review-comet-memory-pr/SKILL.md‎

Lines changed: 27 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -100,19 +100,27 @@ configuration, from the inside out:
100100

101101
- [ ] A new decorator forwards **every** `MemoryPool` method to its inner pool. A partial
102102
implementation makes `reserved()` disagree between levels.
103-
- [ ] **`fair_unified` checks the requesting consumer against its share, and the total against
104-
the pool.** It divides `pool_size` by the number of registered consumers and rejects if what
105-
the consumer holds plus the request exceeds that quotient, or if the pool's total reserved
106-
plus the request exceeds `pool_size`. With two consumers and an 8 GiB pool, once one holds
107-
3 GiB the other can still reserve up to 4 GiB. The pool keeps a running total for each
108-
consumer id, so the sibling reservations that `new_empty()`, `split()` and `take()` create
109-
count against one share. A PR that checks `reservation.size()` instead lets an operator with
110-
several reservations, such as a sort's streaming merge, take other consumers' shares. It
111-
also depends on when DataFusion updates the size, which is after it calls `try_grow` but
112-
before it calls `shrink`. If the PR changes either comparison it changes when every query
113-
spills, so it needs benchmark evidence, not reasoning.
114-
- [ ] **`num_consumers` counts every consumer in the task**, across every native plan, because the
115-
pool is task-shared.
103+
- [ ] **`fair_unified` holds the consumers that can spill to a share of the task's share.** The
104+
task's share is `spark.memory.offHeap.size` divided by the tasks on the executor running
105+
native plans, which `RunningTasks` counts while each task holds its task-shared pool, or
106+
`pool_size` if that is smaller. A consumer that can spill is
107+
refused if what it holds plus the request exceeds what the consumers that cannot spill leave
108+
of the task's share, divided by the registered consumers that can spill, or if the task's
109+
total plus the request exceeds the task's share. A consumer that cannot spill is refused only
110+
if the task's total plus the request exceeds `pool_size`, which binds before Spark's own
111+
limit only while the deprecated fraction is below `1 / N`. With eight running tasks, a 16 GiB
112+
pool and two consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge holds
113+
512 MiB. The pool keeps a running total for each consumer id, so the sibling reservations
114+
that `new_empty()`, `split()` and `take()` create count against one share. A PR that checks
115+
`reservation.size()` instead lets an operator with several reservations, such as a sort's
116+
streaming merge, take other consumers' shares. It also depends on when DataFusion updates the
117+
size, which is after it calls `try_grow` but before it calls `shrink`. A PR that limits a
118+
consumer that cannot spill to a share, or sizes the shares from the whole pool, brings back
119+
#7088: one operator takes the task's share from Spark and a merge or hash join build fails
120+
the task. If the PR changes any of these comparisons it changes when every query spills, so
121+
it needs benchmark evidence, not reasoning.
122+
- [ ] **The consumers that divide the task's share are every consumer that can spill in the
123+
task**, across every native plan, because the pool is task-shared.
116124
- [ ] **Task-shared pool lifetime.** `acquire_task_shared_pool` keeps a process-wide
117125
`HashMap<task_attempt_id, Weak<TaskSharedMemoryPool>>`. The returned `Arc` is the only
118126
lifetime handle, so there is no explicit release to forget and a `createPlan` that fails
@@ -143,14 +151,16 @@ reclamation, that is a significant change and needs to say what it does to in-fl
143151
## 5. Budget and Configuration
144152

145153
`CometExecIterator.getMemoryConfig` computes the budget once per executor and passes it across JNI
146-
as `memory_limit`:
154+
as `memory_limit`, along with `spark.memory.offHeap.size` itself as `off_heap_size`, which
155+
`fair_unified` divides among the tasks running native plans:
147156

148157
```text
149158
memory_limit = spark.memory.offHeap.size * spark.comet.exec.memoryPool.fraction
159+
off_heap_size = spark.memory.offHeap.size
150160
```
151161

152162
- [ ] A new memory config goes through `getMemoryConfig` rather than being read separately in
153-
native code.
163+
native code, and a new value crossing JNI is added to `createPlan` on both sides.
154164
- [ ] Changing `memoryPool.fraction` semantics affects every deployment that tuned it as a haircut
155165
for the accounting gap.
156166
- [ ] On Kubernetes, `spark.memory.offHeap.size` is **part of** the pod limit, not headroom on top
@@ -204,8 +214,8 @@ will not catch. Check:
204214
- The **pool stack** diagram, if a decorator is added, removed, or reordered
205215
- The **pool type** table, if a pool type is added, removed, or changes how it is sized
206216
- The `memory_limit` formula, if the budget calculation changes
207-
- The **fair pool** description, which spells out the share and pool-total checks and their
208-
consequences
217+
- The **fair pool** description, which spells out the task share, the share and task-total
218+
checks, and their consequences
209219
- The **Crossing the FFI boundary** section, which names the operators that reserve for imported
210220
batches
211221
- The **Open problems** list. If the PR closes one of these gaps, the entry must be removed or

‎benchmarks/tpc/README.md‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ python3 run.py --engine <engine> --benchmark <tpch|tpcds> [options]
4545
| `--iterations` | Number of iterations (default: 1) |
4646
| `--output` | Output directory (default: `.`) |
4747
| `--query` | Run a single query number |
48+
| `--plan-dir` | Optional directory to write formatted physical plans per query |
4849
| `--no-restart` | Skip Spark master/worker restart |
4950
| `--dry-run` | Print the spark-submit command without executing |
5051
| `--jfr` | Enable Java Flight Recorder profiling |
@@ -54,7 +55,7 @@ python3 run.py --engine <engine> --benchmark <tpch|tpcds> [options]
5455
| `--async-profiler-event` | Event type: `cpu`, `wall`, `alloc`, `lock`, etc. (default: `cpu`) |
5556
| `--async-profiler-format` | Output format: `flamegraph`, `jfr`, `collapsed`, `text` (default: `flamegraph`) |
5657

57-
Available engines: `spark`, `comet`, `comet-iceberg`, `gluten`
58+
Available engines: `spark`, `spark-iceberg`, `comet`, `comet-iceberg`, `gluten`
5859

5960
## Example usage
6061

@@ -168,21 +169,35 @@ $SPARK_HOME/bin/spark-submit \
168169
--warehouse $ICEBERG_WAREHOUSE
169170
```
170171

171-
### Run Iceberg benchmark
172+
### Run Iceberg benchmarks
172173

173174
```shell
174175
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
175176
export COMET_JAR=/opt/comet/comet-spark-spark3.5_2.12-0.10.0.jar
176177
export ICEBERG_JAR=/path/to/iceberg-spark-runtime-3.5_2.12-1.8.1.jar
177178
export ICEBERG_WAREHOUSE=/mnt/bigdata/iceberg-warehouse
178179
sudo ./drop-caches.sh
180+
181+
python3 run.py --engine spark-iceberg --benchmark tpch
179182
python3 run.py --engine comet-iceberg --benchmark tpch
180183
```
181184

185+
Use `spark-iceberg` as the Spark JVM Iceberg scan baseline and `comet-iceberg`
186+
as the Comet native Iceberg scan comparison. Both engines use `ICEBERG_JAR`,
187+
`ICEBERG_WAREHOUSE`, optional `ICEBERG_CATALOG`, and optional `ICEBERG_DATABASE`.
188+
182189
The benchmark uses Comet's native iceberg-rust integration, which is enabled by default.
183190
Verify native scanning is active by checking for `CometIcebergNativeScanExec` in the
184191
physical plan output.
185192

193+
### Plan capture
194+
195+
Pass `--plan-dir <dir>` to write each query's formatted physical plan to
196+
`<dir>/<engine-name>-q<query-label>.plan.txt`. Plan capture is opt-in and
197+
repeated iterations overwrite the same plan files. TPC-H writes up to 22 files per engine.
198+
TPC-DS writes up to 103 files per engine because q14, q23, q24, and q39 each contain two
199+
SELECT statements.
200+
186201
### create-iceberg-tables.py options
187202

188203
| Option | Required | Default | Description |

‎benchmarks/tpc/create-iceberg-tables.py‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,7 +118,26 @@ def main(benchmark: str, parquet_path: str, warehouse: str, catalog: str, databa
118118
pass
119119

120120
for table in table_names:
121-
parquet_table_path = f"{parquet_path}/{table}.parquet"
121+
parquet_file_path = f"{parquet_path}/{table}.parquet"
122+
parquet_dir_path = f"{parquet_path}/{table}"
123+
parquet_file_exists = os.path.exists(parquet_file_path)
124+
parquet_dir_exists = os.path.exists(parquet_dir_path)
125+
126+
if parquet_file_exists:
127+
parquet_table_path = parquet_file_path
128+
if parquet_dir_exists:
129+
print(
130+
f"Warning: both '{parquet_file_path}' and '{parquet_dir_path}' exist; "
131+
f"using '{parquet_file_path}'",
132+
file=sys.stderr,
133+
)
134+
elif parquet_dir_exists:
135+
parquet_table_path = parquet_dir_path
136+
else:
137+
raise FileNotFoundError(
138+
f"Could not find Parquet data for table '{table}'. "
139+
f"Expected '{parquet_file_path}' or '{parquet_dir_path}'."
140+
)
122141
iceberg_table = f"{catalog}.{database}.{table}"
123142

124143
print(f"Converting {parquet_table_path} -> {iceberg_table}")
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
# Spark + Iceberg JVM scan (no Comet).
19+
# Baseline for comparing against comet-iceberg (native Rust scan).
20+
21+
[engine]
22+
name = "spark-iceberg"
23+
24+
[env]
25+
required = ["ICEBERG_JAR", "ICEBERG_WAREHOUSE"]
26+
27+
[env.defaults]
28+
ICEBERG_CATALOG = "local"
29+
30+
[spark_submit]
31+
jars = ["$ICEBERG_JAR"]
32+
driver_class_path = ["$ICEBERG_JAR"]
33+
34+
[spark_conf]
35+
"spark.driver.extraClassPath" = "$ICEBERG_JAR"
36+
"spark.executor.extraClassPath" = "$ICEBERG_JAR"
37+
"spark.sql.catalog.${ICEBERG_CATALOG}" = "org.apache.iceberg.spark.SparkCatalog"
38+
"spark.sql.catalog.${ICEBERG_CATALOG}.type" = "hadoop"
39+
"spark.sql.catalog.${ICEBERG_CATALOG}.warehouse" = "$ICEBERG_WAREHOUSE"
40+
"spark.sql.defaultCatalog" = "${ICEBERG_CATALOG}"
41+
42+
[tpcbench_args]
43+
use_iceberg = true

‎benchmarks/tpc/run.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -337,6 +337,9 @@ def build_spark_submit_cmd(config, benchmark, args):
337337
if args.query is not None:
338338
cmd += ["--query", str(args.query)]
339339

340+
if args.plan_dir is not None:
341+
cmd += ["--plan-dir", args.plan_dir]
342+
340343
if profile["format"] and not use_iceberg:
341344
cmd += ["--format", profile["format"]]
342345

@@ -397,6 +400,10 @@ def main():
397400
parser.add_argument(
398401
"--query", type=int, default=None, help="Run a single query number"
399402
)
403+
parser.add_argument(
404+
"--plan-dir",
405+
help="Optional directory to write formatted physical plans per query",
406+
)
400407
parser.add_argument(
401408
"--no-restart",
402409
action="store_true",

‎benchmarks/tpc/tpcbench.py‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ def main(
8383
query_num: int = None,
8484
write_path: str = None,
8585
options: Dict[str, str] = None,
86+
plan_dir: str = None,
8687
):
8788
if options is None:
8889
options = {}
@@ -190,6 +191,19 @@ def main(
190191
df = spark.sql(sql)
191192
df.explain("formatted")
192193

194+
if plan_dir is not None and is_query:
195+
os.makedirs(plan_dir, exist_ok=True)
196+
plan_path = os.path.join(plan_dir, f"{name}-q{query_label}.plan.txt")
197+
try:
198+
plan_str = spark._jvm.PythonSQLUtils.explainString(
199+
df._jdf.queryExecution(), "formatted"
200+
)
201+
with open(plan_path, "w") as pf:
202+
pf.write(plan_str)
203+
print(f"Plan saved to {plan_path}")
204+
except Exception as e:
205+
print(f"Warning: could not save plan: {e}")
206+
193207
if is_query and write_path is not None:
194208
if len(df.columns) > 0:
195209
output_path = f"{write_path}/q{query_label}"
@@ -275,6 +289,10 @@ def main(
275289
"--name", required=True,
276290
help="Prefix for result file"
277291
)
292+
parser.add_argument(
293+
"--plan-dir",
294+
help="Optional directory to write formatted plans per query"
295+
)
278296
parser.add_argument(
279297
"--query", type=int,
280298
help="Specific query number (1-based). If omitted, run all."
@@ -297,4 +315,5 @@ def main(
297315
args.query,
298316
args.write,
299317
args.options,
318+
args.plan_dir,
300319
)

‎common/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ under the License.
2626
<parent>
2727
<groupId>org.apache.datafusion</groupId>
2828
<artifactId>comet-parent-spark${spark.version.short}_${scala.binary.version}</artifactId>
29-
<version>1.2.0-SNAPSHOT</version>
29+
<version>1.3.0-SNAPSHOT</version>
3030
<relativePath>../pom.xml</relativePath>
3131
</parent>
3232

‎contrib/delta-spark/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
<parent>
2525
<groupId>org.apache.datafusion</groupId>
2626
<artifactId>comet-parent-spark${spark.version.short}_${scala.binary.version}</artifactId>
27-
<version>1.2.0-SNAPSHOT</version>
27+
<version>1.3.0-SNAPSHOT</version>
2828
<relativePath>../../pom.xml</relativePath>
2929
</parent>
3030

‎contrib/delta/native/Cargo.lock‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎contrib/delta/native/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@
3030
[package]
3131
name = "comet-contrib-delta"
3232
description = "delta-kernel-rs integration for Comet (build-gate stub; native read path lands in a later unit). Linked into libcomet via core's `contrib-delta` Cargo feature flag."
33-
version = "1.2.0"
33+
version = "1.3.0"
3434
edition = "2021"
3535
rust-version = "1.86.0"
3636
publish = false

0 commit comments

Comments
 (0)