Skip to content

Commit 545735c

Browse files
committed
Merge remote-tracking branch 'origin/main' into fix/6746-multi-store-scan-partitions
2 parents cbf3b34 + 019f3be commit 545735c

45 files changed

Lines changed: 1457 additions & 279 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/source/contributor-guide/jvm_shuffle.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,9 @@ JVM shuffle (`CometColumnarExchange`) is used instead of native shuffle (`CometE
5252
range key always falls back here. `HashPartitioning` keys must be primitive only by default:
5353
setting `spark.comet.shuffle.native.partitioning.hash.nested.enabled` to `true` keeps struct and
5454
array keys, and map keys on Spark 4.0 and later, on the native path. The config defaults to
55-
`false`, so a complex hash key falls back to JVM columnar shuffle unless it is enabled. See
55+
`false`, so a complex hash key falls back to JVM columnar shuffle unless it is enabled.
56+
Decimal hash keys with precision greater than 18, including nested decimal leaves, also use
57+
JVM shuffle when there is more than one partition and the shuffle mode allows it. See
5658
[Supported partition key types](native_shuffle.md#when-native-shuffle-is-used) for the exact
5759
rules. Complex types are fully supported as data columns in both implementations.
5860

‎docs/source/contributor-guide/native_shuffle.md‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -83,12 +83,9 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
8383
Spark's `mapsort` normalization makes physical entry order irrelevant. A collated string at any
8484
depth still disqualifies the key. The config defaults to `false` pending measurement of the
8585
nested hashing paths, so by default a complex hash key falls back to JVM shuffle.
86-
- A hash key that is or contains a decimal wider than 18 digits stays on JVM shuffle when the
87-
shuffle's stage starts at a typed `Dataset` conversion
88-
(`spark.comet.convert.typedDataset.enabled`) and the shuffle has more than one partition.
89-
Native shuffle hashes such decimals differently from Spark
90-
([#5994](https://github.com/apache/datafusion-comet/issues/5994)). Without the conversion
91-
this shuffle would have used JVM shuffle, and a join partner may still use it.
86+
Decimal leaves with precision greater than 18 also disqualify a hash key when there is more
87+
than one output partition: native decimal hashing differs from Spark's. A one-partition hash
88+
exchange stays native because the writer serializes it as `SinglePartition` without hashing.
9289

9390
## Architecture
9491

@@ -531,7 +528,7 @@ independently compressed, allowing parallel decompression during reads.
531528
| Input format | Columnar (direct from Comet operators) | Row-based (via ColumnarToRowExec) |
532529
| Partitioning logic | Rust implementation | Spark's partitioner |
533530
| Supported schemes | Hash, Range, Single, RoundRobin | Hash, Range, Single, RoundRobin |
534-
| Partition key types | Primitives only (Hash, Range) | Any type |
531+
| Partition key types | See the hash and range rules above | Any type |
535532
| Performance | Higher (no format conversion) | Lower (columnar→row→columnar) |
536533
| Writer variants | Single path | Bypass (hash) and sort-based |
537534

‎docs/source/user-guide/latest/tuning/shuffle.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,17 @@ scalar types. Hash partitioning keys must be scalar types unless
5353
and later) map keys. That setting is disabled by default until the performance of the nested hashing paths has been
5454
measured. Columns that are not partitioning keys may contain complex types like maps, structs, and arrays.
5555

56+
Hash partitioning into multiple partitions on decimal keys with precision greater than 18 falls back because native
57+
hashing does not match Spark's partition assignments. Mixing the two partitioners can silently lose join rows.
58+
The restriction also applies to decimal leaves in nested hash keys. Wider decimals remain supported as payload
59+
columns, range partitioning keys, and in single-partition shuffles, including one-partition hash exchanges.
60+
61+
With `spark.comet.shuffle.mode=auto`, Comet uses Columnar Shuffle when eligible, adding a columnar-to-row-to-columnar
62+
conversion. With `native`, it uses Spark shuffle, which can also restore aggregates such as `collect_list` and
63+
`collect_set` to Spark to preserve buffer compatibility. Celeborn uses Spark/Celeborn shuffle for unsupported native
64+
keys because it has no Comet columnar fallback. Spark-compatible native decimal hashing is tracked in
65+
[#5994](https://github.com/apache/datafusion-comet/issues/5994).
66+
5667
### Columnar (JVM) Shuffle
5768

5869
Comet Columnar shuffle is JVM-based and supports `HashPartitioning`, `RoundRobinPartitioning`, `RangePartitioning`, and

‎native/common/src/lib.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,9 @@ mod utils;
3030
pub use error::{decimal_overflow_error, SparkError, SparkErrorWithContext, SparkResult};
3131
pub use ffi_offsets::zero_offsets;
3232
pub use query_context::{create_query_context_map, QueryContext, QueryContextMap};
33-
pub use schema::{cast_and_stamp_schema, widen_nested_nullability};
33+
pub use schema::{
34+
cast_and_stamp_schema, describe_timestamp_label_mismatch, widen_nested_nullability,
35+
};
3436
pub use struct_nulls::{child_with_parent_nulls, children_with_parent_nulls};
3537
pub use utf8::decode_string_arrays;
3638
pub use utils::{bytes_to_i128, decode_utf8_spark_lossy};

‎native/common/src/schema.rs‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,55 @@ fn nullability(field: &FieldRef) -> String {
154154
format!("{qualifier} {}", field.data_type())
155155
}
156156

157+
/// Describes the first timestamp whose timezone label differs between `expected` and `actual`, as
158+
/// a dotted path rooted at `path`. Returns `None` when every timestamp in `actual` carries the
159+
/// label `expected` declares, whatever else differs. A dictionary on either side is looked through,
160+
/// since boundaries unpack dictionaries anyway.
161+
///
162+
/// A label never changes the values, which are UTC microseconds whatever it says, so the cast at a
163+
/// boundary repairs it and nothing past the boundary can tell. Inside a native plan the label does
164+
/// matter: any other zone breaks comparisons and type coercion against Comet's `"UTC"`, and a
165+
/// missing one makes kernels read the values as wall-clock time. Strict test runs
166+
/// (`spark.comet.testing.strictTimestampLabels.enabled`) fail at the boundary instead of casting,
167+
/// so the test that produced the drift reports it.
168+
pub fn describe_timestamp_label_mismatch(
169+
path: &str,
170+
expected: &DataType,
171+
actual: &DataType,
172+
) -> Option<String> {
173+
match (expected, actual) {
174+
(DataType::Timestamp(_, e), DataType::Timestamp(_, a)) if e != a => {
175+
Some(format!("{path}: expected {expected}, found {actual}"))
176+
}
177+
(_, DataType::Dictionary(_, a)) => describe_timestamp_label_mismatch(path, expected, a),
178+
(DataType::Dictionary(_, e), _) => describe_timestamp_label_mismatch(path, e, actual),
179+
(DataType::List(e), DataType::List(a))
180+
| (DataType::LargeList(e), DataType::LargeList(a))
181+
| (DataType::FixedSizeList(e, _), DataType::FixedSizeList(a, _)) => {
182+
describe_timestamp_label_mismatch(
183+
&format!("{path}.element"),
184+
e.data_type(),
185+
a.data_type(),
186+
)
187+
}
188+
(DataType::Map(e, _), DataType::Map(a, _)) => describe_timestamp_label_mismatch(
189+
&format!("{path}.entries"),
190+
e.data_type(),
191+
a.data_type(),
192+
),
193+
(DataType::Struct(e), DataType::Struct(a)) if e.len() == a.len() => {
194+
e.iter().zip(a.iter()).find_map(|(e, a)| {
195+
describe_timestamp_label_mismatch(
196+
&format!("{path}.{}", e.name()),
197+
e.data_type(),
198+
a.data_type(),
199+
)
200+
})
201+
}
202+
_ => None,
203+
}
204+
}
205+
157206
/// Returns `base` with nested field nullability widened to also cover `other`, so that arrays of
158207
/// either type can be stamped with the result. Shapes that do not line up (different struct field
159208
/// counts, different list flavours, ...) are left as `base`; those are real type differences and
@@ -416,6 +465,91 @@ mod tests {
416465
assert!(msg.contains("payload: expected Struct"), "{msg}");
417466
}
418467

468+
fn timestamp(tz: Option<&str>) -> DataType {
469+
DataType::Timestamp(arrow::datatypes::TimeUnit::Microsecond, tz.map(Into::into))
470+
}
471+
472+
#[test]
473+
fn describes_timestamp_label_mismatches() {
474+
let utc = timestamp(Some("UTC"));
475+
assert_eq!(
476+
describe_timestamp_label_mismatch("ts", &utc, &timestamp(Some("Etc/UTC"))).unwrap(),
477+
r#"ts: expected Timestamp(µs, "UTC"), found Timestamp(µs, "Etc/UTC")"#
478+
);
479+
// The worst case: an LTZ value labelled as NTZ is read as wall-clock time downstream.
480+
assert!(describe_timestamp_label_mismatch("ts", &utc, &timestamp(None)).is_some());
481+
assert!(describe_timestamp_label_mismatch("ts", &timestamp(None), &utc).is_some());
482+
assert_eq!(describe_timestamp_label_mismatch("ts", &utc, &utc), None);
483+
assert_eq!(
484+
describe_timestamp_label_mismatch("ts", &timestamp(None), &timestamp(None)),
485+
None
486+
);
487+
}
488+
489+
#[test]
490+
fn describes_nested_timestamp_label_paths() {
491+
let wrap = |tz: Option<&str>| {
492+
DataType::Struct(Fields::from(vec![
493+
Field::new("id", DataType::Int64, true),
494+
Field::new(
495+
"events",
496+
DataType::List(Arc::new(Field::new_list_field(timestamp(tz), true))),
497+
true,
498+
),
499+
]))
500+
};
501+
assert_eq!(
502+
describe_timestamp_label_mismatch("c0", &wrap(Some("UTC")), &wrap(None)).unwrap(),
503+
r#"c0.events.element: expected Timestamp(µs, "UTC"), found Timestamp(µs)"#
504+
);
505+
506+
let map = |tz: Option<&str>| {
507+
DataType::Map(
508+
Arc::new(Field::new(
509+
"entries",
510+
DataType::Struct(Fields::from(vec![
511+
Field::new("key", DataType::Utf8, false),
512+
Field::new("value", timestamp(tz), true),
513+
])),
514+
false,
515+
)),
516+
false,
517+
)
518+
};
519+
assert_eq!(
520+
describe_timestamp_label_mismatch("m", &map(Some("UTC")), &map(Some("+00:00")))
521+
.unwrap(),
522+
r#"m.entries.value: expected Timestamp(µs, "UTC"), found Timestamp(µs, "+00:00")"#
523+
);
524+
}
525+
526+
/// Only the label is checked. A dictionary, a nullability flag or a different leaf type is
527+
/// left for the boundary cast, and a dictionary is looked through to the timestamp inside.
528+
#[test]
529+
fn timestamp_label_check_ignores_other_differences() {
530+
let utc = timestamp(Some("UTC"));
531+
let dict =
532+
|value: DataType| DataType::Dictionary(Box::new(DataType::Int32), Box::new(value));
533+
assert_eq!(
534+
describe_timestamp_label_mismatch("ts", &utc, &dict(utc.clone())),
535+
None
536+
);
537+
assert!(
538+
describe_timestamp_label_mismatch("ts", &utc, &dict(timestamp(Some("Etc/UTC"))))
539+
.is_some()
540+
);
541+
assert_eq!(
542+
describe_timestamp_label_mismatch("c0", &DataType::Int64, &DataType::Int32),
543+
None
544+
);
545+
let list =
546+
|nullable: bool| DataType::List(Arc::new(Field::new_list_field(utc.clone(), nullable)));
547+
assert_eq!(
548+
describe_timestamp_label_mismatch("c0", &list(true), &list(false)),
549+
None
550+
);
551+
}
552+
419553
#[test]
420554
fn error_on_column_count_mismatch() {
421555
let schema = Arc::new(Schema::new(vec![Field::new("c0", DataType::Int32, true)]));

‎native/core/src/execution/jni_api.rs‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,8 @@ use crate::execution::spark_config::{
135135
SparkConfig, COMET_DEBUG_ENABLED, COMET_DEBUG_MEMORY,
136136
COMET_EXEC_AGGREGATE_SKIP_PARTIAL_ENABLED, COMET_EXEC_SPILL_COMPRESSION_CODEC,
137137
COMET_EXPLAIN_NATIVE_ENABLED, COMET_MAX_TEMP_DIRECTORY_SIZE,
138-
COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED, COMET_TRACING_ENABLED, SPARK_EXECUTOR_CORES,
138+
COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED, COMET_STRICT_TIMESTAMP_LABELS,
139+
COMET_TRACING_ENABLED, SPARK_EXECUTOR_CORES,
139140
};
140141
use crate::parquet::encryption_support::{CometEncryptionFactory, ENCRYPTION_FACTORY_ID};
141142
use crate::parquet::parquet_support::CometObjectStoreRegistry;
@@ -555,6 +556,8 @@ struct ExecutionContext {
555556
pub session_ctx: Arc<SessionContext>,
556557
/// Whether to enable additional debugging checks & messages
557558
pub debug_native: bool,
559+
/// Whether boundaries fail on a relabelled timestamp instead of casting it (test-only)
560+
pub strict_timestamp_labels: bool,
558561
/// Whether to write native plans with metrics to stdout
559562
pub explain_native: bool,
560563
/// Whether to log memory usage on each call to execute_plan
@@ -706,6 +709,7 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_createPlan(
706709
struct PlanSettings {
707710
spark_config: HashMap<String, String>,
708711
debug_native: bool,
712+
strict_timestamp_labels: bool,
709713
explain_native: bool,
710714
tracing_enabled: bool,
711715
max_temp_directory_size: u64,
@@ -726,6 +730,7 @@ fn prepare_plan_settings(serialized_spark_configs: &[u8]) -> CometResult<PlanSet
726730
// Access Comet configs
727731
Ok(PlanSettings {
728732
debug_native: spark_config.get_bool(COMET_DEBUG_ENABLED),
733+
strict_timestamp_labels: spark_config.get_bool(COMET_STRICT_TIMESTAMP_LABELS),
729734
explain_native: spark_config.get_bool(COMET_EXPLAIN_NATIVE_ENABLED),
730735
tracing_enabled: spark_config.get_bool(COMET_TRACING_ENABLED),
731736
max_temp_directory_size: spark_config
@@ -877,6 +882,7 @@ fn create_execution_context(
877882
plan_creation_time,
878883
session_ctx: session,
879884
debug_native: settings.debug_native,
885+
strict_timestamp_labels: settings.strict_timestamp_labels,
880886
explain_native: settings.explain_native,
881887
tracing_enabled: settings.tracing_enabled,
882888
rust_thread_id,
@@ -963,7 +969,7 @@ fn configure_skip_partial_aggregation(config: &mut SessionConfig, plan: &Operato
963969
if count.children.len() == 1)
964970
})
965971
}
966-
// PartialMerge is represented as native Partial, but consumes states, not rows.
972+
// PartialMerge consumes states, not rows.
967973
_ => false,
968974
},
969975
_ => true,
@@ -1457,6 +1463,7 @@ fn execute_plan(
14571463
let start = Instant::now();
14581464
let planner = PhysicalPlanner::new(Arc::clone(&exec_context.session_ctx), partition)
14591465
.with_exec_id(exec_context_id)
1466+
.with_strict_timestamp_labels(exec_context.strict_timestamp_labels)
14601467
.with_sql_text_pool(&exec_context.spark_plan)
14611468
.with_task_context(exec_context.task_context.clone())
14621469
.with_class_loader(exec_context.class_loader.clone())

‎native/core/src/execution/memory_pools/spill_replay.rs‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,8 @@
2121
//! replays them through an `OrderedFinalAggregateStream` that has no way to spill, so a refused
2222
//! memory request there fails the task. `FinalHashAggregateStream` does this, and so does
2323
//! `OrderedFinalAggregateStream` itself, which DataFusion uses when the input is sorted on some of
24-
//! the grouping keys. The merge reserves read buffers for as many spill files as fit, and those
24+
//! the grouping keys. So does `SingleHashAggregateStream`, which Comet runs for a Spark
25+
//! PartialMerge (#6857). The merge reserves read buffers for as many spill files as fit, and those
2526
//! buffers belong to the same consumer, so the replay often finds the consumer's share already
2627
//! taken.
2728
//!
@@ -31,9 +32,9 @@
3132
//!
3233
//! This relies on the following in DataFusion 55.1, which a DataFusion upgrade has to re-check:
3334
//!
34-
//! - The two aggregates name their consumers `FinalHashAggregateStream[{partition}]` and
35-
//! `OrderedFinalAggregateStream[{partition}]`, and their merge and replay reserve through sibling
36-
//! reservations of that consumer.
35+
//! - The three aggregates name their consumers `FinalHashAggregateStream[{partition}]`,
36+
//! `OrderedFinalAggregateStream[{partition}]` and `SingleHashAggregateStream[{partition}]`, and
37+
//! their merge and replay reserve through sibling reservations of that consumer.
3738
//! - A final aggregate grows one reservation while another of its reservations holds memory only
3839
//! during the replay, while the merge holds its read buffers. Before that, the aggregate's table
3940
//! is its only reservation holding memory, so a refusal stands and makes it spill. The merge
@@ -45,7 +46,8 @@
4546
//!
4647
//! Remove this, as #6583 describes, once Comet's DataFusion has apache/datafusion#25383, which
4748
//! leaves the replay room when the merge picks its files. The #6254 tests in `CometAggregateSuite`
48-
//! fail without this on DataFusion 55.1, so they show whether the replay still needs it.
49+
//! fail without this on DataFusion 55.1, and its PartialMerge read-back test for #6857 fails
50+
//! without it on 55.2, so they show whether the replay still needs it.
4951
5052
use std::collections::HashMap;
5153
use std::fmt;
@@ -174,11 +176,12 @@ pub(super) fn unwrap_spill_replay(pool: &Arc<dyn MemoryPool>) -> Option<&Arc<dyn
174176
.map(|replay| &replay.inner)
175177
}
176178

177-
/// Whether `consumer` belongs to a final aggregate whose spill replay can't spill.
179+
/// Whether `consumer` belongs to a final or single aggregate, whose spill replay can't spill.
178180
fn is_final_aggregate(consumer: &MemoryConsumer) -> bool {
179181
let name = consumer.name();
180182
name.starts_with("FinalHashAggregateStream[")
181183
|| name.starts_with("OrderedFinalAggregateStream[")
184+
|| name.starts_with("SingleHashAggregateStream[")
182185
}
183186

184187
#[cfg(test)]
@@ -353,14 +356,14 @@ mod tests {
353356
for name in [
354357
"FinalHashAggregateStream[3]",
355358
"OrderedFinalAggregateStream[0]",
359+
"SingleHashAggregateStream[0]",
356360
] {
357361
assert!(is_final_aggregate(&MemoryConsumer::new(name)), "{name}");
358362
}
359-
// Partial aggregates emit early instead of spilling, and Comet never plans single ones.
363+
// Partial aggregates emit early instead of spilling.
360364
for name in [
361365
"PartialHashAggregateStream[0]",
362366
"OrderedPartialAggregateStream[0]",
363-
"SingleHashAggregateStream[0]",
364367
"ExternalSorterMerge[0]",
365368
] {
366369
assert!(!is_final_aggregate(&MemoryConsumer::new(name)), "{name}");

0 commit comments

Comments
 (0)