Repository navigation
Audit Spark SQL configs that affect query semantics and ensure Comet honors them (or falls back) #4180
Description
What is the problem the feature request solves?
yes, this is all AI generated ... I asked Claude to find all the Spark configs that we should ensure Comet honors
Spark exposes hundreds of configuration options, many of which alter query semantics (result correctness, not just performance). Comet's test matrix almost exclusively runs with default configs, which means non-default settings may silently produce results that diverge from Spark.
A user running Spark with, e.g., spark.sql.legacy.timeParserPolicy=LEGACY or spark.sql.parquet.datetimeRebaseModeInRead=LEGACY has no guarantee that Comet will produce the same results as vanilla Spark. In the worst case, results are silently wrong with no warning.
Today, Comet reads and respects only a small subset of semantic configs:
| Config | Handling |
|---|---|
spark.sql.ansi.enabled |
Native (cast eval mode) |
spark.sql.session.timeZone |
Native (Parquet + timestamp expressions) |
spark.sql.parquet.int96AsTimestamp |
Native |
spark.sql.parquet.int96TimestampConversion |
Disables Comet entirely when true |
spark.sql.caseSensitiveAnalysis |
Native (Parquet) |
spark.sql.parquet.filterPushdown (+ subflags) |
Native |
spark.sql.nestedSchemaPruningEnabled |
Native |
spark.sql.decimalOperationsAllowPrecisionLoss |
Native |
| Non-default string collations (Spark 4.0+) | Selective operator fallback |
Everything else is effectively "hope the default works for you."
Describe the potential solution
Audit all semantic-affecting Spark SQL configs and, for each one, pick one of three states:
- Respected natively — Comet reads the config and produces Spark-compatible results. Covered by tests that run with non-default values.
- Fallback trigger — Non-default value disables Comet for the affected query (or entirely, like
int96TimestampConversion). - Known-safe to ignore — Config doesn't change results in code paths Comet runs.
Each config needs an explicit decision, a test, and documentation. Below is a starting inventory organized by risk.
Tier 1 — Correctness bombs (silent wrong results)
These most likely produce divergent results today:
-
spark.sql.parquet.datetimeRebaseModeInRead(EXCEPTION / LEGACY / CORRECTED) — Julian↔Gregorian rebase for pre-1582 dates. Comet hasspark.comet.exceptionOnDatetimeRebasebut it's unclear whether all three modes are honored end-to-end. -
spark.sql.parquet.int96RebaseModeInRead— same as above for INT96. -
spark.sql.parquet.datetimeRebaseModeInWrite/int96RebaseModeInWrite— write path. -
spark.sql.legacy.timeParserPolicy(CORRECTED / LEGACY / EXCEPTION) — changesto_date,to_timestamp,unix_timestamp, CSV/JSON date parsing entirely (SimpleDateFormat vs DateTimeFormatter). -
spark.sql.parquet.binaryAsString— changes returned schema (binary columns read as string).
Tier 2 — Function / expression-level semantics
-
spark.sql.legacy.sizeOfNull—size(null)returns-1vsnull. -
spark.sql.legacy.allowNegativeScaleOfDecimal— decimal parsing and arithmetic. -
spark.sql.legacy.typeCoercion.datetimeToString.enabled— implicit date↔string coercion in concat/coalesce. -
spark.sql.legacy.allowCastNumericToTimestamp—CAST(bigint AS timestamp)permitted or not. -
spark.sql.legacy.followThreeValuedLogicInArrayExists—exists(array, pred)null handling. -
spark.sql.function.concatBinaryAsString—concat(binary, binary)return type. -
spark.sql.function.eltOutputAsString— same forelt(). -
spark.sql.legacy.integerGroupingIdEnabled—grouping_id()return type. -
spark.sql.legacy.json.allowEmptyString.enabled— JSON parsing behavior.
Tier 3 — Schema / type behavior
-
spark.sql.legacy.charVarcharAsString— CHAR/VARCHAR padding/truncation. -
spark.sql.readSideCharPadding— CHAR(n) read-side padding. -
spark.sql.legacy.createEmptyCollectionUsingStringType— emptyarray()/map()element type.
Tier 4 — Already read, but verify coverage
Confirm that these are honored in all Comet code paths, not just the ones where they were originally wired up:
-
spark.sql.ansi.enabled— beyond cast, does ANSI propagate to division-by-zero, integer overflow in+/-/*, array index out-of-bounds, etc.? -
spark.sql.caseSensitiveAnalysis— confirmed for Parquet scan; what about nested struct field access in expressions? -
spark.sql.session.timeZone— confirmed for Parquet and casts; verify for all temporal expressions (date_format,from_utc_timestamp,to_utc_timestamp,date_trunc, etc.). -
spark.sql.decimalOperationsAllowPrecisionLoss— confirmed for the serde path; verify arithmetic kernels match.
Suggested process per config
For each item:
- Identify the Spark code paths that branch on the config.
- Check whether Comet's corresponding code path reads the config.
- Add a test that runs with the non-default value and asserts Spark-vs-Comet parity (or explicit fallback).
- Decide: native-respect, fallback-trigger, or known-safe-to-ignore. Document the decision.
Optional: config whitelist gate
Once the audit is done, we could add an opt-in mode (e.g., spark.comet.strictConfigCompatibility=true) that falls back to Spark whenever a semantic config is set to a non-default value not on the whitelist. This would give users a way to trade coverage for correctness guarantees without forcing the tradeoff on everyone.
Additional context
Motivation: the current "default configs only" testing posture is a significant compatibility risk as Comet sees broader adoption. Users routinely tune legacy.timeParserPolicy, rebase modes, and ANSI behavior in production — these are not edge cases.
Describe the potential solution
No response
Additional context
No response
Activity
andygrove commentedon Jul 23, 2026
Audit results
I worked through the config inventory above against
apache/main@0761e549a(default Maven profile, Spark 4.1.2, macOS aarch64, debug build). Method was a differential harness rather than code reading: each case runs the same query withspark.comet.enabledfalse and then true under the config, and records the result, the exception, and the executed plan.Two caveats on the method, because both bit me on the first pass and would bite anyone repeating this:
Row.toStringproduce[B@<identity hash>, sobinaryAsString,concatBinaryAsStringandeltOutputAsStringall looked like mismatches until I compared them viahex(). They are fine.Tier 1
spark.sql.parquet.datetimeRebaseModeInReadspark.sql.parquet.int96RebaseModeInReadspark.sql.parquet.datetimeRebaseModeInWrite,int96RebaseModeInWritespark.comet.parquet.write.enableddefaults tofalseand is marked experimentalspark.sql.legacy.timeParserPolicyspark.sql.parquet.binaryAsString||,lengthandupperTier 2
spark.sql.legacy.sizeOfNullCometSizereadsexpr.legacySizeOfNulland emits the matchingCaseWhenelse-branchspark.sql.legacy.allowNegativeScaleOfDecimalspark.sql.legacy.typeCoercion.datetimeToString.enabledspark.sql.legacy.allowCastNumericToTimestampspark.sql.legacy.followThreeValuedLogicInArrayExistsspark.sql.function.concatBinaryAsStringspark.sql.function.eltOutputAsStringspark.sql.legacy.integerGroupingIdEnabledCometExpand+CometHashAggregatewithCUBEspark.sql.legacy.json.allowEmptyString.enabledTier 3
spark.sql.legacy.charVarcharAsStringc = 'ab'predicatespark.sql.readSideCharPaddingspark.sql.legacy.createEmptyCollectionUsingStringTypeTier 4
spark.sql.ansi.enabledabs, array index andelement_atout of bounds, map key not found,castto every numeric target, decimal overflow,add_months,make_date,conv,substring,round,date_add, andsumoverflow all raise correctly, and all the same expressions match with ANSI offspark.sql.caseSensitiveAnalysisspark.sql.session.timeZonedate_format,hour,minute,date_trunc,from_utc_timestamp,to_utc_timestamp,cast(ts as date),cast(ts as string)andunix_timestamp, against two non-UTC zonesspark.sql.decimalOperationsAllowPrecisionLosssumandavg, under both valuesTwo things worth flagging beyond the pass/fail
"Respected, but incidentally" is not the same as respected.
timeParserPolicyis honoured only becauseCometGetTimestampandCometToUnixTimestampareCometCodegenDispatch, andfollowThreeValuedLogicInArrayExistsonly becauseCometArrayExistsis. Nothing reads either config. Giving any of those expressions a native implementation, which is squarely in scope for #4936 and #4942, would silently break the config with no test to catch it. These are the cases where a regression test earns its keep.ANSI error text. Results are correct, but several messages are not:
bigint + bigintoverflow reports "integer overflow" where Spark says "long overflow",absreports "Int32 overflow", several omit Spark'stry_add/try_multiplysuggestion, and decimal overflow surfaces as a bareCometNativeException: Compute error: Arithmetic overflowinstead ofNUMERIC_VALUE_OUT_OF_RANGE. Related to #4967. Not filed separately since these are error fidelity rather than correctness, but they would fail Spark SQL tests that match on message text.What is still outstanding
The process in the description asks for "an explicit decision, a test, and documentation" per config. This comment provides the decision. The tests do not exist: my harness was a throwaway diagnostic and nothing under
spark/src/testcurrently exercisesspark.sql.legacy.*or the rebase configs. Given the "incidentally respected" cases above, that gap is the substantive remaining work, so I would suggest keeping this issue open until:The optional
spark.comet.strictConfigCompatibilitywhitelist gate from the description is largely subsumed by #4799'sspark.comet.legacyConfFallback.enabled.andygrove commentedon Aug 1, 2026
Re-audit against current
apache/mainThe earlier audit
ran against
0761e549a. Main has moved since then, so I re-ran it againstba21f02de(defaultMaven profile, Spark 4.1.2, macOS aarch64, debug build) and widened the sweep from the inventory in
the description to the whole
spark.sql.legacy.*family.Scope this time: all 104
spark.sql.legacy.*keys declared in Spark 4.1.1'sSQLConfweretriaged, and every one that can reach a code path Comet executes was tested differentially at both
values, plus the rebase-mode family and the list in #5087. Method unchanged: run the same query
with
spark.comet.enabledfalse and then true under the config, over Parquet-backed columns ratherthan literals, and compare results, exceptions and plans.
Result: two categories still diverge, and both already have open issues
ba21f02despark.sql.parquet.datetimeRebaseModeInRead,spark.sql.parquet.int96RebaseModeInRead, and theirspark.sql.legacy.parquet.*aliases1000-01-01reads back as1000-01-06;count(*) ... WHERE d = date'1000-01-01'returns 0 against Spark's 1. Identical in all three read-mode values and at defaults, so the config is simply not consulted. INT96 behaves the samespark.sql.legacy.allowNegativeScaleOfDecimalcore/src/num/mod.rs:475, multiply overflow), decimal arithmetic (planner.rs:917, subtract overflow), and the same cast inside aFilterSince both categories are already filed and assigned, filing per-config issues would only create
duplicates. One genuinely separate item was carved out of #5010 instead:
spark.comet.exceptionOnDatetimeRebaseis dead code. It is declared, non-internal,published to the configs page, and read nowhere. Its intended destination
(
SparkParquetOptions::use_legacy_date_timestamp_or_ntz) is only ever initialized tofalseandnever consulted. A user who sets it precisely to be protected from the limitation above gets
silence and wrong results. It has a self-contained fix that does not depend on rebase support
landing, which is why it is now its own issue.
Everything else matched
No divergence was found for any of the following. Each was run at both values with Comet enabled
and disabled.
Verified respected by design (Comet reads the config or the serde routes around it):
castComplexTypesToString.enabled,negativeIndexInArrayInsert,nullInEmptyListBehavior,truncateForEmptyRegexSplit,sizeOfNull,parquet.returnNullStructIfAllFieldsMissing.Verified matching, not read by Comet (no reader in the tree, but parity holds because the
affected path is Spark's or the values coincide):
timeParserPolicyatLEGACYandCORRECTEDacross
to_date,to_timestamp,unix_timestamp,date_formatandcast(string as date);followThreeValuedLogicInArrayExists;lpadRpadAlwaysReturnString;allowHashOnMapType(bothhashandxxhash64over maps);collationAwareHashFunctions(hash, group-by and shuffle join ona
UTF8_LCASEcolumn);disableMapKeyNormalization(map,map_from_arraysandstr_to_map,with genuine
-0.0andNaNkeys read from Parquet);allowNullComparisonResultInArraySort(including a comparator that returns
NULL);javaCharsets;codingErrorAction;allowZeroIndexInFormatString;raiseErrorWithoutErrorClass;json.allowEmptyString.enabled;json.enableDateTimeParsingFallback;csv.enableDateTimeParsingFallback;nullValueWrittenAsQuotedEmptyStringCsv;charVarcharAsString;noCharPaddingInPredicate;readSideCharPadding;integerGroupingId;groupingIdWithAppendedUserGroupBy;statisticalAggregate;percentileDiscCalculation;histogramNumericPropagateInputType;decimal.retainFractionDigitsOnTruncate;literal.pickMinimumPrecision;bucketedTableScan.outputOrdering;exponentLiteralAsDecimal.enabled;interval.enabled;fromDayTimeString.enabled;viewSchemaCompensation;storeAnalyzedPlanForView;useCurrentConfigsForView;readFileSourceTableCacheIgnoreOptions;doLooseUpcast;duplicateBetweenInput;inSubqueryNullability;scalarSubqueryCountBugBehavior;setopsPrecedence.enabled;allowParameterlessCount;typeCoercion.datetimeToString.enabled;createEmptyCollectionUsingStringType;compareDateTimestampInTimestamp.The per-config verdicts for the #5087 subset are posted on that issue.
Two limits on the above, stated plainly
spark.sql.legacy.parquet.nanosAsLongcould not be tested this way. Spark cannot writenanosecond-precision Parquet, so producing an input requires a non-Spark writer. Comet has no
reader for the config, so it is unverified rather than verified-safe.
executedPlanand does not descend into AQEquery stages. For aggregate and subquery shapes it under-reports Comet operators. The parity
verdicts are unaffected, but for those shapes I can confirm Spark and Comet agree without
confirming which engine ran the expression. Anything reported above as respected-by-design was
confirmed on a non-AQE plan with the Comet operator visible.
What remains open on this issue
Unchanged from the previous audit, and still the substantive work: this comment provides decisions,
not tests. Nothing under
spark/src/testexercises thespark.sql.legacy.*family, so everyverdict above is a point-in-time observation that no regression test defends.
That matters most for the configs in the second list.
timeParserPolicyholds only becauseCometGetTimestampandCometToUnixTimestampareCometCodegenDispatch, andfollowThreeValuedLogicInArrayExistsonly becauseCometArrayExistsis. Nothing reads eitherconfig. Giving any of those expressions a native implementation, which is in scope for #4936 and
#4942, would silently break the config with no test to catch it.
Linked issues: #5010, #5013, #5195, #5087, and #4799 for the per-expression and per-scan handling.