Repository navigation
Closed
Closed
Copy link
Labels
area:expressionsExpression evaluationExpression evaluationpriority:mediumFunctional bugs, performance regressions, broken featuresFunctional bugs, performance regressions, broken features
Description
Describe the bug
CometCaseConversionBase (parent of CometUpper and CometLower) gates compatibility inside convert(...):
override def convert(expr: T, inputs: Seq[Attribute], binding: Boolean): Option[Expr] = {
if (!CometConf.COMET_CASE_CONVERSION_ENABLED.get()) {
withInfo(expr, "Comet is not compatible with Spark for case conversion in " +
s"locale-specific cases. Set ${CometConf.COMET_CASE_CONVERSION_ENABLED.key}=true " +
"to enable it anyway.")
return None
}
super.convert(expr, inputs, binding)
}It never overrides getSupportLevel, so the class-level support level defaults to Compatible(None) even though getIncompatibleReasons() is populated. The Step 5 consistency rule in the audit-comet-expression skill calls this out specifically as the canonical antipattern to avoid. Surfaced by the string-expressions audit in #4461.
Concrete consequences:
- The dispatcher's
allowIncompatiblemachinery is bypassed. A user who flips the standardspark.comet.expr.allowIncompatible=true(or the per-expressionspark.comet.expression.Upper.allowIncompatible) still haslower/upperfall back, because the gate is the older bespokespark.comet.caseConversion.enabled. - EXPLAIN and the auto-generated compatibility docs disagree. EXPLAIN shows the bespoke message above; the compat doc lists the populated
getIncompatibleReasons()text instead. initcapuses the standardIncompatible(Some(...))flow;lower/upperuse the bespoke flow. The asymmetry is confusing for users who have just opted into all string incompatibilities.
Expected behavior
Override getSupportLevel to return Incompatible(Some(reason)) from a shared private val, drop the in-convert conf check, and either deprecate spark.comet.caseConversion.enabled in favor of the standard per-expression allowIncompatible flag or have the support level read the standard flag.
Additional context
- Serde:
CometCaseConversionBaseinspark/src/main/scala/org/apache/comet/serde/strings.scala - Conf:
CometConf.COMET_CASE_CONVERSION_ENABLED(defaultfalse) - Related: Initcap behaves differently in Spark and in DataFusion (also Comet) #1052 (initcap divergence, already follows the standard pattern)
Metadata
Metadata
Assignees
Labels
area:expressionsExpression evaluationExpression evaluationpriority:mediumFunctional bugs, performance regressions, broken featuresFunctional bugs, performance regressions, broken features
Activity
docs(audit): cross-reference filed follow-up issues from string audit…
docs: require audit skill to file issues for prose-only findings
chore(audit): address PR review feedback on string-expressions audit
[-][Bug] CometCaseConversionBase gates compat inside convert() instead of getSupportLevel[/-][+]CometCaseConversionBase gates compat inside convert() instead of getSupportLevel[/+]dwsmith1983 commentedon Sep 11, 2026
#4721 (46b5841) reworked this.
CometCaseConversionBasein spark/src/main/scala/org/apache/comet/serde/strings.scala now overridesgetSupportLeveland the support level isCompatiblewith aNativeOptInpointing atspark.comet.caseConversion.enabled;convertno longer gates or returnsNone, it only chooses between the native kernel (opt-in) and the codegen dispatcher (default), solower/uppernever fall back to Spark and EXPLAIN, the COMET-INFO hint and the generated docs all describe the same thing. The three consequences listed here (bypassedallowIncompatiblemachinery, EXPLAIN/doc disagreement, asymmetry withinitcap) no longer apply, andinitcap,lowerandupperare all documented as Hybrid in docs/source/user-guide/latest/expressions.md. Suggest closing.andygrove commentedon Sep 28, 2026
#4721 fixed this: CometCaseConversionBase now overrides getSupportLevel and reports Compatible with a native opt-in, and convert only picks between the native kernel and the codegen dispatcher, so upper and lower never fall back and EXPLAIN matches the docs. Moving the opt-in onto the standard allowIncompatible key was dropped in #4853 as a breaking config change.