Skip to content

CometCaseConversionBase gates compat inside convert() instead of getSupportLevel #4467

Closed

Description

@andygrove

Describe the bug

CometCaseConversionBase (parent of CometUpper and CometLower) gates compatibility inside convert(...):

override def convert(expr: T, inputs: Seq[Attribute], binding: Boolean): Option[Expr] = {
  if (!CometConf.COMET_CASE_CONVERSION_ENABLED.get()) {
    withInfo(expr, "Comet is not compatible with Spark for case conversion in " +
      s"locale-specific cases. Set ${CometConf.COMET_CASE_CONVERSION_ENABLED.key}=true " +
      "to enable it anyway.")
    return None
  }
  super.convert(expr, inputs, binding)
}

It never overrides getSupportLevel, so the class-level support level defaults to Compatible(None) even though getIncompatibleReasons() is populated. The Step 5 consistency rule in the audit-comet-expression skill calls this out specifically as the canonical antipattern to avoid. Surfaced by the string-expressions audit in #4461.

Concrete consequences:

  1. The dispatcher's allowIncompatible machinery is bypassed. A user who flips the standard spark.comet.expr.allowIncompatible=true (or the per-expression spark.comet.expression.Upper.allowIncompatible) still has lower/upper fall back, because the gate is the older bespoke spark.comet.caseConversion.enabled.
  2. EXPLAIN and the auto-generated compatibility docs disagree. EXPLAIN shows the bespoke message above; the compat doc lists the populated getIncompatibleReasons() text instead.
  3. initcap uses the standard Incompatible(Some(...)) flow; lower/upper use the bespoke flow. The asymmetry is confusing for users who have just opted into all string incompatibilities.

Expected behavior

Override getSupportLevel to return Incompatible(Some(reason)) from a shared private val, drop the in-convert conf check, and either deprecate spark.comet.caseConversion.enabled in favor of the standard per-expression allowIncompatible flag or have the support level read the standard flag.

Additional context

Activity

  1. added
    priority:mediumFunctional bugs, performance regressions, broken features
    and removed on Jun 1, 2026
  2. changed the title [-][Bug] CometCaseConversionBase gates compat inside convert() instead of getSupportLevel[/-] [+]CometCaseConversionBase gates compat inside convert() instead of getSupportLevel[/+] on Jun 30, 2026
  3. dwsmith1983 commented on Sep 11, 2026

    @dwsmith1983
    Contributor

    #4721 (46b5841) reworked this. CometCaseConversionBase in spark/src/main/scala/org/apache/comet/serde/strings.scala now overrides getSupportLevel and the support level is Compatible with a NativeOptIn pointing at spark.comet.caseConversion.enabled; convert no longer gates or returns None, it only chooses between the native kernel (opt-in) and the codegen dispatcher (default), so lower/upper never fall back to Spark and EXPLAIN, the COMET-INFO hint and the generated docs all describe the same thing. The three consequences listed here (bypassed allowIncompatible machinery, EXPLAIN/doc disagreement, asymmetry with initcap) no longer apply, and initcap, lower and upper are all documented as Hybrid in docs/source/user-guide/latest/expressions.md. Suggest closing.

  4. andygrove commented on Sep 28, 2026

    @andygrove
    MemberAuthor

    #4721 fixed this: CometCaseConversionBase now overrides getSupportLevel and reports Compatible with a native opt-in, and convert only picks between the native kernel and the codegen dispatcher, so upper and lower never fall back and EXPLAIN matches the docs. Moving the opt-in onto the standard allowIncompatible key was dropped in #4853 as a breaking config change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:expressionsExpression evaluationpriority:mediumFunctional bugs, performance regressions, broken features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions