Repository navigation
[Doc] decode does not appear in auto-generated compatibility docs #4466
Description
Describe the bug
decode is handled in Comet via CommonStringExprs.stringDecode called from versionSpecificExprToProtoInternal in the per-Spark-version CometExprShim. There is no CometExpressionSerde[StringDecode] (or equivalent) registered in QueryPlanSerde.scala's stringExpressions map.
Consequence: decode never appears in the auto-generated docs/source/user-guide/compatibility.md, neither under "incompatible" nor "unsupported" reasons. Users have no docs-side way to discover what Comet supports for decode (only literal 'utf-8' charset, case-insensitive) or what it doesn't (every other charset, all four-arg Spark 4.0 cases described in #4465).
Surfaced by the string-expressions audit in #4461.
Expected behavior
Register a CometExpressionSerde[StringDecode] (or a per-version equivalent that the shims dispatch to) so the doc generator picks the expression up and includes both the supported charset restriction and the legacy-flag gap as getUnsupportedReasons() entries.
Additional context
- Shim:
spark/src/main/spark-{3.4,3.5,4.0,...}/org/apache/comet/shims/CometExprShim.scala - Helper:
CommonStringExprs.stringDecodeinspark/src/main/scala/org/apache/comet/serde/strings.scala - Doc generator:
GenerateDocsreadsget*Reasons()from registeredCometExpressionSerdeinstances; helpers called from shims are invisible to it.
Activity
docs(audit): cross-reference filed follow-up issues from string audit…
dwsmith1983 commentedon Sep 11, 2026
#4639 (7b279f8) replaced the shim helper with a registered serde.
CometStringDecode(spark/src/main/spark-3.x/org/apache/comet/serde/CometStringDecode.scala) is aCometCodegenDispatchregistered throughsparkVersionSpecificStringExpressionson Spark 3.x, and on Spark 4.0+ theStaticInvoke(StringDecode.decode, ...)rewrite is mapped in spark/src/main/scala/org/apache/comet/serde/statics.scala, so the doc generator sees it.decodenow appears in docs/source/user-guide/latest/expressions.md (which replaced the compatibility.md tables this issue refers to). Suggest closing.andygrove commentedon Sep 28, 2026
This was fixed by #4639. decode now runs Spark's own decoder through the codegen dispatcher on every Spark version, via CometStringDecode on 3.x and the StaticInvoke mapping on 4.x, so there are no charset or legacy-flag restrictions left to document, and it is listed in the supported expressions guide. The dash in its Implementation column comes from the same GenerateDocs lookup gap as the builder-registered functions in #6169.