You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e6694b1
Browse filesBrowse the repository at this point in the historyBrowse files
chore(audit): address PR review feedback on string-expressions audit
Apply the recurring patterns surfaced in PR review comments rather than
documenting them as Known Limitation prose.
- Remove the unreachable `CometScalarFunction("btrim")` serde mapping.
`StringTrimBoth` is `RuntimeReplaceable` and rewritten to `StringTrim`
before serde runs, so the entry was dead code. Updated the support-doc
sub-bullet for `btrim` to point readers at the `trim` entry.
- Rewrite `CometCaseConversionBase` to use the standard `Incompatible`
opt-in instead of the bespoke `spark.comet.caseConversion.enabled`
conf. Override `getSupportLevel` to return
`Incompatible(Some(reason))` and drop the in-`convert` config check.
Remove `COMET_CASE_CONVERSION_ENABLED` and update the two callers and
the string-expression benchmark to use the per-expression
`spark.comet.expression.{Upper,Lower,InitCap}.allowIncompatible`
keys. Closes the gating asymmetry called out in apache#4467.
- Mark `replace` as `Incompatible` when `search` is a literal empty
string. DataFusion's `replace` inserts the replacement between every
character whereas Spark short-circuits to return `src` unchanged.
Refile under apache#4497 (apache#3344 was closed previously with the bug body
describing the divergence in the wrong direction).
- Drop the spurious `CometStringRepeat.getCompatibleNotes()` claim that
DataFusion `repeat` throws on negative counts. Verified in
`expressions/string/string_repeat.sql` that negative counts produce
the empty string Spark expects. The audit's prose claim was wrong, so
apache#4462 was closed as already-fixed.
- Update the `replace` SQL fixture to assert the new fallback message
rather than the prior `ignore(...)` shape. Update `lower.sql` /
`upper.sql` `expect_fallback(...)` strings to match the new reason
text. Switch `lower_enabled.sql` / `upper_enabled.sql` from
`spark.comet.caseConversion.enabled` to the per-expression
`allowIncompatible` keys.
- Cross-reference the collation umbrella issue
(apache#4496) from every support-doc sub-bullet that previously documented
"non-default collations may diverge silently" without an issue link.
Add the cross-reference to the entries that route through Spark 4.0
`CollationSupport.X.exec` but had no collation note at all
(`bit_length`, `concat_ws`, `endswith`, `initcap`, `instr`, `lpad`,
`octet_length`, `repeat`, `replace`, `right`, `rpad`, `rtrim`,
`split`, `startswith`, `substring`, `substring_index`, `translate`).
- Replace the stale "Known limitation: enforced inside `convert`" prose
on `left`, `right`, `substring`, and `length`. The literal-only
argument restriction has already been moved into `getSupportLevel`;
the prose just hadn't been updated.
Tests:
- `./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite expressions/string/"` (42 pass)
- `./mvnw test -Dsuites="org.apache.comet.CometStringExpressionSuite"` (33 pass)
0 commit comments