You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 571477c
Browse filesBrowse the repository at this point in the historyBrowse files
feat(cli): add opt-in Spark function registration (apache#24179)
## Which issue does this PR close?
- Closesapache#24146.
## Rationale for this change
`datafusion-cli` does not currently enable the Spark SQL dialect or
register the `datafusion-spark` function library, so Spark-compatible
syntax and functions cannot be explored interactively. Enabling them by
default could override core DataFusion functions with the same names and
change existing CLI behavior.
## What changes are included in this PR?
- Add a `--spark` CLI flag.
- Select the Spark SQL dialect while constructing the session
configuration.
- Register Spark-compatible scalar, aggregate, and window functions only
when the flag is enabled.
- Add CLI integration coverage for both default and opt-in behavior.
- Document the flag and its function-override behavior in the CLI and
Spark documentation.
## Are these changes tested?
Yes. The following checks pass:
- `cargo test -p datafusion-cli --test cli_integration
spark_features_require_opt_in`
- `cargo test -p datafusion-cli --all-targets`
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `./ci/scripts/doc_prettier_check.sh`
## Are there any user-facing changes?
Yes. Passing `--spark` enables the Spark SQL dialect, Spark expression
planner, and Spark-compatible functions for the CLI session. The option
is disabled by default, so existing CLI behavior remains unchanged.
0 commit comments