chore: remove the `datafusion-spark` crate (moving to apache/datafusion-spark) by andygrove · Pull Request #27337 · apache/datafusion

andygrove · 2026-11-11T18:47:33Z

Which issue does this PR close?

Part of [DISCUSSION] How should we test datafusion-spark against Spark? datafusion-spark#87.

Rationale for this change

As agreed in apache/datafusion-spark#87, datafusion-spark is moving to the new apache/datafusion-spark repository, along with its git history. datafusion-spark 56.0.0 is the last release from this repository (branch-56 keeps the crate), and 57.0.0 will be released from the new one.

This stays a draft until the code has been imported into apache/datafusion-spark. Open PRs that touch these files have a comment pointing to apache/datafusion-spark#87.

What changes are included in this PR?

Removes the crate (datafusion/spark), its sqllogictest files (datafusion/sqllogictest/test_files/spark) and the audit-datafusion-spark-expression agent skill, all of which move to the new repository.
datafusion-cli: removes the --spark flag and its test. Without the crate, the flag could only switch the SQL dialect, which SET datafusion.sql_parser.dialect = 'spark' already does. The CLI builds its context with SessionContext::new_with_config_rt again, as it did before feat(cli): add opt-in Spark function registration #24179.
datafusion-sqllogictest: removes the Spark session setup for files under spark/. The Spark result formatting in the DataFusion engine stays, so the new repository can reuse the published engine.
CI: removes the cargo check datafusion-spark features job, its required check in .asf.yaml, its entries in ci/scripts/check_rust_workflow_skips.py, and the labeler rule.
Docs: points the Spark functions page and the contributor how-to at the new repository, removes --spark from the CLI usage page and the crate from the release publish steps, and adds a 57.0.0 upgrade note.

SparkOptions (datafusion.spark.*) stays in datafusion-common, because datafusion-spark reads it. Moving it into the crate as a ConfigExtension can be a follow-up.

What is the testing strategy for this PR?

This PR removes code and adds no tests. I ran these locally with stable Rust 1.97, because the pinned 1.99.1 isn't available on my machine:

cargo check --workspace --all-targets
both clippy passes from ci/scripts/rust_clippy.sh with -D warnings. I allowed clippy::from_iter_instead_of_collect, unfulfilled_lint_expectations and unknown_lints, which only fire on 1.97 and only in code this PR doesn't touch.
cargo fmt, taplo, typos, prettier, the generated and examples docs checks, and the .asf.yaml and workflow-skip validators
the extended test suite from CLAUDE.md: 13,332 passed, 0 failed, including all 304 sqllogictest files
cargo test --profile ci -p datafusion-cli: 115 passed, 0 failed

Are there any user-facing changes?

Yes:

datafusion-cli --spark is removed.
datafusion-spark is published from apache/datafusion-spark starting with 57.0.0. The crate name doesn't change.

…on-spark) Part of #27103.

comphead · 2026-11-11T19:11:41Z

@@ -2249,8 +2249,8 @@ impl<'a> TryFrom<&'a FormatOptions> for arrow::util::display::FormatOptions<'a>
 }

 config_namespace! {


wondering can we move this as well

it can be separate PR, it doesn't look straight forward and critical

comphead

Thanks @andygrove good negative PR.
Nice followup would be to move SparkOptions from DF config as well

comphead · 2026-11-11T19:14:58Z

@@ -2249,8 +2249,8 @@ impl<'a> TryFrom<&'a FormatOptions> for arrow::util::display::FormatOptions<'a>
 }

 config_namespace! {


it can be separate PR, it doesn't look straight forward and critical

alamb · 2026-11-12T16:54:17Z

Looks like we need to update the required checks to merge

Co-Authored-By: Claude Opus 5.5 <[email protected]>

andygrove · 2026-11-12T17:05:12Z

Branch protection comes from .asf.yaml on main, so the removed cargo check datafusion-spark features check is still required here. I opened #27375 to drop it from the required checks first. Once that merges I'll rebase this PR.

The check is no longer required, so check_rust_workflow_skips.py no longer allows it to skip on pushes to main. Co-Authored-By: Claude Opus 5.5 <[email protected]>

# Conflicts: # .github/workflows/rust.yml

…hecks (apache#27375) ## Which issue does this PR close? - Related to apache#27337. ## Rationale for this change apache#27337 removes the `datafusion-spark` crate, along with the `cargo check datafusion-spark features` CI job. ASF applies branch protection from the `.asf.yaml` on `main`, so while that check is still listed there, apache#27337 can't merge: the check it's waiting on never runs. ## What changes are included in this PR? - Removes `cargo check datafusion-spark features` from `required_status_checks` in `.asf.yaml`. - Removes the job from `POST_MERGE_SKIPPED_JOBS` in `check_rust_workflow_skips.py`. The script only lets required checks skip on pushes to `main`, because the merge queue only runs required checks. - Drops the post-merge clause from the job's `if:` in `rust.yml`. The same script requires each job's `if:` to match its skip sets exactly, so leaving the job out of `POST_MERGE_SKIPPED_JOBS` means its `if:` keeps only the fork-PR clause. As a result, the job runs on pushes to `main` again until apache#27337 removes it. It is no longer required, so it can't block anything. ## What is the testing strategy for this PR? The `Validate required_status_checks in .asf.yaml` CI check, which runs `check_asf_yaml_status_checks.py`, `check_rust_workflow_skips.py` and `test_rust_workflow_skips.py`. All three pass locally. ## Are there any user-facing changes? No. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>

alamb · 2026-11-12T20:52:33Z

🤞 EPIC

Builds on apache#7249, which tracks DataFusion branch-56 (56.0.0-rc1), with current main merged in and the DataFusion dependencies moved to DataFusion's main branch. - datafusion crates on apache/datafusion's main branch (57.0.0-dev), locked at 6c42917 - datafusion-spark on apache/datafusion-spark's main branch, locked at 4cd969d, because apache/datafusion#27337 moved the crate out of DataFusion. That repository builds against DataFusion branch-56, so a [patch] section moves its DataFusion dependencies to DataFusion main - arrow/parquet 60.1.0, object_store 0.14.2 - iceberg and iceberg-storage-opendal on apache/iceberg-rust@6c97391, the apache/iceberg-rust#3257 (Arrow 60) merge, instead of the andygrove/iceberg-rust fork. It contains the revision main uses (de0a6ac), and main's opendal-hdfs-native feature is kept - native/native-executors stays as on main, and its CI job stays off as in apache#7249 - source changes for the Arrow 60, Parquet 60 and DataFusion API changes as in apache#7249 Co-authored-by: Andy Grove <[email protected]>

Builds on apache#7249, which tracks DataFusion branch-56 (56.0.0-rc1), with current main merged in and the DataFusion dependencies moved to DataFusion's main branch. - datafusion crates on apache/datafusion's main branch (57.0.0-dev), locked at 73e0b03 - datafusion-spark on apache/datafusion-spark's main branch, locked at 21334bb, because apache/datafusion#27337 moved the crate out of DataFusion. That repository builds against DataFusion branch-56, so a [patch] section moves its DataFusion dependencies to DataFusion main - arrow/parquet 60.1.0, object_store 0.14.2 - iceberg and iceberg-storage-opendal on apache/iceberg-rust@6c97391, the apache/iceberg-rust#3257 (Arrow 60) merge, instead of the andygrove/iceberg-rust fork. It contains the revision main uses (de0a6ac), and main's opendal-hdfs-native feature is kept - native/native-executors stays as on main, and its CI job stays off as in apache#7249 - source changes for the Arrow 60, Parquet 60 and DataFusion API changes as in apache#7249 Co-authored-by: Andy Grove <[email protected]>

chore: remove the datafusion-spark crate (moving to apache/datafusi…

ec4119b

…on-spark) Part of #27103.

andygrove added the api change Changes the API exposed to users of the crate label Nov 11, 2026

github-actions Bot added documentation Improvements or additions to documentation development-process Related to development process of DataFusion sqllogictest SQL Logic Tests (.slt) common Related to common crate spark Changes to datafusion-spark crate labels Nov 11, 2026

andygrove mentioned this pull request Nov 11, 2026

[DISCUSSION] How should we test datafusion-spark against Spark? apache/datafusion-spark#87

Open

comphead reviewed Nov 11, 2026

View reviewed changes

andygrove mentioned this pull request Nov 11, 2026

Move SparkOptions and enable_ansi_mode from datafusion-common to datafusion-spark #27342

Open

alamb approved these changes Nov 11, 2026

View reviewed changes

andygrove marked this pull request as ready for review November 11, 2026 20:24

andygrove mentioned this pull request Nov 11, 2026

[EPIC] Finish moving datafusion-spark into this repository apache/datafusion-spark#56

Open

30 tasks

andygrove enabled auto-merge November 12, 2026 12:54

Merge remote-tracking branch 'apache/main' into remove-datafusion-spark

0f05092

andygrove mentioned this pull request Nov 12, 2026

fix(spark): return NaN and -Infinity from pow where Spark does #26382

Merged

alamb mentioned this pull request Nov 12, 2026

fix(spark): parse_url returns null for bad urls when ansi mode is off #27158

Closed

chore: stop requiring the datafusion-spark features check

9bf1941

Co-Authored-By: Claude Opus 5.5 <[email protected]>

andygrove mentioned this pull request Nov 12, 2026

chore: remove cargo check datafusion-spark features from required checks #27375

Merged

ci: run the datafusion-spark features check after merge

b5905a6

The check is no longer required, so check_rust_workflow_skips.py no longer allows it to skip on pushes to main. Co-Authored-By: Claude Opus 5.5 <[email protected]>

andygrove disabled auto-merge November 12, 2026 17:11

andygrove added 2 commits November 12, 2026 10:12

Merge branch 'remove-spark-required-check' into remove-datafusion-spark

39bca47

# Conflicts: # .github/workflows/rust.yml

Merge remote-tracking branch 'apache/main' into remove-datafusion-spark

eea52a6

andygrove enabled auto-merge November 12, 2026 20:23

andygrove added this pull request to the merge queue Nov 12, 2026

Merged via the queue into apache:main with commit 0302193 Nov 12, 2026
45 checks passed

andygrove deleted the remove-datafusion-spark branch November 12, 2026 21:46

github-actions Bot added the v57.0.0 label Nov 12, 2026

andygrove mentioned this pull request Nov 13, 2026

moved SparkOptions to datafusion-spark #27371

Open

Navigation Menu

Repository navigation

chore: remove the `datafusion-spark` crate (moving to apache/datafusion-spark) - #27337

chore: remove the `datafusion-spark` crate (moving to apache/datafusion-spark)#27337
andygrove merged 6 commits into
apache:mainfrom
andygrove:remove-datafusion-spark

andygrove commented Nov 11, 2026

Uh oh!

comphead Nov 11, 2026

Uh oh!

comphead Nov 11, 2026 •

edited

Loading

Uh oh!

comphead left a comment

Uh oh!

comphead Nov 11, 2026 •

edited

Loading

Uh oh!

alamb commented Nov 12, 2026

Uh oh!

andygrove commented Nov 12, 2026

Uh oh!

alamb commented Nov 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -2249,8 +2249,8 @@ impl<'a> TryFrom<&'a FormatOptions> for arrow::util::display::FormatOptions<'a>
		}

		config_namespace! {

Conversation

andygrove commented Nov 11, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

What is the testing strategy for this PR?

Are there any user-facing changes?

Uh oh!

comphead Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

comphead Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

comphead Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Nov 12, 2026

Uh oh!

andygrove commented Nov 12, 2026

Uh oh!

alamb commented Nov 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

comphead Nov 11, 2026 •

edited

Loading

comphead Nov 11, 2026 •

edited

Loading