Skip to content

Commit 284f211

Browse files
adriangbclaude
andauthored
fix: Remove serde_json/preserve_order feature from library crates (apache#25884)
## Which issue does this PR close? - Closes apache#25883. ## Rationale for this change `datafusion-physical-plan` enables the `serde_json/preserve_order` feature. Cargo feature unification turns this feature on for the whole application that embeds DataFusion. Then every `serde_json::Map` in that application is backed by `IndexMap` instead of `BTreeMap`. This changes the key order of JSON that the application writes (REST responses, signed JSON, snapshot tests), and the application did not ask for this change. DataFusion only needs a fixed key order for the `EXPLAIN FORMAT pgjson` output (`"Node Type"` first, `"Plans"` last, as in PostgreSQL). A library must not change a global serde behavior for this. There is also a small latent bug: `datafusion-expr` enabled `preserve_order` only as a dev-dependency. So in real builds, the key order of the logical-plan pgjson output depended on whether some other crate in the dependency graph enabled the feature. ## What changes are included in this PR? - Physical plan pgjson (`datafusion/physical-plan/src/display.rs`): the nodes are now `#[derive(Serialize)]` structs. The field order is the output order. `Extras` is an ordered list of pairs that is serialized as a map. - Logical plan pgjson (`datafusion/expr/src/logical_plan/display.rs`): there are many node shapes, so a small private `pg_fields!` macro builds an ordered field list. It replaces the `json!({...})` calls almost one-for-one. A short manual `Serialize` impl writes the fields in order. - Remove `features = ["preserve_order"]` from `datafusion-physical-plan` and from the `datafusion-expr` dev-dependencies. `datafusion-cli` and `datafusion-sqllogictest` keep the feature, because they are binaries or test crates. - Add `serde` (with `derive`) as a workspace dependency. It was already in the dependency tree through `serde_json`. `cargo tree -e features -i serde_json -p datafusion` now shows no `preserve_order`. ## What is the testing strategy for this PR? The existing tests cover this change, and the output is byte-identical: - `pgjson_snapshot_of_sample_plan` in `datafusion-physical-plan` and `test_display_pg_json` in `datafusion-expr`. These tests now build without `preserve_order`, so they really check the key order. With sorted keys, `"Details"` would come before `"Node Type"` and the snapshot would fail. - `explain.slt` and `explain_analyze.slt` pass. ## Are there any user-facing changes? - Applications that depend on DataFusion no longer get `serde_json/preserve_order` from DataFusion. - The `EXPLAIN FORMAT pgjson` output does not change, with one small exception. With `ShowMetrics::Full`, `"Actual Rows"` now always comes before `"Actual Total Time"`. Before, the order followed the order in which the operator registered its metrics. - Note: the third-party `substrait` crate also enables `serde_json/preserve_order`. Users of `datafusion-substrait` still get this feature from that crate. No public API changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent afe4a63 commit 284f211

6 files changed

Lines changed: 240 additions & 146 deletions

File tree

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,7 @@ rand = "0.9"
197197
recursive = "0.1.1"
198198
regex = "1.12"
199199
rstest = "0.27.0"
200+
serde = { version = "1", features = ["derive"] }
200201
serde_json = "1"
201202
sha2 = "^0.11.0"
202203
sqlparser = { version = "0.63.0", default-features = false, features = ["std", "visitor"] }

‎datafusion/expr/Cargo.toml‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ datafusion-proto-models = { workspace = true, optional = true }
6565
indexmap = { workspace = true }
6666
itertools = { workspace = true }
6767
recursive = { workspace = true, optional = true }
68+
serde = { workspace = true }
6869
serde_json = { workspace = true }
6970
sqlparser = { workspace = true, optional = true }
7071

@@ -73,9 +74,6 @@ criterion = { workspace = true }
7374
ctor = { workspace = true }
7475
env_logger = { workspace = true }
7576
insta = { workspace = true }
76-
# Makes sure `test_display_pg_json` behaves in a consistent way regardless of
77-
# feature unification with dependencies
78-
serde_json = { workspace = true, features = ["preserve_order"] }
7977

8078
[[bench]]
8179
name = "normalize_columns"

0 commit comments

Comments
 (0)