Skip to content

Commit 62057df

Browse files
ryzhykryzhyk
committed
[adapters] Pin a datafusion fork that preserves INT96 field metadata
Coercing INT96 timestamps rebuilds every struct, list and map field with `Field::new_*`, which start from empty metadata, so reading a file that holds an INT96 column strips `PARQUET:field_id` from every container field in it. A Delta table with column mapping pairs its columns to the read schema by that id, so a struct column pairs with nothing and reads back as NULL, silently, on exactly the Databricks tables that write INT96. That happens inside datafusion's Parquet opener, before any adapter of ours is constructed, so the snapshot scan cannot be fixed from this repo. Pin a fork carrying the one-line fix until it is released upstream: issue: apache/datafusion#24786 branch: https://github.com/ryzhyk/datafusion/tree/int96-preserve-field-metadata repro: https://github.com/ryzhyk/coerce_int96_bug Three manifests need the patch, because each resolves its own graph: the platform workspace, the SQL compiler's `temp` crate, which the workspace excludes, and the workspace the SQL compiler emits for a pipeline. The last is the one that matters in production, since a pipeline is what links the Delta connector; without it the fix would never reach a running pipeline. Patching one datafusion crate pulls its siblings in from the fork, so all 32 in the graph are patched, or the same crate resolves twice and the build fails on mismatched types. `delta_table_input_int96_column_mapping_snapshot_test` guards the pin: remove the `[patch.crates-io]` section and the struct column comes back NULL.
1 parent be3f074 commit 62057df

6 files changed

Lines changed: 358 additions & 67 deletions

File tree

‎Cargo.lock‎

Lines changed: 33 additions & 66 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -374,3 +374,57 @@ debug = true
374374

375375
[profile.release]
376376
debug = "line-tables-only"
377+
378+
# Feldera fork of datafusion: release 53.1.0 + 1 patch, on branch
379+
# `int96-preserve-field-metadata-53.1.0`.
380+
#
381+
# - Preserve field metadata when coercing INT96 timestamps.
382+
# `Int96Coercer` rebuilds struct, list and map fields with `Field::new_*`,
383+
# which start from empty metadata, so reading a file that holds an INT96
384+
# column strips `PARQUET:field_id` from every container field. A Delta
385+
# table with column mapping pairs its columns to the read schema by that
386+
# id, so a struct column silently reads as NULL.
387+
# Upstream issue: https://github.com/apache/datafusion/issues/24786
388+
# Upstream branch: https://github.com/ryzhyk/datafusion/tree/int96-preserve-field-metadata
389+
# Repro: https://github.com/ryzhyk/coerce_int96_bug
390+
#
391+
# Patching one crate pulls its siblings in from the fork, so every datafusion
392+
# crate in the graph must be patched or the same crate resolves twice, once
393+
# from the fork and once from crates.io, and the build fails on mismatched
394+
# types. `datafusion-functions-json` is a third-party crate, not part of the
395+
# datafusion workspace, so it stays on crates.io.
396+
#
397+
# Drop this section once the fix is released upstream.
398+
[patch.crates-io]
399+
datafusion = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
400+
datafusion-catalog = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
401+
datafusion-catalog-listing = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
402+
datafusion-common = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
403+
datafusion-common-runtime = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
404+
datafusion-datasource = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
405+
datafusion-datasource-arrow = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
406+
datafusion-datasource-csv = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
407+
datafusion-datasource-json = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
408+
datafusion-doc = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
409+
datafusion-execution = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
410+
datafusion-expr = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
411+
datafusion-expr-common = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
412+
datafusion-functions = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
413+
datafusion-functions-aggregate = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
414+
datafusion-functions-aggregate-common = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
415+
datafusion-functions-nested = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
416+
datafusion-functions-table = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
417+
datafusion-functions-window = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
418+
datafusion-functions-window-common = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
419+
datafusion-macros = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
420+
datafusion-optimizer = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
421+
datafusion-physical-expr = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
422+
datafusion-physical-expr-adapter = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
423+
datafusion-physical-expr-common = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
424+
datafusion-physical-optimizer = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
425+
datafusion-physical-plan = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
426+
datafusion-proto = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
427+
datafusion-proto-common = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
428+
datafusion-pruning = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
429+
datafusion-session = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }
430+
datafusion-sql = { git = "https://github.com/feldera/datafusion.git", rev = "bee5811ee4c1cfea159bc2e6ece0309c59a66b9f" }

0 commit comments

Comments
 (0)