You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 19ef150
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](datalake) Preserve nanos schemas and safe Iceberg filters
### What problem does this PR solve?
Related PR: #68934
Problem Summary: TVF schema discovery rounded nanosecond files before V2 decoding. Generated Iceberg overwrite/conflict predicates bypassed retained DATE-bound safeguards, and partial predicates under NOT could prune matching files. Historical ORC producer semantics, metadata MIN/MAX, dropped partition sources, per-stripe reader setup, and nullable Parquet nanos dispatch also needed correction.
Route ORC/Parquet schema discovery through V2, require exact negated predicates, reject unsafe static overwrite, and preserve conservative conflict filtering. Keep Java/C++ historical carriers distinct, fall back from unsafe metadata aggregates, coalesce equal-zone stripes, and restore fused nullable decoding. SDK sources and legacy LOAD/V1 decoding are unchanged.
### Release note
Preserve nanosecond precision in ORC/Parquet TVF schema discovery and correct affected Iceberg filtering and ORC timestamp reads.
### Check List (For Author)
- Test: 425 FE unit tests; 949 BE unit tests; Iceberg nanosecond regression with real S3 ORC/Parquet TVF reads; BE/FE builds; FE Checkstyle; clang-format 16; clang-tidy; build hygiene; GCC 15 syntax check.
- Behavior changed: Yes, preserve TVF nanos precision, reject unsafe static overwrites, and avoid incorrect pruning or historical timestamp results.
- Does this need documentation: No, review corrections to the existing feature.
0 commit comments