Skip to content

[fix](arrow) Reject out-of-range Flight timestamps - #68604

Merged
Gabriel39 merged 3 commits into
apache:branch-4.1from
Gabriel39:fix/arrow-flight-timestamp-range
Sep 29, 2026
Merged

Gabriel39 merged 3 commits into
apache:branch-4.1from
Gabriel39:fix/arrow-flight-timestamp-range

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Flight SQL can publish out-of-range timestamps from legacy INT96 date materialization. For example, a non-null zero DATETIME becomes timestamp[us] with value -62169984000000000: Arrow validation succeeds, but PyArrow scalar conversion raises OverflowError.

Validate timestamps in ArrowFlightArrowBlockConvertor before publishing each batch. Check the actual timestamp unit and both UTC/local calendar bounds for zoned values. Recursively check arrays, map keys/values, and structs while skipping values masked by a NULL parent. Reject unsupported timestamps with an error identifying the field path and block row. Other Arrow consumers and Parquet scan compatibility remain unchanged.

Related PR: #68596 adds separate UTF-8 validation. This PR is independent of that change; both checks must be retained when integrating the shared Flight conversion entry point.

Testing

  • Added seven BE tests covering zero dates, years 0/10000, seconds/milliseconds/microseconds, valid calendar boundaries, pre-epoch fractions, timezone offsets, slices/subsequent batches, nested timestamps, NULL parents, and other consumers.
  • Before the fix: four new test groups failed because malformed timestamps were published; three valid/NULL groups passed.
  • After the fix: all 36 tests in ArrowFlightTimestampTest.*:DataTypeSerDeArrowTest.* passed under ASAN. Recompiled the changed converter and new tests against existing ASAN BE libraries, with C++ access control enabled. Test dates are constructed through the public unchecked setter so invalid calendar values reach the validator.
  • Added a Flight JDBC regression suite and a separate HDFS INT96 suite: four existing group4 fixtures must be read successfully with microsecond truncation at the upper calendar boundary; the zero-year fixture must return a range error. Both suites preserve the configured Flight credentials and avoid the TLS-dependent connect shortcut. The HDFS suite discovers the live FE Flight endpoint through MySQL because external clusters can override the configured port. The HDFS suite requires enableHiveTest and the external test environment.
  • Fixed the nested NULL assertion that caused the P0/Cloud P0 Groovy getAt() failure by consuming the JDBC array directly.
  • Clang-format 16 passed for all affected C++ files. Groovy 4.0.19 executed both scripts against mock JDBC connections in 12 checks covering TLS on/off, successful assertions, missing expected errors, and unrelated errors. These are script-level checks; the live external validation below additionally exercises real SQL and HDFS data.

Live external regression validation

  • Used the CI-built FE/BE binaries for 8d1158d3f5 in an isolated local cluster and rebuilt the regression framework from this branch. This follow-up changes only test scripts and expected results.
  • Before the update: all four affected suites failed locally, reproducing the stale Flight port and zero-year timestamp failures.
  • After the update: all four suites passed, with zero failures and zero skips: test_flight_int96_timestamp_range, test_remote_doris_all_types_select, test_remote_doris_statistics, and test_remote_doris_table_stats.
  • The configured Flight port intentionally remained incorrect. The INT96 suite discovered the running FE endpoint, consumed all four valid HDFS fixtures, and asserted the expected error for the invalid fixture.
  • Remote Doris success fixtures now use year 0001 as the minimum timestamp, with matching golden results. A separate assertion preserves coverage of zero-year rejection through a Remote Doris catalog.

Release note

Flight SQL now returns a server-side error for timestamps outside the supported 0001–9999 calendar range instead of publishing values that fail client conversion. Valid timestamps and NULLs retain their existing values.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
  • Behavior changed:
    • Yes: reject out-of-range Flight timestamps before publishing a batch.
  • Does this need documentation?
    • No.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.1.5-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants