[fix](arrow) Validate Flight result arrays before publishing batches - #68596

### What problem does this PR solve?

Flight SQL can publish out-of-range timestamps from legacy INT96 date
materialization. For example, a non-null zero DATETIME becomes
`timestamp[us]` with value `-62169984000000000`: Arrow validation
succeeds, but PyArrow scalar conversion raises `OverflowError`.

Validate timestamps in `ArrowFlightArrowBlockConvertor` before
publishing each batch. Check the actual timestamp unit and both
UTC/local calendar bounds for zoned values. Recursively check arrays,
map keys/values, and structs while skipping values masked by a NULL
parent. Reject unsupported timestamps with an error identifying the
field path and block row. Other Arrow consumers and Parquet scan
compatibility remain unchanged.

Related PR: #68596 adds separate UTF-8 validation. This PR is
independent of that change; both checks must be retained when
integrating the shared Flight conversion entry point.

### Testing

- Added seven BE tests covering zero dates, years 0/10000,
seconds/milliseconds/microseconds, valid calendar boundaries, pre-epoch
fractions, timezone offsets, slices/subsequent batches, nested
timestamps, NULL parents, and other consumers.
- Before the fix: four new test groups failed because malformed
timestamps were published; three valid/NULL groups passed.
- After the fix: all 36 tests in
`ArrowFlightTimestampTest.*:DataTypeSerDeArrowTest.*` passed under ASAN.
Recompiled the changed converter and new tests against existing ASAN BE
libraries, with C++ access control enabled. Test dates are constructed
through the public unchecked setter so invalid calendar values reach the
validator.
- Added a Flight JDBC regression suite and a separate HDFS INT96 suite:
four existing group4 fixtures must be read successfully with microsecond
truncation at the upper calendar boundary; the zero-year fixture must
return a range error. Both suites preserve the configured Flight
credentials and avoid the TLS-dependent `connect` shortcut. The HDFS
suite discovers the live FE Flight endpoint through MySQL because
external clusters can override the configured port. The HDFS suite
requires `enableHiveTest` and the external test environment.
- Fixed the nested NULL assertion that caused the P0/Cloud P0 Groovy
`getAt()` failure by consuming the JDBC array directly.
- Clang-format 16 passed for all affected C++ files. Groovy 4.0.19
executed both scripts against mock JDBC connections in 12 checks
covering TLS on/off, successful assertions, missing expected errors, and
unrelated errors. These are script-level checks; the live external
validation below additionally exercises real SQL and HDFS data.

### Live external regression validation

- Used the CI-built FE/BE binaries for `8d1158d3f5` in an isolated local
cluster and rebuilt the regression framework from this branch. This
follow-up changes only test scripts and expected results.
- Before the update: all four affected suites failed locally,
reproducing the stale Flight port and zero-year timestamp failures.
- After the update: all four suites passed, with zero failures and zero
skips: `test_flight_int96_timestamp_range`,
`test_remote_doris_all_types_select`, `test_remote_doris_statistics`,
and `test_remote_doris_table_stats`.
- The configured Flight port intentionally remained incorrect. The INT96
suite discovered the running FE endpoint, consumed all four valid HDFS
fixtures, and asserted the expected error for the invalid fixture.
- Remote Doris success fixtures now use year 0001 as the minimum
timestamp, with matching golden results. A separate assertion preserves
coverage of zero-year rejection through a Remote Doris catalog.

### Release note

Flight SQL now returns a server-side error for timestamps outside the
supported 0001–9999 calendar range instead of publishing values that
fail client conversion. Valid timestamps and NULLs retain their existing
values.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
- Behavior changed:
- [x] Yes: reject out-of-range Flight timestamps before publishing a
batch.
- Does this need documentation?
    - [x] No.

…68596)

### What problem does this PR solve?

Arrow Flight SQL can return a string array containing invalid UTF-8, for
example from `SELECT unhex(encoded) FROM input_table` when a row
contains the hex string `84`. Arrow string builders accept the bytes,
but clients such as PyArrow fail when decoding the result.

Validate each completed array in `ArrowFlightArrowBlockConvertor` before
publishing the batch. Reuse Arrow's recursive validation to cover nested
strings and per-value UTF-8 boundaries. Return an invalid-argument error
identifying the column ordinal, field name and Arrow validation failure.
Binary values retain their byte representation, and NULL payloads are
handled through the serialized validity bitmap.

Add six BE unit tests and a Flight SQL regression suite covering
malformed encodings, separate rows whose concatenation is valid UTF-8,
subsequent batches, nested arrays/structs/map keys and values, large
strings, valid Unicode, NULLs and binary results.

### Validation

- Reproduced four failing tests before the fix; the two valid-data/NULL
tests passed.
- ASAN: all 46 tests in `ArrowBlockConvertorTest` and
`DataTypeSerDeArrowTest` passed after the fix.
- clang-format 16 passed for all three affected C++ files.
- Updated the Flight regression suite to bypass constant folding for the
constant-input check, reuse the configured Flight connection, assert
valid results directly, and clean up in a finally block.
- Groovy 4.0.19: six mock JDBC execution checks passed, covering TLS
on/off, absent expected errors, unrelated errors, and cleanup. These
checks do not replace live Flight SQL execution; the updated regression
is pending CI.
- Re-ran the six focused Flight BE tests under ASAN after the
regression-only update: all passed.

The local BE unit-test build selected the relevant suites and their test
support files; the build configuration was restored and is not part of
this PR.

### Release note

Arrow Flight SQL now rejects invalid UTF-8 string results with a
descriptive server-side error instead of returning malformed Arrow data.

### Check List (For Author)

- Test
    - [x] Regression test added (live execution pending CI)
    - [x] Unit Test
- Behavior changed:
- [x] Yes. Invalid Arrow Flight string payloads fail on the server
before the affected batch is returned.
- Does this need documentation?
    - [x] No.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

### What problem does this PR solve?

Flight SQL can publish out-of-range timestamps from legacy INT96 date
materialization. For example, a non-null zero DATETIME becomes
`timestamp[us]` with value `-62169984000000000`: Arrow validation
succeeds, but PyArrow scalar conversion raises `OverflowError`.

Validate timestamps in `ArrowFlightArrowBlockConvertor` before
publishing each batch. Check the actual timestamp unit and both
UTC/local calendar bounds for zoned values. Recursively check arrays,
map keys/values, and structs while skipping values masked by a NULL
parent. Reject unsupported timestamps with an error identifying the
field path and block row. Other Arrow consumers and Parquet scan
compatibility remain unchanged.

Related PR: #68596 adds separate UTF-8 validation. This PR is
independent of that change; both checks must be retained when
integrating the shared Flight conversion entry point.

### Testing

- Added seven BE tests covering zero dates, years 0/10000,
seconds/milliseconds/microseconds, valid calendar boundaries, pre-epoch
fractions, timezone offsets, slices/subsequent batches, nested
timestamps, NULL parents, and other consumers.
- Before the fix: four new test groups failed because malformed
timestamps were published; three valid/NULL groups passed.
- After the fix: all 36 tests in
`ArrowFlightTimestampTest.*:DataTypeSerDeArrowTest.*` passed under ASAN.
Recompiled the changed converter and new tests against existing ASAN BE
libraries, with C++ access control enabled. Test dates are constructed
through the public unchecked setter so invalid calendar values reach the
validator.
- Added a Flight JDBC regression suite and a separate HDFS INT96 suite:
four existing group4 fixtures must be read successfully with microsecond
truncation at the upper calendar boundary; the zero-year fixture must
return a range error. Both suites preserve the configured Flight
credentials and avoid the TLS-dependent `connect` shortcut. The HDFS
suite discovers the live FE Flight endpoint through MySQL because
external clusters can override the configured port. The HDFS suite
requires `enableHiveTest` and the external test environment.
- Fixed the nested NULL assertion that caused the P0/Cloud P0 Groovy
`getAt()` failure by consuming the JDBC array directly.
- Clang-format 16 passed for all affected C++ files. Groovy 4.0.19
executed both scripts against mock JDBC connections in 12 checks
covering TLS on/off, successful assertions, missing expected errors, and
unrelated errors. These are script-level checks; the live external
validation below additionally exercises real SQL and HDFS data.

### Live external regression validation

- Used the CI-built FE/BE binaries for `8d1158d3f5` in an isolated local
cluster and rebuilt the regression framework from this branch. This
follow-up changes only test scripts and expected results.
- Before the update: all four affected suites failed locally,
reproducing the stale Flight port and zero-year timestamp failures.
- After the update: all four suites passed, with zero failures and zero
skips: `test_flight_int96_timestamp_range`,
`test_remote_doris_all_types_select`, `test_remote_doris_statistics`,
and `test_remote_doris_table_stats`.
- The configured Flight port intentionally remained incorrect. The INT96
suite discovered the running FE endpoint, consumed all four valid HDFS
fixtures, and asserted the expected error for the invalid fixture.
- Remote Doris success fixtures now use year 0001 as the minimum
timestamp, with matching golden results. A separate assertion preserves
coverage of zero-year rejection through a Remote Doris catalog.

### Release note

Flight SQL now returns a server-side error for timestamps outside the
supported 0001–9999 calendar range instead of publishing values that
fail client conversion. Valid timestamps and NULLs retain their existing
values.

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Unit Test
- Behavior changed:
- [x] Yes: reject out-of-range Flight timestamps before publishing a
batch.
- Does this need documentation?
    - [x] No.

…ht results (#69771)

### What problem does this PR solve?

Issue Number: None

Related PR: #68596, #68604

Problem Summary:

#### In short

This ports branch-4.x's #68596 and #68604 to master. An Arrow Flight
result can carry values that Arrow clients cannot decode: a `STRING`
value that is not valid UTF-8 (for example `unhex('84')`), and a
`DATETIME` outside 0001-01-01..9999-12-31 (Doris's own range starts at
`0000-01-01 00:00:00`). The Arrow writers accept both, so the BE
publishes the batch and the failure surfaces later in the client:
PyArrow rejects the string, and raises `OverflowError` when it converts
the timestamp. This PR checks every batch in the Flight convertor before
it is handed out and fails the query with a server-side error that names
the column, the nested path and the row. It also makes a failed
`_query_plan` scan of the Spark and Flink connectors report its error:
the BE used to end such a scan as a complete result, so a rejected batch
would have become silently missing rows.

#### Background

- `ArrowFlightArrowBlockConvertor`
(`be/src/format/arrow/arrow_block_convertor.h`) builds the Arrow batches
of an Arrow Flight result: `ArrowFlightBatchReader` uses it for Flight
SQL and ADBC clients, which also covers a remote Doris catalog with
`use_arrow_flight = true`, and the memory scratch sink uses it for the
`_query_plan` scan API of the Spark and Flink Doris connectors.
- The STRING serde appends the bytes of each value to an Arrow `utf8`
(or `large_utf8`) builder without looking at them. Arrow requires these
values to be valid UTF-8, but only `Array::ValidateFull()` checks it;
building and publishing the array does not.
- The memory scratch sink queues the batches that `get_next` of the BE
Thrift scanner hands to those connectors, and ends the queue with a
`nullptr` sentinel. Its `close()` put the sentinel whatever the
execution status, and nothing has recorded a status in the queue since
the non-pipeline engine was removed (#35803), so a fragment that failed
(a scan error, a cancellation) ended the scan as if it were complete.
- `DATETIME(p)` is published as `timestamp[s]`, `timestamp[ms]` or
`timestamp[us]` by its scale. Arrow accepts any `int64` there, so a
year-0 value such as `0000-12-31 00:00:00` (`-62135683200000000` us) is
published as it is, while Python's `datetime`, which PyArrow converts
to, only covers years 1..9999.

#### The problem, and what it cost

| Result value | Before this PR |
|---|---|
| `unhex('84')`, `unhex('C0AF')`, a surrogate or a code point above
U+10FFFF, also inside an ARRAY / STRUCT / MAP key or value | Published
as is; the client fails decoding the batch, after earlier batches were
already delivered |
| Two rows `unhex('C2')` and `unhex('A2')` | Published; each row is
invalid UTF-8 on its own, though the concatenated buffer is valid |
| `CAST('0000-12-31 00:00:00' AS DATETIME(6))`, or a zero date read from
a table or an INT96 Parquet file | Published as `-62135683200000000`;
PyArrow raises `OverflowError` converting it |
| A `_query_plan` scan (Spark / Flink connector) whose fragment fails |
`get_next` reports the end of the scan after the rows fetched so far;
with the validation below, a table with one invalid UTF-8 row would read
zero rows and no error |

The query looks successful on the server, so neither the error nor the
offending column shows up in Doris; the Flight client gets a failure
that does not say which value caused it.

#### How this PR fixes it

- `ArrowBlockConvertor::convert_to_arrow` becomes virtual, and
`ArrowFlightArrowBlockConvertor` overrides it: after the base
conversion, it runs `ValidateFull()` on every column and returns
`InvalidArgument("Invalid Arrow Flight result in column <n> ('<name>'):
...")` on failure. `ValidateFull()` checks each value, nested children
included, so the row boundaries of the `C2` / `A2` case are respected.
Binary columns keep their bytes; NULL values are skipped through the
validity bitmap.
- A new `FlightTimestampValidator` walks each column recursively (list
values, map keys and values, struct fields), reads the actual timestamp
unit, and checks each value against 0001-01-01..9999-12-31, also against
the local calendar when the Arrow type has a time zone. Values under a
NULL list / map / struct parent are not observable and are not checked.
A value outside the range returns `InvalidArgument("Arrow Flight
timestamp in column '<path>' at row <n> is outside the supported
0001-9999 range (type: ...)")`, with the row counted in the block.
- For a timestamp with a time zone, the local bounds are computed with
the exact offsets (cctz, Python's zoneinfo) and with the minute-rounded
offsets of pytz, which PyArrow uses for zone names when it is installed
(`(offset + 30) // 60 * 60`), and the stricter bound wins. pytz turns
America/Chicago's local mean time -05:50:36 into -05:51, so 0001-01-01
05:50:36Z, local 00:00:00 for cctz, would be 0000-12-31 23:59:36 in
PyArrow; the lower bound there is 05:51:00Z. In such zones up to 30
seconds at the start of year 1 are rejected.
- `MemoryScratchSinkLocalState::close` records a failed execution status
in the result queue before it puts the sentinel, and
`ResultQueueMgr::fetch_result` checks the queue status again when it
takes the sentinel, because a fetch already waiting for data passed the
first check before the failure. `get_next` then returns the error to the
connector.
- Of the Arrow convertors, only the Flight convertor changes. The
ordinary Doris Arrow convertor (Python UDFs) and the table-format
convertors (Parquet, Iceberg, Paimon writers) keep their current
behavior; `DorisArrowBlockConvertor` still publishes the same values in
the new tests.
- The code is branch-4.2's final form of the two PRs; master's Flight
convertor already overrides `init()` and `write_column()`, and the new
override sits next to them.

#### Results

| Query over Arrow Flight | Before | After |
|---|---|---|
| `SELECT unhex(encoded) AS payload` with an invalid UTF-8 row | Batch
published, client fails decoding | Query fails: `Invalid Arrow Flight
result in column 1 ('payload'): Invalid: Invalid UTF8 sequence at string
index 0` |
| `SELECT array(unhex(encoded)) AS nested` ... | Batch published | `...
('nested'): Invalid: List child array invalid: Invalid: Invalid UTF8
sequence at string index 0` |
| `SELECT CAST(value AS DATETIME(6)) AS event_time` over `0000-12-31
00:00:00` | Batch published, client fails converting | Query fails:
`Arrow Flight timestamp in column 'event_time' at row 1 is outside the
supported 0001-9999 range (type: timestamp[us])`; nested values report
paths such as `nested[]`, `nested.key`, `nested.child` |
| Remote Doris catalog over Flight reading a `0000-01-01 00:00:00`
DATETIME | Value returned | `Arrow Flight timestamp in column
'datetime_0' at row 1 is outside the supported 0001-9999 range (type:
timestamp[s])` |
| `_query_plan` scan through `open_scanner` / `get_next` over a table
with an invalid UTF-8 row | 0 rows, then the end of the scan |
`get_next` returns `[INVALID_ARGUMENT]Invalid Arrow Flight result in
column 2 (''): Invalid: Invalid UTF8 sequence at string index 1` |
| The same scan over a `0000-12-31 00:00:00` DATETIME | 0 rows, then the
end of the scan | `Arrow Flight timestamp in column '' at row 2 is
outside the supported 0001-9999 range (type: timestamp[us])` |
| Valid Unicode text, NULLs, binary bytes, `0001-01-01`, `9999-12-31
23:59:59.999999` | Returned | Returned unchanged |

Behavior change: a Flight query (Flight SQL, ADBC, a remote Doris
catalog with `use_arrow_flight = true`, or a `_query_plan` read of the
Spark / Flink connector) whose result contains a `STRING` value that is
not valid UTF-8, or a `DATETIME` outside years 0001..9999, now fails
with the error above instead of returning data. A remote Doris catalog
that used to round-trip the zero date now reports this error; the three
remote Doris suites that inserted `0000-01-01 00:00:00` as their minimum
value now use `0001-01-01 00:00:00`, and
`test_remote_doris_all_types_select` asserts the zero-date error
separately. A `_query_plan` read whose fragment fails for any reason,
also a scan error or a cancellation, now gets the error from `get_next`
instead of ending as a complete result.

Tests:

- BE UT `ArrowBlockConvertorTest`: six new cases for invalid UTF-8 (four
malformed encodings, per-row boundaries across a slice and the next
batch, nested array / struct / map key / map value, large strings, bytes
under a NULL), and valid text, NULLs and binary.
- BE UT `ArrowFlightTimestampTest` (new file): zero date, year 0 and
year 10000 at scales 0 / 3 / 6, calendar boundaries and pre-epoch
fractions, slices and later batches, nested values and map keys, UTC and
zoned bounds, the America/Chicago lower bound with pytz's rounded
offset, naive bounds that do not depend on the session time zone, and
values under NULL parents; the ordinary Doris convertor still publishes
the same values.
- BE UT `ResultQueueMgrTest`: a fetch already waiting for data reports a
failure recorded before the sentinel; an OK status keeps a normal end.
- Regression `arrow_flight_sql_p0/test_query_plan_scan_validation` reads
`_query_plan` scans the way the connectors do (FE `_query_plan`, then BE
`open_scanner` / `get_next` over Thrift): a valid scan reads all rows,
and the invalid UTF-8 and year-0 scans get the errors above.
- Regression `arrow_flight_sql_p0/test_flight_utf8_validation`,
`test_flight_timestamp_range` (both red before this PR on master, green
with it), `test_flight_int96_timestamp_range` (reads the INT96 fixtures
of `group4` over HDFS; needs the Hive docker environment), and the three
remote Doris suites.