Skip to content

[fix](external) Backport SCHEMATA, ORC and JSON/WAL fixes to 4.1 - #70319

Merged
Gabriel39 merged 4 commits into
apache:branch-4.1from
Gabriel39:fix/backport-schemata-orc-json-wal-4.1
Nov 9, 2026
Merged

Gabriel39 merged 4 commits into
apache:branch-4.1from
Gabriel39:fix/backport-schemata-orc-json-wal-4.1

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Backport three master fixes to branch-4.1: avoid unnecessary database-object loading for SCHEMATA scans, prevent ORC condition-cache seek crashes or missing rows, and preserve JSON/WAL row counts when no file columns are projected.

The commits retain their original authors and are applied in master merge order:

  1. [fix](fe) Avoid database object loading for schemata scans #70010 — Avoid database-object enumeration when SCHEMATA only needs names.
  2. [fix](be) Fix ORC condition-cache seeks past filtered stripe suffixes #70079 — Advance safely past SARG-filtered ORC stripe suffixes during condition-cache seeks.
  3. [fix](be) Preserve JSON and WAL rows with empty file projections #70154 — Track logical JSON records independently of projected columns and capture WAL row counts before column ownership is transferred.

Branch adaptations:

  • Keep Thrift field 7 reserved and add the optional include_db_ids flag at upstream field 8 with its default true. Preserve existing privilege checks, pattern matching and schema-name formatting for both enumeration paths.
  • Use the upstream fixed ORC dependency revision 5962bb071d6345cc29c5353d64d85a203525e4d9, which descends from the branch's existing revision.
  • Resolve the JSON reader conflicts by retaining 4.1's error handling and adding logical row accounting without importing unrelated missing-field/load-rejection changes. Remove the rooted-array pre-count that consumes the ondemand cursor before JSON-path materialization; retain the upstream rooted-array coverage.

Validation

  • The branch's FE production and test sources compiled with the existing Java 8 target settings. All 7 targeted FE tests passed, including the existing FrontendServiceImplTest#testGetDBNames, the 100,000-namespace case, filtering and legacy-request compatibility. FE Checkstyle passed.
  • A fresh ASAN build of the exact pinned ORC dependency passed 56 focused tests, including 32 filtered-stripe-suffix boundary cases.
  • All 7 affected C++ implementation/test/generator translation units compiled. Clang-format 16 passed for all 8 affected C++ source/header files.
  • The ORC fixture generator reproduced the committed fixture byte for byte. All 3 added Groovy suites compiled.
  • Full Doris BE unit-test execution and external SQL regression execution remain for CI; they were not run locally for this backport.

Release note

Improve SCHEMATA enumeration for large catalogs. Fix potential crashes or missing rows during ORC condition-cache seeks and incorrect JSON/WAL row counts with empty file projections.

Check List (For Author)

  • Test
    • Unit tests and focused dependency tests
    • Regression tests included and compiled; full SQL execution pending CI
  • Behavior changed:
    • Yes. Apply the upstream fixes while preserving branch-specific validation and protocol compatibility.
  • Does this need documentation?
    • No.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Add branch pick label

ARM64 link dependency follow-up

The ORC revision enables libdeflate ZLIB decoding on every architecture. Remove the old x86-only guard around add_thirdparty(deflate) so ARM64 BE links also receive the required library, matching master. An adjacent comment records this dependency invariant.

Validation: probes include each branch's actual be/cmake/thirdparty.cmake and select aarch64, arm64, x86 and x86_64 dependency configurations. Before the fix, both ARM configurations fail to link with missing libdeflate symbols while both x86 controls pass. After the fix, all eight repository/configuration combinations link and pass a DEFLATE compression/decompression round trip. These probes run natively on x86_64 and validate CMake dependency selection; a native ARM64/full BE build remains for CI. Clang-format 16 passed for the PR-affected C++ files.

)

### What problem does this PR solve?

Issue Number: N/A

Problem Summary:

Queries such as:

```sql
SELECT COUNT(*) FROM example_catalog.information_schema.schemata;
```

only need database and catalog names. However, `SchemaSchemataScanner`
uses the default `getDbNames` RPC, whose FE handler calls
`catalog.getAllDbs()` to also return database IDs. For external
catalogs, this resolves every database object even though SCHEMATA never
reads those IDs. A large namespace list can therefore churn a much
smaller database-object cache and allocate substantial temporary
metadata while waiting inside a synchronous BE-to-FE RPC. A warm
database-name cache does not remove this work.

This PR adds the optional `include_db_ids` request field, defaulting to
`true`, and lets SCHEMATA set it to `false`. The FE then calls
`catalog.getDbNames()` and omits the unused database IDs. Both paths
share the existing privilege checks, pattern matching, and schema-name
formatting. Callers that need IDs keep the object-enumeration path
without additional intermediate lists.

The new field uses ID 8; field 7 remains reserved. Definitions were
checked on master and branch-3.1/4.0/4.1/4.2. Older FE nodes ignore the
optional field and keep serving the existing response; older BE callers
retain the default ID-inclusive behavior on a new FE.

### Validation

- Added a 100,000-namespace test that rejects any database-object
enumeration and verifies the result-array lengths and omitted IDs.
- Covered privilege/pattern filtering, qualified schema names,
empty/unavailable catalogs, existing ID-returning callers, and binary
deserialization of requests without the new field.
- Confirmed the new behavior tests fail on the original handler before
implementing the change. All 7 targeted FE tests pass, including the
existing `FrontendServiceImplTest#testGetDBNames`.

```bash
bash run-fe-ut.sh --run 'org.apache.doris.service.FrontendServiceGetDbNamesTest,org.apache.doris.service.FrontendServiceImplTest#testGetDBNames'
```

- FE Checkstyle: passed with zero violations.
- Changed BE scanner: clang-format 16 and C++ syntax-only compilation
with regenerated Thrift headers passed.

An in-process FE-handler benchmark uses the actual `ExternalCatalog` /
`TestExternalCatalog` name and object-cache implementations, a synthetic
metadata provider, and stubbed catalog routing and privilege checks. It
compares the original handler source at master commit
`bdb90856e6e468eb495e5bfd67010de90570bfbc` with the patched handler,
using the same dependencies and fixture. Each trial starts with a warm
name cache and cold object cache, whose capacity is 1,000. There are
100,000 synthetic namespaces plus the two system databases. Settings:
JDK 17, 2 GiB heap, 4 active processors; discard one warm-up pair and
report medians from three measured trials.

| Metric | Original master | Patched SCHEMATA path |
| --- | ---: | ---: |
| FE handler elapsed time | 2,984.2 ms | 1,331.1 ms |
| Cumulative current-thread allocation | 2,117.2 MiB | 726.2 MiB |
| Database objects constructed | 100,002 | 0 |
| Name-provider invocations | 1 | 1 |
| Returned rows | 100,002 | 100,002 |

In this fixture, handler time falls by 55.4% (2.24x faster) and
cumulative allocation falls by 65.7%. Allocation is total bytes
allocated during the call, not peak live heap. This benchmark excludes
network transport, Thrift serialization, and SQL execution; it does not
measure a deployed PostgreSQL/Iceberg environment. Full BE linking and a
deployed end-to-end SQL test were not run locally.

### Release note

Improve `information_schema.schemata` enumeration for large catalogs by
avoiding unnecessary database-object loading.

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
    - [x] Manual test (FE-handler benchmark described above)
    - [ ] No need to test or manual test.
- Behavior changed:
- [x] No. SQL result semantics and existing ID-returning requests are
preserved.
    - [ ] Yes.
- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

(cherry picked from commit 74b477d)
…apache#70079)

### What problem does this PR solve?

Related PR: apache/doris-thirdparty#431 (merge
the ORC dependency fix first)

A condition-cache granule can start in a stripe's SARG-filtered suffix
while containing a matching row in the next stripe. On an initialized
stripe, ORC's `seekToRow()` uses the stripe-end sentinel as a row-group
index. This can access past the index when the stripe ends on an index
boundary, or prematurely return EOF when the last row group is partial.

The same-stripe fast path originates in [Apache ORC
ORC-1170](apache/orc#1114); the same missing
stripe-end handling remains in the current upstream C++ reader.

Pin the ORC fix that initializes the next selected stripe before
positioning its column readers. Preserve SARG pruning, lazy reads and
split boundaries.

Add a Doris reader regression and a 1.5 KB synthetic ORC fixture with
its generator. The fixture has two 20,000-row stripes, a 10,000-row
index stride, and one matching row at the start of each stripe. A cache
hit seeks to row 18,432 in the first stripe's filtered suffix. The SQL
regression compares cache-disabled and cached results with lazy reads
and min/max pruning enabled or disabled, and checks the profile for
actual cache hits and skipped rows.

### Release note

Fix potential crashes and missing rows when ORC condition-cache seeks
reach a SARG-filtered stripe suffix.

### Check List (For Author)

- Test:
- [x] Unit Test: 247 ORC-related BE tests passed under AddressSanitizer,
including the new Doris condition-cache case and 32 dependency boundary
cases.
- [x] Unit Test: 56 focused standalone ORC tests passed under
AddressSanitizer. Reverting the fix makes all 16 initialized-stripe
boundary cases fail.
- [ ] Regression test: SQL suite added; Groovy compilation checked. The
external SQL suite has not been executed locally.
  - [x] clang-format 16 and BE build-hygiene checks passed.
- [ ] Local full-file clang-tidy analysis did not complete within ten
minutes; no completed result is available.
- Behavior changed:
  - [x] Yes. Safely advance to the next matching stripe or EOF.
- Does this need documentation?
  - [x] No.

(cherry picked from commit 75ab5ba)
…che#70154)

### What problem does this PR solve?

Issue Number: close apache#69926

Problem Summary:

Hive JSON queries that only need partition columns can return zero rows
even when the selected files contain records. For example, `SELECT * ...
WHERE dt = 'p1'` returns data while `SELECT COUNT(*) ... WHERE dt =
'p1'` returns zero. The V2 JSON reader derived its row count from the
output block, which has no columns after physical field pruning.
Writer-backed reader tests reproduce the same row-count loss for empty
WAL projections.

Track accepted JSON records independently of materialized columns,
including batch progress, array expansion, error rollback, and
ignore-malformed NULL rows. Preserve JSON-path record selection for
empty projections, and let the existing TableReader materialize
partition values using the returned row count.

For WAL, save the deserialized source row count before materialization
transfers column ownership, then pass it through the existing filter
pipeline. This preserves the row count when no file columns are
requested. The empty-projection reproduction is at the reader interface;
production replay reachability of an empty projection remains
unverified.

Add ten JSON BE tests covering empty file schemas and projections,
arrays, JSON roots/paths, malformed input, batch boundaries, EOF, and
actual TableReader partition materialization. Add eight writer-backed
WAL cases covering empty, single-column, and reordered full projections
across multiple blocks, with and without schema discovery, plus repeated
EOF reads. Add a WAL SQL regression that forces two original async-load
transactions to abort, then verifies row counts, NULLs, aggregates,
filters, and actual values after replay. Add a Hive SQL regression using
the existing partitioned JSON fixture for COUNT, partition projections,
grouping, and file-column filters with external COUNT pushdown enabled
and disabled.

### Release note

Fix incorrect zero counts and missing rows when querying only partition
columns of Hive JSON tables. Preserve logical row counts for empty WAL
reader projections.

### Check List (For Author)

- Test
- [x] Unit Test: 166 ASAN tests passed, including JSON reader,
TableReader, WAL V2 reader, WAL reader/writer, and CSV/Text
empty-projection controls. Before the JSON fix, nine of ten new tests
failed; before the WAL fix, both empty-projection cases failed while six
projection controls passed.
- [x] WAL SQL regression: `test_group_commit_wal_row_count` passed twice
on isolated Doris 4.1.5 with scanner V2 enabled. Generated the `.out`
file with `run-regression-test.sh`, then reran for result comparison.
Execution on master is pending CI.
- [x] Hive SQL regression: added `test_hive_json_partition_only`; Groovy
compilation passed. Full execution against the external Hive fixture is
pending CI.
- Behavior changed:
- [x] Yes. Empty JSON and WAL file projections preserve logical row
counts. Hive JSON partition-only queries produce the correct rows and
aggregates.
- Does this need documentation?
    - [x] No.

(cherry picked from commit 4bde8d0)
@Gabriel39
Gabriel39 requested a review from yiguolei as a code owner November 6, 2026 03:43
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes for one P1: the ORC revision makes libdeflate mandatory on ARM64, but this branch still links it only on x86. The inline finding identifies the missing companion dependency change. No existing blocking inline findings were present.

Reviewed all 17 changed paths and all three commits included by the ORC revision, with independent JSON/WAL, SCHEMATA/ORC-integration and risk-focused coverage. The other intended fixes look consistent with their caller contracts under static inspection.

Critical checkpoints:

  • Goal and proof: names-only SCHEMATA avoids database-object enumeration; JSON/WAL preserve logical rows with empty projections; the intended ORC seek fix handles filtered stripe suffixes. Added tests target each behavior; they were inspected, not executed.
  • Scope and clarity: local changes are focused; the dependency pin includes two additional decompression changes. The all-architecture libdeflate change needs the companion linkage fix reported inline.
  • Concurrency: no new threads, shared mutable state or locking. Reader state remains scanner-owned; catalog name enumeration uses the existing cache/snapshot synchronization.
  • Lock cost and order: no new critical sections, heavy work under locks or lock-order changes identified.
  • Lifecycle and static initialization: logical row state resets per batch; malformed rollback restores it with columns. WAL captures row count before ownership moves. ORC stripe transitions reset readers/indexes/followers; decompression scratch retains its owning pool. No new ownership cycle or cross-translation-unit initializer dependency.
  • Configuration: no configuration items added or dynamic-update contract changed.
  • Compatibility: optional Thrift field 8 defaults to true, preserving legacy ID-returning requests; old FEs can ignore it. No storage/WAL format change. ARM64 link compatibility is the reported defect.
  • Thrift/protobuf identity: compared the touched definition on master and branches 4.2, 4.1, 4.0 and 3.1. Master/4.2 have the identical optional field 8; older branches have no conflicting field. Reserved field 7 remains intact; no required field or enum change.
  • Parallel paths: eager/lazy ORC and SARG on/off are covered by added tests; the dependency affects both reader generations. JSON roots/paths/outer arrays and WAL empty/subset/reordered/full projections were traced through table materialization.
  • Conditional checks: JSON path absence, root selection, malformed-null handling and ORC stripe-end/range/EOF conditions preserve their documented boundaries; no additional defect verified.
  • Test coverage: inspected FE name/privilege/pattern/default-wire tests; JSON malformed/empty/root/path/batch/partition tests; WAL multiple-block/projection/EOF/replay tests; ORC filtered-suffix, lazy/range/backward-seek and decompression tests. ARM64 linkage needs coverage after fixing the finding.
  • Expected results: WAL golden rows, null counts, sums and predicate count match input; ORC expected IDs/payloads match the generator and static fixture stripe metadata. Query comparisons are ordered. Generation/execution of expected artifacts was not repeated.
  • Observability: existing row, parse/materialization/filter, ORC seek/cache and I/O counters remain on the affected paths; no additional logging/metrics defect identified.
  • Transactions and persistence: no EditLog, visible-version, publish/delete-bitmap or WAL-byte-format change. Replay transaction handling and failover contracts are unchanged by the row-count capture.
  • Data writes, atomicity and crash behavior: production edits change enumeration/read accounting, not commit/write atomicity; no new crash-cleanup, leak or deadlock path identified in the changed lifecycle.
  • FE-BE propagation: only SCHEMATA opts out of database IDs; its consumer does not read IDs. Other senders retain the default behavior; no further transmitted state introduced.
  • Performance and memory: name enumeration avoids per-database object loading; row counting adds constant state. Lazy ORC compressed-input scratch still grows before split-chunk copies and is reused. No additional verified allocation or complexity regression.
  • Other issues: the complete dependency comparison exposed the ARM64 link defect; no second actionable issue survived independent rechecking.

No additional user focus was supplied. All four initial risk questions have explicit dispositions in the ledger; there are no unresolved candidates or static coverage gaps. Review converged in round 2. Validation limits: static review only; no builds, tests, benchmarks or external SQL were executed, and submodules were left uninitialized.

Review converged after 2 round(s).

Comment thread contrib/apache-orc Resolved
@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.53% (1964/2501)
Line Coverage 65.55% (35456/54092)
Region Coverage 53.20% (34133/64158)
Branch Coverage 55.88% (10496/18783)

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #70319 at head 47c2235 found no new actionable P0-P3 findings across all changed paths. The previously reported P1 on contrib/apache-orc (comment 4488893681) is resolved at this head: be/cmake/thirdparty.cmake now adds deflate on every architecture, so it is not reposted.

Critical checkpoints: JSON v2 now tracks accepted logical rows independently of empty file-column blocks; its array/root/path, malformed rollback, batch, EOF, filter, and TableReader paths remain aligned. WAL captures rectangular source-block rows before projection moves ownership, then passes that count through the existing materialized-filter pipeline. The FE name-only request uses optional Thrift field 8 with default true, so old or unset callers retain database IDs; only SCHEMATA opts out, and its non-ID response lists stay aligned with privileges, patterns, display-name formatting, and empty-catalog handling. The ORC revision advances from a filtered stripe suffix to the next selected stripe and the added tests cover initialized/fresh seeks, split ranges, lazy reads, min/max combinations, both columns, EOF, and backward seeks.

No new threads, locks, configuration items, storage/WAL wire formats, persistence paths, or ownership cycles were introduced. Added unit and regression tests were inspected for deterministic assertions and cleanup; they were not executed. This was a static review only: no builds, tests, benchmarks, or external SQL were run, and repository submodules remained uninitialized. No additional user-provided review focus was supplied.

Review converged after 1 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 89.29% (25/28) 🎉
Increment coverage report
Complete coverage report

@Gabriel39
Gabriel39 merged commit 5d46ecf into apache:branch-4.1 Nov 9, 2026
34 of 37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants