Repository navigation
[fix](fe) Avoid database object loading for schemata scans - #70010
Conversation
### What problem does this PR solve? Issue Number: N/A Problem Summary: SCHEMATA only consumes database names, but its metadata RPC materializes every database object to compute unused IDs. Let this scanner opt out of IDs through an optional request flag while preserving existing ID-inclusive callers and privilege filtering. ### Release note Improve information_schema.schemata enumeration for large catalogs. ### Check List (For Author) - Test: 7 targeted FE tests; FE Checkstyle; clang-format 16; BE scanner syntax check; synthetic 100,000-namespace FE-handler benchmark. - Behavior changed: No SQL result-semantic changes. - Does this need documentation: No.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
/review |
There was a problem hiding this comment.
Automated review completed for PR 70010 at head 6ce6ae7. No verified correctness, compatibility, concurrency, memory, or error-handling findings remain, so no inline comments are submitted.
The change meets its stated goal: SCHEMATA sets include_db_ids=false, the FE uses the catalog name snapshot without materializing every DatabaseIf, and all output lists remain aligned. Existing callers keep the optional Thrift field's default true and continue receiving db_ids. Field 8 is unused on the checked maintained branches and is optional, so mixed-version FE/BE behavior remains compatible.
Critical checkpoints: the implementation is focused and the added FE tests cover names-only behavior, legacy defaults, privileges, patterns, empty/unavailable catalogs, and display names; tests and builds were inspected but not executed under the static-only review contract. The new lambda mutates only request-local result lists and captures effectively-final per-request state; no new shared concurrency or lifecycle hazard is present. No configuration, transaction, persistence, data-write, or observability changes are involved. All parallel TGetDbs consumers and the BE schema scanners were checked, and only SCHEMATA uses the false flag. There is no additional user-provided review focus.
The review converged in round 1: all assigned scopes returned NO_NEW_VALUABLE_FINDINGS, every risk focus was verified and dismissed with evidence, and there are no existing blocking inline findings to carry forward.
Review converged after 1 round(s).
TPC-H: Total hot run time: 28557 ms |
TPC-H: Total hot run time: 28817 ms |
TPC-DS: Total hot run time: 151509 ms |
ClickBench: Total hot run time: 24.41 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Queries such as:
```sql
SELECT COUNT(*) FROM example_catalog.information_schema.schemata;
```
only need database and catalog names. However, `SchemaSchemataScanner`
uses the default `getDbNames` RPC, whose FE handler calls
`catalog.getAllDbs()` to also return database IDs. For external
catalogs, this resolves every database object even though SCHEMATA never
reads those IDs. A large namespace list can therefore churn a much
smaller database-object cache and allocate substantial temporary
metadata while waiting inside a synchronous BE-to-FE RPC. A warm
database-name cache does not remove this work.
This PR adds the optional `include_db_ids` request field, defaulting to
`true`, and lets SCHEMATA set it to `false`. The FE then calls
`catalog.getDbNames()` and omits the unused database IDs. Both paths
share the existing privilege checks, pattern matching, and schema-name
formatting. Callers that need IDs keep the object-enumeration path
without additional intermediate lists.
The new field uses ID 8; field 7 remains reserved. Definitions were
checked on master and branch-3.1/4.0/4.1/4.2. Older FE nodes ignore the
optional field and keep serving the existing response; older BE callers
retain the default ID-inclusive behavior on a new FE.
### Validation
- Added a 100,000-namespace test that rejects any database-object
enumeration and verifies the result-array lengths and omitted IDs.
- Covered privilege/pattern filtering, qualified schema names,
empty/unavailable catalogs, existing ID-returning callers, and binary
deserialization of requests without the new field.
- Confirmed the new behavior tests fail on the original handler before
implementing the change. All 7 targeted FE tests pass, including the
existing `FrontendServiceImplTest#testGetDBNames`.
```bash
bash run-fe-ut.sh --run 'org.apache.doris.service.FrontendServiceGetDbNamesTest,org.apache.doris.service.FrontendServiceImplTest#testGetDBNames'
```
- FE Checkstyle: passed with zero violations.
- Changed BE scanner: clang-format 16 and C++ syntax-only compilation
with regenerated Thrift headers passed.
An in-process FE-handler benchmark uses the actual `ExternalCatalog` /
`TestExternalCatalog` name and object-cache implementations, a synthetic
metadata provider, and stubbed catalog routing and privilege checks. It
compares the original handler source at master commit
`bdb90856e6e468eb495e5bfd67010de90570bfbc` with the patched handler,
using the same dependencies and fixture. Each trial starts with a warm
name cache and cold object cache, whose capacity is 1,000. There are
100,000 synthetic namespaces plus the two system databases. Settings:
JDK 17, 2 GiB heap, 4 active processors; discard one warm-up pair and
report medians from three measured trials.
| Metric | Original master | Patched SCHEMATA path |
| --- | ---: | ---: |
| FE handler elapsed time | 2,984.2 ms | 1,331.1 ms |
| Cumulative current-thread allocation | 2,117.2 MiB | 726.2 MiB |
| Database objects constructed | 100,002 | 0 |
| Name-provider invocations | 1 | 1 |
| Returned rows | 100,002 | 100,002 |
In this fixture, handler time falls by 55.4% (2.24x faster) and
cumulative allocation falls by 65.7%. Allocation is total bytes
allocated during the call, not peak live heap. This benchmark excludes
network transport, Thrift serialization, and SQL execution; it does not
measure a deployed PostgreSQL/Iceberg environment. Full BE linking and a
deployed end-to-end SQL test were not run locally.
### Release note
Improve `information_schema.schemata` enumeration for large catalogs by
avoiding unnecessary database-object loading.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (FE-handler benchmark described above)
- [ ] No need to test or manual test.
- Behavior changed:
- [x] No. SQL result semantics and existing ID-returning requests are
preserved.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
…#70010 (#70104) Cherry-picked from #70010 Co-authored-by: Gabriel <[email protected]>
) ### What problem does this PR solve? Backport three master fixes to `branch-4.1`: avoid unnecessary database-object loading for SCHEMATA scans, prevent ORC condition-cache seek crashes or missing rows, and preserve JSON/WAL row counts when no file columns are projected. The commits retain their original authors and are applied in master merge order: 1. #70010 — Avoid database-object enumeration when SCHEMATA only needs names. 2. #70079 — Advance safely past SARG-filtered ORC stripe suffixes during condition-cache seeks. 3. #70154 — Track logical JSON records independently of projected columns and capture WAL row counts before column ownership is transferred. Branch adaptations: - Keep Thrift field 7 reserved and add the optional `include_db_ids` flag at upstream field 8 with its default `true`. Preserve existing privilege checks, pattern matching and schema-name formatting for both enumeration paths. - Use the upstream fixed ORC dependency revision `5962bb071d6345cc29c5353d64d85a203525e4d9`, which descends from the branch's existing revision. - Resolve the JSON reader conflicts by retaining 4.1's error handling and adding logical row accounting without importing unrelated missing-field/load-rejection changes. Remove the rooted-array pre-count that consumes the ondemand cursor before JSON-path materialization; retain the upstream rooted-array coverage. ### Validation - The branch's FE production and test sources compiled with the existing Java 8 target settings. All **7 targeted FE tests passed**, including the existing `FrontendServiceImplTest#testGetDBNames`, the 100,000-namespace case, filtering and legacy-request compatibility. FE Checkstyle passed. - A fresh ASAN build of the exact pinned ORC dependency passed **56 focused tests**, including **32 filtered-stripe-suffix boundary cases**. - All **7 affected C++ implementation/test/generator translation units compiled**. Clang-format 16 passed for all **8 affected C++ source/header files**. - The ORC fixture generator reproduced the committed fixture byte for byte. All **3 added Groovy suites compiled**. - Full Doris BE unit-test execution and external SQL regression execution remain for CI; they were not run locally for this backport. ### Release note Improve SCHEMATA enumeration for large catalogs. Fix potential crashes or missing rows during ORC condition-cache seeks and incorrect JSON/WAL row counts with empty file projections. ### Check List (For Author) - Test - [x] Unit tests and focused dependency tests - [x] Regression tests included and compiled; full SQL execution pending CI - Behavior changed: - [x] Yes. Apply the upstream fixes while preserving branch-specific validation and protocol compatibility. - Does this need documentation? - [x] No. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Add branch pick label ### ARM64 link dependency follow-up The ORC revision enables libdeflate ZLIB decoding on every architecture. Remove the old x86-only guard around `add_thirdparty(deflate)` so ARM64 BE links also receive the required library, matching master. An adjacent comment records this dependency invariant. Validation: probes include each branch's actual `be/cmake/thirdparty.cmake` and select `aarch64`, `arm64`, `x86` and `x86_64` dependency configurations. Before the fix, both ARM configurations fail to link with missing libdeflate symbols while both x86 controls pass. After the fix, all eight repository/configuration combinations link and pass a DEFLATE compression/decompression round trip. These probes run natively on x86_64 and validate CMake dependency selection; a native ARM64/full BE build remains for CI. Clang-format 16 passed for the PR-affected C++ files.
What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Queries such as:
only need database and catalog names. However,
SchemaSchemataScanneruses the defaultgetDbNamesRPC, whose FE handler callscatalog.getAllDbs()to also return database IDs. For external catalogs, this resolves every database object even though SCHEMATA never reads those IDs. A large namespace list can therefore churn a much smaller database-object cache and allocate substantial temporary metadata while waiting inside a synchronous BE-to-FE RPC. A warm database-name cache does not remove this work.This PR adds the optional
include_db_idsrequest field, defaulting totrue, and lets SCHEMATA set it tofalse. The FE then callscatalog.getDbNames()and omits the unused database IDs. Both paths share the existing privilege checks, pattern matching, and schema-name formatting. Callers that need IDs keep the object-enumeration path without additional intermediate lists.The new field uses ID 8; field 7 remains reserved. Definitions were checked on master and branch-3.1/4.0/4.1/4.2. Older FE nodes ignore the optional field and keep serving the existing response; older BE callers retain the default ID-inclusive behavior on a new FE.
Validation
FrontendServiceImplTest#testGetDBNames.bash run-fe-ut.sh --run 'org.apache.doris.service.FrontendServiceGetDbNamesTest,org.apache.doris.service.FrontendServiceImplTest#testGetDBNames'An in-process FE-handler benchmark uses the actual
ExternalCatalog/TestExternalCatalogname and object-cache implementations, a synthetic metadata provider, and stubbed catalog routing and privilege checks. It compares the original handler source at master commitbdb90856e6e468eb495e5bfd67010de90570bfbcwith the patched handler, using the same dependencies and fixture. Each trial starts with a warm name cache and cold object cache, whose capacity is 1,000. There are 100,000 synthetic namespaces plus the two system databases. Settings: JDK 17, 2 GiB heap, 4 active processors; discard one warm-up pair and report medians from three measured trials.In this fixture, handler time falls by 55.4% (2.24x faster) and cumulative allocation falls by 65.7%. Allocation is total bytes allocated during the call, not peak live heap. This benchmark excludes network transport, Thrift serialization, and SQL execution; it does not measure a deployed PostgreSQL/Iceberg environment. Full BE linking and a deployed end-to-end SQL test were not run locally.
Release note
Improve
information_schema.schemataenumeration for large catalogs by avoiding unnecessary database-object loading.Check List (For Author)
Check List (For Reviewer who merge this PR)