Repository navigation
branch-4.2: [fix](fe) Avoid database object loading for schemata scans #70010 - #70104
Merged
Merged
Conversation
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Queries such as:
```sql
SELECT COUNT(*) FROM example_catalog.information_schema.schemata;
```
only need database and catalog names. However, `SchemaSchemataScanner`
uses the default `getDbNames` RPC, whose FE handler calls
`catalog.getAllDbs()` to also return database IDs. For external
catalogs, this resolves every database object even though SCHEMATA never
reads those IDs. A large namespace list can therefore churn a much
smaller database-object cache and allocate substantial temporary
metadata while waiting inside a synchronous BE-to-FE RPC. A warm
database-name cache does not remove this work.
This PR adds the optional `include_db_ids` request field, defaulting to
`true`, and lets SCHEMATA set it to `false`. The FE then calls
`catalog.getDbNames()` and omits the unused database IDs. Both paths
share the existing privilege checks, pattern matching, and schema-name
formatting. Callers that need IDs keep the object-enumeration path
without additional intermediate lists.
The new field uses ID 8; field 7 remains reserved. Definitions were
checked on master and branch-3.1/4.0/4.1/4.2. Older FE nodes ignore the
optional field and keep serving the existing response; older BE callers
retain the default ID-inclusive behavior on a new FE.
### Validation
- Added a 100,000-namespace test that rejects any database-object
enumeration and verifies the result-array lengths and omitted IDs.
- Covered privilege/pattern filtering, qualified schema names,
empty/unavailable catalogs, existing ID-returning callers, and binary
deserialization of requests without the new field.
- Confirmed the new behavior tests fail on the original handler before
implementing the change. All 7 targeted FE tests pass, including the
existing `FrontendServiceImplTest#testGetDBNames`.
```bash
bash run-fe-ut.sh --run 'org.apache.doris.service.FrontendServiceGetDbNamesTest,org.apache.doris.service.FrontendServiceImplTest#testGetDBNames'
```
- FE Checkstyle: passed with zero violations.
- Changed BE scanner: clang-format 16 and C++ syntax-only compilation
with regenerated Thrift headers passed.
An in-process FE-handler benchmark uses the actual `ExternalCatalog` /
`TestExternalCatalog` name and object-cache implementations, a synthetic
metadata provider, and stubbed catalog routing and privilege checks. It
compares the original handler source at master commit
`bdb90856e6e468eb495e5bfd67010de90570bfbc` with the patched handler,
using the same dependencies and fixture. Each trial starts with a warm
name cache and cold object cache, whose capacity is 1,000. There are
100,000 synthetic namespaces plus the two system databases. Settings:
JDK 17, 2 GiB heap, 4 active processors; discard one warm-up pair and
report medians from three measured trials.
| Metric | Original master | Patched SCHEMATA path |
| --- | ---: | ---: |
| FE handler elapsed time | 2,984.2 ms | 1,331.1 ms |
| Cumulative current-thread allocation | 2,117.2 MiB | 726.2 MiB |
| Database objects constructed | 100,002 | 0 |
| Name-provider invocations | 1 | 1 |
| Returned rows | 100,002 | 100,002 |
In this fixture, handler time falls by 55.4% (2.24x faster) and
cumulative allocation falls by 65.7%. Allocation is total bytes
allocated during the call, not peak live heap. This benchmark excludes
network transport, Thrift serialization, and SQL execution; it does not
measure a deployed PostgreSQL/Iceberg environment. Full BE linking and a
deployed end-to-end SQL test were not run locally.
### Release note
Improve `information_schema.schemata` enumeration for large catalogs by
avoiding unnecessary database-object loading.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (FE-handler benchmark described above)
- [ ] No need to test or manual test.
- Behavior changed:
- [x] No. SQL result semantics and existing ID-returning requests are
preserved.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Nov 4, 2026
Contributor
Author
|
PR approved by anyone and no changes requested. |
Contributor
Author
|
PR approved by at least one committer and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #70010