Repository navigation
Commit 74b477d
authored
[fix](fe) Avoid database object loading for schemata scans (apache#70010)
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Queries such as:
```sql
SELECT COUNT(*) FROM example_catalog.information_schema.schemata;
```
only need database and catalog names. However, `SchemaSchemataScanner`
uses the default `getDbNames` RPC, whose FE handler calls
`catalog.getAllDbs()` to also return database IDs. For external
catalogs, this resolves every database object even though SCHEMATA never
reads those IDs. A large namespace list can therefore churn a much
smaller database-object cache and allocate substantial temporary
metadata while waiting inside a synchronous BE-to-FE RPC. A warm
database-name cache does not remove this work.
This PR adds the optional `include_db_ids` request field, defaulting to
`true`, and lets SCHEMATA set it to `false`. The FE then calls
`catalog.getDbNames()` and omits the unused database IDs. Both paths
share the existing privilege checks, pattern matching, and schema-name
formatting. Callers that need IDs keep the object-enumeration path
without additional intermediate lists.
The new field uses ID 8; field 7 remains reserved. Definitions were
checked on master and branch-3.1/4.0/4.1/4.2. Older FE nodes ignore the
optional field and keep serving the existing response; older BE callers
retain the default ID-inclusive behavior on a new FE.
### Validation
- Added a 100,000-namespace test that rejects any database-object
enumeration and verifies the result-array lengths and omitted IDs.
- Covered privilege/pattern filtering, qualified schema names,
empty/unavailable catalogs, existing ID-returning callers, and binary
deserialization of requests without the new field.
- Confirmed the new behavior tests fail on the original handler before
implementing the change. All 7 targeted FE tests pass, including the
existing `FrontendServiceImplTest#testGetDBNames`.
```bash
bash run-fe-ut.sh --run 'org.apache.doris.service.FrontendServiceGetDbNamesTest,org.apache.doris.service.FrontendServiceImplTest#testGetDBNames'
```
- FE Checkstyle: passed with zero violations.
- Changed BE scanner: clang-format 16 and C++ syntax-only compilation
with regenerated Thrift headers passed.
An in-process FE-handler benchmark uses the actual `ExternalCatalog` /
`TestExternalCatalog` name and object-cache implementations, a synthetic
metadata provider, and stubbed catalog routing and privilege checks. It
compares the original handler source at master commit
`bdb90856e6e468eb495e5bfd67010de90570bfbc` with the patched handler,
using the same dependencies and fixture. Each trial starts with a warm
name cache and cold object cache, whose capacity is 1,000. There are
100,000 synthetic namespaces plus the two system databases. Settings:
JDK 17, 2 GiB heap, 4 active processors; discard one warm-up pair and
report medians from three measured trials.
| Metric | Original master | Patched SCHEMATA path |
| --- | ---: | ---: |
| FE handler elapsed time | 2,984.2 ms | 1,331.1 ms |
| Cumulative current-thread allocation | 2,117.2 MiB | 726.2 MiB |
| Database objects constructed | 100,002 | 0 |
| Name-provider invocations | 1 | 1 |
| Returned rows | 100,002 | 100,002 |
In this fixture, handler time falls by 55.4% (2.24x faster) and
cumulative allocation falls by 65.7%. Allocation is total bytes
allocated during the call, not peak live heap. This benchmark excludes
network transport, Thrift serialization, and SQL execution; it does not
measure a deployed PostgreSQL/Iceberg environment. Full BE linking and a
deployed end-to-end SQL test were not run locally.
### Release note
Improve `information_schema.schemata` enumeration for large catalogs by
avoiding unnecessary database-object loading.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (FE-handler benchmark described above)
- [ ] No need to test or manual test.
- Behavior changed:
- [x] No. SQL result semantics and existing ID-returning requests are
preserved.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label1 parent 1aa505c commit 74b477d
4 files changed
Lines changed: 229 additions & 15 deletions
File tree
- be/src/information_schema
- fe/fe-core/src
- main/java/org/apache/doris/service
- test/java/org/apache/doris/service
- gensrc/thrift
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
| 56 | + | |
55 | 57 | | |
56 | 58 | | |
57 | 59 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
362 | 362 | | |
363 | 363 | | |
364 | 364 | | |
365 | | - | |
366 | 365 | | |
367 | 366 | | |
368 | 367 | | |
| |||
379 | 378 | | |
380 | 379 | | |
381 | 380 | | |
| 381 | + | |
382 | 382 | | |
383 | 383 | | |
384 | 384 | | |
| |||
512 | 512 | | |
513 | 513 | | |
514 | 514 | | |
515 | | - | |
| 515 | + | |
516 | 516 | | |
517 | 517 | | |
518 | 518 | | |
519 | 519 | | |
520 | 520 | | |
521 | 521 | | |
522 | 522 | | |
| 523 | + | |
| 524 | + | |
523 | 525 | | |
524 | 526 | | |
525 | 527 | | |
| |||
533 | 535 | | |
534 | 536 | | |
535 | 537 | | |
| 538 | + | |
536 | 539 | | |
537 | | - | |
| 540 | + | |
| 541 | + | |
538 | 542 | | |
539 | | - | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
540 | 549 | | |
541 | 550 | | |
542 | 551 | | |
543 | 552 | | |
| 553 | + | |
544 | 554 | | |
545 | | - | |
| 555 | + | |
546 | 556 | | |
547 | | - | |
| 557 | + | |
548 | 558 | | |
549 | 559 | | |
550 | 560 | | |
551 | | - | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
552 | 564 | | |
553 | 565 | | |
554 | | - | |
| 566 | + | |
555 | 567 | | |
556 | 568 | | |
557 | | - | |
| 569 | + | |
558 | 570 | | |
559 | 571 | | |
560 | 572 | | |
561 | 573 | | |
562 | 574 | | |
563 | | - | |
564 | | - | |
| 575 | + | |
565 | 576 | | |
566 | 577 | | |
567 | | - | |
| 578 | + | |
568 | 579 | | |
569 | 580 | | |
570 | 581 | | |
571 | | - | |
| 582 | + | |
572 | 583 | | |
573 | 584 | | |
574 | 585 | | |
575 | 586 | | |
576 | 587 | | |
577 | | - | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
578 | 600 | | |
579 | 601 | | |
580 | 602 | | |
581 | 603 | | |
582 | 604 | | |
583 | 605 | | |
584 | | - | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
585 | 609 | | |
586 | 610 | | |
587 | 611 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| 107 | + | |
| 108 | + | |
107 | 109 | | |
108 | 110 | | |
109 | 111 | | |
| |||
0 commit comments