Repository navigation
Commit 3c6af92
authored
[feature](inverted index) Add index_disk_usage table function (apache#67977)
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Doris reports the total inverted index size of a rowset or table (for
example `information_schema.rowsets.INDEX_DISK_SIZE` and `SHOW DATA`),
but SQL users cannot see how much space each index takes, or how that
space splits between the term dictionary, postings, positions and
statistics. Deciding whether to drop `support_phrase`, change a parser
or switch the index storage format therefore requires inspecting index
files by hand on each backend.
This PR adds the table function `index_disk_usage`:
```sql
SELECT index_name, column_name, structure, storage_format,
SUM(total_bytes), SUM(dict_bytes), SUM(posting_bytes), SUM(position_bytes), SUM(stats_bytes)
FROM index_disk_usage("database" = "db1", "table" = "t1")
GROUP BY 1, 2, 3, 4;
```
- Parameters: `database` and `table` (required), `partitions` and
`indexes` (comma-separated names), `level` (`tablet` by default,
`rowset` or `segment`), and `position_detail` (default `false`).
- Columns: `PARTITION_NAME`, `MATERIALIZED_INDEX_NAME`, `TABLET_ID`,
`BACKEND_ID`, `ROWSET_ID`, `SEGMENT_ID`, `INDEX_ID`, `INDEX_NAME`,
`INDEX_TYPE`, `COLUMN_NAME`, `INDEX_SUFFIX`, `STRUCTURE`
(`TERM`/`BKD`/`ANN`/`CONTAINER`), `STORAGE_FORMAT`
(`V1`/`V2`/`V3`/`SNII`), `SEGMENT_COUNT`, `ROW_COUNT`, `TOTAL_BYTES`,
`DICT_BYTES`, `POSTING_BYTES`, `POSITION_BYTES`, `STATS_BYTES`,
`OTHER_BYTES`, `STATS_SOURCE`.
How it works:
- FE resolves the table, its formal partitions and the tablets of the
base and rollup indexes, and checks the `SHOW` privilege. In local mode
the partition visible versions are read together with the tablets under
the table lock; in cloud mode they are fetched from meta-service after
the lock is released, with the batched snapshot version API. Each tablet
is sent to one queryable replica through a new `INDEX_DISK_USAGE`
metadata scan range; a cloud replica routed to a smooth upgrade source
backend fails the query, because that backend cannot run this scan. A
`partitions` or `indexes` filter that names nothing is rejected.
- BE captures the rowsets visible at that version (syncing rowsets first
in cloud mode) and reads only index file metadata of each segment:
- V1/V2/V3: CLucene compound directory entries are attributed by
extension (`tis`/`tii` to dictionary, `frq` to postings, `prx` to
positions, `nrm` to statistics, everything else to other). BKD files
form a `BKD` row with only `TOTAL_BYTES`. The shared V2/V3 compound
header is reported as a `CONTAINER` row.
- SNII: the logical index directory and core metadata are read. The
dictionary region, sampled term index, dictionary block directory and
BSBF count as dictionary; the posting region as postings; core metadata
and norms as statistics; the null bitmap as other. SNII interleaves
positions with postings, so `POSITION_BYTES` is NULL unless
`position_detail=true` decodes the posting region to split them out.
Docs-only indexes report 0 positions.
- Rows are aggregated to the requested level. Within a segment,
`TOTAL_BYTES` of all rows adds up to the index file sizes; unknown
components are NULL.
- Every index file read carries the IO context of the query (query id,
reader type and the remote scan cache write limiter), so file cache
accounting and `file_cache_query_limit_bytes` apply to this scan like to
any other scan of the query.
- A request covering more tablets than the session variable
`index_disk_usage_max_tablets` (default 20000) is rejected, and
`position_detail=true` is additionally limited by
`index_disk_usage_position_detail_max_tablets` (default 10000).
Collecting these statistics at write time and an FE-aggregated
`information_schema` table are left to follow-up PRs.
### Release note
Add the table function `index_disk_usage` to show the on-disk size of
each inverted or ANN index, broken down into dictionary, posting,
position and statistics bytes, at tablet, rowset or segment level.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- `external_table_p0/tvf/test_index_disk_usage_tvf`: V2, V3 and SNII
tables with two rowsets each; component presence per structure,
per-rowset totals reconciled with
`information_schema.rowsets.INDEX_DISK_SIZE`, all three levels, SNII
`position_detail`, index filtering and parameter errors. Passed on a
local single FE/BE cluster.
- [x] Unit Test
- BE: `IndexDiskUsage*` (V1/V2/V3/SNII collectors, docs-only and
positional indexes, `position_detail`, BKD, index filtering, missing
files, level aggregation) and `SniiSegmentReaderLogicalIndexesTest`: 22
passed.
- FE: `IndexDiskUsageTableValuedFunctionTest` (parameters, privileges,
partition/index resolution, visible version read outside the table lock,
`position_detail` tablet limit) and `IndexDiskUsageScanNodeTest`
(replica selection and grouping by backend): 20 passed.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
- The new table function needs a page in doris-website; not submitted
yet.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->1 parent 07887d2 commit 3c6af92
31 files changed
Lines changed: 5374 additions & 14 deletions
File tree
- be
- src
- exec/scan
- format/table
- storage
- index
- inverted/snii/reader
- segment
- test
- format/table
- storage/index
- inverted/snii/reader
- fe/fe-core/src
- main/java/org/apache/doris
- catalog
- datasource/tvf/source
- nereids/trees/expressions
- functions/table
- visitor
- planner
- qe
- tablefunction
- test/java/org/apache/doris
- datasource/tvf/source
- qe
- tablefunction
- gensrc/thrift
- regression-test
- data/external_table_p0/tvf
- suites/external_table_p0/tvf
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
| |||
69 | 70 | | |
70 | 71 | | |
71 | 72 | | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
72 | 78 | | |
73 | 79 | | |
74 | 80 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
59 | 62 | | |
60 | 63 | | |
61 | 64 | | |
| |||
0 commit comments