Skip to content

Commit 3c6af92

Browse files
authored
[feature](inverted index) Add index_disk_usage table function (apache#67977)
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Doris reports the total inverted index size of a rowset or table (for example `information_schema.rowsets.INDEX_DISK_SIZE` and `SHOW DATA`), but SQL users cannot see how much space each index takes, or how that space splits between the term dictionary, postings, positions and statistics. Deciding whether to drop `support_phrase`, change a parser or switch the index storage format therefore requires inspecting index files by hand on each backend. This PR adds the table function `index_disk_usage`: ```sql SELECT index_name, column_name, structure, storage_format, SUM(total_bytes), SUM(dict_bytes), SUM(posting_bytes), SUM(position_bytes), SUM(stats_bytes) FROM index_disk_usage("database" = "db1", "table" = "t1") GROUP BY 1, 2, 3, 4; ``` - Parameters: `database` and `table` (required), `partitions` and `indexes` (comma-separated names), `level` (`tablet` by default, `rowset` or `segment`), and `position_detail` (default `false`). - Columns: `PARTITION_NAME`, `MATERIALIZED_INDEX_NAME`, `TABLET_ID`, `BACKEND_ID`, `ROWSET_ID`, `SEGMENT_ID`, `INDEX_ID`, `INDEX_NAME`, `INDEX_TYPE`, `COLUMN_NAME`, `INDEX_SUFFIX`, `STRUCTURE` (`TERM`/`BKD`/`ANN`/`CONTAINER`), `STORAGE_FORMAT` (`V1`/`V2`/`V3`/`SNII`), `SEGMENT_COUNT`, `ROW_COUNT`, `TOTAL_BYTES`, `DICT_BYTES`, `POSTING_BYTES`, `POSITION_BYTES`, `STATS_BYTES`, `OTHER_BYTES`, `STATS_SOURCE`. How it works: - FE resolves the table, its formal partitions and the tablets of the base and rollup indexes, and checks the `SHOW` privilege. In local mode the partition visible versions are read together with the tablets under the table lock; in cloud mode they are fetched from meta-service after the lock is released, with the batched snapshot version API. Each tablet is sent to one queryable replica through a new `INDEX_DISK_USAGE` metadata scan range; a cloud replica routed to a smooth upgrade source backend fails the query, because that backend cannot run this scan. A `partitions` or `indexes` filter that names nothing is rejected. - BE captures the rowsets visible at that version (syncing rowsets first in cloud mode) and reads only index file metadata of each segment: - V1/V2/V3: CLucene compound directory entries are attributed by extension (`tis`/`tii` to dictionary, `frq` to postings, `prx` to positions, `nrm` to statistics, everything else to other). BKD files form a `BKD` row with only `TOTAL_BYTES`. The shared V2/V3 compound header is reported as a `CONTAINER` row. - SNII: the logical index directory and core metadata are read. The dictionary region, sampled term index, dictionary block directory and BSBF count as dictionary; the posting region as postings; core metadata and norms as statistics; the null bitmap as other. SNII interleaves positions with postings, so `POSITION_BYTES` is NULL unless `position_detail=true` decodes the posting region to split them out. Docs-only indexes report 0 positions. - Rows are aggregated to the requested level. Within a segment, `TOTAL_BYTES` of all rows adds up to the index file sizes; unknown components are NULL. - Every index file read carries the IO context of the query (query id, reader type and the remote scan cache write limiter), so file cache accounting and `file_cache_query_limit_bytes` apply to this scan like to any other scan of the query. - A request covering more tablets than the session variable `index_disk_usage_max_tablets` (default 20000) is rejected, and `position_detail=true` is additionally limited by `index_disk_usage_position_detail_max_tablets` (default 10000). Collecting these statistics at write time and an FE-aggregated `information_schema` table are left to follow-up PRs. ### Release note Add the table function `index_disk_usage` to show the on-disk size of each inverted or ANN index, broken down into dictionary, posting, position and statistics bytes, at tablet, rowset or segment level. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - `external_table_p0/tvf/test_index_disk_usage_tvf`: V2, V3 and SNII tables with two rowsets each; component presence per structure, per-rowset totals reconciled with `information_schema.rowsets.INDEX_DISK_SIZE`, all three levels, SNII `position_detail`, index filtering and parameter errors. Passed on a local single FE/BE cluster. - [x] Unit Test - BE: `IndexDiskUsage*` (V1/V2/V3/SNII collectors, docs-only and positional indexes, `position_detail`, BKD, index filtering, missing files, level aggregation) and `SniiSegmentReaderLogicalIndexesTest`: 22 passed. - FE: `IndexDiskUsageTableValuedFunctionTest` (parameters, privileges, partition/index resolution, visible version read outside the table lock, `position_detail` tablet limit) and `IndexDiskUsageScanNodeTest` (replica selection and grouping by backend): 20 passed. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [x] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - The new table function needs a page in doris-website; not submitted yet. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
1 parent 07887d2 commit 3c6af92

31 files changed

Lines changed: 5374 additions & 14 deletions

‎be/src/exec/scan/meta_scanner.cpp‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
#include "core/column/column_vector.h"
3838
#include "core/data_type/define_primitive_type.h"
3939
#include "core/types.h"
40+
#include "format/table/index_disk_usage_reader.h"
4041
#include "format/table/parquet_metadata_reader.h"
4142
#include "runtime/cluster_info.h"
4243
#include "runtime/descriptors.h"
@@ -69,6 +70,11 @@ Status MetaScanner::_open_impl(RuntimeState* state) {
6970
_scan_range.meta_scan_range);
7071
RETURN_IF_ERROR(reader->init_reader());
7172
_reader = std::move(reader);
73+
} else if (_scan_range.meta_scan_range.metadata_type == TMetadataType::INDEX_DISK_USAGE) {
74+
auto reader = IndexDiskUsageReader::create_unique(_tuple_desc->slots(), state, _profile,
75+
_scan_range.meta_scan_range);
76+
RETURN_IF_ERROR(reader->init_reader());
77+
_reader = std::move(reader);
7278
} else {
7379
RETURN_IF_ERROR(_fetch_metadata(_scan_range.meta_scan_range));
7480
}

‎be/src/format/table/index_disk_usage_reader.cpp‎

Lines changed: 436 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#pragma once
19+
20+
#include <gen_cpp/PlanNodes_types.h>
21+
22+
#include <cstdint>
23+
#include <vector>
24+
25+
#include "common/factory_creator.h"
26+
#include "common/status.h"
27+
#include "format/generic_reader.h"
28+
#include "io/io_common.h"
29+
#include "runtime/descriptors.h"
30+
#include "storage/index/index_disk_usage.h"
31+
#include "storage/tablet/tablet_fwd.h"
32+
33+
namespace doris {
34+
class Block;
35+
class RuntimeProfile;
36+
class RuntimeState;
37+
38+
// Produces the rows of the index_disk_usage table function for the tablets of one scan range.
39+
// Each block holds the rows of one tablet, so memory stays bounded by the largest tablet.
40+
class IndexDiskUsageReader : public GenericReader {
41+
ENABLE_FACTORY_CREATOR(IndexDiskUsageReader);
42+
43+
public:
44+
// Output columns, in the order the FE schema declares them.
45+
enum class Column : uint8_t {
46+
kPartitionName,
47+
kMaterializedIndexName,
48+
kTabletId,
49+
kBackendId,
50+
kRowsetId,
51+
kSegmentId,
52+
kIndexId,
53+
kIndexName,
54+
kIndexType,
55+
kColumnName,
56+
kIndexSuffix,
57+
kStructure,
58+
kStorageFormat,
59+
kSegmentCount,
60+
kRowCount,
61+
kTotalBytes,
62+
kDictBytes,
63+
kPostingBytes,
64+
kPositionBytes,
65+
kStatsBytes,
66+
kOtherBytes,
67+
kStatsSource,
68+
};
69+
70+
IndexDiskUsageReader(std::vector<SlotDescriptor*> slots, RuntimeState* state,
71+
RuntimeProfile* profile, TMetaScanRange scan_range);
72+
~IndexDiskUsageReader() override = default;
73+
74+
Status init_reader();
75+
Status _do_get_next_block(Block* block, size_t* read_rows, bool* eof) override;
76+
Status close() override { return Status::OK(); }
77+
const segment_v2::IndexDiskUsageOptions& options() const { return _options; }
78+
79+
// The schema that labels the rows of a tablet: the newest schema among its rowsets, because a
80+
// cached cloud tablet may not know an index that later rowsets already carry.
81+
static TabletSchemaSPtr label_schema(const TabletSchemaSPtr& tablet_schema,
82+
const std::vector<RowsetSharedPtr>& rowsets);
83+
// The IO context of one tablet: the query context plus the tablet TTL that classifies its
84+
// reads in the file cache.
85+
static io::IOContext tablet_io_context(const io::IOContext& query_io_ctx, int64_t ttl_seconds);
86+
// A tablet and its rowsets through the scan version.
87+
struct TabletRowsets {
88+
BaseTabletSPtr tablet;
89+
std::vector<RowsetSharedPtr> rowsets;
90+
};
91+
// Looks up a tablet and captures its rowsets through `version`. A cloud tablet is brought up
92+
// to date with meta-service, so rowsets that a compaction on another backend replaced are not
93+
// reported, and a tablet that was just loaded is only synchronized if it lacks `version`.
94+
static Result<TabletRowsets> capture_tablet_rowsets(int64_t tablet_id, int64_t version);
95+
96+
protected:
97+
Status _do_init_reader(ReaderInitContext* /*ctx*/) override { return init_reader(); }
98+
99+
private:
100+
Status _collect_tablet(const TIndexDiskUsageTablet& target,
101+
std::vector<segment_v2::IndexDiskUsageRow>* rows,
102+
TabletSchemaSPtr* current_schema) const;
103+
Status _fill_block(Block* block, const TIndexDiskUsageTablet& target,
104+
const TabletSchema& schema,
105+
const std::vector<segment_v2::IndexDiskUsageRow>& rows) const;
106+
void _append_rows(std::vector<MutableColumnPtr>& columns, const TIndexDiskUsageTablet& target,
107+
const TabletSchema& schema,
108+
const std::vector<segment_v2::IndexDiskUsageRow>& rows) const;
109+
110+
RuntimeState* _state = nullptr;
111+
std::vector<SlotDescriptor*> _slots;
112+
TMetaScanRange _scan_range;
113+
std::vector<Column> _slot_columns;
114+
segment_v2::IndexDiskUsageLevel _level = segment_v2::IndexDiskUsageLevel::kTablet;
115+
// Referenced by _options, so the reader must not move after init_reader().
116+
io::IOContext _io_ctx;
117+
segment_v2::IndexDiskUsageOptions _options;
118+
size_t _next_tablet = 0;
119+
};
120+
121+
} // namespace doris

‎be/src/storage/CMakeLists.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,9 @@ pch_reuse(Storage)
5656
# - the three heaviest template-instantiation TUs (predicate creators) which
5757
# would dominate any batch they join
5858
set(STORAGE_UNITY_SKIP
59+
# Keep storage batches stable: null_predicate's using directive makes
60+
# later CLucene sse_util references ambiguous.
61+
${CMAKE_CURRENT_SOURCE_DIR}/index/index_disk_usage.cpp
5962
# Keep the spill helpers independent: the PRX encoder requests ZSTD's static
6063
# workspace API, and inserting these files into the existing unity batches
6164
# exposes an unrelated LIMIT_REACH enum collision in olap_scan_common.cpp.

0 commit comments

Comments
 (0)