Skip to content

Commit 96fd1e9

Browse files
authored
[feature](inverted index) Add SNII inverted index storage format (apache#66052)
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Add **SNII**, a Doris-native inverted index storage format designed for object storage, as an opt-in alternative to the CLucene-based V1/V2/V3. On cold queries served directly from S3, SNII is about an order of magnitude faster than V3. Cold object-storage latency is dominated by the number of round trips, not by the bytes read. CLucene spreads a single term lookup across several index streams that sit far apart in the file, so one lookup becomes many small scattered reads. SNII keeps each segment's indexes in one container, packs the sections of an index contiguously, aligns them to file-cache block boundaries, and coalesces adjacent ranges into a single batched read that is served in parallel — a lookup that costs CLucene many requests costs SNII a few. Positions are decoded on demand rather than read in full. Enable it per table; the cluster default stays V3, so existing tables and queries are unaffected: ```sql PROPERTIES ("inverted_index_storage_format" = "SNII") ``` The format covers the full index lifecycle: writing and querying (exact, analyzed, phrase and sloppy phrase), BM25 scoring, BKD and ANN indexes, VARIANT sub-columns, the SEARCH DSL, native index compaction, `BUILD INDEX`, and local plus cloud index changes. Beyond the format itself, this PR also fixes shared inverted index behaviour: reader selection no longer aborts the BE process when a caller has no runtime column type, and match-analyzer resolution is made consistent (custom analyzers now work with the `none` parser). ### Release note Add SNII, a new opt-in inverted index storage format designed for object storage, giving roughly an order of magnitude faster cold queries against S3. Enable it per table with `"inverted_index_storage_format" = "SNII"`; the cluster default remains V3, so existing tables are unaffected. **Upgrade / rollback constraint:** once a table uses the SNII format, the cluster can no longer be rolled back to a version that does not support SNII. A pre-SNII backend reads the new storage-format enum as the field's default (V1) rather than rejecting it, so it would silently interpret SNII indexes as V1. Drop or convert such tables to V3 before rolling back. Upgrades must follow the standard order: upgrade BEs first, then FEs.
1 parent 07f9112 commit 96fd1e9

534 files changed

Lines changed: 130732 additions & 2636 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/cloud/cloud_index_change_compaction.cpp‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,12 @@ Status CloudIndexChangeCompaction::prepare_compact() {
8383
_input_rowsets_total_size += rs->total_disk_size();
8484
}
8585

86-
_enable_inverted_index_compaction = false;
86+
// Non-SNII compaction cannot reuse index files when the index schema changes. SNII validates
87+
// each destination logical index independently and rebuilds ineligible columns from raw data.
88+
_enable_inverted_index_compaction =
89+
_enable_inverted_index_compaction &&
90+
input_rowset->tablet_schema()->get_inverted_index_storage_format() ==
91+
InvertedIndexStorageFormatPB::SNII;
8792
LOG_INFO("[index_change]prepare CloudIndexChangeCompaction, tablet_id={}, range=[{}-{}]",
8893
_tablet->tablet_id(), _input_rowsets.front()->start_version(),
8994
_input_rowsets.back()->end_version())

‎be/src/cloud/cloud_internal_service.cpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1055,7 +1055,7 @@ void record_warmup_ed_fail_index(const std::string& job_id_str, int64_t idx_size
10551055
void record_warmup_ed_skipped_rowset_as_finished(RowsetMeta& rs_meta,
10561056
const std::string& job_id_str) {
10571057
auto schema_ptr = rs_meta.tablet_schema();
1058-
bool has_inverted_index = schema_ptr->has_inverted_index() || schema_ptr->has_ann_index();
1058+
bool has_inverted_index = schema_ptr->has_inverted_or_ann_index();
10591059
auto idx_version = schema_ptr->get_inverted_index_storage_format();
10601060
for (int64_t segment_id = 0; segment_id < rs_meta.num_segments(); segment_id++) {
10611061
record_warmup_ed_finish_segment(job_id_str, rs_meta.segment_file_size(segment_id));
@@ -1359,7 +1359,7 @@ void CloudInternalServiceImpl::warm_up_rowset(google::protobuf::RpcController* c
13591359
auto schema_ptr = rs_meta.tablet_schema();
13601360
auto idx_version = schema_ptr->get_inverted_index_storage_format();
13611361

1362-
if (schema_ptr->has_inverted_index() || schema_ptr->has_ann_index()) {
1362+
if (schema_ptr->has_inverted_or_ann_index()) {
13631363
if (idx_version == InvertedIndexStorageFormatPB::V1) {
13641364
auto&& inverted_index_info = rs_meta.inverted_index_file_info(segment_id);
13651365
std::unordered_map<int64_t, int64_t> index_size_map;

‎be/src/cloud/cloud_rowset_writer.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ Status CloudRowsetWriter::build(RowsetSharedPtr& rowset) {
148148
} else {
149149
_rowset_meta->add_segments_file_size(seg_file_size.value());
150150
}
151-
if (_context.tablet_schema->has_inverted_index() || _context.tablet_schema->has_ann_index()) {
151+
if (_context.tablet_schema->has_inverted_or_ann_index()) {
152152
if (auto idx_files_info = _idx_files.inverted_index_file_info(_segment_start_id);
153153
!idx_files_info.has_value()) [[unlikely]] {
154154
LOG(ERROR) << "expected inverted index files info, but none presents: "

‎be/src/cloud/cloud_snapshot_mgr.cpp‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -250,8 +250,7 @@ Status CloudSnapshotMgr::_create_rowset_meta(
250250
file_mapping[src_index_file] = dst_index_file;
251251
}
252252
} else {
253-
if (context.tablet_schema->has_inverted_index() ||
254-
context.tablet_schema->has_ann_index()) {
253+
if (context.tablet_schema->has_inverted_or_ann_index()) {
255254
std::string src_index_file = InvertedIndexDescriptor::get_index_file_path_v2(
256255
InvertedIndexDescriptor::get_index_file_path_prefix(src_segment_file));
257256
std::string dst_index_file = InvertedIndexDescriptor::get_index_file_path_v2(

‎be/src/cloud/cloud_tablet.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1981,7 +1981,7 @@ void CloudTablet::_add_rowsets_directly(std::vector<RowsetSharedPtr>& rowsets,
19811981
expiration_time);
19821982
}
19831983
} else {
1984-
if (schema_ptr->has_inverted_index() || schema_ptr->has_ann_index()) {
1984+
if (schema_ptr->has_inverted_or_ann_index()) {
19851985
auto&& inverted_index_info = rowset_meta->inverted_index_file_info(seg_id);
19861986
int64_t idx_size = 0;
19871987
if (inverted_index_info.has_index_size()) {

‎be/src/cloud/cloud_warm_up_manager.cpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -375,7 +375,7 @@ void CloudWarmUpManager::handle_jobs() {
375375
tablet_id);
376376
}
377377
} else {
378-
if (schema_ptr->has_inverted_index() || schema_ptr->has_ann_index()) {
378+
if (schema_ptr->has_inverted_or_ann_index()) {
379379
auto idx_path =
380380
storage_resource.value()->remote_idx_v2_path(*rs, seg_id);
381381
file_size = idx_file_info.has_index_size() ? idx_file_info.index_size()
@@ -848,7 +848,7 @@ Status CloudWarmUpManager::_do_warm_up_rowset(RowsetMeta& rs_meta, int64_t table
848848
g_file_cache_event_driven_warm_up_requested_segment_size << seg_size;
849849
g_warmup_ed_requested_segment_size.put({job_id_str}, seg_size);
850850

851-
if (schema_ptr->has_inverted_index() || schema_ptr->has_ann_index()) {
851+
if (schema_ptr->has_inverted_or_ann_index()) {
852852
if (idx_version == InvertedIndexStorageFormatPB::V1) {
853853
auto&& inverted_index_info =
854854
rs_meta.inverted_index_file_info(cast_set<int>(segment_id));

‎be/src/common/config.cpp‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@
3030
#include <fstream> // IWYU pragma: keep
3131
#include <functional>
3232
#include <iostream>
33+
#include <limits>
3334
#include <map>
3435
#include <memory>
3536
#include <mutex>
@@ -1296,6 +1297,25 @@ DEFINE_Int32(inverted_index_query_cache_shards, "256");
12961297
// inverted index match bitmap cache size
12971298
DEFINE_String(inverted_index_query_cache_limit, "10%");
12981299

1300+
namespace {
1301+
1302+
bool valid_common_grams_cost_ratio(int32_t value) {
1303+
return value >= 0 && value <= 100;
1304+
}
1305+
1306+
bool valid_common_grams_verify_factor(int32_t value) {
1307+
return value >= 0;
1308+
}
1309+
1310+
} // namespace
1311+
1312+
DEFINE_mBool(enable_common_grams_query_plan, "false");
1313+
DEFINE_mBool(enable_common_grams_index_build, "true");
1314+
DEFINE_mInt32(common_grams_plan_cost_ratio_percent, "85");
1315+
DEFINE_Validator(common_grams_plan_cost_ratio_percent, valid_common_grams_cost_ratio);
1316+
DEFINE_mInt32(common_grams_position_verify_factor, "0");
1317+
DEFINE_Validator(common_grams_position_verify_factor, valid_common_grams_verify_factor);
1318+
12991319
// condition cache limit
13001320
DEFINE_Int16(condition_cache_limit, "512");
13011321

@@ -1309,6 +1329,69 @@ DEFINE_mDouble(inverted_index_ram_buffer_size, "512");
13091329
// -1 indicates not working.
13101330
// Normally we should not change this, it's useful for testing.
13111331
DEFINE_mInt32(inverted_index_max_buffered_docs, "-1");
1332+
// G16-c: whether plain positions-tier (non-scoring) SNII indexes lay out freq
1333+
// regions. Freq bytes serve ONLY BM25 scoring, which the Doris integration
1334+
// does not reach yet (scoring_query has no production caller), so the default
1335+
// drops them (textbench: -2.2 GB index). Scoring-config indexes always write
1336+
// freq regardless. Applies at segment build (write side only); existing
1337+
// segments keep whatever layout they were written with (self-describing).
1338+
DEFINE_mBool(snii_positions_index_write_freq, "false");
1339+
// G16-h: zstd levels for the SNII dict-block compression and the .prx window
1340+
// auto mode. Level 9 (vs the historical 3) shrinks the two largest compressed
1341+
// sections -- textbench: index -457 MB (0.918x -> 0.891x V3) -- for an import
1342+
// CPU cost inside the run-to-run variance band; zstd decode speed does not
1343+
// depend on the level, and warm/cold benches measured no query change.
1344+
// Write side only; segments self-describe their compression.
1345+
// Default 3 since the all-level-3 evaluation (2026-07-11, 4 corpora): vs
1346+
// level 9 the settled index grows only +0.6%..+6.3% (whole table
1347+
// +0.3%..+1.9%) while import index CPU drops 17-24% and full-compaction CPU
1348+
// 8-24%; settled cold-query latency is unchanged (interleaved A/B). The
1349+
// delta+varint-encoded payloads are high-entropy, so level 9's extra search
1350+
// buys almost no ratio. Raise only for size-critical deployments.
1351+
DEFINE_mInt32(snii_dict_block_zstd_level, "3");
1352+
DEFINE_mInt32(snii_prx_zstd_level, "3");
1353+
// Patch C prx tiering: zstd level for the prx region of DIRECT-LOAD segments
1354+
// only (stream/broker load, see IndexColumnWriter::set_direct_load). Inert at
1355+
// the defaults (both levels 3); it exists for size-critical deployments that
1356+
// RAISE snii_prx_zstd_level (e.g. 9) and still want cheap loads: compaction
1357+
// rewrites every segment at snii_prx_zstd_level, so SETTLED data (and the
1358+
// cold-query path over it) is unaffected by the load tier -- measured -290s
1359+
// (httplogs) / -204s (agentlogs) of import index CPU at 3 vs 9. Same clamp
1360+
// [3, 19]. Read at index flush like snii_prx_zstd_level (a mid-load change
1361+
// lands on in-flight segments); the direct-load BIT itself is captured once.
1362+
DEFINE_mInt32(snii_prx_zstd_level_direct_load, "3");
1363+
// G16-d: target SNII dict block size in bytes; 0 uses the format default
1364+
// (64 KiB). Larger blocks compress better under the per-block zstd (the dict
1365+
// is the dominant physical section on high-cardinality corpora) at the cost
1366+
// of a larger fetch+decompress unit per cold dict-block miss. Write side
1367+
// only; the block size is self-described by the on-disk directory.
1368+
DEFINE_mInt32(snii_target_dict_block_bytes, "0");
1369+
// SNII's index-build share of the process memory limit, as a percent -- the
1370+
// index-build analogue of load_process_max_memory_limit_percent. Once live SNII
1371+
// index-build memory crosses this share, the largest reclaimable posting arenas
1372+
// are asked to spill early. Derived from the process limit rather than an
1373+
// absolute number so it scales with the BE. Only the RECLAIMABLE population
1374+
// counts against it: index-merge compaction charges the same observation
1375+
// tracker but registers no spillable writer, so its bytes are excluded from the
1376+
// comparison (its own hard reservation cap bounds them instead).
1377+
//
1378+
// 0 disables SNII's OWN share trigger; the process-level backstops (system
1379+
// available memory below its warning water mark, process usage above the soft
1380+
// limit) still apply. The share is deliberately well below those backstops so
1381+
// SNII sheds its own memory before the global valve -- the global valve trips
1382+
// late by design and would be a worse trigger than none. The derived share is
1383+
// floored at four times inverted_index_ram_buffer_size so a small BE is not
1384+
// permanently over it the moment two writers exist.
1385+
DEFINE_mInt32(snii_index_build_max_memory_limit_percent, "10");
1386+
// Minimum reclaimable posting-arena bytes before a G09 forced spill is honored
1387+
// (and before a writer is eligible as a spill victim): forced spills reclaim
1388+
// ONLY the arena, so smaller triggers cut tiny runs for near-zero relief.
1389+
// Default 64 MiB.
1390+
DEFINE_mInt64(snii_forced_spill_min_arena_bytes, "67108864");
1391+
// Max spill-run files one SNII writer accumulates before its runs are
1392+
// merge-compacted into one (bounds the k-way merge fan-in and its open fds;
1393+
// every run is held open for the whole merge). 0 = uncapped. Default 64.
1394+
DEFINE_mInt32(snii_spill_max_run_files_per_buffer, "64");
13121395
// dict path for chinese analyzer
13131396
DEFINE_String(inverted_index_dict_path, "${DORIS_HOME}/dict");
13141397
DEFINE_Int32(inverted_index_read_buffer_size, "4096");
@@ -2270,6 +2353,33 @@ bool init(const char* conf_file, bool fill_conf_map, bool must_exist, bool set_t
22702353
return Status::OK(); \
22712354
}
22722355

2356+
namespace {
2357+
2358+
// UPDATE_FIELD invokes registered validators before assigning the candidate value. Validate the two
2359+
// mutable planner coefficients explicitly so their startup and runtime constraints stay identical.
2360+
Status validate_common_grams_runtime_config(const std::string& field, const std::string& value) {
2361+
bool (*validator)(int32_t) = nullptr;
2362+
if (field == "common_grams_plan_cost_ratio_percent") {
2363+
validator = valid_common_grams_cost_ratio;
2364+
} else if (field == "common_grams_position_verify_factor") {
2365+
validator = valid_common_grams_verify_factor;
2366+
} else {
2367+
return Status::OK();
2368+
}
2369+
2370+
int32_t candidate = 0;
2371+
if (!convert(value, candidate)) {
2372+
return Status::OK();
2373+
}
2374+
if (!validator(candidate)) {
2375+
return Status::Error<ErrorCode::INVALID_ARGUMENT, false>("validate {}={} failed", field,
2376+
candidate);
2377+
}
2378+
return Status::OK();
2379+
}
2380+
2381+
} // namespace
2382+
22732383
// write config to be_custom.conf
22742384
// the caller need to make sure that the given config is valid
22752385
Status persist_config(const std::string& field, const std::string& value) {
@@ -2300,6 +2410,8 @@ Status set_config(const std::string& field, const std::string& value, bool need_
23002410
"'{}' is not support to modify", field);
23012411
}
23022412

2413+
RETURN_IF_ERROR(validate_common_grams_runtime_config(field, value));
2414+
23032415
UPDATE_FIELD(it->second, value, bool, need_persist);
23042416
UPDATE_FIELD(it->second, value, int16_t, need_persist);
23052417
UPDATE_FIELD(it->second, value, int32_t, need_persist);

‎be/src/common/config.h‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1363,6 +1363,15 @@ DECLARE_Int32(inverted_index_query_cache_shards);
13631363
// inverted index match bitmap cache size
13641364
DECLARE_String(inverted_index_query_cache_limit);
13651365

1366+
// Process-wide emergency switch for CommonGrams query plans.
1367+
DECLARE_mBool(enable_common_grams_query_plan);
1368+
// Build-only CommonGrams kill switch. Logical index writers snapshot it at construction; changing
1369+
// it affects only writers created after the transition and never changes query/cache semantics.
1370+
DECLARE_mBool(enable_common_grams_index_build);
1371+
// Release-calibrated query-planner coefficients. Both remain mutable for controlled recalibration.
1372+
DECLARE_mInt32(common_grams_plan_cost_ratio_percent);
1373+
DECLARE_mInt32(common_grams_position_verify_factor);
1374+
13661375
// condition cache limit
13671376
DECLARE_Int16(condition_cache_limit);
13681377

@@ -1374,6 +1383,68 @@ DECLARE_Int32(ann_index_result_cache_stale_sweep_time_sec);
13741383
// inverted index
13751384
DECLARE_mDouble(inverted_index_ram_buffer_size);
13761385
DECLARE_mInt32(inverted_index_max_buffered_docs);
1386+
// G16-c: whether plain positions-tier (non-scoring) SNII indexes lay out freq
1387+
// regions. Freq serves ONLY BM25 scoring (no production caller yet), so the
1388+
// default (false) drops the layout; scoring-config indexes always keep freq.
1389+
// Write-side only; segments are self-describing either way.
1390+
DECLARE_mBool(snii_positions_index_write_freq);
1391+
// G16-h: zstd levels for SNII dict blocks / prx windows. Default 3 (the
1392+
// all-level-3 evaluation showed level 9 buys <=6.3% index size for 17-24%
1393+
// import CPU; see the DEFINEs in config.cpp).
1394+
DECLARE_mInt32(snii_dict_block_zstd_level);
1395+
DECLARE_mInt32(snii_prx_zstd_level);
1396+
// Patch C: prx zstd level for DIRECT-LOAD segments only (default 3, cheaper
1397+
// import); compaction rewrites at snii_prx_zstd_level so settled segments are
1398+
// unaffected. Full contract at the DEFINE in config.cpp.
1399+
DECLARE_mInt32(snii_prx_zstd_level_direct_load);
1400+
// G16-d: target SNII dict block size in bytes; 0 = format default (64 KiB).
1401+
// Bigger blocks -> better per-block zstd on the dict region, larger cold
1402+
// fetch+decompress unit per dict-block miss. Write side only.
1403+
DECLARE_mInt32(snii_target_dict_block_bytes);
1404+
// PROCESS-WIDE share for SNII index-build RAM, as a PERCENT of the process
1405+
// memory limit -- the index-build analogue of
1406+
// load_process_max_memory_limit_percent. The per-writer
1407+
// inverted_index_ram_buffer_size is a reclaimable-buffer spill threshold, not a
1408+
// hard cap on persistent vocabulary bytes: a concurrent load keeps (tablets x
1409+
// concurrency) writers alive at once, none of which may reach that threshold,
1410+
// while their SUM can still be large. Once live SNII index-build memory
1411+
// (ingestion plus index-merge compaction) crosses this share, the writers
1412+
// holding the largest reclaimable posting arenas are asked to spill early
1413+
// (async-safe advisory requests, honored on each writer's own thread; output
1414+
// stays byte-identical). Read at every decision, so a change takes effect
1415+
// immediately for writers that are already running.
1416+
//
1417+
// 0 disables SNII's own share trigger; the process-level backstops (system
1418+
// available memory below its warning water mark, process usage above the soft
1419+
// limit) still apply.
1420+
//
1421+
// FLOORED AGAINST inverted_index_ram_buffer_size: the share is never less than
1422+
// four writers' worth of the per-writer spill threshold. A smaller share would
1423+
// put a small BE permanently over it as soon as two writers exist -- unrelievable
1424+
// back-pressure rather than a limit -- because the per-writer threshold is what
1425+
// one writer may hold before it spills on its own.
1426+
DECLARE_mInt32(snii_index_build_max_memory_limit_percent);
1427+
// G09 forced-spill floor: minimum reclaimable posting-arena bytes a SNII
1428+
// writer must hold before a process-wide forced-spill request is honored, and
1429+
// before the global limiter selects it as a spill victim. A forced spill
1430+
// reclaims ONLY the posting arena -- the persistent vocab / pair-map
1431+
// structures survive it -- so honoring below a real floor degenerates into a
1432+
// storm of tiny runs whenever the memory over the share is dominated by
1433+
// persistent bytes (each run then costs a file, a sort and a merge-fd for
1434+
// near-zero memory relief). THIS FLOOR, not any judgement about whether the
1435+
// overage is reachable, is what bounds forced spilling: it caps the cost at one
1436+
// >= floor-sized run per floor of arena growth per writer. Forced spilling
1437+
// therefore reclaims SPILLABLE memory only, never persistent memory.
1438+
// Default 64 MiB.
1439+
DECLARE_mInt64(snii_forced_spill_min_arena_bytes);
1440+
// G09 run-file cap: maximum spill-run files one SNII writer may accumulate;
1441+
// on the next spill past the cap, the existing runs are merge-compacted into
1442+
// a single run first (term stream unchanged). Bounds the final k-way merge's
1443+
// fan-in and, decisively, its simultaneously-open file descriptors -- every
1444+
// run of a buffer is reopened and held open for the whole merge, so unbounded
1445+
// run counts across ~100 concurrent writers can exhaust the BE nofile rlimit
1446+
// ("Too many open files" at run reopen). 0 disables the cap. Default 64.
1447+
DECLARE_mInt32(snii_spill_max_run_files_per_buffer);
13771448
// dict path for chinese analyzer
13781449
DECLARE_String(inverted_index_dict_path);
13791450
DECLARE_Int32(inverted_index_read_buffer_size);

‎be/src/common/metrics/doris_metrics.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,7 @@ class DorisMetrics {
232232
UIntGauge* load_mem_consumption = nullptr;
233233
UIntGauge* load_channel_mem_consumption = nullptr;
234234
UIntGauge* memtable_memory_limiter_mem_consumption = nullptr;
235+
UIntGauge* snii_index_build_mem_consumption = nullptr;
235236
UIntGauge* query_mem_consumption = nullptr;
236237
UIntGauge* schema_change_mem_consumption = nullptr;
237238
UIntGauge* storage_migration_mem_consumption = nullptr;

‎be/src/common/status.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -299,6 +299,7 @@ namespace ErrorCode {
299299
E(INVERTED_INDEX_COMPACTION_ERROR, -6010, false); \
300300
E(INVERTED_INDEX_ANALYZER_ERROR, -6011, false); \
301301
E(INVERTED_INDEX_FILE_CORRUPTED, -6012, false); \
302+
E(INVERTED_INDEX_SNII_NOT_FOUND, -6013, false); \
302303
E(KEY_NOT_FOUND, -7000, false); \
303304
E(KEY_ALREADY_EXISTS, -7001, false); \
304305
E(ENTRY_NOT_FOUND, -7002, false); \

0 commit comments

Comments
 (0)