Skip to content

Commit 7219c67

Browse files
authored
[feat](cloud) Support Cloud partition inverted format rollout (apache#66475)
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Cloud tables previously had one table-level inverted_index_storage_format, so an online V2-to-V3 migration required all partitions to move together. This prevents a gradual rollout and cannot retain the format selected when an existing partition was created while new partitions use a new default. This PR adds the Cloud-only table property partition.inverted_index_storage_format. It is the default for partitions created after the property is set. The property supports V2 and V3; V1 is rejected. Existing partitions and their tablet metadata are not rewritten by a property update. - Persist the resolved base-index inverted-index storage format in PartitionInfo for every Cloud partition. The value is durable in journal replay and is transferred through partition creation, recycle, recovery, and replacement metadata. - New explicit, automatic, and dynamic partitions use the current partition format default. Existing partitions retain their recorded format, allowing V2 and V3 to coexist in one table. - Tablet and rowset metadata persist the physical format independently of shared schema metadata so that mixed-format partitions remain distinguishable throughout a rolling upgrade. - SHOW PARTITIONS, SHOW PARTITION, SHOW PARTITION ID, and the partitions TVF expose InvertedIndexStorageFormat from PartitionInfo rather than reading tablet metadata from MetaService for this value. Basic usage, with `enable_partition_inverted_index_storage_format_rollout` enabled: ```sql CREATE TABLE partition_format_demo ( k DATE NOT NULL, v STRING, INDEX idx_v (v) USING INVERTED ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_v2 VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE partition_format_demo SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE partition_format_demo ADD PARTITION p_v3 VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM partition_format_demo; ``` Expected result: > the existing p_v2 partition remains V2, while the newly created p_v3 partition is V3. The InvertedIndexStorageFormat column returned by SHOW PARTITIONS therefore contains V2 for p_v2 and V3 for p_v3. Changing the default again affects only partitions created afterwards. | Coverage area | Implementation coverage | Test coverage | | --- | --- | --- | | Property resolution and partition metadata | partition.inverted_index_storage_format takes precedence, the legacy table property remains the fallback, and the resolved format is persisted per Cloud partition. | FE unit coverage for property analysis, table creation, table properties, and partition persistence. | | Partition creation paths | Explicit range/list partitions, auto partitions, and dynamic partitions use the current resolved partition-format default. | Cloud rollout regression coverage. | | Mixed-format storage compatibility | Coexisting V2/V3 partitions retain their own tablet and rowset metadata format. | BE rowset/tablet metadata unit coverage and Cloud rowset-schema regression coverage. | | Recycle, recovery, and replacement paths | The persisted partition format survives recycle/recover, truncate, and insert-overwrite metadata paths. | FE recycle/recover unit coverage and Cloud recycle/truncate regression coverage. | | Metadata observability | Partition format is served from FE PartitionInfo, avoiding per-partition tablet-meta reads for SHOW and TVF output. | Cloud format regression coverage and partitions TVF coverage. | ### Release note Cloud OLAP tables can roll out inverted-index file storage formats by partition. New partitions can use V3 while existing V2 partitions remain online and queryable.
1 parent a10c799 commit 7219c67

51 files changed

Lines changed: 2703 additions & 33 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/cloud/cloud_rowset_writer.cpp‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,10 @@ Status CloudRowsetWriter::init(const RowsetWriterContext& rowset_writer_context)
7878
_rowset_meta->set_newest_write_timestamp(_context.newest_write_timestamp);
7979
}
8080
_rowset_meta->set_tablet_schema(_context.tablet_schema);
81+
if (_context.persist_inverted_index_storage_format &&
82+
_context.inverted_index_storage_format.has_value()) {
83+
_rowset_meta->set_inverted_index_storage_format(*_context.inverted_index_storage_format);
84+
}
8185
_rowset_meta->set_job_id(_context.job_id);
8286
if (_context.write_binlog_opt().enable) {
8387
_rowset_meta->mark_row_binlog();

‎be/src/cloud/cloud_tablet.cpp‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -849,6 +849,9 @@ Result<std::unique_ptr<RowsetWriter>> CloudTablet::create_rowset_writer(
849849
context.write_binlog_opt().set_need_before(
850850
tablet_meta()->binlog_config().need_historical_value());
851851
}
852+
context.inverted_index_storage_format = tablet_meta()->inverted_index_storage_format();
853+
context.persist_inverted_index_storage_format =
854+
tablet_meta()->has_inverted_index_storage_format();
852855
return RowsetFactory::create_rowset_writer(_engine, context, vertical);
853856
}
854857

@@ -898,6 +901,9 @@ Result<std::unique_ptr<RowsetWriter>> CloudTablet::create_transient_rowset_write
898901
context.enable_unique_key_merge_on_write = enable_unique_key_merge_on_write();
899902
context.txn_expiration = txn_expiration;
900903
context.encrypt_algorithm = tablet_meta()->encryption_algorithm();
904+
context.inverted_index_storage_format = tablet_meta()->inverted_index_storage_format();
905+
context.persist_inverted_index_storage_format =
906+
tablet_meta()->has_inverted_index_storage_format();
901907
// TODO(liaoxin) enable packed file for transient rowset
902908
context.allow_packed_file = false;
903909

‎be/src/service/http/action/show_nested_index_file_action.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
#include <exception>
2323
#include <string>
2424

25+
#include "cloud/cloud_tablet.h"
2526
#include "common/status.h"
2627
#include "service/http/action/action_constants.h"
2728
#include "service/http/http_channel.h"
@@ -53,6 +54,10 @@ Status ShowNestedIndexFileAction::_handle_show_nested_index_file(HttpRequest* re
5354
}
5455

5556
auto tablet = DORIS_TRY(ExecEnv::get_tablet(tablet_id));
57+
if (auto cloud_tablet = std::dynamic_pointer_cast<CloudTablet>(tablet)) {
58+
// The debug endpoint must inspect all visible cloud rowsets, not only the local cache.
59+
RETURN_IF_ERROR(cloud_tablet->sync_rowsets());
60+
}
5661
RETURN_IF_ERROR(tablet->show_nested_index_file(json_meta));
5762
return Status::OK();
5863
}

‎be/src/storage/rowset/beta_rowset_writer.cpp‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -395,6 +395,10 @@ Status BaseBetaRowsetWriter::init(const RowsetWriterContext& rowset_writer_conte
395395
}
396396
_rowset_meta->set_tablet_uid(_context.tablet_uid);
397397
_rowset_meta->set_tablet_schema(_context.tablet_schema);
398+
if (_context.persist_inverted_index_storage_format &&
399+
_context.inverted_index_storage_format.has_value()) {
400+
_rowset_meta->set_inverted_index_storage_format(*_context.inverted_index_storage_format);
401+
}
398402
_rowset_meta->set_compaction_level(_context.compaction_level);
399403
if (_context.write_binlog_opt().enable) {
400404
_rowset_meta->mark_row_binlog();

‎be/src/storage/rowset/rowset_meta.cpp‎

Lines changed: 48 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,19 @@ bool RowsetMeta::init(const RowsetMeta* rowset_meta) {
7272
}
7373

7474
bool RowsetMeta::init_from_pb(const RowsetMetaPB& rowset_meta_pb) {
75+
if (rowset_meta_pb.has_inverted_index_storage_format()) {
76+
_rowset_meta_pb.set_inverted_index_storage_format(
77+
rowset_meta_pb.inverted_index_storage_format());
78+
} else {
79+
_rowset_meta_pb.clear_inverted_index_storage_format();
80+
}
7581
if (rowset_meta_pb.has_tablet_schema()) {
76-
set_tablet_schema(rowset_meta_pb.tablet_schema());
82+
TabletSchemaPB schema_pb = rowset_meta_pb.tablet_schema();
83+
if (rowset_meta_pb.has_inverted_index_storage_format()) {
84+
schema_pb.set_inverted_index_storage_format(
85+
rowset_meta_pb.inverted_index_storage_format());
86+
}
87+
set_tablet_schema(schema_pb);
7788
}
7889
// Release ownership of TabletSchemaPB from `rowset_meta_pb` and then set it back to `rowset_meta_pb`,
7990
// this won't break const semantics of `rowset_meta_pb`, because `rowset_meta_pb` is not changed
@@ -211,6 +222,10 @@ void RowsetMeta::to_rowset_pb(RowsetMetaPB* rs_meta_pb, bool skip_schema) const
211222
if (!skip_schema) {
212223
// For cloud, separate tablet schema from rowset meta to reduce persistent size.
213224
_schema->to_schema_pb(rs_meta_pb->mutable_tablet_schema());
225+
if (rs_meta_pb->has_inverted_index_storage_format()) {
226+
rs_meta_pb->mutable_tablet_schema()->set_inverted_index_storage_format(
227+
rs_meta_pb->inverted_index_storage_format());
228+
}
214229
}
215230
}
216231
rs_meta_pb->set_has_variant_type_in_schema(has_variant_type_in_schema());
@@ -223,6 +238,12 @@ RowsetMetaPB RowsetMeta::get_rowset_pb(bool skip_schema) const {
223238
}
224239

225240
void RowsetMeta::set_tablet_schema(const TabletSchemaSPtr& tablet_schema) {
241+
if (_rowset_meta_pb.has_inverted_index_storage_format()) {
242+
TabletSchemaPB schema_pb;
243+
tablet_schema->to_schema_pb(&schema_pb);
244+
set_tablet_schema(schema_pb);
245+
return;
246+
}
226247
if (_handle) {
227248
TabletSchemaCache::instance()->release(_handle);
228249
}
@@ -232,22 +253,42 @@ void RowsetMeta::set_tablet_schema(const TabletSchemaSPtr& tablet_schema) {
232253
}
233254

234255
void RowsetMeta::set_tablet_schema(const TabletSchemaPB& tablet_schema) {
256+
TabletSchemaPB resolved_schema = tablet_schema;
257+
if (_rowset_meta_pb.has_inverted_index_storage_format()) {
258+
resolved_schema.set_inverted_index_storage_format(
259+
_rowset_meta_pb.inverted_index_storage_format());
260+
}
235261
if (_handle) {
236262
TabletSchemaCache::instance()->release(_handle);
237263
}
238264
auto pair = TabletSchemaCache::instance()->insert(
239-
TabletSchema::deterministic_string_serialize(tablet_schema));
265+
TabletSchema::deterministic_string_serialize(resolved_schema));
240266
_handle = pair.first;
241267
_schema = pair.second;
242268
}
243269

270+
void RowsetMeta::set_inverted_index_storage_format(InvertedIndexStorageFormatPB format) {
271+
_rowset_meta_pb.set_inverted_index_storage_format(format);
272+
if (_schema && _schema->get_inverted_index_storage_format() != format) {
273+
TabletSchemaPB schema_pb;
274+
_schema->to_schema_pb(&schema_pb);
275+
schema_pb.set_inverted_index_storage_format(format);
276+
set_tablet_schema(schema_pb);
277+
}
278+
}
279+
244280
bool RowsetMeta::_deserialize_from_pb(std::string_view value) {
245281
if (!_rowset_meta_pb.ParseFromArray(value.data(), cast_set<int32_t>(value.size()))) {
246282
_rowset_meta_pb.Clear();
247283
return false;
248284
}
249285
if (_rowset_meta_pb.has_tablet_schema()) {
250-
set_tablet_schema(_rowset_meta_pb.tablet_schema());
286+
TabletSchemaPB schema_pb = _rowset_meta_pb.tablet_schema();
287+
if (_rowset_meta_pb.has_inverted_index_storage_format()) {
288+
schema_pb.set_inverted_index_storage_format(
289+
_rowset_meta_pb.inverted_index_storage_format());
290+
}
291+
set_tablet_schema(schema_pb);
251292
_rowset_meta_pb.set_allocated_tablet_schema(nullptr);
252293
}
253294
return true;
@@ -260,6 +301,10 @@ bool RowsetMeta::_serialize_to_pb(std::string* value) {
260301
RowsetMetaPB rowset_meta_pb = _rowset_meta_pb;
261302
if (_schema) {
262303
_schema->to_schema_pb(rowset_meta_pb.mutable_tablet_schema());
304+
if (rowset_meta_pb.has_inverted_index_storage_format()) {
305+
rowset_meta_pb.mutable_tablet_schema()->set_inverted_index_storage_format(
306+
rowset_meta_pb.inverted_index_storage_format());
307+
}
263308
}
264309
return rowset_meta_pb.SerializeToString(value);
265310
}

‎be/src/storage/rowset/rowset_meta.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,16 @@ class RowsetMeta : public MetadataAdder<RowsetMeta> {
113113

114114
void set_index_id(int64_t index_id) { _rowset_meta_pb.set_index_id(index_id); }
115115

116+
bool has_inverted_index_storage_format() const {
117+
return _rowset_meta_pb.has_inverted_index_storage_format();
118+
}
119+
120+
InvertedIndexStorageFormatPB inverted_index_storage_format() const {
121+
return _rowset_meta_pb.inverted_index_storage_format();
122+
}
123+
124+
void set_inverted_index_storage_format(InvertedIndexStorageFormatPB format);
125+
116126
TabletUid tablet_uid() const { return _rowset_meta_pb.tablet_uid(); }
117127

118128
void set_tablet_uid(TabletUid tablet_uid) {

‎be/src/storage/rowset/rowset_writer_context.h‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,11 @@ struct RowsetWriterContext {
7373
RowsetTypePB rowset_type {BETA_ROWSET};
7474

7575
TabletSchemaSPtr tablet_schema;
76-
76+
// Immutable inverted-index file format inherited from the owner tablet.
77+
std::optional<InvertedIndexStorageFormatPB> inverted_index_storage_format;
78+
// Whether the owner tablet persists the format in its top-level metadata.
79+
// This is derived from TabletMeta and keeps rowset metadata consistent with it.
80+
bool persist_inverted_index_storage_format = false;
7781
// PREPARED/COMMITTED for pending rowset
7882
// VISIBLE for non-pending rowset
7983
RowsetStatePB rowset_state {PREPARED};

‎be/src/storage/tablet/tablet.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2058,6 +2058,7 @@ void Tablet::_init_context_common_fields(RowsetWriterContext& context) {
20582058
context.tablet_id = tablet_id();
20592059
context.partition_id = partition_id();
20602060
context.tablet_schema_hash = schema_hash();
2061+
context.inverted_index_storage_format = tablet_meta()->inverted_index_storage_format();
20612062
context.rowset_type = tablet_meta()->preferred_rowset_type();
20622063
// Alpha Rowset will be removed in the future, so that if the tablet's default rowset type is
20632064
// alpha rowset, then set the newly created rowset to storage engine's default rowset.

‎be/src/storage/tablet/tablet_meta.cpp‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -278,8 +278,8 @@ TabletMeta::TabletMeta(const TabletMeta& b)
278278
_time_series_compaction_empty_rowsets_threshold(
279279
b._time_series_compaction_empty_rowsets_threshold),
280280
_time_series_compaction_level_threshold(b._time_series_compaction_level_threshold),
281-
_vertical_compaction_num_columns_per_group(
282-
b._vertical_compaction_num_columns_per_group) {};
281+
_vertical_compaction_num_columns_per_group(b._vertical_compaction_num_columns_per_group),
282+
_inverted_index_storage_format(b._inverted_index_storage_format) {};
283283

284284
void TabletMeta::init_column_from_tcolumn(uint32_t unique_id, const TColumn& tcolumn,
285285
ColumnPB* column) {
@@ -799,6 +799,10 @@ void TabletMeta::init_from_pb(const TabletMetaPB& tablet_meta_pb) {
799799
_cumulative_layer_point = tablet_meta_pb.cumulative_layer_point();
800800
_tablet_uid = TabletUid(tablet_meta_pb.tablet_uid());
801801
_ttl_seconds = tablet_meta_pb.ttl_seconds();
802+
_inverted_index_storage_format.reset();
803+
if (tablet_meta_pb.has_inverted_index_storage_format()) {
804+
_inverted_index_storage_format = tablet_meta_pb.inverted_index_storage_format();
805+
}
802806
if (tablet_meta_pb.has_tablet_type()) {
803807
_tablet_type = tablet_meta_pb.tablet_type();
804808
} else {
@@ -829,7 +833,11 @@ void TabletMeta::init_from_pb(const TabletMetaPB& tablet_meta_pb) {
829833

830834
// init _schema
831835
TabletSchemaSPtr schema = std::make_shared<TabletSchema>();
832-
schema->init_from_pb(tablet_meta_pb.schema());
836+
TabletSchemaPB schema_pb = tablet_meta_pb.schema();
837+
if (_inverted_index_storage_format.has_value()) {
838+
schema_pb.set_inverted_index_storage_format(*_inverted_index_storage_format);
839+
}
840+
schema->init_from_pb(schema_pb);
833841
if (_handle) {
834842
TabletSchemaCache::instance()->release(_handle);
835843
}
@@ -955,6 +963,11 @@ void TabletMeta::to_meta_pb(TabletMetaPB* tablet_meta_pb, bool cloud_get_rowset_
955963
}
956964

957965
_schema->to_schema_pb(tablet_meta_pb->mutable_schema());
966+
if (_inverted_index_storage_format.has_value()) {
967+
tablet_meta_pb->set_inverted_index_storage_format(*_inverted_index_storage_format);
968+
tablet_meta_pb->mutable_schema()->set_inverted_index_storage_format(
969+
*_inverted_index_storage_format);
970+
}
958971

959972
tablet_meta_pb->set_in_restore_mode(in_restore_mode());
960973

‎be/src/storage/tablet/tablet_meta.h‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -334,6 +334,15 @@ class TabletMeta : public MetadataAdder<TabletMeta> {
334334

335335
EncryptionAlgorithmPB encryption_algorithm() const { return _encryption_algorithm; }
336336

337+
bool has_inverted_index_storage_format() const {
338+
return _inverted_index_storage_format.has_value();
339+
}
340+
341+
InvertedIndexStorageFormatPB inverted_index_storage_format() const {
342+
return _inverted_index_storage_format.value_or(
343+
_schema->get_inverted_index_storage_format());
344+
}
345+
337346
private:
338347
Status _save_meta(DataDir* data_dir);
339348
void _check_mow_rowset_cache_version_size(size_t rowset_cache_version_size);
@@ -403,6 +412,9 @@ class TabletMeta : public MetadataAdder<TabletMeta> {
403412
// Persisted storage format for this tablet (e.g. V2, V3). Used to derive
404413
// schema-level defaults such as external ColumnMeta usage.
405414
TStorageFormat::type _storage_format = TStorageFormat::V2;
415+
// The schema KV is shared by (index_id, schema_version). Keep the tablet's
416+
// immutable file format outside that shared identity.
417+
std::optional<InvertedIndexStorageFormatPB> _inverted_index_storage_format;
406418

407419
mutable std::shared_mutex _meta_lock;
408420
};

0 commit comments

Comments
 (0)