You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 7219c67
Browse filesBrowse the repository at this point in the historyBrowse files
[feat](cloud) Support Cloud partition inverted format rollout (apache#66475)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Cloud tables previously had one table-level
inverted_index_storage_format, so an online V2-to-V3 migration required
all partitions to move together. This prevents a gradual rollout and
cannot retain the format selected when an existing partition was created
while new partitions use a new default.
This PR adds the Cloud-only table property
partition.inverted_index_storage_format. It is the default for
partitions created after the property is set. The property supports V2
and V3; V1 is rejected. Existing partitions and their tablet metadata
are not rewritten by a property update.
- Persist the resolved base-index inverted-index storage format in
PartitionInfo for every Cloud partition. The value is durable in journal
replay and is transferred through partition creation, recycle, recovery,
and replacement metadata.
- New explicit, automatic, and dynamic partitions use the current
partition format default. Existing partitions retain their recorded
format, allowing V2 and V3 to coexist in one table.
- Tablet and rowset metadata persist the physical format independently
of shared schema metadata so that mixed-format partitions remain
distinguishable throughout a rolling upgrade.
- SHOW PARTITIONS, SHOW PARTITION, SHOW PARTITION ID, and the partitions
TVF expose InvertedIndexStorageFormat from PartitionInfo rather than
reading tablet metadata from MetaService for this value.
Basic usage, with
`enable_partition_inverted_index_storage_format_rollout` enabled:
```sql
CREATE TABLE partition_format_demo (
k DATE NOT NULL,
v STRING,
INDEX idx_v (v) USING INVERTED
) ENGINE=OLAP
DUPLICATE KEY(k)
PARTITION BY RANGE(k) (
PARTITION p_v2 VALUES LESS THAN ("2024-01-01")
)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
"replication_num" = "1",
"inverted_index_storage_format" = "V2"
);
ALTER TABLE partition_format_demo
SET ("partition.inverted_index_storage_format" = "V3");
ALTER TABLE partition_format_demo
ADD PARTITION p_v3 VALUES [("2024-01-01"), ("2025-01-01"));
SHOW PARTITIONS FROM partition_format_demo;
```
Expected result:
> the existing p_v2 partition remains V2, while the newly created p_v3
partition is V3. The InvertedIndexStorageFormat column returned by SHOW
PARTITIONS therefore contains V2 for p_v2 and V3 for p_v3. Changing the
default again affects only partitions created afterwards.
| Coverage area | Implementation coverage | Test coverage |
| --- | --- | --- |
| Property resolution and partition metadata |
partition.inverted_index_storage_format takes precedence, the legacy
table property remains the fallback, and the resolved format is
persisted per Cloud partition. | FE unit coverage for property analysis,
table creation, table properties, and partition persistence. |
| Partition creation paths | Explicit range/list partitions, auto
partitions, and dynamic partitions use the current resolved
partition-format default. | Cloud rollout regression coverage. |
| Mixed-format storage compatibility | Coexisting V2/V3 partitions
retain their own tablet and rowset metadata format. | BE rowset/tablet
metadata unit coverage and Cloud rowset-schema regression coverage. |
| Recycle, recovery, and replacement paths | The persisted partition
format survives recycle/recover, truncate, and insert-overwrite metadata
paths. | FE recycle/recover unit coverage and Cloud recycle/truncate
regression coverage. |
| Metadata observability | Partition format is served from FE
PartitionInfo, avoiding per-partition tablet-meta reads for SHOW and TVF
output. | Cloud format regression coverage and partitions TVF coverage.
|
### Release note
Cloud OLAP tables can roll out inverted-index file storage formats by
partition. New partitions can use V3 while existing V2 partitions remain
online and queryable.
0 commit comments