Skip to content

Commit 660c4ef

Browse files
authored
fix(mito2): fence writes before a graceful downgrade (#9716)
* test(mito2): cover graceful role changes racing with writes A graceful role change flips the region state in a background task while the worker keeps handling writes. Add bulk and row write tests that check a write handled after the role change request is either rejected or covered by the returned last_entry_id. Add tests for DowngradingLeader while an ALTER keeps the region in Altering, alone and in the downgrade sequence the datanode runs (DowngradingLeader, Flush, Follower) with a migration candidate on its own local WAL. Add a test that the downgrade sequence covers a write in flight when a lease renewal sets Downgrading. Signed-off-by: jeremyhi <[email protected]> * fix(mito2): fence writes before a graceful downgrade set_region_role_state_gracefully hands the role change to a background task that waits for the manifest lock. The worker keeps handling writes until the task flips the state, so a write could be acknowledged with an entry id above the returned last_entry_id. Set Downgrading in the worker before spawning the task for Follower and DowngradingLeader requests, as lease renewal already does, so every write the worker handles afterwards is rejected. DowngradingLeader also did nothing and returned success when the region was in another leader state such as Altering. The region became writable again when the operation finished, and the datanode then flushed and turned it into a follower in one step. Return a retryable error instead, so metasrv retries the downgrade. Signed-off-by: jeremyhi <[email protected]> * test(mito2): assert the retryable downgrade error during ALTER Check that DowngradingLeader during an ALTER returns a retryable error, and release the paused ALTER on that path. The local WAL test now retries the downgrade after the ALTER, as metasrv does. Signed-off-by: jeremyhi <[email protected]> * test(mito2): write before the retried downgrade in the local WAL test Rows written before an ALTER were already flushed, so the candidate check did not depend on the downgrade's final flush. Write rows after the ALTER and before the downgrade is retried, which only that flush can carry to the candidate. Signed-off-by: jeremyhi <[email protected]> --------- Signed-off-by: jeremyhi <[email protected]>
1 parent b3dc567 commit 660c4ef

5 files changed

Lines changed: 564 additions & 10 deletions

File tree

‎src/mito2/src/engine/basic_test.rs‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1882,7 +1882,10 @@ async fn restart_without_region_close(
18821882

18831883
/// Builds a bulk insert request for a region created by [CreateRequestBuilder]
18841884
/// with its default columns `(tag_0, field_0, ts)`.
1885-
fn build_bulk_insert_request(region_id: RegionId, num_rows: usize) -> RegionBulkInsertsRequest {
1885+
pub(crate) fn build_bulk_insert_request(
1886+
region_id: RegionId,
1887+
num_rows: usize,
1888+
) -> RegionBulkInsertsRequest {
18861889
let schema = Arc::new(Schema::new(vec![
18871890
Field::new("tag_0", DataType::Utf8, true),
18881891
Field::new("field_0", DataType::Float64, true),

‎src/mito2/src/engine/object_store_wal_test.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,13 +79,13 @@ fn open_request(table_dir: String) -> RegionOpenRequest {
7979
}
8080
}
8181

82-
async fn new_engine(env: &mut TestEnv) -> (MitoEngine, Arc<ObjectStoreLogStore>) {
82+
pub(crate) async fn new_engine(env: &mut TestEnv) -> (MitoEngine, Arc<ObjectStoreLogStore>) {
8383
env.create_engine_with_object_store_wal(MitoConfig::default(), &wal_config())
8484
.await
8585
}
8686

8787
/// Creates a region with object store WAL options and returns its table dir and row schema.
88-
async fn create_region(
88+
pub(crate) async fn create_region(
8989
engine: &MitoEngine,
9090
region_id: RegionId,
9191
) -> (String, Vec<api::v1::ColumnSchema>) {

0 commit comments

Comments
 (0)