Repository navigation
Commit 660c4ef
authored
fix(mito2): fence writes before a graceful downgrade (#9716)
* test(mito2): cover graceful role changes racing with writes
A graceful role change flips the region state in a background task while
the worker keeps handling writes. Add bulk and row write tests that check a
write handled after the role change request is either rejected or covered
by the returned last_entry_id.
Add tests for DowngradingLeader while an ALTER keeps the region in
Altering, alone and in the downgrade sequence the datanode runs
(DowngradingLeader, Flush, Follower) with a migration candidate on its own
local WAL. Add a test that the downgrade sequence covers a write in flight
when a lease renewal sets Downgrading.
Signed-off-by: jeremyhi <[email protected]>
* fix(mito2): fence writes before a graceful downgrade
set_region_role_state_gracefully hands the role change to a background task
that waits for the manifest lock. The worker keeps handling writes until the
task flips the state, so a write could be acknowledged with an entry id above
the returned last_entry_id. Set Downgrading in the worker before spawning the
task for Follower and DowngradingLeader requests, as lease renewal already
does, so every write the worker handles afterwards is rejected.
DowngradingLeader also did nothing and returned success when the region was
in another leader state such as Altering. The region became writable again
when the operation finished, and the datanode then flushed and turned it
into a follower in one step. Return a retryable error instead, so metasrv
retries the downgrade.
Signed-off-by: jeremyhi <[email protected]>
* test(mito2): assert the retryable downgrade error during ALTER
Check that DowngradingLeader during an ALTER returns a retryable error, and
release the paused ALTER on that path. The local WAL test now retries the
downgrade after the ALTER, as metasrv does.
Signed-off-by: jeremyhi <[email protected]>
* test(mito2): write before the retried downgrade in the local WAL test
Rows written before an ALTER were already flushed, so the candidate check
did not depend on the downgrade's final flush. Write rows after the ALTER
and before the downgrade is retried, which only that flush can carry to the
candidate.
Signed-off-by: jeremyhi <[email protected]>
---------
Signed-off-by: jeremyhi <[email protected]>1 parent b3dc567 commit 660c4ef
5 files changed
Lines changed: 564 additions & 10 deletions
File tree
- src/mito2/src
- engine
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1882 | 1882 | | |
1883 | 1883 | | |
1884 | 1884 | | |
1885 | | - | |
| 1885 | + | |
| 1886 | + | |
| 1887 | + | |
| 1888 | + | |
1886 | 1889 | | |
1887 | 1890 | | |
1888 | 1891 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
82 | | - | |
| 82 | + | |
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
88 | | - | |
| 88 | + | |
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| |||
0 commit comments