In some cases killed mutation lead to unfinished alter #16953
Description
One of our users just killed mutations produced by ALTER MODIFY queries on 21 servers. After that new alters on three servers stuck with:
Cannot execute alter metadata with version: 8 because another alter 7 must be executed before
However, there is no corresponding alter №7 in the replication queue and no undone mutations:
select * from system.mutations where is_done = 0
However, in the ZooKeeper, we have the correct state of metadata_version:
Row 13:
───────
name: metadata_version
value: 7
czxid: 23078663291
mzxid: 29401964979
ctime: 2020-09-02 17:52:08
mtime: 2020-11-13 00:43:18
version: 7
cversion: 0
aversion: 0
ephemeralOwner: 0
dataLength: 1
numChildren: 0
pzxid: 23078663291
SYSTEM RESTART REPLICA dbname.table_name solves the issue, but it seems like KILL MUTATION in some cases can lead to unfinished alter.