High amount of ZooKeeper requests/CPU Load after upgrade to 22.11.1 from 22.3 #43647
Description
We are using clickhouse-operator for our clickhouse cluster. We have 5 shards, 2 replicas each. Also we have one raw table and 5 MV (TO) with aggregation. Each table has no more than 10 columns. After upgrade from clickhouse version 22.3 to 22.11, we see huge difference in Zookeeper CPU consumption, and also amount of ZooKeeper requests. Even if I stop data ingestion, I see significant > 50% cpu usage on Zookeeper Nodes.
I tried to downgrade from 22.11 to 22.10.3.37 but with no luck, zookeeper cpu consumption and amount of requests was the same.
SELECT *
FROM system.events
WHERE event ILIKE '%zoo%'
Query id: 2059d338-c985-45f7-a497-b41516fed7ae
┌─event─────────────────────┬─────────value─┬─description───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ZooKeeperInit │ 1 │ Number of times connection with ZooKeeper has been established. │
│ ZooKeeperTransactions │ 62272213 │ Number of ZooKeeper operations, which include both read and write operations as well as multi-transactions. │
│ ZooKeeperList │ 59566267 │ Number of 'list' (getChildren) requests to ZooKeeper. │
│ ZooKeeperCreate │ 47371 │ Number of 'create' requests to ZooKeeper. │
│ ZooKeeperRemove │ 163665 │ Number of 'remove' requests to ZooKeeper. │
│ ZooKeeperExists │ 566333 │ Number of 'exists' requests to ZooKeeper. │
│ ZooKeeperGet │ 1333409 │ Number of 'get' requests to ZooKeeper. │
│ ZooKeeperMulti │ 595168 │ Number of 'multi' requests to ZooKeeper (compound transactions). │
│ ZooKeeperWatchResponse │ 141901 │ Number of times watch notification has been received from ZooKeeper. │
│ ZooKeeperWaitMicroseconds │ 2370348154687 │ Number of microseconds spent waiting for responses from ZooKeeper after creating a request, summed across all the requesting threads. │
│ ZooKeeperBytesSent │ 7371458424 │ Number of bytes send over network while communicating with ZooKeeper. │
│ ZooKeeperBytesReceived │ 9085809562 │ Number of bytes received over network while communicating with ZooKeeper. │
└───────────────────────────┴───────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
It looks like we start to send a lot of ZooKeeperList requests, is it normal behaviour?