Kafka storage's shutdown hangs if no messages are ever fetched #5615
Description
Using ClickHouse 19.8.3.8 on Debian 9 on WSL, installed from Yandex repository. The problem also appeared on version 19.7.5.27.
Queries were executed in ClickHouse CLI.
Test table is created with this query:
CREATE TABLE kafka_test2 (
timestamp UInt64,
assetId String,
dimension1 Nullable(String),
dimension2 Nullable(String),
dimension3 Nullable(String),
dimension4 Nullable(String),
dimension5 Nullable(String),
metric1 Nullable(Float64)
) ENGINE = Kafka('localhost:9092', 'test_metrics', 'test-ch-group-100', 'JSONEachRow');
Table name kafka_test2 is unique and was never used, as well as consumer group test-ch-group-100.
Trace log:
2019.06.14 13:20:39.192913 [ 49 ] {3f3f4d3b-105c-4a3b-886b-b9c4a0cb4c53} <Debug> executeQuery: (from 127.0.0.1:34618) CREATE TABLE kafka_test2 ( timestamp UInt64, assetId String, dimension1 Nullable(String),
dimension2 Nullable(String), dimension3 Nullable(String), dimension4 Nullable(String), dimension5 Nullable(String), metric1 Nullable(Float64) ) ENGINE = Kafka('localhost:9092', 'test_metrics', 'test-ch-group-100', 'JSONEachRow');
2019.06.14 13:20:39.203243 [ 49 ] {3f3f4d3b-105c-4a3b-886b-b9c4a0cb4c53} <Trace> StorageKafka (kafka_test2): Setting brokers: localhost:9092
2019.06.14 13:20:39.203310 [ 49 ] {3f3f4d3b-105c-4a3b-886b-b9c4a0cb4c53} <Trace> StorageKafka (kafka_test2): Setting Group ID: test-ch-group-100 Client ID: clickhouse
2019.06.14 13:20:39.204507 [ 49 ] {3f3f4d3b-105c-4a3b-886b-b9c4a0cb4c53} <Debug> MemoryTracker: Peak memory usage (for query): 397.00 B.
2019.06.14 13:20:39.204674 [ 49 ] {3f3f4d3b-105c-4a3b-886b-b9c4a0cb4c53} <Information> TCPHandler: Processed in 0.012 sec.
Then I run DROP TABLE. Messages from CLI:
Darkrai.localdomain :) DROP TABLE kafka_test2;
DROP TABLE kafka_test2
Timeout exceeded while receiving data from server. Waited for 300 seconds, timeout is 300 seconds.
Cancelling query.
It hangs on cancelling.
In server log only one message:
2019.06.14 13:20:59.621685 [ 49 ] {ff0392c1-0ac8-4dff-94fe-f179e8853d6f} <Debug> executeQuery: (from 127.0.0.1:34618) DROP TABLE kafka_test2;
After stopping the server in linux console it hangs on:
v1rusw0rm@Darkrai:~$ sudo service clickhouse-server stop
Stop clickhouse-server service:
And server log:
2019.06.14 13:43:29.699305 [ 50 ] {} <Information> Application: Received termination signal (Terminated)
2019.06.14 13:43:29.699892 [ 1 ] {} <Debug> Application: Received termination signal.
2019.06.14 13:43:29.700008 [ 1 ] {} <Debug> Application: Waiting for current connections to close.
2019.06.14 13:43:30.936872 [ 1 ] {} <Debug> Application: Closed all listening sockets. Waiting for 2 outstanding connections.
2019.06.14 13:43:35.983936 [ 1 ] {} <Debug> Application: Closed connections. But 2 remains. Tip: To increase wait time add to config: <shutdown_wait_unfinished>60</shutdown_wait_unfinished>
2019.06.14 13:43:55.987254 [ 1 ] {} <Information> Application: Shutting down storages.
Server's error log is clean.