Skip to content

Manually installation of clickhouse keeper without operator create issue - Not found address of host: clickhouse-keeper-0. (DNS_ERROR) #1765

@kushal1794

Description

@kushal1794

Hi Team,

While Installing 3 node clickhouse keeper cluster manually without crd or clickhouse operator, I am getting below error.

clickhouse keeper version - clickhouse-keeper:25.4.2

Error - 2025.07.08 07:00:19.584912 [ 22 ] {} <Error> bool DB::(anonymous namespace)::isLocalhost(const std::string &): Code: 198. DB::NetException: Not found address of host: clickhouse-keeper-0. (DNS_ERROR), Stack trace (when copying this message, always include the lines below):

Full logs


+ source /conf/env.sh
+++ hostname -d
hostname: clickhouse-keeper-0: Address not available
++ export DOMAIN=
++ DOMAIN=
++ export CLIENT_HOST=clickhouse-keeper
++ CLIENT_HOST=clickhouse-keeper
++ export CLIENT_PORT=2181
++ CLIENT_PORT=2181
++ export RAFT_PORT=9444
++ RAFT_PORT=9444
+ source /conf/keeperFunctions.sh
++ set -ex
++ hostname -s
+ HOST=clickhouse-keeper-0
+ [[ clickhouse-keeper-0 =~ (.*)-([0-9]+)$ ]]
+ NAME=clickhouse-keeper
+ ORD=0
+ export MY_ID=1
+ MY_ID=1
+ set +e
+ getent hosts
+ [[ 0 -eq 0 ]]
+ ACTIVE_ENSEMBLE=true
+ set -e
+ mkdir -p /tmp/clickhouse-keeper/config.d/
+ [[ true == \t\r\u\e ]]
++ clickhouse-keeper-client --history-file=/dev/null -h clickhouse-keeper -p 2181 -q 'get '\''/keeper/config'\'''
Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [240b:c0e0:104:544d:b464:2:0:c2ab]:2181
Poco::Exception. Code: 1000, e.code() = 113, Net Exception: No route to host (version 25.4.2.31 (official build)), [240b:c0e0:104:544d:b464:2:0:c2ab]:2181
Poco::Exception. Code: 1000, e.code() = 113, Net Exception: No route to host (version 25.4.2.31 (official build)), [240b:c0e0:104:544d:b464:2:0:c2ab]:2181
Poco::Exception. Code: 1000, e.code() = 113, Net Exception: No route to host (version 25.4.2.31 (official build)), [240b:c0e0:104:544d:b464:2:0:c2ab]:2181

++ true
+ CURRENT_KEEPER_CONFIG=
+ echo '<yandex><keeper_server>'
+ echo '<server_id>1</server_id>'
+ echo '<raft_configuration>'
++ echo ''
++ grep -c clickhouse-keeper-0.
+ [[ 0 == 0 ]]
+ echo '<server><id>1</id><hostname>clickhouse-keeper-0.</hostname><port>9444</port><priority>1</priority><start_as_follower>true</start_as_follower></server>'
+ IFS=
+ read -r line
++ echo ''
++ ++ cut -d cut . -d -f = 2
-f 1
+ id=
+ [[ '' != '' ]]
+ IFS=
+ read -r line
+ echo '</raft_configuration>'
+ echo '</keeper_server></yandex>'
+ cat /tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml
<yandex><keeper_server>
<server_id>1</server_id>
<raft_configuration>
<server><id>1</id><hostname>clickhouse-keeper-0.</hostname><port>9444</port><priority>1</priority><start_as_follower>true</start_as_follower></server>
</raft_configuration>
</keeper_server></yandex>
+ rm -rfv /var/lib/clickhouse-keeper/terminated
+ clickhouse-keeper --config-file=/etc/clickhouse-keeper/keeper_config.xml
Processing configuration file '/etc/clickhouse-keeper/keeper_config.xml'.
Including configuration file '/tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml'.
Couldn't save preprocessed config to /etc/clickhouse-keeper/keeper_config-preprocessed.xml: File is read-only: /etc/clickhouse-keeper/keeper_config-preprocessed.xml
2025.07.08 07:00:19.519681 [ 22 ] {} <Information> SentryWriter: Sending crash reports is disabled
2025.07.08 07:00:19.572026 [ 22 ] {} <Information> Application: Starting ClickHouse Keeper 25.4.2.31 (revision: 54498, git hash: e2ce00b6885efd45f04e2086d9968c1b65dae9ea, build id: 9A21D4161362469DB37DC3BBEED21DE83C963455), PID 22
2025.07.08 07:00:19.572078 [ 22 ] {} <Information> Application: starting up
2025.07.08 07:00:19.572104 [ 22 ] {} <Information> Application: OS Name = Linux, OS Version = 3.10.0-1160.71.1.el7.x86_64, OS Architecture = x86_64
2025.07.08 07:00:19.572264 [ 22 ] {} <Information> Jemalloc: Value for background_thread set to true (from true)
2025.07.08 07:00:19.573966 [ 22 ] {} <Information> Application: keeper_server.max_memory_usage_soft_limit is set to 3.60 GiB
2025.07.08 07:00:19.574141 [ 22 ] {} <Information> CgroupsReader: Will create cgroup reader from '/sys/fs/cgroup/memory' (cgroups version: v1)
2025.07.08 07:00:19.574179 [ 22 ] {} <Information> MemoryWorker: Starting background memory thread with period of 50ms, using Cgroups as source
2025.07.08 07:00:19.574334 [ 22 ] {} <Debug> Application: Initializing DateLUT.
2025.07.08 07:00:19.574362 [ 22 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'.
2025.07.08 07:00:19.578979 [ 22 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/thermal
2025.07.08 07:00:19.579031 [ 22 ] {} <Trace> AsynchronousMetrics: Scanning /sys/block
2025.07.08 07:00:19.579323 [ 22 ] {} <Trace> AsynchronousMetrics: Scanning /sys/devices/system/edac
2025.07.08 07:00:19.579515 [ 22 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/hwmon
2025.07.08 07:00:19.580377 [ 22 ] {} <Information> Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously
2025.07.08 07:00:19.580422 [ 22 ] {} <Debug> KeeperDispatcher: Initializing storage dispatcher
2025.07.08 07:00:19.580570 [ 22 ] {} <Information> KeeperContext: Keeper feature flag FILTERED_LIST: enabled
2025.07.08 07:00:19.580589 [ 22 ] {} <Information> KeeperContext: Keeper feature flag MULTI_READ: enabled
2025.07.08 07:00:19.580608 [ 22 ] {} <Information> KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: disabled
2025.07.08 07:00:19.580624 [ 22 ] {} <Information> KeeperContext: Keeper feature flag CREATE_IF_NOT_EXISTS: disabled
2025.07.08 07:00:19.580640 [ 22 ] {} <Information> KeeperContext: Keeper feature flag REMOVE_RECURSIVE: disabled
2025.07.08 07:00:19.582925 [ 22 ] {} <Trace> KeeperSnapshotManager: Reading from disk LocalSnapshotDisk
2025.07.08 07:00:19.583043 [ 22 ] {} <Trace> KeeperSnapshotManager: No snapshots were found on LocalSnapshotDisk
2025.07.08 07:00:19.584488 [ 22 ] {} <Warning> DNSResolver: Cannot resolve host (clickhouse-keeper-0.), error 0: Host not found.
2025.07.08 07:00:19.584912 [ 22 ] {} <Error> bool DB::(anonymous namespace)::isLocalhost(const std::string &): Code: 198. DB::NetException: Not found address of host: clickhouse-keeper-0. (DNS_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000b4bac5b
1. DB::NetException::NetException<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000b496f84
2. DB::(anonymous namespace)::hostByName(String const&) @ 0x000000000b498ac4
3. DB::DNSResolver::getResolvedIPAdressessWithFiltering(String const&) @ 0x000000000b496972
4. DB::KeeperStateManager::parseServersConfiguration(Poco::Util::AbstractConfiguration const&, bool, bool) const @ 0x000000000dce3156
5. DB::KeeperStateManager::KeeperStateManager(int, String const&, String const&, Poco::Util::AbstractConfiguration const&, std::shared_ptr<DB::KeeperContext>) @ 0x000000000dce69cb
6. DB::KeeperServer::KeeperServer(std::shared_ptr<DB::KeeperConfigurationAndSettings> const&, Poco::Util::AbstractConfiguration const&, ConcurrentBoundedQueue<DB::KeeperResponseForSession>&, ConcurrentBoundedQueue<DB::CreateSnapshotTask>&, std::shared_ptr<DB::KeeperContext>, DB::KeeperSnapshotManagerS3&, std::function<void (unsigned long, DB::KeeperRequestForSession const&)>) @ 0x000000000dc54ef0
7. DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::shared_ptr<DB::Macros const> const&) @ 0x000000000dc3caf9
8. DB::Context::initializeKeeperDispatcher(bool) const @ 0x000000000c747a72
9. DB::Keeper::main(std::vector<String, std::allocator<String>> const&) @ 0x0000000005e22ffa
10. Poco::Util::Application::run() @ 0x0000000011ecbee6
11. DB::Keeper::run() @ 0x0000000005e1fdd0
12. mainEntryClickHouseKeeper(int, char**) @ 0x0000000005e1e9a2
13. main @ 0x0000000005e1d5b3
14. ? @ 0x00007fafe7e5cd90
15. ? @ 0x00007fafe7e5ce40
16. _start @ 0x0000000004f1e02e
 (version 25.4.2.31 (official build))
2025.07.08 07:00:19.585287 [ 22 ] {} <Debug> KeeperDispatcher: Shutting down storage dispatcher
2025.07.08 07:00:19.585419 [ 22 ] {} <Debug> KeeperSnapshotManagerS3: Shutting down KeeperSnapshotManagerS3
2025.07.08 07:00:19.585465 [ 22 ] {} <Information> KeeperSnapshotManagerS3: KeeperSnapshotManagerS3 shut down
2025.07.08 07:00:19.585482 [ 22 ] {} <Debug> KeeperDispatcher: Dispatcher shut down
2025.07.08 07:00:19.585505 [ 22 ] {} <Debug> Context: Destructing remote fs threadpool reader
2025.07.08 07:00:19.585529 [ 22 ] {} <Debug> Context: Destructing local fs threadpool reader
2025.07.08 07:00:19.585673 [ 22 ] {} <Debug> Context: Destructing local fs threadpool reader
2025.07.08 07:00:19.585704 [ 22 ] {} <Trace> Context: Shutting down named sessions
2025.07.08 07:00:19.585738 [ 22 ] {} <Trace> Context: Shutting down database catalog
2025.07.08 07:00:19.585761 [ 22 ] {} <Trace> DatabaseCatalog: Shutting down system databases
2025.07.08 07:00:19.585796 [ 22 ] {} <Trace> Context: Shutting down DDLWorker
2025.07.08 07:00:19.585813 [ 22 ] {} <Trace> Context: Shutting down caches
2025.07.08 07:00:19.585839 [ 22 ] {} <Trace> Context: Shutting down AccessControl
2025.07.08 07:00:19.586234 [ 22 ] {} <Information> Application: Waiting for background threads
2025.07.08 07:00:19.586623 [ 22 ] {} <Information> Application: Background threads finished in 0 ms
2025.07.08 07:00:19.586875 [ 22 ] {} <Error> Application: Code: 568. DB::Exception: At least one of servers should be able to start as leader (without <start_as_follower>). (RAFT_ERROR), Stack trace (when copying this message, always include the lines below):

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions