Description of the Issue (and unexpected/desired result)
Experiencing excessive clock drift on Windows systems when consul >= 0.9.3 is running and connected to the cluster.
We have a mixed cluster with all Linux servers, some Linux clients and some Windows clients. Only the Windows systems experience the drift, Linux clients do not. The clock drift begins as soon as consul is started with an initial drift of 100ms/sec. Anecdotally this maxes out at 180 seconds off at which point NTP may be trying to adjust it back (but not completely understood yet).
Some observations:
- Tested the latest consul version 1.06 and still experienced the issue
- Tested 0.7.0, 0.8.0, 0.90, 0.9.1 and 0.9.2 and none of these experienced the issue
Desired result is that the system clock stays stable relative to the NTP source.
Reproduction steps
I am downgrading my cluster, but I can reproduce this any time by running 0.9.3, so let me know if there is any issue reproducing.
Run consul >= v0.9.3 on a Windows system with either Win2003 or Win2008R2 (not yet tested on Win2012 or Win2016) attached to a domain.
W32Time running as the (S)NTP client with a domain controller as the NTP source (not yet tested under other NTP clients or sources).
Compare time to the source (we used w32tm):
w32tm /stripchart /computer:x.x.x.x /dataonly /samples:10
Tracking x.x.x.x [x.x.x.x:123].
Collecting 10 samples.
The current time is 2/28/2018 6:18:50 PM.
18:18:50, -113.1472053s
18:18:51, -112.7045555s
18:18:53, -112.2945822s
18:18:54, -111.8998771s
18:18:56, -111.5200829s
consul version for both Client and Server
Client (Linux): Consul v0.9.3
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
Client (Windows):Consul v0.9.3
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
Server: Consul v0.9.3
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
consul info for both Client and Server
Client:
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 6
build:
prerelease =
revision = 112c060
version = 0.9.3
consul:
known_servers = 7
server = false
runtime:
arch = amd64
cpu_count = 4
goroutines = 39
max_procs = 4
os = windows
version = go1.9
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 120
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 137796
members = 24
query_queue = 0
query_time = 1
Server:
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 112c060
version = 0.9.3
consul:
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.200.2.162:8300
server = true
raft:
applied_index = 11339377
commit_index = 11339377
fsm_pending = 0
last_contact = 37.0276ms
last_log_index = 11339377
last_log_term = 422005
last_snapshot_index = 11335794
last_snapshot_term = 422005
latest_configuration = [{Suffrage:Voter ID:x.x.x.x:8300 Address:x.x.x.x:8300} {Suffrage:Voter ID:x.x.x.x:8300 Address:x.x.x.x:8300}
{Suffrage:Voter ID:x.x.x.x:8300 Address:x.x.x.x:8300} {Suffrage:Voter ID:x.x.x.x:8300 Address:x.x.x.x:8300} {Suffrage:Voter ID:x.x.x.x:8300 Address:x.x.x.x:8300} {Suffrage:Voter ID:x.x.x.x:8300 Address:x.x.x.x:8300} {Suffrage:Voter ID:x.x.x.x:8300 Address:x.x.x.x:8300}]
latest_configuration_index = 10813000
num_peers = 6
protocol_version = 1
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 422005
runtime:
arch = amd64
cpu_count = 1
goroutines = 143
max_procs = 2
os = linux
version = go1.9
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 120
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 137796
members = 24
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 69
members = 7
query_queue = 0
query_time = 1
Operating system and Environment details
Clients: Windows 2003, 2008R2, Centos 6.6, Centos 7.4 (no issue on Centos systems)
Servers: Centos 7.4
Log Fragments or Link to gist
I would need assistance to even decide what logs are appropriate. Let me know and I will collect whatever is needed.
Description of the Issue (and unexpected/desired result)
Experiencing excessive clock drift on Windows systems when consul >= 0.9.3 is running and connected to the cluster.
We have a mixed cluster with all Linux servers, some Linux clients and some Windows clients. Only the Windows systems experience the drift, Linux clients do not. The clock drift begins as soon as consul is started with an initial drift of 100ms/sec. Anecdotally this maxes out at 180 seconds off at which point NTP may be trying to adjust it back (but not completely understood yet).
Some observations:
Desired result is that the system clock stays stable relative to the NTP source.
Reproduction steps
I am downgrading my cluster, but I can reproduce this any time by running 0.9.3, so let me know if there is any issue reproducing.
Run consul >= v0.9.3 on a Windows system with either Win2003 or Win2008R2 (not yet tested on Win2012 or Win2016) attached to a domain.
W32Time running as the (S)NTP client with a domain controller as the NTP source (not yet tested under other NTP clients or sources).
Compare time to the source (we used w32tm):
consul versionfor both Client and ServerClient (Linux): Consul v0.9.3
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
Client (Windows):Consul v0.9.3
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
Server: Consul v0.9.3
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
consul infofor both Client and ServerClient:
Server:
Operating system and Environment details
Clients: Windows 2003, 2008R2, Centos 6.6, Centos 7.4 (no issue on Centos systems)
Servers: Centos 7.4
Log Fragments or Link to gist
I would need assistance to even decide what logs are appropriate. Let me know and I will collect whatever is needed.