Connect Vault CA token updates are not retained on restart #11363
Description
When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.
Overview of the Issue
When you use the consul connect ca set-config command to update the Vault token the configuration is updated and successfully connects, however when the leadership in the cluster changes or Consul is restarted on a node the vault token in the configuration reverts back to the previous version (as seen by a consul connect ca get-config command.
Reproduction Steps
Steps to reproduce this issue, eg:
- Create a cluster with Vault CA integration
- Revoke the Vault Token and issue a new one
- Update the configuration on Consul with the
consul connect ca set-configcommand - Force an election or restart the leader node
It doesn't always happen but I can reproduce this on all of our clusters (DM me if you want remote access to our SBX environment)
Consul info for both Client and Server
Server info
agent:
check_monitors = 0
check_ttls = 0
checks = 3
services = 3
build:
prerelease =
revision = c976ffd2
version = 1.10.3
consul:
acl = enabled
bootstrap = false
known_datacenters = 3
leader = true
leader_addr = 172.16.18.231:8300
server = true
raft:
applied_index = 3459907
commit_index = 3459907
fsm_pending = 0
last_contact = 0
last_log_index = 3459907
last_log_term = 31964
last_snapshot_index = 3459154
last_snapshot_term = 31964
latest_configuration = [{Suffrage:Voter ID:0db759ad-0007-2ea4-4d41-62aff64515d5 Address:172.16.18.231:8300} {Suffrage:Voter ID:4de69dad-f717-cb90-5c23-011fe0301004 Address:172.16.19.63:8300} {Suffrage:Voter ID:b846fa4a-7c4b-f797-33bb-31cd99211e35 Address:172.16.18.7:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 31964
runtime:
arch = amd64
cpu_count = 2
goroutines = 400
max_procs = 2
os = linux
version = go1.16.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 874
failed = 0
health_score = 0
intent_queue = 0
left = 3
member_time = 103391
members = 16
query_queue = 0
query_time = 13
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 6
intent_queue = 0
left = 0
member_time = 15622
members = 9
query_queue = 0
query_time = 1
Operating system and Environment details
Amazon Linux 2 running on EC2 with Consul version 1.10.3 (before upgrade was 1.9.6)
Log Fragments
2021-10-07T00:53:27.206Z [ERROR] agent.server.connect: CA root replication failed, will retry: routine="secondary CA roots watch" error="Failed to initialize secondary CA provider: error configuring provider: Error making API request.
URL: GET https://vault.sandbox.homexlabs.com/v1/auth/token/lookup-self
Code: 403. Errors:
* permission denied"