Fixed linkerd check not finding Prometheus#4797
Merged
Conversation
## The Problem `linkerd check` run right after install is failing because it can't find the Prometheus Pod. ## The Cause The "control plane pods are ready" check used to verify the existence of all the control plane pods, blocking until all the pods were ready. Since #4724, Prometheus is no longer included in that check because it's checked separately as an add-on. An unintended consequence is that when the ensuing "control plane self-check" is triggered, Prometheus might not be ready yet and the check fails because it doesn't do retries. ## The Fix The "control plane self-check" uses a gRPC call (it's the only check that does that) and those weren't designed with retries in mind. This PR adds retry functionality to the `runCheckRPC()` function, making sure the final output remains the same It also temporarily disables the `upgrade-edge` integration test because after installing edge-20.7.4 `linkerd check` will fail because of this.
kleimkuhler
reviewed
Jul 25, 2020
Contributor
kleimkuhler
left a comment
kleimkuhler
left a comment
There was a problem hiding this comment.
This looks good and I've tested that this works. I do have one question before approving.
Pothulapati
reviewed
Jul 27, 2020
Contributor
Pothulapati
left a comment
Pothulapati
left a comment
There was a problem hiding this comment.
I scaled linkerd-prometheus to 0 and ran the linkerd check, and the check command ran for 5m4s
linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
| waiting for check to complete
but once prometheus comes back up, the check succeeds
| retryDeadline: hc.RetryDeadline, | ||
| description: "control plane self-check", | ||
| hintAnchor: "l5d-api-control-api", | ||
| surfaceErrorOnRetry: false, |
Contributor
There was a problem hiding this comment.
Any reason this has to be false? As its hard to understand which check we are waiting for with just \ waiting for check to complete?
Contributor
There was a problem hiding this comment.
I get the following output, by making it true
- Error calling Prometheus from the control plane: server_error: server error: 503
This seems better than the waiting for check right?
Member
Author
There was a problem hiding this comment.
As you know, prometheus takes a long while to become available, so users that run linker check right after install will always see that error. I thought it was less scary for them to just let them know we're waiting for the check. If prometheus doesn't come up by the time-out, the real error is shown:
linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
× [prometheus] control plane can talk to Prometheus
Error calling Prometheus from the control plane: server_error: server error: 503
see https://linkerd.io/checks/#l5d-api-control-api for hints
Status check results are ×
han-so1omon
pushed a commit
to han-so1omon/linkerd2
that referenced
this pull request
Jul 28, 2020
* Fixed `linkerd check` not finding Prometheus ## The Problem `linkerd check` run right after install is failing because it can't find the Prometheus Pod. ## The Cause The "control plane pods are ready" check used to verify the existence of all the control plane pods, blocking until all the pods were ready. Since linkerd#4724, Prometheus is no longer included in that check because it's checked separately as an add-on. An unintended consequence is that when the ensuing "control plane self-check" is triggered, Prometheus might not be ready yet and the check fails because it doesn't do retries. ## The Fix The "control plane self-check" uses a gRPC call (it's the only check that does that) and those weren't designed with retries in mind. This PR adds retry functionality to the `runCheckRPC()` function, making sure the final output remains the same It also temporarily disables the `upgrade-edge` integration test because after installing edge-20.7.4 `linkerd check` will fail because of this.
han-so1omon
pushed a commit
to han-so1omon/linkerd2
that referenced
this pull request
Jul 28, 2020
* Fixed `linkerd check` not finding Prometheus ## The Problem `linkerd check` run right after install is failing because it can't find the Prometheus Pod. ## The Cause The "control plane pods are ready" check used to verify the existence of all the control plane pods, blocking until all the pods were ready. Since linkerd#4724, Prometheus is no longer included in that check because it's checked separately as an add-on. An unintended consequence is that when the ensuing "control plane self-check" is triggered, Prometheus might not be ready yet and the check fails because it doesn't do retries. ## The Fix The "control plane self-check" uses a gRPC call (it's the only check that does that) and those weren't designed with retries in mind. This PR adds retry functionality to the `runCheckRPC()` function, making sure the final output remains the same It also temporarily disables the `upgrade-edge` integration test because after installing edge-20.7.4 `linkerd check` will fail because of this. Signed-off-by: Eric Solomon <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Problem
linkerd checkrun right after install is failing because it can't find the Prometheus Pod.The Cause
The "control plane pods are ready" check used to verify the existence of all the control plane pods, blocking until all the pods were ready.
Since #4724, Prometheus is no longer included in that check because it's checked separately as an add-on. An unintended consequence is that when the ensuing "control plane self-check" is triggered, Prometheus might not be ready yet and the check fails because it doesn't do retries.
The Fix
The "control plane self-check" uses a gRPC call (it's the only check that does that) and those weren't designed with retries in mind.
This PR adds retry functionality to the
runCheckRPC()function, making sure the final output remains the sameIt also temporarily disables the
upgrade-edgeintegration test because after installing edge-20.7.4linkerd checkwill fail because of this.