Do not consider PENDING state as healthy#1866
Do not consider PENDING state as healthy#1866pracucci wants to merge 1 commit intocortexproject:masterfrom
Conversation
… not consider PENDING state as healthy Signed-off-by: Marco Pracucci <[email protected]>
|
Personally, I would find it easier to read |
gouthamve
left a comment
gouthamve
left a comment
There was a problem hiding this comment.
I I think this is better yeah.
|
I am concerned that this could break queries after a single ingester failure. If an ingester pod vanishes (node issue or something), it will still be unhealthy in the ring, and now the new pending pod will also be considered unhealthy causing all reads to fail. Would you be able to test that? |
|
I think Chris is correct, and I think the solution is not to count pending ingesters as an error. |
|
This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions. |
|
Closing for now |
5 participants
What this PR does:
While working on the PR #1818 I've realized that an ingester is considered healthy for read operations while
PENDINGbut not whileJOINING. From my perspective, an ingester shouldn't be considered healthy whilePENDING, so I'm suggesting to:PENDINGstate as healthyIn the read path, there are some API endpoints like
LabelNamesandLabelValuesForLabelNamefor which we query all healthy ingesters (seedistributor.forAllIngesters()). APENDINGingester shouldn't hold any data yet, so shouldn't be required to be hit.A comment I've received in a previous discussion on this topic is that this change may introduce quorum issues during ingesters rollout. However, I can't see a real difference compared to when the ingester switches from
PENDINGtoJOINING, considered that we already consider theJOININGstate as unhealthy.Which issue(s) this PR fixes:
No issue
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]