TSDB: metrics for queries#1981
Merged
pracucci merged 7 commits intocortexproject:masterfrom Jan 15, 2020
pstibrany:tsdb-metrics
Merged
TSDB: metrics for queries#1981pracucci merged 7 commits intocortexproject:masterfrom pstibrany:tsdb-metrics
pracucci merged 7 commits intocortexproject:masterfrom
pstibrany:tsdb-metrics
Conversation
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
codesome
reviewed
Jan 15, 2020
codesome
reviewed
Jan 15, 2020
Signed-off-by: Peter Štibraný <[email protected]>
pracucci
approved these changes
Jan 15, 2020
Contributor
pracucci
left a comment
pracucci
left a comment
There was a problem hiding this comment.
Thanks Peter! Could you also add a changelog entry, please (we prefix them with TSDB:)?
Ignored: cortex_ingester_memory_series – cannot see easy way to get this, not even in TSDB metrics
I have a working solution for this (I will submit a PR):
master...pracucci:fix-ingester-memory-series-for-blocks-storage
Ignored: cortex_ingester_memory_series_created_total, cortex_ingester_memory_series_removed_total (these are per-user) – again, no obvious/easy way to get this from TSDB
I agree it's quite complicated. I'm crashing my head against it since several hours and trying different approaches. I'm currently work on it.
Signed-off-by: Peter Štibraný <[email protected]>
3 tasks
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Signed-off-by: Peter Štibraný <[email protected]>
Contributor
|
Thanks for doing that! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR updates some existing ingester metrics from blocks code. Specifically:
We already update ingestion metrics from blocks code (
cortex_ingester_ingested_samples_total,cortex_ingester_ingested_samples_failures_total)Querying:
cortex_ingester_queries_total(this was already tracked before this PR)cortex_ingester_queried_samples(added by this PR)cortex_ingester_queried_series(added by this PR)cortex_ingester_queried_chunks– chunks-specificUser state:
cortex_ingester_memory_users(added by this PR[update: now removed again, in favor of PR Fix ingester_memory_series/users metrics for TSDB storage #1982), basically equals to number of open TSDB databases in Cortex)cortex_ingester_memory_series– cannot see easy way to get this, not even in TSDB metricscortex_ingester_memory_series_created_total,cortex_ingester_memory_series_removed_total(these are per-user) – again, no obvious/easy way to get this from TSDBThere are some more metrics that are updated when flushing chunks. This PR doesn't update these metrics and there seems to be no obvious way how we could do that:
cortex_ingester_chunk_age_secondscortex_ingester_chunk_lengthcortex_ingester_chunk_size_bytescortex_ingester_chunk_utilizationcortex_ingester_chunks_created_totalcortex_ingester_dropped_chunks_totalcortex_ingester_memory_chunks(total chunks in memory)Additional cortex_ingester metrics are related to transfer of data between ingesters. This PR doesn't change them, but for completeness, here they are:
cortex_ingester_received_bytes_total,cortex_ingester_received_files,cortex_ingester_sent_bytes_total,cortex_ingester_sent_files- updated when transferring TSDB's between ingesterscortex_ingester_received_chunks,cortex_ingester_sent_chunks- updated when transferring chunks between ingesters