Add supervisor/count metric for health and state monitoring#19114
Merged
abhishekrb19 merged 14 commits intoMar 12, 2026
Conversation
Introduce SupervisorStatsMonitor that exposes supervisor count per state (RUNNING, SUSPENDED, UNHEALTHY_SUPERVISOR, etc.) with dimensions supervisorId, type, and state. Opt-in via druid.monitoring.monitors.
kfaraz
approved these changes
Mar 10, 2026
kfaraz
left a comment
kfaraz
left a comment
Contributor
There was a problem hiding this comment.
Looks good, left a minor suggestion.
Co-authored-by: Abhishek Radhakrishnan <[email protected]>
…rStatsMonitor.java Co-authored-by: Kashif Faraz <[email protected]>
…unt metric - Add dataSource, stream, and detailedState fields to SupervisorStats - Populate dataSource from SupervisorSpec.getDataSources() via DefaultQueryMetrics.getTableNamesAsString() - Populate stream from SupervisorSpec.getSource() (null for non-streaming supervisors) - Populate detailedState from Supervisor.getState().toString() (null only when state is null) - Emit dataSource always, stream and detailedState conditionally via setDimensionIfNotNull - Update tests to cover new dimensions, null stream, multi-datasource, and detailedState != state cases - Update metrics.md to list new dimensions and link to supervisor status report for state/detailedState values
abhishekrb19
approved these changes
Mar 10, 2026
abhishekrb19
left a comment
abhishekrb19
left a comment
Contributor
There was a problem hiding this comment.
Lgtm, left a few small suggestions. Thanks!
…ltMetrics.json Co-authored-by: Abhishek Radhakrishnan <[email protected]>
…rop null checks - Inline supervisorId, stateStr, dataSource, stream, and detailedState - Use spec.getId() instead of entry.getKey() - Remove redundant null check on spec.getDataSources() - Use spec.getSource() in place of spec.getStream() (fix suggested typo) - Replace Collections.emptyList() null fallback with direct getDataSources() usage
Comment on lines
+93
to
+105
| public Collection<SupervisorStatsProvider.SupervisorStats> getSupervisorStats() | ||
| { | ||
| List<SupervisorStatsProvider.SupervisorStats> stats = new ArrayList<>(); | ||
| for (Map.Entry<String, Pair<Supervisor, SupervisorSpec>> entry : supervisors.entrySet()) { | ||
| final SupervisorSpec spec = entry.getValue().rhs; | ||
| final SupervisorStateManager.State state = entry.getValue().lhs.getState(); | ||
|
|
||
| stats.add(new SupervisorStatsProvider.SupervisorStats( | ||
| spec.getId(), | ||
| spec.getType(), | ||
| state == null ? "UNKNOWN" : state.getBasicState().toString(), | ||
| DefaultQueryMetrics.getTableNamesAsString(new HashSet<>(spec.getDataSources())), | ||
| spec.getSource(), |
Contributor
There was a problem hiding this comment.
It looks like jacoco-coverage is failing due to lack of code coverage for this new method, which seems legitimate. Could you please add a few unit tests in SupervisorManagerTest?
Covers empty supervisors, active supervisors with non-null state, and supervisors with null state to satisfy jacoco diff coverage for the new method.
Move SegmentIdWithShardSpec import before SupervisorStatsProvider so segment comes before server as required by alphabetical ordering.
Revert unintended change to DefaultQueryMetrics and update SupervisorManagerTest to expect for supervisors with no data sources, matching getTableNamesAsString behavior.
TestSupervisorSpec.getDataSources() returned an empty list, causing the supervisor stats test to assert [] (stringified empty list) as the dataSource — a confusing and unrealistic value. Return the supervisor id instead, matching real SupervisorSpec implementations.
…or null supervisor entries
abhishekrb19
approved these changes
Mar 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add supervisor/count metric for health and state monitoring
Previously, supervisor health was only exposed via the REST API (
GET /druid/indexer/v1/supervisor/{id}/health) and thesys.supervisorsSQL table. There was no way to scrape supervisor health/state from Prometheus, StatsD, or other metric emitters for monitoring and alerting.This PR adds
SupervisorStatsMonitor, which emits asupervisor/countgauge metric per supervisor. Each active supervisor contributes a value of 1, tagged withsupervisorId,type, andstate(RUNNING, SUSPENDED, UNHEALTHY_SUPERVISOR, UNHEALTHY_TASKS, etc.). Users can aggregate by state for dashboards and alerts (e.g.sum(supervisor_count{state=~"UNHEALTHY_.*"})).Description
Design
TaskCountStatsMonitor/TaskCountStatsProvider: a provider interface inserver/, implemented bySupervisorManagerinindexing-service/, consumed by an Overlord-only monitor.supervisor/count— gauge, value 1 per supervisor; dimensions:supervisorId,type,state.getState().getBasicState()(7 values) instead of detailed states to avoid high cardinality.SupervisorStatsMonitortodruid.monitoring.monitors; no behavioral change without it.getState()and spec metadata (cheap reads); nogetStatus()calls on the monitor interval.Release note
Adds a new
supervisor/countmetric whenSupervisorStatsMonitoris enabled indruid.monitoring.monitors. The metric reports each supervisor’s state (RUNNING, SUSPENDED, UNHEALTHY_SUPERVISOR, etc.) for Prometheus, StatsD, and other metric systems.Key changed/added classes in this PR
SupervisorStatsProvider(new interface)SupervisorStatsProvider.SupervisorStats(new inner class)SupervisorStatsMonitor(new)SupervisorManager(implementsSupervisorStatsProvider)This PR has: