Skip to content

Commit b8526eb

Browse files
authored
fix(server): install the metrics recorder before startup reconciliation (NVIDIA#4524)
The endpoint-status startup sweep ran before the gateway installed its Prometheus recorder, so the lock waits, timeouts, errors and hold times it recorded went to the no-op recorder and only its warnings reached the log. The sweep retries lock timeouts for up to three lock waits before startup fails, and a slow sweep that still succeeds is what operators want to see in the lock metrics. When metrics are enabled, the recorder is now installed before startup reconciliation, and the metrics listener still binds with the other listeners. A new pod's first scrape therefore already holds the sweep's samples, so they show in the raw counter and histogram count values, not in rate() or increase(). A sweep that fails still exits before any listener binds, so its timeouts remain log-only. The startup lock-timeout test now waits for the sweep's timeout counter instead of a fixed sleep and checks its timeout and wait samples, and the debug skill says which startup timeouts reach the metrics and how to read them. Part of NVIDIA#4307 Closes NVIDIA#4523 Signed-off-by: Emilien Macchi <[email protected]>
1 parent 9c6867c commit b8526eb

3 files changed

Lines changed: 35 additions & 9 deletions

File tree

‎crates/openshell-server/src/grpc/policy/endpoint_status_tests.rs‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1591,6 +1591,9 @@ async fn startup_reconciliation_retries_no_lock_timeout_after_the_deadline() {
15911591

15921592
#[tokio::test]
15931593
async fn startup_reconciliation_retries_a_lock_timeout() {
1594+
const WAITS: &str = "openshell_server_mutation_lock_wait_seconds_count{scope=\"sandbox\"}";
1595+
const TIMEOUTS: &str = "openshell_server_mutation_lock_timeouts_total{scope=\"sandbox\"}";
1596+
let metrics = crate::gateway_metrics::MetricsCapture::install();
15941597
let state = test_server_state().await;
15951598
let sandbox_id = "endpoint-startup-lock-timeout";
15961599
let initial = seed_stale_endpoint_status(&state, sandbox_id).await;
@@ -1607,7 +1610,13 @@ async fn startup_reconciliation_retries_a_lock_timeout() {
16071610

16081611
// Release after the first attempt's lock wait timed out, while the retry
16091612
// waits.
1610-
tokio::time::sleep(lock_timeout * 3 / 2).await;
1613+
tokio::time::timeout(std::time::Duration::from_secs(5), async {
1614+
while metrics.value(TIMEOUTS).is_none() {
1615+
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
1616+
}
1617+
})
1618+
.await
1619+
.expect("the first lock wait times out");
16111620
assert!(
16121621
!reconciliation.is_finished(),
16131622
"a lock timeout must not end the scan"
@@ -1623,6 +1632,10 @@ async fn startup_reconciliation_retries_a_lock_timeout() {
16231632
.status
16241633
.expect("status remains present");
16251634
assert_eq!(status.endpoint_statuses, vec![initial]);
1635+
1636+
// The sweep counts the timed-out wait and records the retry's wait.
1637+
assert_eq!(metrics.value(TIMEOUTS), Some(1));
1638+
assert_eq!(metrics.value(WAITS), Some(1));
16261639
}
16271640

16281641
#[tokio::test]

‎crates/openshell-server/src/lib.rs‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -826,6 +826,21 @@ pub(crate) async fn run_server(
826826

827827
let state = Arc::new(state);
828828

829+
// Install the recorder before the endpoint-status startup reconciliation so
830+
// the lock waits and timeouts recorded there are exported. Its listener
831+
// binds with the others.
832+
let metrics_endpoint = config
833+
.metrics_bind_address
834+
.map(|metrics_bind_address| {
835+
gateway_metrics::install_global_recorder(
836+
supervisor_session::RELAY_CAPACITY,
837+
store.mutation_lock_connections(),
838+
)
839+
.map(|prometheus_handle| (metrics_bind_address, prometheus_handle))
840+
})
841+
.transpose()
842+
.map_err(|e| Error::config(format!("failed to install metrics recorder: {e}")))?;
843+
829844
// Reconcile local-driver running intent before watchers spawn so their
830845
// first snapshots observe the post-start backend state. Explicitly stopped
831846
// sandboxes remain stopped.
@@ -874,12 +889,7 @@ pub(crate) async fn run_server(
874889
}
875890

876891
// Bind the Prometheus metrics endpoint on a dedicated port when configured.
877-
if let Some(metrics_bind_address) = config.metrics_bind_address {
878-
let prometheus_handle = gateway_metrics::install_global_recorder(
879-
supervisor_session::RELAY_CAPACITY,
880-
store.mutation_lock_connections(),
881-
)
882-
.map_err(|e| Error::config(format!("failed to install metrics recorder: {e}")))?;
892+
if let Some((metrics_bind_address, prometheus_handle)) = metrics_endpoint {
883893
let metrics_listener = TcpListener::bind(metrics_bind_address).await.map_err(|e| {
884894
Error::transport(format!(
885895
"failed to bind metrics port {metrics_bind_address}: {e}",

‎skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -582,8 +582,11 @@ that scan, then logs
582582
the `sandbox_id` and exits with
583583
`tool server endpoint-status startup reconciliation failed`. A pod that
584584
restarts with that error during a rollout or scale-out is waiting behind a
585-
long holder: find it as above. Those startup timeouts appear in the gateway log
586-
only, because the metrics endpoint starts after the scan.
585+
long holder: find it as above. When the scan finishes, its waits and timeouts
586+
are already in the lock metrics at the pod's first scrape, so read the counter
587+
and `_count` values on the new pod: `rate()` and `increase()` take that first
588+
scrape as their baseline and do not show them. A scan that fails exits before
589+
the metrics endpoint starts, so its timeouts appear in the gateway log only.
587590

588591
A lock connection that PostgreSQL does not open, although at least a second of
589592
the 10-second limit remained, is not a lock timeout. The request fails with

0 commit comments

Comments
 (0)