Repository navigation
Commit b8526eb
authored
fix(server): install the metrics recorder before startup reconciliation (NVIDIA#4524)
The endpoint-status startup sweep ran before the gateway installed its
Prometheus recorder, so the lock waits, timeouts, errors and hold times
it recorded went to the no-op recorder and only its warnings reached
the log. The sweep retries lock timeouts for up to three lock waits
before startup fails, and a slow sweep that still succeeds is what
operators want to see in the lock metrics.
When metrics are enabled, the recorder is now installed before startup
reconciliation, and the metrics listener still binds with the other
listeners. A new pod's first scrape therefore already holds the sweep's
samples, so they show in the raw counter and histogram count values,
not in rate() or increase(). A sweep that fails still exits before any
listener binds, so its timeouts remain log-only.
The startup lock-timeout test now waits for the sweep's timeout counter
instead of a fixed sleep and checks its timeout and wait samples, and
the debug skill says which startup timeouts reach the metrics and how
to read them.
Part of NVIDIA#4307
Closes NVIDIA#4523
Signed-off-by: Emilien Macchi <[email protected]>1 parent 9c6867c commit b8526eb
3 files changed
Lines changed: 35 additions & 9 deletions
File tree
- crates/openshell-server/src
- grpc/policy
- skills/debug-openshell-cluster
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1591 | 1591 | | |
1592 | 1592 | | |
1593 | 1593 | | |
| 1594 | + | |
| 1595 | + | |
| 1596 | + | |
1594 | 1597 | | |
1595 | 1598 | | |
1596 | 1599 | | |
| |||
1607 | 1610 | | |
1608 | 1611 | | |
1609 | 1612 | | |
1610 | | - | |
| 1613 | + | |
| 1614 | + | |
| 1615 | + | |
| 1616 | + | |
| 1617 | + | |
| 1618 | + | |
| 1619 | + | |
1611 | 1620 | | |
1612 | 1621 | | |
1613 | 1622 | | |
| |||
1623 | 1632 | | |
1624 | 1633 | | |
1625 | 1634 | | |
| 1635 | + | |
| 1636 | + | |
| 1637 | + | |
| 1638 | + | |
1626 | 1639 | | |
1627 | 1640 | | |
1628 | 1641 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
826 | 826 | | |
827 | 827 | | |
828 | 828 | | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
829 | 844 | | |
830 | 845 | | |
831 | 846 | | |
| |||
874 | 889 | | |
875 | 890 | | |
876 | 891 | | |
877 | | - | |
878 | | - | |
879 | | - | |
880 | | - | |
881 | | - | |
882 | | - | |
| 892 | + | |
883 | 893 | | |
884 | 894 | | |
885 | 895 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
582 | 582 | | |
583 | 583 | | |
584 | 584 | | |
585 | | - | |
586 | | - | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
587 | 590 | | |
588 | 591 | | |
589 | 592 | | |
| |||
0 commit comments