Add documentation for SLOs and alerting - #2476

Suggest Da Huang (@git286) , Jeff Luo (@JeffLuoo) and/or Julian Gutierrez Oschmann (@juli4n) as reviewers.

• gmp-alerting-rules.yaml:127: The page text says "the 30d 99.9% capacity budget", but the target and the thresholds use 99% (0.01).
• slos-and-alerts.md:43: The text still says "behaviour". Also, docs/observability.md still has no link to this page. The earlier thread marked both as done.
• slos-and-alerts.md:246: No SetWorkerCapacity RPC exists. Workers report through RegisterWorker and UpdateWorker.
• slos-and-alerts.md:152: With the defaults (100ms, 1.1x, 5s budget), parking makes at most about 19 attempts, not 23. Jitter only makes the delays longer.
• slos-and-alerts.md:305 and :309: "Ateom" is now ateworker.
• gmp-alerting-rules.yaml:81: "tighted up" should be "tightened up".

ty, fixed, sorry for missing some of these before.

I made a few final changes:

• Changed the emphasis in the header from "this is technically GMP but you should treat it as stock Prometheus" to "This is a GMP config, here's how you can change it to stock Prometheus"
• In SubstrateWorkerPoolCapacityNotDelivered, I changed from max by (...) (min_over_time) (which is a bit nonsensical) to min by (...) (min_over_time) which is consistent. I added a comment explaining why.
• Several other small typos recommended by Antigravity

From a user's point of view, low-latency suspend/resume is one of Substrate's main selling points. But the scenarios here only cover availability; the only mention of latency is the TODO in the yaml. Should we add a resume-latency SLI, and maybe one for suspend?

From a user's point of view, low-latency suspend/resume is one of Substrate's main selling points. But the scenarios here only cover availability; the only mention of latency is the TODO in the yaml. Should we add a resume-latency SLI, and maybe one for suspend?

I don't think it's high priority for suspend/resume; those are per-node operations which the operator can't easily affect (e.g. by adding more capacity). At most it could detect a bad rollout, but it's also very easy for users to add a custom alert. By contrast, the failure rate of resume is a critical operational factor IMO, and that's covered by this PR.

For latency of the API server operations - users can add more capacity and bring down the latency. It's quite difficult for me to test, and I suspect that different operators would want different rates, and the structure would be ~identical to the availability SLOs. So I'll comment this better but won't add more at this time.

those are per-node operations which the operator can't easily affect (e.g. by adding more capacity)

Most resume latency is snapshot/image transfer, which operators can affect (storage backend, region, node bandwidth). That's what B.4 already says. But line 47 lists snapshot storage as non-primary because it "is visible via the API server", and nothing in the rules watches API latency. Maybe you want line 47 point to B.4 instead of the API server?

those are per-node operations which the operator can't easily affect (e.g. by adding more capacity)

Most resume latency is snapshot/image transfer, which operators can affect (storage backend, region, node bandwidth). That's what B.4 already says. But line 47 lists snapshot storage as non-primary because it "is visible via the API server", and nothing in the rules watches API latency. Maybe you want line 47 point to B.4 instead of the API server?

Good idea, I've added a link to B.4 from line 47. Even though I haven't defined SLO rules for api server latencies, it's still true that the times are visible from the API server, and I've added notes (GMP.yaml line 180, doc.md line 179) indicating how operators could adapt the availability rules for latency.