Repository navigation
docs(batch-serving): add observability and troubleshooting section - #2635
Conversation
|
/assign @shimib |
shimib
left a comment
shimib
left a comment
There was a problem hiding this comment.
Checked every metric name, label, and link against llm-d-async and llm-d-batch-gateway main. All resolve. Two corrections needed inline, plus nits:
inflight_requestsis per queue andpool_worker_limitis per pool, so the utilization ratio needssum by (pool_name)on the numerator.- Shed means HTTP 429 only. 5xx responses count as failures or retries, not sheds.
NumWorkersis the Go field. The user-facing knobs arenum_workersin the processor config andprocessor.config.numWorkersin the chart.
|
|
||
| #### Common failure modes | ||
|
|
||
| - **Backlog grows while workers sit idle**: utilization is well below 1.0 and `llm_d_async_async_dispatch_budget` is at or near 0. The gate is closed. If `gate_metric_value` is above `gate_metric_threshold`, the pool really is saturated and the backlog is expected, since this path fills slack capacity. If `gate_metric_source_available` is 0, the gate cannot read its source (for example Prometheus is unreachable) and is running on its default; fix the source before tuning thresholds. |
There was a problem hiding this comment.
This is inverted. Per the metric help text, the gate closes when gate_metric_value ≤ gate_metric_threshold, and for saturation gates the value is 1 − saturation, not the raw metric. So a value above the threshold means the gate is open. Same issue in the "Gate input" table row: the gauge is the gate's budget reading, not pool saturation or vLLM running requests.
There was a problem hiding this comment.
Still as before at f36952a, and the "Gate input" row above has the same issue. Suggested text for the bullet: "The gate is closed when gate_metric_value ≤ gate_metric_threshold. The value is the gate's budget reading (1 − saturation for saturation gates), not the raw pool metric, so a low value means the pool really is saturated and the backlog is expected." For the table row: "The budget the gate last computed from its source, compared against the threshold; the gate closes at or below it."
There was a problem hiding this comment.
Applied your wording for both the bullet and the Gate input row in 65ed808: the gate closes at or below the threshold, and the value is the gate's budget reading, not the raw pool metric.
| - **Failed jobs** (`BatchGatewayHighJobFailureRate`): check `request_errors_by_model_total` to see whether one model accounts for them, and `file_storage_operations_total{status="exhausted"}` for jobs that failed on input or output storage. | ||
| - **Orphaned jobs**: a non-zero `batch_reconciler_orphans_recovered_total` points at processor crashes, and `BatchReconcilerErrors` means the reconciler itself is failing to recover them. | ||
|
|
||
| The bundled alerting rules cover the Batch Gateway only; there are no bundled alerts for the Async Processor yet. |
There was a problem hiding this comment.
Not quite. The llm-d-async chart ships a PrometheusRule with four AsyncProcessor alerts (HighRetryRate, HighDeadlineExceededRate, LowSuccessRate, HighShedRate). Suggest: "llm-d's bundled rules cover the Batch Gateway only; the Async Processor's alerts ship in the llm-d-async chart."
There was a problem hiding this comment.
Still as before at f36952a. Suggested: "llm-d's bundled alerting rules cover the Batch Gateway only; the Async Processor's alerts (AsyncProcessorHighRetryRate, HighDeadlineExceededRate, LowSuccessRate, HighShedRate) ship in the llm-d-async chart's PrometheusRule."
There was a problem hiding this comment.
Applied in 65ed808, and the note now names the four AsyncProcessor alerts in the llm-d-async chart's PrometheusRule.
|
@UgaTheDev This PR has merge conflicts with gh pr checkout 2635
git fetch origin main
git rebase origin/main
# resolve conflicts
git push --force-with-lease |
Add per-guide Observability & Troubleshooting for batch serving (llm-d#1982), covering the Async Processor and Batch Gateway separately: key signals, how to tell a saturated pool from a batch layer that is not dispatching, and common failure modes. Batch Gateway links to the shared metric reference and alert rules; Async Processor links to the llm-d-async metrics reference, since the shared reference does not list it yet. Adds a short pointer from the batch-serving workload doc. Rebased onto main after the Operations-guides consolidation (llm-d#2691) moved docs/well-lit-paths/workloads/batch-serving/README.md to docs/architecture/advanced/batch/README.md; the pointer paragraph moved with it. Signed-off-by: Kush Zingade <[email protected]>
a794512 to
3235c31
Compare
|
Rebased onto main; the overlap was the Operations-guides consolidation (#2691), which moved the workload doc this PR touched and split Batch Gateway metrics into their own doc. Re-approve when green. |
…num_workers, metric names) - Utilization row: aggregate per-queue inflight to pool_name before dividing by the per-pool worker limit. - Narrow the shed failure mode to HTTP 429 only; 5xx now routes to the retries/failures guidance instead. - Replace NumWorkers (Go field name) with the actual config keys: num_workers (processor config) / processor.config.numWorkers (Helm). - Verified the four Batch Gateway metric names and the two llm-d-async metric names cited in the doc against current main in both repos; all match exactly, no renames needed. Signed-off-by: Kush Zingade <[email protected]>
|
Thanks @shimib, applied all three in f36952a:
Since #2691 moved the Batch Gateway metrics to their own page and its table doesn't list the four I cite under backpressure and per-model load, I checked them against llm-d-batch-gateway main: |
|
Thanks for the nits. The two inline threads (gate value direction at L128 and the table row above it, bundled alerts at L158) are still unchanged at f36952a. Suggested wording is in each thread. |
… note Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01G8SG4bhupnZ9HRuc5cotSH Signed-off-by: Kush Zingade <[email protected]>
… dashboard The batch-serving observability section (llm-d#2635) tells operators to read the dispatch budget, gate decisions by reason, in-flight requests against the pool worker limit, and the metric gates' last reading and threshold, but the ported dashboard had none of them. Adds four panels for those signals below the existing ones, using the metric names and labels the processor exports (dispatch_budget per queue, gate_decisions_total by reason, inflight_requests and pool_worker_limit by pool, gate_metric_value / _threshold / _source_available). Signed-off-by: Shimi Bandiel <[email protected]>
… dashboard The batch-serving observability section (llm-d#2635) tells operators to read the dispatch budget, gate decisions by reason, in-flight requests against the pool worker limit, and the metric gates' last reading and threshold, but the ported dashboard had none of them. Adds four panels for those signals below the existing ones, using the metric names and labels the processor exports (dispatch_budget per queue, gate_decisions_total by reason, inflight_requests and pool_worker_limit by pool, gate_metric_value / _threshold / _source_available). Signed-off-by: Shimi Bandiel <[email protected]>
… dashboard set (#2785) * observability: add the Async Processor Grafana dashboard to the llm-d dashboard set llm-d-async#505: the Async Processor had no dashboard in guides/recipes/observability/grafana/dashboards, so the llm-d Grafana install (load-llm-d-dashboards.sh) could not show it, and operators had to enable the llm-d-async chart's own ConfigMap, against the #1685 rule that every user-facing dashboard lives in llm-d. Ports the chart's dashboard (13 panels: request rate and outcomes, success and retry ratios, message latency, exceeded deadlines, token rate, deadline proximity, broker backlog, in-process queue depth) to the conventions the other dashboards here follow: a DS_PROMETHEUS datasource variable on every panel and target, a namespace variable bound to it, llm-d tags, and the uid llm-d-async-processor so it does not collide with the chart's copy when both are loaded. Panel queries are unchanged. The dashboards page lists it, notes it needs the chart's PodMonitor enabled, and links the Async Processor metric reference. Signed-off-by: Shimi Bandiel <[email protected]> * observability: add gate and worker-pool panels to the Async Processor dashboard The batch-serving observability section (#2635) tells operators to read the dispatch budget, gate decisions by reason, in-flight requests against the pool worker limit, and the metric gates' last reading and threshold, but the ported dashboard had none of them. Adds four panels for those signals below the existing ones, using the metric names and labels the processor exports (dispatch_budget per queue, gate_decisions_total by reason, inflight_requests and pool_worker_limit by pool, gate_metric_value / _threshold / _source_available). Signed-off-by: Shimi Bandiel <[email protected]> * observability: plot latencies the Async Processor records on every transport The ported dashboard's two latency panels read llm_d_async_async_message_latency_time_millis, which the processor only observes on GCP Pub/Sub, where the broker stamps a publish time (SupportsMessageLatency is false for both Redis transports). On Redis Sorted Set, the transport the guides recommend, the histogram is never observed and the panels stay empty; verified against a v0.11.0 deployment, whose /metrics has no message_latency series at all. The panels now plot inference latency (router and model server time per attempt) and queue residence time by queue (the delay the processor adds), which every transport records and which the batch-serving observability section tells operators to compare. End-to-end message latency keeps a panel of its own, labelled as Pub/Sub only. Signed-off-by: Shimi Bandiel <[email protected]> --------- Signed-off-by: Shimi Bandiel <[email protected]>
Part of #1982 (Phase 3, Batch serving). Adds an Observability & Troubleshooting section to
guides/batch-serving/README.mdcovering the Async Processor and Batch Gateway separately: key signals, how to tell a saturated pool from a batch layer that is not dispatching, and common failure modes. Batch Gateway links to the shared metric reference and alert rules; the Async Processor links to the llm-d-async metrics reference, since the shared reference does not list it yet. Also adds a short pointer fromdocs/well-lit-paths/workloads/batch-serving/README.md.