Skip to content

The router reports a shed request as unavailable after a full fleet #2127

Closed

Description

@JeffLuoo

Follow-up to #1896. Parent: #1664.

Context

On the router's atenet.router.route.duration histogram, the ate.router.outcome (ateattr.RouterOutcomeKey) attribute is computed by classifyOutcome(err) in cmd/atenet/internal/router/extproc/metrics.go.

Prior to PR #1929, classifyOutcome(err) mapped every envoy_type.StatusCode_ServiceUnavailable (HTTP 503) *extproc.ReqError without a wrapped gRPC status code to no_capacity (ateattr.RouterOutcomeNoCapacity). Because both the ingress and egress handlers record atenet.router.route.duration, non-capacity 503s (such as egress denials or policy/credential lookup failures) were falsely reported as no_capacity.

PR #1929 fixed that over-reporting by updating classifyOutcome so that no_capacity is reported only when status.Code(err) == codes.ResourceExhausted (unwrapping through ReqError.Cause), while bare 503 ReqErrors without a ResourceExhausted cause fall through to unavailable (ateattr.RouterOutcomeUnavailable).

Expected Behavior

A request that the router sheds while the resume of its actor is retrying because of ResourceExhausted reports no_capacity. A shed request with a different cause continues to report unavailable. The unavailable brief in docs/metrics/registry/metrics.yaml and the comment on the router outcome values in internal/ateattr/ateattr.go describe the new behavior.

Discussion: #1929 (comment)

Activity

  1. added this to the M3 milestone on Oct 2, 2026
  2. JeffLuoo commented on Oct 6, 2026

    @JeffLuoo
    CollaboratorAuthor

    TODO: Follow up after #2244 to map store.ErrVersionConflict to codes.Aborted before recordLifecycleOp so ate.actor.lifecycle.operation.duration reports error.type="Aborted" instead of "Internal" across resume/suspend/pause.

  3. added a commit that references this issue on Oct 7, 2026
    858f354
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions