Skip to content

[Task][HTTP Client] Add optional observations for upstream attempts and response completion #8209

Open

Description

@Aias00

Part of #8199.

Goal and verified gap

Add optional, read-only observations for each actual upstream attempt in the standard HTTP forwarding path, including its corresponding response body. Reuse existing tracing/metrics/logging facilities. Do not change routing, retry eligibility, circuit-breaker policy or response-writing behavior.

Reviewed master 750673e3f619e335333150ff1b2d14070d5d5c31. AbstractHttpClientPlugin prepares the initial call. CURRENT retry can resubscribe that publisher; another-target execution creates a separate transport call. A single enclosing-plugin observation cannot describe each attempt.

Netty doRequest returns response metadata and retains the connection; WebClient doRequest returns an entity containing a body publisher. Headers readiness is not response completion. The corresponding body is consumed later by NettyClientMessageWriter or WebClientMessageWriter.

#8203 owns inclusive plugin-publisher observation; #7566 owns tracing and outbound propagation; #8125/#7955 own AI attempt/usage integration. This task supplies the missing standard-HTTP transport/body adapter, not another observation platform.

First delivery

  • Cover both NettyHttpClientPlugin and WebClientPlugin. Allocate attempt identity per transport subscription and associate the actual response/connection with that identity through body consumption and cleanup. A subscription means an admitted transport call, not proof that bytes reached the backend.
  • Provide a small optional adapter with a no-op default. Report configured target identity, method, headers-ready timing/status and terminal completion/error/cancellation. Reuse logical request/trace context and existing observation infrastructure where applicable.
  • Distinguish upstream/body outcome from downstream delivery outcome when observable. Handle pre-header failures and rejected-status body draining without declaring headers completion a successful full attempt.
  • Keep callbacks read-only, nonblocking and failure-isolated. They cannot mutate requests, select targets, restart work or subscribe to bodies. Use existing bounded delivery facilities; expose unsupported facts explicitly.

Acceptance criteria

  • Real local HTTP servers exercise both clients: success, connection/header failure, delayed/failed body, and downstream cancellation. Headers followed by a delayed body do not finish the observation early.
  • CURRENT retry subscriptions and attempts against another selected target receive separate identities. Later success cannot overwrite an earlier attempt's facts; each actual attempt has one terminal record.
  • Response-writer completion, failure and cancellation finish the correct observation under concurrent requests. State belongs to the request/subscription, not singleton mutable fields.
  • Local request/body rejection is distinguishable from upstream transport, status and body failure. A selection or validation rejection before transport subscription is not reported as an upstream call.
  • Rejected-status body draining and cleanup terminate their observation once. Existing retry eligibility, response bytes and body-subscription counts remain unchanged; no extra buffering or response-writer replacement is added.
  • Disabled instrumentation preserves behavior with a reproducible overhead comparison. Observer failure or a slow sink cannot block EventLoops or fail requests; enabled output is bounded.
  • Reuse trace context without duplicating gateway server spans or automatically duplicating Java-agent client spans. Exclude credentials, raw URLs/queries and request IDs from metric labels. Document the observation layer for byte counts.

Coordination and limits

#8155/#8156/#8158 retain ownership of breaker classification, initial-target attribution and inflight lifetime fixes. Reuse their scenarios as regression inputs without implementing those fixes again. #8063 remains the shared supporting-extension contract owner; this is one focused integration. #8203 durations remain inclusive plugin scopes, distinct from HTTP attempt timing.

AI token accounting, new retry/breaker algorithms, RPC aggregation, WebSocket session observation, a universal protocol client and a second telemetry backend are out of scope.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions