Skip to content

storage: no log or metric when a BidiReadObject stream is ended by the service and resumed #7345

Open

Description

@james-media-anywhere

What is missing

When the service ends a BidiReadObject stream and the worker resumes it with the
read_handle, nothing records that it happened: no tracing event, no counter, nothing on the
ObjectDescriptor. Only a failed resume is visible, and then only as an error on the next read.

Why it matters

Operators cannot measure the reset rate, and cannot tell a healthy idle prune from a failure they
are chasing. We spent several weeks on a dead-stream problem (#6688) before adding a
tracing::warn! and two counters to a vendored copy and discovering how ordinary the resumes
are. From that instrumented copy, over a 68-hour soak with 6.3 M reads against a real bucket:

  • 64 service-ended streams, all resumed;
  • 7 under load: 4 Unavailable "Cancelling all calls" (the h2 connection closed under the
    stream), 3 Unknown "error reading a body from connection" after an
    RST_STREAM(INTERNAL_ERROR) on streams idle for under 90 s;
  • 57 after the load stopped, in waves every six minutes or so, the idle-prune cadence;
  • one resume happened with a range outstanding and the read completed in under 200 ms.

Without the instrumentation none of that is observable from any log.

Request

  • A tracing event (warn or info) when the worker enters the resume path, carrying the status
    and the number of outstanding ranges, and another when the resume fails permanently.
  • Counts of successful and failed resumes, either as a per-descriptor accessor or as metrics.

Our vendored patch (inline in the comment on #6688) does the minimal version: two AtomicU64 statics and
object_descriptor::resume_count() / resume_failure_count(). Happy to adapt to whatever shape
fits the crate, for example a stats struct on the descriptor or OpenTelemetry metrics.

Also noticed while here

The doc comment on retry_policy::storage_default() says the client stops retrying after 10
seconds; the code sets with_time_limit(Duration::from_secs(300)).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: storageIssues related to the Cloud Storage API.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions