You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
storage: no log or metric when a BidiReadObject stream is ended by the service and resumed #7345
When the service ends a BidiReadObject stream and the worker resumes it with the read_handle, nothing records that it happened: no tracing event, no counter, nothing on the ObjectDescriptor. Only a failed resume is visible, and then only as an error on the next read.
Why it matters
Operators cannot measure the reset rate, and cannot tell a healthy idle prune from a failure they
are chasing. We spent several weeks on a dead-stream problem (#6688) before adding a tracing::warn! and two counters to a vendored copy and discovering how ordinary the resumes
are. From that instrumented copy, over a 68-hour soak with 6.3 M reads against a real bucket:
64 service-ended streams, all resumed;
7 under load: 4 Unavailable "Cancelling all calls" (the h2 connection closed under the
stream), 3 Unknown "error reading a body from connection" after an RST_STREAM(INTERNAL_ERROR) on streams idle for under 90 s;
57 after the load stopped, in waves every six minutes or so, the idle-prune cadence;
one resume happened with a range outstanding and the read completed in under 200 ms.
Without the instrumentation none of that is observable from any log.
Request
A tracing event (warn or info) when the worker enters the resume path, carrying the status
and the number of outstanding ranges, and another when the resume fails permanently.
Counts of successful and failed resumes, either as a per-descriptor accessor or as metrics.
Our vendored patch (inline in the comment on #6688) does the minimal version: two AtomicU64 statics and object_descriptor::resume_count() / resume_failure_count(). Happy to adapt to whatever shape
fits the crate, for example a stats struct on the descriptor or OpenTelemetry metrics.
Also noticed while here
The doc comment on retry_policy::storage_default() says the client stops retrying after 10
seconds; the code sets with_time_limit(Duration::from_secs(300)).
Activity