Skip to content

storage: Recommended read-resume policy does not resume on ABORTED, the status the service uses to close an idle BidiReadObject stream #7344

Open

Description

@james-media-anywhere

What happens

read_resume_policy::is_transient_code (src/storage/src/read_resume_policy.rs line 154 on
main and in 1.21.0) treats Unavailable, ResourceExhausted, Internal and
DeadlineExceeded as resumable. Aborted is not in the list, so Recommended returns
ResumeResult::Permanent for it and the bidi worker stops.

Why it matters

Cloud Storage engineering told us, through our support case, that an idle BidiReadObject
stream is pruned after roughly six minutes and that the intended signal is gRPC ABORTED with a
message like "Idle stream closed...". What the client receives today is an HTTP/2
RST_STREAM(INTERNAL_ERROR) with no trailers, which surfaces as a transport error and is
resumed. The day the service sends the documented status, every client on the default policy
stops resuming idle-pruned streams: the worker parks in its permanent-error arm and every later
read on that descriptor fails (with #6688 unfixed, by a panic on the next read_range).

An idle-prune ABORTED on a descriptor with a valid read_handle is resumable by construction:
the resume is the same RPC carrying the handle, and the object has not changed.

Reproduction

Fake google.storage.v2.Storage (tonic) implementing BidiReadObject. Open a descriptor, read
one range, leave it idle, have the fake end the stream with Status::aborted("Idle stream closed"). Observed on 1.21.0: zero reconnect attempts reach the fake; the descriptor is dead.
With the fake ending the same idle stream with Unavailable instead, the client reconnects and
the next read succeeds.

Expected

Add Code::Aborted to is_transient_code, or match the idle-close status specifically if its
message or error details are stable. Either way Recommended should resume it.

Workaround

open_object(...).with_read_resume_policy(...) with a policy that continues on Aborted. We
recover at the pool level instead (evict the descriptor on a failed read and retry once on a
fresh stream), so the cost today is one failed read.

Environment

google-cloud-storage 1.21.0, gRPC transport, BidiReadObject via open_object; main has the
same code as of 2 Nov 2026.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: storageIssues related to the Cloud Storage API.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions