Skip to content

Should object store retry on connection reset by peer? #5378

@kszlim

Description

@kszlim

Which part is this question about
Object store

Describe your question
Should object store's retry logic also retry when the connection is reset by a peer?

Additional context
I'm querying an s3 bucket via polars (which uses object store under the hood) and I'm encountering this issue, this only happens when I'm querying many (~10k files)

Generic S3 error: Error after 0 retries in 221.182308ms, max_retries:10, retry_timeout:10s, source:error sending request for url (https://s3.us-west-2.amazonaws.com/bucket/some_parquet_id.parquet): connection error: Connection reset by peer (os error 104)

This seems to be because this error isn't covered under object store's retry policy.

It'd be very handy if this was, though I'm not certain if it'd be strictly correct behavior (ie. Should it be handled by the caller of object store instead? Which in this case is polars).

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions