-
Notifications
You must be signed in to change notification settings - Fork 715
Description
Feature Description
Add support for HTTP 429 "Too Many Requests" rate limiting errors in OpenDAL's S3 service to improve compatibility with few S3-compatible storage like Oracle Cloud Infrastructure .
Problem and Solution
OpenDAL's S3 service currently only handles AWS S3's rate limiting pattern (HTTP 503 with "SlowDown" error code). Many S3-compatible services like Oracle Cloud Infrastructure Object Storage, Scaleway, and others return HTTP 429 with "TooManyRequests" error codes rather than logging SlowDown.
When these services return HTTP 429, OpenDAL classifies them as non-retryable ErrorKind::Unexpected errors, causing applications to fail permanently instead of implementing proper retry logic. This breaks rate limiting functionality for users of these S3-compatible services.
2025-09-23T14:22:32.748030Z ERROR opendal_load_test: read/stat error: error=Unexpected (permanent) at read, context: { uri: https://{$namespace}.compat.objectstorage.{$region}.oraclecloud.com/{$objectstore}/{$prefix}/125/667837.data, response: Parts { status: 429, version: HTTP/1.1, headers: {"x-amz-request-id": "ams-1:NAkuDFmfxxxxxxxxxxxxxxxxxX1HQ8VQ3", "content-type": "application/xml; charset=utf-8", "content-length": "119", "date": "Tue, 23 Sep 2025 14:22:32 GMT", "opc-request-id": "ams-1:NAkuDFmfxxxxxxxxxxxxxxxxxX1HQ8VQ3", "x-api-id": "s3-compatible", "x-content-type-options": "nosniff", "strict-transport-security": "max-age=31536000; includeSubDomains", "access-control-allow-origin": "*", "access-control-allow-methods": "POST,PUT,GET,HEAD,DELETE,OPTIONS", "access-control-allow-credentials": "true", "access-control-expose-headers": "access-control-allow-credentials,access-control-allow-methods,access-control-allow-origin,content-length,content-type,date,opc-client-info,opc-request-id,strict-transport-security,x-amz-request-id,x-api-id,x-content-type-options"} }, service: s3, path: {$prefix}/125/667837.data, range: 0-15 } => S3Error { code: "TooManyRequests", message: "rate-limit exceeded", resource: "", request_id: "" } key={$prefix}/125/667837.data
Additional Context
No response
Are you willing to contribute to the development of this feature?
- Yes, I am willing to contribute to the development of this feature.