[Bug] BE loses S3 access once the EKS Pod Identity token rotates; token file is never re-read #66554

Activity

1. Doris-Breakwater commented on Aug 6, 2026

   

   Doris-Breakwater

   on Aug 6, 2026

   Collaborator

   More actions

   Breakwater-GitHub-Analysis-Slot: slot_fc3e45704055

   Initial triage

   Judgment: valid BE credential-provider compatibility bug; high confidence. The issue is open with no labels, assignee, milestone, or earlier comments, and the author is willing to submit a PR. The unsupported behavior is present at the current remote heads of both master (8e4bd4d5f11d68ddd5fd4ae2ee153ad8d6779323) and branch-4.1 (e48fc7e97b98909199d27bfd3aba3924f5588c51).

   Verified facts

   • Both branches still vendor AWS SDK for C++ 1.11.219. AWS lists 1.11.263 as the minimum C++ SDK version for EKS Pod Identity.
   • EKS Pod Identity injects AWS_CONTAINER_CREDENTIALS_FULL_URI and AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE; the projected token file is refreshed by EKS. This is the documented Pod Identity environment and volume setup.
   • Doris's CustomAwsCredentialsProviderChain recognizes the full URI but only reads AWS_CONTAINER_AUTHORIZATION_TOKEN, once while constructing TaskRoleCredentialsProvider. It never reads AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE.
   • In SDK 1.11.219, TaskRoleCredentialsProvider::Reload() refreshes the returned temporary AWS credentials, but its ECSCredentialsClient retains the authorization token supplied at construction. Therefore a token copied into AWS_CONTAINER_AUTHORIZATION_TOKEN at BE startup becomes stale after the projected file rotates.
   • Upstream SDK 1.11.263 added the intended mechanism: the default chain passes the token-file path into GeneralHTTPCredentialsProvider, and Reload() reads that file again before each credential-endpoint request.
   • There is a second affected route: an explicitly selected CredProviderType::Container reads only AWS_CONTAINER_CREDENTIALS_RELATIVE_URI. EKS supplies the full URI, so this route also needs coverage. The cloud recycler has the same construction pattern.

   One detail of the reported timeline is not yet explained. With the standard EKS injection there is no raw AWS_CONTAINER_AUTHORIZATION_TOKEN, so the current code can fail on the first credential fetch, not only after rotation. The stated "works initially, then fails" sequence is consistent if an entrypoint or another component copied the initial file contents into the raw-token environment variable, or if another credential source was initially selected. This does not weaken the confirmed missing token-file support, but it should be clarified before treating the exact timeline as reproduced.

   Information requested from the reporter

   1. Exact Doris build/commit for each tested version, plus the BE value of aws_credentials_provider_version and the catalog's s3.credentials_provider_type (or confirmation that it is unset/default).
   2. Presence and paths of the AWS_CONTAINER_* variables inside the BE container. Please redact the authorization token itself; only indicate whether the raw-token variable is present and how it is populated.
   3. The exact first failure and surrounding BE credential-provider/S3 log lines, including timestamps relative to the credential expiry and projected-token rotation. Please redact credentials, account IDs, bucket names, and other sensitive values.
   4. Confirmation that the first S3 query really used the Pod Identity role rather than a node role, profile, or another earlier provider in the chain.

   These details are useful to reconcile the timing, but they are not blockers to fixing the verified incompatibility.

   Recommended implementation and validation

   • Prefer upgrading the C++ SDK to at least 1.11.263 (or a newer vetted 1.11.x release) and porting the upstream general/container HTTP provider logic into CustomAwsCredentialsProviderChain while preserving Doris's intentional provider ordering. An SDK bump alone is insufficient because the custom chain currently bypasses the new default-chain token-file logic.
   • Apply the same helper/provider construction to explicit CredProviderType::Container paths in BE and cloud code. Audit other C++ container-credential call sites so behavior is consistent.
   • Add a focused unit test with a loopback credential endpoint and a temporary authorization-token file: accept token A for the first short-lived credential response, replace the file with token B, force/await provider refresh, and verify the next request sends B and returns refreshed credentials. Also retain coverage for ECS relative URI and the direct AWS_CONTAINER_AUTHORIZATION_TOKEN form.
   • After the master fix is verified, backport it to 4.1. Restarting the BE pod is only a temporary operational workaround because it reloads the initial token but does not make rotation safe.

   Suggested triage: kind/fix plus the repository's BE/object-storage label, and assign the S3 credential-provider owner.
2. hello-stephen

   added a commit that references this issue on Aug 21, 2026

   [chore](thirdparty) Upgrade aws-sdk-cpp from 1.11.219 to 1.11.221 (#6…

   ...

   5202d06
3. morningman

   closed this as completedin #67076on Sep 3, 2026
4. morningman

   added a commit that references this issue on Sep 3, 2026

   [fix](be): fix BE access to aws under eks pod identity (#67076)

   ...

   This issue will close once commit d34f06b is merged into the 'master' branch.

   d34f06b