Java SDK does not support EKS IAM for service accounts

EKS IAM Service Account Role introduces a new environment variable "AWS_WEB_IDENTITY_TOKEN_FILE" and based on the documentation on these two pages, the Java SDK should use "AWS_WEB_IDENTITY_TOKEN_FILE" for credentials if exists.

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#pod-configuration

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

I have tried the latest Java SDK 2.9.19 and it doesn't seem to work. There is a similar discussion on DotNet SDK at here:

aws/aws-sdk-net#1413

I couldn't find docs saying "AWS_WEB_IDENTITY_TOKEN_FILE" is in the list of credentials chain for SDK Java 2. I am wondering if this is implemented or not.

Expected Behavior

Based on the EKS doc link above, Java SDK should recognize the environment variable "AWS_WEB_IDENTITY_TOKEN_FILE" and use it to call AssumeRoleWithWebIdentity for access/secret/session tokens.

Current Behavior

I am getting Access Denied with Java SDK in a correctly setup EKS pod with service account. In the same pod, I was able to run aws s3 ls and it worked, which means the token is correct.

Exception in thread "main" software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: 467C66AAEAD354A9)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleErrorResponse(HandleResponseStage.java:115)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.handleResponse(HandleResponseStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:58)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:41)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.doExecute(RetryableStage.java:113)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:86)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:240)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:96)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:120)
        at software.amazon.awssdk.core.client.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:73)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:44)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.s3.DefaultS3Client.listBuckets(DefaultS3Client.java:2035)
        at com.example.s3.S3BucketOps.main(S3BucketOps.java:63)

Steps to Reproduce (for bugs)

The code I am using to test is from sample S3 code at here:
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/S3BucketOps.java

I set up a pod in EKS with service account and IAM role properly setup. In the pod, I used aws cli to test the token and it worked. However, the Java SDK didn't work.

Your Environment

AWS Java SDK version used: 2.9.19
JDK version used: 1.8
Operating System and version: Linux running from a container (pod on EKS)

Having the same issue using the AWS Java SDK version: 1.11.653.

JDK version used: 11
Operating System and version: Linux running from a container (pod on EKS)
Kubernetes version 1.14

According to the documentation this version of the SDK should also work fine. My application uses the role of the node instead of the one I have added through a service account.

@endre-synnes

@endre-synnes The issue is due to WebIdentityTokenFileCredentialsProvider is not in the default credentials provider chain

The workaround for now is to specify the provider in the client initialization:

import software.amazon.awssdk.auth.credentials.WebIdentityTokenFileCredentialsProvider;
...
S3Client s3 = S3Client.builder()
                              .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create())
                              .region(region)
                              .build();

Looking forward that WebIdentityTokenFileCredentialsProvider to be added to the default credentials provider chain.

Address confusion due to some SDKs don't support this feature out of box

@starchx

@starchx Thank you! That solved the issue😄 Hope that the WebIdentityTokenFileCredentialsProvider will be added as a default soon!

Hi all, this should be part of the default chain. We will prepare a change for this.

@endre-synnes

@endre-synnes I am also experiencing the same problem despite the documentation saying it should work. We should file an issue at https://github.com/aws/aws-sdk-java/issues

Fixed via #1501 and released as part of 2.10.11

Hi,
It still doesn't work for me with 2.10.57. The sdk uses node role instead of service account role.

UPDATE
It works. I just had to add

      securityContext:
        fsGroup: 65534

to the pod spec in order to make sure containers have access to the service token file.

Hi, It still doesn't work for me with 2.10.72.
It seems that java sdk doesn't get credentials from AWS_WEB_IDENTITY_TOKEN_FILE, AWS_ROLE_ARN envs.
(FYI, I already added security context to pods to be able to access the token file)

@imcheck

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

@imcheck

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

Not really.

When I try to run s3 client, it says WebIdentityCredentialProvider needs sts been loaded into class path
So I searched and get StsWebIdentityCredentialsProviderFactory.java

Then I included

		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>sts</artifactId>
		</dependency>

It just works :)

Hi Jiang,

Could you please let me the code snippet you have used with StsWebIdentityCredentialsProviderFactory.java

Hi Jiang,

Could you please let me the code snippet you have used with StsWebIdentityCredentialsProviderFactory.java

S3Client s3 = S3Client.builder()
                              .region(region)
                              .build();

and include sts with the latest version from maven repo, today it is 2.11.14

		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>sts</artifactId>
		</dependency>

Thank you Jiang. This was conflicting with aws sdk jar I was using. Latest versions of sdk and sts helped me.

This is an interesting discussion and somewhat related to a use case I am trying to figure out. My service needs to access resources in a different AWS account from EKS, so I want to use profiles and assume role using a config like this:

[profile eks-account-role]
role_arn = <eks-account-role-arn>
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token

[profile other-account-role]
role_arn = <other-account-role-arn>
source_profile = eks-account-role
role_session_name = x-account-session

Using the following code to access the STS client (for other purposes than assuming roles).

stsClient = StsClient.builder().build();

Then I set the environment variable AWS_PROFILE=other-account-role in the container.

But when I go to run it, the service is still running as the eks-account-role. I'm assuming the default credential provider chain will pick up the AWS profile defined in the config and use that method for authentication, but it doesn't seem to be happening. However, when I try this with the AWS CLI it works as expected (I have properly setup the other role to be assumed by the eks role).

I have found a workaround where if I specify to use profile credentials explicitly it works as expected.

stsClient = StsClient.builder()
                .credentialsProvider(ProfileCredentialsProvider.create())
                .build();

Now this isn't a big deal as it is a small amount of additional code, but I am curious what the SDK is doing under the hood and why it is different from the CLI. A couple things to consider, EKS seems to be injecting two environment variables into the container that could be effecting the behavior AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. However, I am running the CLI from within the container with the same variables present.

Thoughts?

@imcheck

@imcheck make sure you have aws-java-sdk-sts dependency packaged for your application as well.

Not really.

When I try to run s3 client, it says WebIdentityCredentialProvider needs sts been loaded into class path
So I searched and get StsWebIdentityCredentialsProviderFactory.java

Then I included
		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>sts</artifactId>
		</dependency>
It just works :)

Surprised that this works, but it does. Any idea why the separate declaration of sts is required?

I still get an error trying to connect to Secrets Manager from EKS using Service Account role.
I'm using sdk v2.16.52.

software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Not authorized to perform sts:AssumeRoleWithWebIdentity (Service: Sts, Status Code: 403, Request ID: 13e02bf6-082c-47a2-ace4-900b02792b72, Extended Request ID: null), ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.] at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:112) at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45) at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:104) at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.createExecutionContext(AwsSyncClientHandler.java:68) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:99) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:169) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:95) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) at software.amazon.awssdk.services.secretsmanager.DefaultSecretsManagerClient.getSecretValue(DefaultSecretsManagerClient.java:1098) at com.hlpp.csw.util.SecretManagerUtil.getSecretFromSM(SecretManagerUtil.java:51) at com.hlpp.csw.config.ShiroCustomEnvironment.setIniOverrides(ShiroCustomEnvironment.java:23) at com.hlpp.csw.config.ShiroCustomEnvironment.init(ShiroCustomEnvironment.java:15) at org.apache.shiro.util.LifecycleUtils.init(LifecycleUtils.java:45) at org.apache.shiro.util.LifecycleUtils.init(LifecycleUtils.java:40) at org.apache.shiro.web.env.EnvironmentLoader.createEnvironment(EnvironmentLoader.java:313) at org.apache.shiro.web.env.EnvironmentLoader.initEnvironment(EnvironmentLoader.java:139) at org.apache.shiro.web.env.EnvironmentLoaderListener.contextInitialized(EnvironmentLoaderListener.java:58) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4716) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5172) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:692) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1023) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1903) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:824) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:474) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1611) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:319) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936) at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:829) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardService.startInternal(StandardService.java:433) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) at org.apache.catalina.startup.Catalina.start(Catalina.java:772) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)

still happens

SDK version: software.amazon.awssdk:ssm:2.17.19
JDK version used: Java 11
Operating System and version: openjdk:11.0.11-jdk-slim
EKS version: 1.21

only after I added software.amazon.awssdk:sts:2.17.19 fixed

Hi,
We're still on sdk v1, anyone who's able to workaround this on v1? Using the following dependencies:

aws-java-sdk-core (v1.12.239)
aws-java-sdk-sns (v1.11.125)

@itowk

@itowk. i don't know if it'll solve your issue, but definitely don't use misaligned versions of the sdk. there's a bom for a reason.

Hi, We're still on sdk v1, anyone who's able to workaround this on v1? Using the following dependencies:

aws-java-sdk-core (v1.12.239)

aws-java-sdk-sns (v1.11.125)

Adding the latest v1 version (1.11.1034 today's date) for the aws-java-sdk-sts dependency solved my issue (no code needed to be touched using the method DefaultAWSCredentialsProviderChain(). I hope it helps

Navigation Menu

Repository navigation

Java SDK does not support EKS IAM for service accounts #1470

Expected Behavior

Current Behavior

Steps to Reproduce (for bugs)

Your Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Java SDK does not support EKS IAM for service accounts #1470

Description

Expected Behavior

Current Behavior

Steps to Reproduce (for bugs)

Your Environment

Activity

endre-synnes commented on Oct 17, 2019

starchx commented on Oct 18, 2019

endre-synnes commented on Oct 21, 2019

dagnir commented on Oct 23, 2019

thundergolfer commented on Oct 29, 2019

zoewangg commented on Nov 7, 2019

thereisnospoon commented on Feb 25, 2020

imcheck commented on Mar 19, 2020

thereisnospoon commented on Mar 19, 2020

jiang-wei commented on Apr 20, 2020

gargkshitiz commented on Apr 20, 2020

jiang-wei commented on Apr 20, 2020

gargkshitiz commented on Apr 20, 2020

campbel commented on Jun 10, 2020

rod-dvla commented on Dec 9, 2020

StealthyDev commented on May 28, 2021

llnformer commented on Aug 17, 2021

itowk commented on Aug 1, 2022

madorb commented on Aug 26, 2022

maufranchini commented on Nov 13, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions