Repository navigation
Support for EKS 'IAM for service accounts' not default [1.11.623, 1.11.653] #2136
Description
Description
Despite the documentation saying a minimum version of 1.11.623 should allow IAM Service Accounts to works, I'm running pods and seeing that that run with the IAM role of the node and not the role linked to the IAM ServiceAccount I am running as.
Another using describes the same problem with 1.11.653 here.
I have discovered a hack that fixes the issue:
// NOTE: For some unknown reason if this call isn't made then the following code
// uses the K8s node's role and not the IAM ServiceAccount role.
// TODO(Jonathon): Remove when the above weirdness is fixed.
val client = AWSSecurityTokenServiceClientBuilder.standard.build
val request = new GetCallerIdentityRequest()
val _ = client.getCallerIdentity(request)
That above identity request returns the correct role, and then subsequently doing something like this, AmazonSNSAsyncClientBuilder.defaultClient(), will get me a client authenticated to the IAM role associated with the service account.
Reacted by Roman Frey, Malte , Jhonatan Morais, Jonathon Belotti and Iury Alves de Souza
Activity
thundergolfer commentedon Oct 30, 2019
Something like this works:
So it's just a problem that
WebIdentityTokenCredentialsProviderisn't in the default toolchain.[-]Java SDK support for EKS 'IAM for service accounts' not working [1.11.623, 1.11.653][/-][+]Support for EKS 'IAM for service accounts' not default [1.11.623, 1.11.653][/+]zoewangg commentedon Nov 7, 2019
Hi @thundergolfer
WebIdentityTokenCredentialsProvideris in the default credential provider chain in v1 though.aws-sdk-java/aws-java-sdk-core/src/main/java/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.java
Lines 45 to 50 in 5bafd0d
Could it be possible that you had some other credentials providers available on your credential chain and took precedence?
thundergolfer commentedon Nov 7, 2019
Hey @zoewangg thanks for the reply,
Yeh you're right seems it landed in the default chain in
.620. I'm 95% sure I had no code that was configuring any providers. I was just requesting the default clients for SNS, S3 and leaving it up to the default chain.Given that someone else has seen this is, I'm confident something is going on here.
Do you have any advice on how I would best track this down? Perhaps simply enabling
DEBUGlevel logs would show behaviour of the provider chain?zoewangg commentedon Nov 7, 2019
Yeah, it'd be great if you could enable debug logging and provide the logs here (Be sure to redact any sensitive info from the logs)
anoopk19 commentedon Nov 10, 2019
I am debugging a similar issue. AWS SDK fails to recognize
AWS_WEB_IDENTITY_TOKEN_FILEenvironment variable and defaults toEC2ContainerCredentialsProviderWrapper.SDK version used is :
aws-sdk-java/1.11.671Debug logs captured below,
thundergolfer commentedon Nov 10, 2019
In those logs
WebIdentityisn't present at all, and it seems you're on the latest version. It's skipping over it fromProfileCredentialstoEC2ContainerCredentials...What's that on the end here ->
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@44e06fc8. Is it a commit hash?chriswilding commentedon Nov 14, 2019
I think I'm experiencing the same issue or something similar. Yesterday I updated my application to use a service account and also updated the aws-java-sdk to v1.11.673. Adding my logs here in case they help.
The app fails to deploy to EKS and hits CrashLoopBackOff. In the logs I can't see WebIdentityTokenCredentialsProvider -
{"@timestamp":"2019-11-14T10:47:56.466+00:00","@version":"1","message":"Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.466+00:00","@version":"1","message":"Unable to load credentials from SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.466+00:00","@version":"1","message":"Unable to load credentials from com.amazonaws.auth.profile.ProfileCredentialsProvider@4287d447: profile file cannot be null","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.472+00:00","@version":"1","message":"Unable to load configuration from com.amazonaws.monitoring.EnvironmentVariableCsmConfigurationProvider@54e7391d: Unable to load Client Side Monitoring configurations from environment variables!","logger_name":"com.amazonaws.monitoring.CsmConfigurationProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.472+00:00","@version":"1","message":"Unable to load configuration from com.amazonaws.monitoring.SystemPropertyCsmConfigurationProvider@50b8ae8d: Unable to load Client Side Monitoring configurations from system properties variables!","logger_name":"com.amazonaws.monitoring.CsmConfigurationProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.472+00:00","@version":"1","message":"Unable to load configuration from com.amazonaws.monitoring.ProfileCsmConfigurationProvider@3c8bdd5b: Unable to load config file","logger_name":"com.amazonaws.monitoring.CsmConfigurationProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.479+00:00","@version":"1","message":"Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.479+00:00","@version":"1","message":"Unable to load credentials from SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.480+00:00","@version":"1","message":"Unable to load credentials from com.amazonaws.auth.profile.ProfileCredentialsProvider@4287d447: profile file cannot be null","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.480+00:00","@version":"1","message":"Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.480+00:00","@version":"1","message":"Unable to load credentials from SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} {"@timestamp":"2019-11-14T10:47:56.480+00:00","@version":"1","message":"Unable to load credentials from com.amazonaws.auth.profile.ProfileCredentialsProvider@4287d447: profile file cannot be null","logger_name":"com.amazonaws.auth.AWSCredentialsProviderChain","thread_name":"main","level":"DEBUG","level_value":10000} ... this repeats many timesI also get a StackOverflowError with the stack trace after the
AWSCredentialsProviderChainrepeated many times. The stack trace does include reference the WebIdentityTokenCredentialsProvider -ConstantineXVI commentedon Nov 15, 2019
When we implemented IRSA for our Java apps, we found that as configured in the default chain instance profile creds outrank WebIdentity; whereas in the Ruby and Go SDKs WebIdentity took precedence. Since the kubelet auths with IAM there's no way to not have an instance profile, and we don't have the option of blocking the metadata API for pods until every last service in our cluster is using IRSA. Unless the Java SDK changes its precedence order to match the other SDKs we're going to need to provide our own chains in all of our Java apps, which is less than ideal.
chriswilding commentedon Nov 26, 2019
I was experiencing a StackOverflowError becuase another library was pulling in an older version of aws-java-sdk-sts which used to initialise a default credentials provider internally, updating this resolved the StackOverflowError.
Once that issue was resolved, as mentioned above we had to provide a credential chain to give WebIdentityTokenCredentialsProvider precedence. It would be helpful if the java sdk's default credentials provider precedence order was changed to give the WebIdentityTokenCredentialsProvider higher precedence.
11 remaining items
jhonatanmorais commentedon Jan 23, 2020
@debora-ito:
bqnguyen94 commentedon Jan 28, 2020
@jhonatanmorais so I assume you have tried
WebIdentityTokenCredentialsProvider.create()and it did not work as well?jhonatanmorais commentedon Jan 28, 2020
@bqnguyen94 I've tried only those two scenarios mentioned above.
jhonatanmorais commentedon Feb 3, 2020
@bqnguyen94, did you test using
defaultClient()implementation?bqnguyen94 commentedon Feb 4, 2020
We built
AWSSecretsManagerclients withstandard()so that we can specify the region, thenbuild(), so I assume it is the same asdefaultClient()jhonatanmorais commentedon Feb 8, 2020
I've just upgraded to
1.11.717and everything is fine now!SMR39 commentedon Feb 19, 2020
@jhonatanmorais
1.11.717worked without code changes ?we use standard() which is same as default() as you said but the app still doesnt use service account iam role but IAM role of the node instead :/
bqnguyen94 commentedon Feb 19, 2020
Actually I had to also include
aws-java-sdk-stsin the pom file for it to work. I'm still at 1.11.704 though.jhonatanmorais commentedon Feb 26, 2020
Yes. Did you follow the last recommendation?
bump aws sdk version (#77)
Add aws-sdk-java-sts, for container auth
Add dependency on aws sts to support IAM roles for service accounts
Add dependency on aws sts to support IAM roles for service accounts
Add dependency on aws sts to support IAM roles for service accounts
Add dependency on aws sts to support IAM roles for service accounts