Environment details
- OS: Ubuntu 14.04.5 LTS
- Java version: build 1.8.0_151-b12
- google-auth-library-java version(s): 0.10.0
We get the auth libraries as a transitive dependency from com.google.cloud:google-cloud-nio:jar:shaded:0.59.0-alpha
Here's the relevant section from mvn depedency:tree
com.google.cloud:google-cloud-nio:jar:shaded:0.59.0-alpha:test
[INFO] +- com.google.cloud:google-cloud-storage:jar:1.41.0:test
[INFO] | +- com.google.cloud:google-cloud-core:jar:1.41.0:test
[INFO] | | +- joda-time:joda-time:jar:2.9.2:test
[INFO] | | +- com.google.http-client:google-http-client:jar:1.24.1:test
[INFO] | | +- com.google.api:api-common:jar:1.7.0:test
[INFO] | | +- com.google.api:gax:jar:1.30.0:test
[INFO] | | | - org.threeten:threetenbp:jar:1.3.3:test
[INFO] | | +- com.google.protobuf:protobuf-java-util:jar:3.6.0:test
[INFO] | | +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:test
[INFO] | | - com.google.api.grpc:proto-google-iam-v1:jar:0.12.0:test
[INFO] | +- com.google.cloud:google-cloud-core-http:jar:1.41.0:test
[INFO] | | +- com.google.auth:google-auth-library-credentials:jar:0.10.0:test
[INFO] | | +- com.google.auth:google-auth-library-oauth2-http:jar:0.10.0:test
[INFO] | | +- com.google.oauth-client:google-oauth-client:jar:1.24.1:test
[INFO] | | +- com.google.api-client:google-api-client:jar:1.24.1:test
[INFO] | | +- com.google.http-client:google-http-client-appengine:jar:1.24.1:test
[INFO] | | +- com.google.http-client:google-http-client-jackson:jar:1.24.1:test
[INFO] | | +- com.google.http-client:google-http-client-jackson2:jar:1.24.1:test
[INFO] | | +- com.google.api:gax-httpjson:jar:0.47.0:test
[INFO] | | +- io.opencensus:opencensus-api:jar:0.15.0:test
[INFO] | | | - io.grpc:grpc-context:jar:1.12.0:test
[INFO] | | - io.opencensus:opencensus-contrib-http-util:jar:0.15.0:test
[INFO] | - com.google.apis:google-api-services-storage:jar:v1-rev135-1.24.1:test
[INFO] +- com.google.guava:guava:jar:20.0:provided
[INFO] - javax.inject:javax.inject:jar:1:test
Steps to reproduce
This may be hard for you to reproduce without a lot of effort, we found it by running our test suite for disq on travis-ci
- Run test suite from disq on travis without "NO_GCE_TEST=true"
Stacktrace
com.google.cloud.storage.StorageException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:220)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:415)
at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:198)
at com.google.cloud.storage.StorageImpl$5.call(StorageImpl.java:195)
at shaded.cloud_nio.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:195)
at com.google.cloud.storage.StorageImpl.get(StorageImpl.java:209)
at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.readAttributes(CloudStorageFileSystemProvider.java:718)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.Files.isDirectory(Files.java:2192)
at org.disq_bio.disq.impl.file.NioFileSystemWrapper.isDirectory(NioFileSystemWrapper.java:69)
at org.disq_bio.disq.HtsjdkReadsRddStorage.read(HtsjdkReadsRddStorage.java:105)
at org.disq_bio.disq.HtsjdkReadsRddStorage.read(HtsjdkReadsRddStorage.java:84)
at org.disq_bio.disq.HtsjdkReadsRddTest.testReadAndWrite(HtsjdkReadsRddTest.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at junitparams.internal.InvokeParameterisedMethod.evaluate(InvokeParameterisedMethod.java:234)
at junitparams.internal.ParameterisedTestMethodRunner.runMethodInvoker(ParameterisedTestMethodRunner.java:47)
at junitparams.internal.ParameterisedTestMethodRunner.runTestMethod(ParameterisedTestMethodRunner.java:40)
at junitparams.internal.ParameterisedTestClassRunner.runParameterisedTest(ParameterisedTestClassRunner.java:146)
at junitparams.JUnitParamsRunner.runChild(JUnitParamsRunner.java:446)
at junitparams.JUnitParamsRunner.runChild(JUnitParamsRunner.java:393)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.io.IOException: Error code 404 trying to get security access token from Compute Engine metadata for the default service account. This may be because the virtual machine instance does not have permission scopes specified.
at shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials.refreshAccessToken(ComputeEngineCredentials.java:151)
at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:179)
at shaded.cloud_nio.com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata(OAuth2Credentials.java:165)
at shaded.cloud_nio.com.google.auth.http.HttpCredentialsAdapter.initialize(HttpCredentialsAdapter.java:96)
at com.google.cloud.http.HttpTransportOptions$1.initialize(HttpTransportOptions.java:161)
at com.google.cloud.http.CensusHttpModule$CensusHttpRequestInitializer.initialize(CensusHttpModule.java:120)
at shaded.cloud_nio.com.google.api.client.http.HttpRequestFactory.buildRequest(HttpRequestFactory.java:93)
at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest(AbstractGoogleClientRequest.java:300)
at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
at shaded.cloud_nio.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.get(HttpStorageRpc.java:412)
... 47 more
Code snippet
I don't have a self contained code snippet to try, we can probably produce a small test case if you need one. I'm not sure if the right resolution is "add a note about NO_GCE_CHECK" to the error message or if it needs something more involved.
Any additional information below
Our tests access public data on google cloud without authenticating. This was working fine for a while and then suddenly stopped working. I'm not sure if it was a change in the google auth environment or the travis environment that caused it to stop.
We noticed the tests starting to fail here: disq-bio/disq#47 and resolved the problem by adding NO_GCE_TEST=true in disq-bio/disq#54.
Based on #110 and #109 it seems like the NO_GCE_CHECK option was added to deal with intermittent failures and this wasn't expected to be a repeatable problem. We see the problem consistently so there may be a deeper issue.
One potentially interesting thing here is that the travis VM's may be running on google cloud:
5b3eb94a-00c5-440a-ad5a-06c17b321e5e@1.production-2-worker-org-gce-8znl sounds like it might be a gce node to me?
Since we have a workaround this isn't a critical issue. Finding the workaround took us some time though. It would have been useful if the error message included a note about NO_GCE_CHECK.
Environment details
We get the auth libraries as a transitive dependency from com.google.cloud:google-cloud-nio:jar:shaded:0.59.0-alpha
Here's the relevant section from
mvn depedency:treecom.google.cloud:google-cloud-nio:jar:shaded:0.59.0-alpha:test
[INFO] +- com.google.cloud:google-cloud-storage:jar:1.41.0:test
[INFO] | +- com.google.cloud:google-cloud-core:jar:1.41.0:test
[INFO] | | +- joda-time:joda-time:jar:2.9.2:test
[INFO] | | +- com.google.http-client:google-http-client:jar:1.24.1:test
[INFO] | | +- com.google.api:api-common:jar:1.7.0:test
[INFO] | | +- com.google.api:gax:jar:1.30.0:test
[INFO] | | | - org.threeten:threetenbp:jar:1.3.3:test
[INFO] | | +- com.google.protobuf:protobuf-java-util:jar:3.6.0:test
[INFO] | | +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:test
[INFO] | | - com.google.api.grpc:proto-google-iam-v1:jar:0.12.0:test
[INFO] | +- com.google.cloud:google-cloud-core-http:jar:1.41.0:test
[INFO] | | +- com.google.auth:google-auth-library-credentials:jar:0.10.0:test
[INFO] | | +- com.google.auth:google-auth-library-oauth2-http:jar:0.10.0:test
[INFO] | | +- com.google.oauth-client:google-oauth-client:jar:1.24.1:test
[INFO] | | +- com.google.api-client:google-api-client:jar:1.24.1:test
[INFO] | | +- com.google.http-client:google-http-client-appengine:jar:1.24.1:test
[INFO] | | +- com.google.http-client:google-http-client-jackson:jar:1.24.1:test
[INFO] | | +- com.google.http-client:google-http-client-jackson2:jar:1.24.1:test
[INFO] | | +- com.google.api:gax-httpjson:jar:0.47.0:test
[INFO] | | +- io.opencensus:opencensus-api:jar:0.15.0:test
[INFO] | | | - io.grpc:grpc-context:jar:1.12.0:test
[INFO] | | - io.opencensus:opencensus-contrib-http-util:jar:0.15.0:test
[INFO] | - com.google.apis:google-api-services-storage:jar:v1-rev135-1.24.1:test
[INFO] +- com.google.guava:guava:jar:20.0:provided
[INFO] - javax.inject:javax.inject:jar:1:test
Steps to reproduce
This may be hard for you to reproduce without a lot of effort, we found it by running our test suite for disq on travis-ci
Stacktrace
Code snippet
I don't have a self contained code snippet to try, we can probably produce a small test case if you need one. I'm not sure if the right resolution is "add a note about NO_GCE_CHECK" to the error message or if it needs something more involved.
Any additional information below
Our tests access public data on google cloud without authenticating. This was working fine for a while and then suddenly stopped working. I'm not sure if it was a change in the google auth environment or the travis environment that caused it to stop.
We noticed the tests starting to fail here: disq-bio/disq#47 and resolved the problem by adding
NO_GCE_TEST=truein disq-bio/disq#54.Based on #110 and #109 it seems like the
NO_GCE_CHECKoption was added to deal with intermittent failures and this wasn't expected to be a repeatable problem. We see the problem consistently so there may be a deeper issue.One potentially interesting thing here is that the travis VM's may be running on google cloud:
5b3eb94a-00c5-440a-ad5a-06c17b321e5e@1.production-2-worker-org-gce-8znlsounds like it might be a gce node to me?Since we have a workaround this isn't a critical issue. Finding the workaround took us some time though. It would have been useful if the error message included a note about
NO_GCE_CHECK.