Problem

Cloud Client Libraries (including Cloud Monitoring API v3) support Application Default Credentials and should use the App Engine service account to authenticate against other services. This does not appear to work as intended|documented:

com.google.api.gax.rpc.UnauthenticatedException: io.grpc.StatusRuntimeException: UNAUTHENTICATED

A colleague provided a hack that obtains an access token using the App Engine service account and injects this into the credentials provided to the Cloud Monitoring API. This works but is unwieldy (see below) and should be unnecessary.

Repro

-- Maven generate an App Engine Standard J8 app
-- Mash-up w/ Google provided Custom Metric sample
-- Observe UNAUTHENTICATED problems auth'ing with MetricServiceClient.create()

Solution

Caveat: I do not understand why this code works... it does

-- Revise MetricServiceClient.create() with:

List<String> scopes = Arrays.asList("https://www.googleapis.com/auth/monitoring");

AppIdentityService appIdentityService = AppIdentityServiceFactory
    .getAppIdentityService();

String access_token = appIdentityService
    .getAccessToken(scopes)
    .getAccessToken();

AppEngineCredentials credentials = AppEngineCredentials
    .newBuilder()
    .setAppIdentityService(appIdentityService)
    .setScopes(scopes)
    .build();

CredentialsProvider credentialsProvider = FixedCredentialsProvider.create(credentials);

MetricServiceSettings metricServiceSettings =
    MetricServiceSettings.newBuilder()
        .setCredentialsProvider(credentialsProvider)
        .build();
MetricServiceClient metricServiceClient = MetricServiceClient
    .create(metricServiceSettings);        

Here:
https://gist.github.com/DazWilkin/05b1a2ed702e78019e20e862df274129#file-testservlet-java-L52-L74

Thanks @salrashid123 for providing the solution.