Opening and closing several Spanner instances during the lifetime of an application can cause a thread leak, as the method Spanner.close() does not clean up all threads that are created.

A Spanner instance contains three underlying gRPC stubs: SpannerStub, InstanceAdminStub and DatabaseAdminStub. In order to prevent these from using the same thread pool for gRPC calls, the stubs are handed a custom ExecutorProvider. This happens here:

The InstantiatingGrpcChannelProvider will create the underlying transport channels and assign these executors that are gotten from the above ExecutorProvider here: https://github.com/googleapis/gax-java/blob/13564bbe38e5496821e4686ab629181c5bb4ac66/gax-grpc/src/main/java/com/google/api/gax/grpc/InstantiatingGrpcChannelProvider.java#L178

This executor is then passed on to a ManagedChannelBuilder (or actually a NettyChannelBuilder which extends AbstractManagedChannelImplBuilder). This latter builder wraps the executor in a FixedObjectPool which always returns this executor when one is requested from the 'pool', and does nothing when it is returned to the pool. This means that the underlying executor is never shutdown when the transport channel is shutdown.

Removing the setExecutorProvider call on line 193 of GapicSpannerRpc solves the resource leak, but causes the underlying gRPC stubs to use a shared executor for their transport channels, possibly causing race conditions.