[Parquet] Support async Parquet encryption key retrieval #8238

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The current KeyRetriever trait is synchronous, but it is common for implementations to need to make network calls when decrypting keys, for example to a key management service (KMS). Currently this will require blocking a thread which is not desirable in an async application.

For compatibility with the synchronous API, we should continue to allow the use of a synchronous KeyRetriever.

Describe the solution you'd like

My proposed solution is to add a new interface, AsyncKeyRetriever, and allow FileDecryptionProperties to hold either a KeyRetriever or an AsyncKeyRetriever. New code paths will be added to use the async retriever. When used via synchronous code, an error will be raised if an AsyncKeyRetriever has been set.

I'm concerned this might require duplicating a lot of existing synchronous code, but hopefully that can be minimized.

Describe alternatives you've considered

• Don't change anything and require blocking. The impact of this may not be too bad when using double wrapping, so interactions with the KMS are infrequent.
• Make KeyRetriever async and don't allow using it from sync code (suggested in Add Key Management Tools API for Parquet encryption #7387 (comment) by @tustvold). This is a bit tidier by not having two similar traits, but I don't think it would simplify the implementation all that much and it would be nice to continue to allow a synchronous KeyRetriever for users using the synchronous APIs.