Native Parquet S3 scan fails on credential provider classes that Spark/Hadoop accept #6022

What is the problem the feature request solves?

When Comet's native Parquet reader is enabled, an S3 query fails if fs.s3a.aws.credentials.provider names a provider class that Comet doesn't recognize, even though plain Spark runs the same query fine. The error looks like:

org.apache.comet.CometNativeException: General execution error with reason:
Generic S3 error: Unsupported credential provider:
com.amazonaws.auth.DefaultAWSCredentialsProviderChain

Seen on Spark 3.4.3 / Hadoop 3.3.4. The task retries and the stage aborts.

Why it happens

Comet's native readers don't go through Hadoop S3A. They resolve S3 credentials in Rust and only understand a fixed list of provider class names. Any class not on that list is rejected. Plenty of valid, widely-used classes aren't on it — the AWS default chain, the web-identity wrapper, custom providers — so config that works in Spark breaks under Comet.

Iceberg reads don't hit this because that path falls back to a default credential chain instead of failing. So on the same cluster, Iceberg reads succeed while Parquet reads fail.

Impact

• Clusters using a provider class outside Comet's list can't use the native Parquet scan.
• Behavior differs from Spark, and differs between Comet's own Parquet and Iceberg paths.

Describe the potential solution

Comet already has an SPI (CometS3CredentialProvider) for calling JVM credential code from native, but ships no ready-made implementations. Add two built-in ones so standard providers work with a one-line config change, and forward the fs.s3a.* config to the native side so those implementations can read it. This is for the Parquet path only, and is opt-in.

Additional context

• Spark: 3.4.3
• Hadoop: 3.3.4 (AWS SDK v1)
• fs.s3a.aws.credentials.provider = com.amazonaws.auth.DefaultAWSCredentialsProviderChain