[FEATURE REQUEST] Add Support for S3A prefix #214
Description
Is your feature request related to a problem? Please describe.
I have set the allowed location of the created catalog to S3 storage type using s3:// prefix. When I run remove_orphan_files procedure in Spark, it results in an error message: No FileSystem for scheme "s3". To solve this problem, I attempted to create the catalog with the s3a:// prefix, but I received a 400 Bad Request error with the message: Location prefix not allowed.
Here's my spark configuration
spark = SparkSession.builder \
.config("spark.jars.packages","org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2,org.apache.iceberg:iceberg-aws-bundle:1.5.2,org.apache.hadoop:hadoop-aws:3.4.0,org.apache.hadoop:hadoop-common:3.4.0") \
.config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
.config('spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation', 'true') \
.config("spark.sql.catalog.polaris.uri", POLARIS_URI) \
.config("spark.sql.catalog.polaris.type", "rest") \
.config("spark.sql.catalog.polaris", "org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.polaris.warehouse", POLARIS_CATALOG_NAME) \
.config("spark.sql.catalog.polaris.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \
.config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
.config('spark.sql.catalog.polaris.credential', POLARIS_CREDENTIALS) \
.config('spark.sql.catalog.polaris.scope', POLARIS_SCOPE) \
.config('spark.sql.catalog.polaris.token-refresh-enabled', 'true') \
.getOrCreate()
Describe the solution you'd like
Probably add the s3a:// prefix as an alternative for the S3 storage type.
Describe alternatives you've considered
No response
Additional context
No response