You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg write silently drops S3 settings it cannot honour (credentials provider, SSE, ACL, tags, remote signing) instead of falling back #6139
The native Iceberg write gate admits s3 / s3a data locations without checking whether the S3 settings in effect can be honoured by the native writer. Settings it cannot honour are dropped silently.
What reaches the native writer:
From the Hadoop configuration, only six fs.s3a.* settings are translated: access key, secret key, session token, endpoint, region and path-style access (NativeConfig.scala:62-68, via hadoopToIcebergS3Properties at CometIcebergNativeWrite.scala:684-687).
FileIO properties are forwarded as-is (CometIcebergNativeWrite.scala:594-595), and load_file_io passes on every s3. / client. key (iceberg_common.rs:39, :122). iceberg-rust ignores keys it does not know.
Credentials otherwise come from Comet's own credential-provider bridge when one is configured, or from opendal's default chain.
Settings that are silently lost include:
Hadoop S3A: fs.s3a.aws.credentials.provider, fs.s3a.encryption.algorithm / fs.s3a.encryption.key (SSE-KMS, SSE-C), fs.s3a.assumed.role.*, and EMRFS settings for s3://
A write that works with iceberg-java fails with an access error, because the native writer uses different credentials.
The write succeeds, but the objects are missing the encryption, ACL, tags or storage class the table owner configured. Nothing reports this.
The gs:// path already fails closed for this reason (#5637). S3 does not.
Steps to reproduce
Configure an S3A write that needs a non-default credentials provider or SSE-KMS through fs.s3a.encryption.*, enable spark.comet.iceberg.write.enabled, and insert into an Iceberg table on s3a://. The plan shows CometIcebergWriteExec, and the resulting objects are written without the configured encryption (or the write fails on credentials).
Expected behavior
Fail closed. The gate should decline the native write, with a reason, when any Hadoop fs.s3a.* or FileIO s3.* / client.* setting outside an allow-list of settings the native writer honours is set. Settings could then be added to the allow-list one at a time as the native writer gains support.
Additional context
Found in an audit of the native Iceberg write path before enabling it by default. Part of #5649. Related: #5643 (keep-or-lift per eligibility restriction), #5637.
Activity
sam-1112 commentedon Sep 24, 2026
take