Skip to content

Commit 31b3819

Browse files
authored
fix: decline native Iceberg writes with a custom location provider (#6216)
* fix: decline native Iceberg writes with a custom location provider Inspect table.locationProvider() so a TableOperations-supplied provider cannot bypass the property gate. * docs: explain why the location-provider gate is version-independent Record Iceberg 1.11 executor semantics in the comment and name the three instantiated-state rules in the contributor guide. --------- Co-authored-by: liupoyi-1031 <[email protected]>
1 parent a12fd59 commit 31b3819

6 files changed

Lines changed: 152 additions & 16 deletions

File tree

‎docs/source/contributor-guide/iceberg-writes.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,8 +133,9 @@ The gate has these properties, and a new rule has to keep them:
133133
overlaid with `SparkWrite.writeProperties`, which is where iceberg-java resolves per-write options
134134
and `spark.sql.iceberg.*` session overrides. Reading table properties alone misses those.
135135
- **It checks instantiated state as well as properties.** A catalog or a custom `TableOperations`
136-
can install a `FileIO` or an `EncryptionManager` without any property changing, so
137-
`requireRecognizedTableFileIO` and `requirePlaintextEncryptionManager` look at `table.io()` and
136+
can install a `LocationProvider`, `FileIO`, or `EncryptionManager` without any property changing,
137+
so `requireDefaultLocationProvider`, `requireRecognizedTableFileIO`, and
138+
`requirePlaintextEncryptionManager` inspect `table.locationProvider()`, `table.io()`, and
138139
`table.encryption()` themselves.
139140
- **It fails closed.** A reflection lookup that cannot answer returns a reason, not `None`, and
140141
`getSupportLevel` turns any non-fatal exception into `Unsupported`. `convert` does the same for

‎docs/source/user-guide/latest/iceberg-writes.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,7 @@ A write is eligible only when ALL of the following hold:
172172
| `write.spark.fanout.enabled` | any value (the native writer implements both clustered and fanout modes) |
173173
| `write.target-file-size-bytes` | any value (the two writers can choose different roll points; see accepted divergences) |
174174
| data location URI scheme | `file`, `memory`, `s3`, `s3a`, `gs` (`gs` only when the `FileIO` opening the data location is a `GCSFileIO`; see below) |
175+
| resolved `table.locationProvider()` | Iceberg's built-in `DefaultLocationProvider` |
175176
| partition spec | any |
176177
| column types | any except `uuid` (Spark plans it as a string; no Arrow cast reaches `fixed(16)`) |
177178

@@ -184,12 +185,13 @@ key in the session Hadoop configuration other than the reader-only
184185
iceberg-java's writer but not the native one). Also gated explicitly: any `encryption.*` key,
185186
`write.object-storage.enabled=true`, `write.location-provider.impl`, and `io-impl`.
186187

187-
Two checks look past properties at the table's instantiated state, because both can be
188+
Three checks look past properties at the table's instantiated state, because they can be
188189
configured at the catalog level (or by a custom `TableOperations`) without any table or write
189-
property changing: `table.io()` must be a recognized `FileIO` (the same allowlist the native
190-
scan uses, minus the `EncryptingFileIO` family — the native writer produces plaintext files,
191-
so an encrypting `FileIO` is rejected on the write side), and `table.encryption()` must be
192-
Iceberg's `PlaintextEncryptionManager`. Anything else falls back.
190+
property changing: `table.locationProvider()` must be Iceberg's `DefaultLocationProvider`;
191+
`table.io()` must be a recognized `FileIO` (the same allowlist the native scan uses, minus the
192+
`EncryptingFileIO` family — the native writer produces plaintext files, so an encrypting `FileIO`
193+
is rejected on the write side); and `table.encryption()` must be Iceberg's
194+
`PlaintextEncryptionManager`. Anything else falls back.
193195

194196
A `gs` data location additionally requires that the `FileIO` actually opening it is a
195197
`GCSFileIO` (for a `ResolvingFileIO`, the delegate it instantiates for that location,

‎spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala‎

Lines changed: 27 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,8 @@ object IcebergReflection extends Logging {
5757
val SPARK_STAGED_SCAN = "org.apache.iceberg.spark.source.SparkStagedScan"
5858
val SPARK_SCHEMA_UTIL = "org.apache.iceberg.spark.SparkSchemaUtil"
5959
val TABLE = "org.apache.iceberg.Table"
60+
val DEFAULT_LOCATION_PROVIDER =
61+
"org.apache.iceberg.LocationProviders$DefaultLocationProvider"
6062
val RESOLVING_FILE_IO = "org.apache.iceberg.io.ResolvingFileIO"
6163
val GCS_FILE_IO = "org.apache.iceberg.gcp.gcs.GCSFileIO"
6264
val PARTITIONING = "org.apache.iceberg.Partitioning"
@@ -1371,23 +1373,41 @@ object IcebergReflection extends Logging {
13711373
}
13721374
}
13731375

1374-
def getDataLocation(table: Any): Option[String] =
1376+
/**
1377+
* The table's resolved `LocationProvider` (`table.locationProvider()`). Inspecting the
1378+
* instantiated provider catches custom `TableOperations` that supply one without setting
1379+
* `write.location-provider.impl`. Returns `None` on reflection failure so callers fail closed.
1380+
*/
1381+
def getLocationProvider(table: Any): Option[AnyRef] =
13751382
try {
13761383
val locationProviderMethod =
13771384
findMethodInHierarchy(table.getClass, "locationProvider").getOrElse(
13781385
throw new NoSuchMethodException(
13791386
s"locationProvider() not found on ${table.getClass.getName}"))
1380-
val provider = locationProviderMethod.invoke(table)
1381-
val newDataLocMethod = provider.getClass.getMethod("newDataLocation", classOf[String])
1382-
newDataLocMethod.setAccessible(true)
1383-
val location = newDataLocMethod.invoke(provider, "").asInstanceOf[String]
1384-
Some(location.stripSuffix("/"))
1387+
Option(locationProviderMethod.invoke(table).asInstanceOf[AnyRef])
13851388
} catch {
13861389
case e: Exception =>
1387-
logError(s"Iceberg reflection failure: Failed to get data location: ${e.getMessage}", e)
1390+
logError(
1391+
"Iceberg reflection failure: Failed to get LocationProvider from table: " +
1392+
s"${e.getMessage}",
1393+
e)
13881394
None
13891395
}
13901396

1397+
def getDataLocation(table: Any): Option[String] =
1398+
getLocationProvider(table).flatMap { provider =>
1399+
try {
1400+
val newDataLocMethod = provider.getClass.getMethod("newDataLocation", classOf[String])
1401+
newDataLocMethod.setAccessible(true)
1402+
val location = newDataLocMethod.invoke(provider, "").asInstanceOf[String]
1403+
Some(location.stripSuffix("/"))
1404+
} catch {
1405+
case e: Exception =>
1406+
logError(s"Iceberg reflection failure: Failed to get data location: ${e.getMessage}", e)
1407+
None
1408+
}
1409+
}
1410+
13911411
/**
13921412
* Returns the top-level column names of an Iceberg `Schema`, in declared order. Used by the
13931413
* native write serde to project Spark 4.x `ReplaceData` row streams (which carry an

‎spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeWrite.scala‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,7 @@ object CometIcebergNativeWrite extends CometOperatorSerde[IcebergWriteExec] {
173173
requirePropertyAbsent(
174174
PropertyKeys.WriteLocationProviderImpl,
175175
"custom location provider unsupported"),
176+
requireDefaultLocationProvider,
176177
requireFormatVersionAtMostTwo,
177178
requireNoUuidColumns,
178179
requireNoEncryptionPrefix,
@@ -217,6 +218,29 @@ object CometIcebergNativeWrite extends CometOperatorSerde[IcebergWriteExec] {
217218
if (ctx.properties.contains(key)) Some(s"$key is set ($reason)") else None
218219
}
219220

221+
// The property rule above only sees providers configured through table/write properties. A
222+
// custom TableOperations can return a LocationProvider directly, while the native writer always
223+
// generates `<data location>/<partition path>/<file>`. Admit only Iceberg's default provider;
224+
// object-storage layout is already declined by the preceding property rule.
225+
//
226+
// Before Iceberg 1.11, iceberg-java does not preserve that TableOperations-supplied provider on
227+
// executors: it reconstructs the provider from the table location and properties, so those
228+
// writes use the default layout anyway. From 1.11 on, iceberg-java keeps and uses the custom
229+
// provider. This gate stays unconditional and fail-closed on every Iceberg version Comet pins,
230+
// so a non-default provider always falls back.
231+
private val requireDefaultLocationProvider: TriggerRule = ctx =>
232+
IcebergReflection.getLocationProvider(ctx.table) match {
233+
case None =>
234+
Some("could not resolve table.locationProvider() for native write compatibility checking")
235+
case Some(provider)
236+
if provider.getClass.getName == IcebergReflection.ClassNames.DEFAULT_LOCATION_PROVIDER =>
237+
None
238+
case Some(provider) =>
239+
Some(
240+
s"table.locationProvider() is ${provider.getClass.getName}, " +
241+
"which the native write path would bypass")
242+
}
243+
220244
private val requireFormatVersionAtMostTwo: TriggerRule = ctx =>
221245
IcebergReflection.getFormatVersion(ctx.table) match {
222246
case Some(v) if v >= MinUnsupportedFormatVersion => Some(s"format-version=$v unsupported")

‎spark/src/test/scala/org/apache/comet/CometIcebergWriteDetectionSuite.scala‎

Lines changed: 73 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,11 @@ import org.scalactic.source.Position
2525
import org.scalatest.Tag
2626

2727
import org.apache.hadoop.conf.Configuration
28-
import org.apache.iceberg.hadoop.{HadoopConfigurable, HadoopFileIO}
29-
import org.apache.iceberg.io.{FileIO, InputFile, OutputFile, ResolvingFileIO}
28+
import org.apache.iceberg.{PartitionSpec, StructLike, TableMetadata, TableOperations}
29+
import org.apache.iceberg.catalog.TableIdentifier
30+
import org.apache.iceberg.encryption.EncryptionManager
31+
import org.apache.iceberg.hadoop.{HadoopCatalog, HadoopConfigurable, HadoopFileIO}
32+
import org.apache.iceberg.io.{FileIO, InputFile, LocationProvider, OutputFile, ResolvingFileIO}
3033
import org.apache.iceberg.util.SerializableSupplier
3134
import org.apache.spark.SparkConf
3235
import org.apache.spark.rdd.RDD
@@ -126,6 +129,12 @@ class CometIcebergWriteDetectionSuite extends CometTestBase with CometIcebergTes
126129
"writeProperties")
127130
assert(IcebergReflection.getMetadataLocation(table).isDefined, "metadataLocation")
128131
assert(IcebergReflection.getDataLocation(table).isDefined, "dataLocation")
132+
val provider = IcebergReflection
133+
.getLocationProvider(table)
134+
.getOrElse(fail("locationProvider"))
135+
assert(
136+
provider.getClass.getName == IcebergReflection.ClassNames.DEFAULT_LOCATION_PROVIDER,
137+
provider.getClass.getName)
129138
assert(IcebergReflection.getTableProperties(table).isDefined, "tableProperties")
130139
}
131140
}
@@ -185,6 +194,32 @@ class CometIcebergWriteDetectionSuite extends CometTestBase with CometIcebergTes
185194
}
186195
}
187196

197+
test("fall-back: custom TableOperations LocationProvider that no property reveals") {
198+
withTempIcebergDir { warehouseDir =>
199+
val locationCat = "location_provider_probe_cat"
200+
withSQLConf(
201+
s"spark.sql.catalog.$locationCat" -> "org.apache.iceberg.spark.SparkCatalog",
202+
s"spark.sql.catalog.$locationCat.catalog-impl" ->
203+
classOf[DetectionCustomLocationHadoopCatalog].getName,
204+
s"spark.sql.catalog.$locationCat.warehouse" -> warehouseDir.getAbsolutePath) {
205+
spark.sql(s"""
206+
CREATE TABLE $locationCat.$ns.custom_location_provider (
207+
id INT,
208+
region STRING,
209+
amount DOUBLE
210+
) USING iceberg
211+
""")
212+
val writeExec =
213+
planInsertWriteExec(s"$locationCat.$ns.custom_location_provider")
214+
assertUnsupportedContains(
215+
writeExec,
216+
"custom_location_provider",
217+
"table.locationProvider()",
218+
classOf[DetectionDelegatingLocationProvider].getName)
219+
}
220+
}
221+
}
222+
188223
test("fall-back: format-version=3") {
189224
assume(isSpark35Plus, "V3 tables require Iceberg 1.8.1+ (Spark 3.5 profile)")
190225
withDetectionCatalog { dir =>
@@ -1150,3 +1185,39 @@ class DetectionDelegatingFileIO extends FileIO with HadoopConfigurable {
11501185
SerializableSupplier[Configuration]]): Unit =
11511186
delegate.serializeConfWith(confSerializer)
11521187
}
1188+
1189+
/**
1190+
* A catalog whose TableOperations supplies a custom LocationProvider directly, without setting
1191+
* `write.location-provider.impl`. This is the path the native write gate must inspect explicitly.
1192+
*/
1193+
class DetectionCustomLocationHadoopCatalog extends HadoopCatalog {
1194+
override protected def newTableOps(identifier: TableIdentifier): TableOperations =
1195+
new DetectionLocationProviderTableOperations(super.newTableOps(identifier))
1196+
}
1197+
1198+
class DetectionLocationProviderTableOperations(delegate: TableOperations)
1199+
extends TableOperations {
1200+
override def current(): TableMetadata = delegate.current()
1201+
override def refresh(): TableMetadata = delegate.refresh()
1202+
override def commit(base: TableMetadata, metadata: TableMetadata): Unit =
1203+
delegate.commit(base, metadata)
1204+
override def io(): FileIO = delegate.io()
1205+
override def encryption(): EncryptionManager = delegate.encryption()
1206+
override def metadataFileLocation(fileName: String): String =
1207+
delegate.metadataFileLocation(fileName)
1208+
override def locationProvider(): LocationProvider =
1209+
new DetectionDelegatingLocationProvider(delegate.locationProvider())
1210+
override def temp(uncommittedMetadata: TableMetadata): TableOperations =
1211+
new DetectionLocationProviderTableOperations(delegate.temp(uncommittedMetadata))
1212+
override def newSnapshotId(): Long = delegate.newSnapshotId()
1213+
override def requireStrictCleanup(): Boolean = delegate.requireStrictCleanup()
1214+
}
1215+
1216+
class DetectionDelegatingLocationProvider(delegate: LocationProvider) extends LocationProvider {
1217+
override def newDataLocation(filename: String): String = delegate.newDataLocation(filename)
1218+
override def newDataLocation(
1219+
spec: PartitionSpec,
1220+
partitionData: StructLike,
1221+
filename: String): String =
1222+
delegate.newDataLocation(spec, partitionData, filename)
1223+
}

‎spark/src/test/scala/org/apache/comet/iceberg/IcebergReflectionSuite.scala‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,24 @@ class IcebergReflectionSuite extends AnyFunSuite {
230230
assert(IcebergReflection.getEncryptionManager(new NoEncryptionMethodTable).isEmpty)
231231
}
232232

233+
class CustomLocationProviderTable {
234+
def locationProvider(): AnyRef = new Object
235+
}
236+
237+
class NoLocationProviderTable
238+
239+
test("getLocationProvider resolves the provider the table actually installed") {
240+
val custom = IcebergReflection.getLocationProvider(new CustomLocationProviderTable)
241+
assert(custom.isDefined)
242+
assert(custom.get.getClass.getName != IcebergReflection.ClassNames.DEFAULT_LOCATION_PROVIDER)
243+
}
244+
245+
test("getLocationProvider returns None when locationProvider() cannot be resolved") {
246+
// The write gate treats None as fail-closed, so a table type without the accessor (or a
247+
// future rename) declines the native write rather than assuming DefaultLocationProvider.
248+
assert(IcebergReflection.getLocationProvider(new NoLocationProviderTable).isEmpty)
249+
}
250+
233251
test("executor-side reflection surface resolves against the linked Iceberg") {
234252
// The eligibility gate declines a native write when any class, method, or constructor used
235253
// by the executor-side commit-message assembly fails to resolve (it would otherwise be a

0 commit comments

Comments
 (0)