You can check https://iceberg.apache.org/docs/latest/spark-procedures/#remove_orphan_files for some instructions. This procedure uses the FileIO implementation that is configured for the underlying table, so the statement that it only supports the Hadoop filesystem isn't correct.

How does your catalog configuration look like and how do are you calling the procedure?

Thanks for answering this question.When i use s3FileIo, i get this error:
org.apache.iceberg.exceptions.RuntimeIOException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "S3"
at org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.listDirRecursively(DeleteOrphanFilesSparkAction.java:298) ~[iceberg-spark-runtime-0.14.0.jar:?]
at org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.buildActualFileDF(DeleteOrphanFilesSparkAction.java:243) ~[iceberg-spark-runtime-0.14.1.jar:?]
at org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.doExecute(DeleteOrphanFilesSparkAction.java:214) ~[iceberg-spark-runtime-0.14.1.jar:?]
at org.apache.iceberg.spark.actions.BaseSparkAction.withJobGroupInfo(BaseSparkAction.java:112) ~[iceberg-spark-runtime-0.14.1.jar:?]
at org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.execute(DeleteOrphanFilesSparkAction.java:198) ~[iceberg-spark-runtime-0.14.1.jar:?]
at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "S3"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3585) ~[hadoop-client-api-3.3.6.jar:?]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3608) ~[hadoop-client-api-3.3.6.jar:?]
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) ~[hadoop-client-api-3.3.6.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3712) ~[hadoop-client-api-3.3.6.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3663) ~[hadoop-client-api-3.3.6.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557) ~[hadoop-client-api-3.3.6.jar:?]
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) ~[hadoop-client-api-3.3.6.jar:?]
at org.apache.iceberg.spark.actions.DeleteOrphanFilesSparkAction.listDirRecursively(DeleteOrphanFilesSparkAction.java:275) ~[polars-iceberg-spark-runtime-0.14.01.jar:?]
... 13 more

How to clean s3/oss orphan files with spark? #8368

Description

Query engine

Question

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions