Repository navigation
Missing size rewrite in rewrite_table_path for delete file #12554
Description
Apache Iceberg version
1.8.1 (latest release)
Query engine
None
Please describe the bug 🐞
I think this a bug produced by 11555
While running rewrite_table_path and moving the data with a delete file i got the error
Error: TrinoExternalError(type=EXTERNAL, name=ICEBERG_CANNOT_OPEN_SPLIT,
message="Error opening Iceberg split gs://buket/dir/data/id_trunc_2=3e/20250305_141051_06175_fjupa-4a03af2a-12c4-4f27-881b-bc15f459b13c.parquet
(offset=0, length=1187): Read 1187 tail bytes of file
gs://buket/dir/data/id_trunc_2=3e/20250305_141051_06175_fjupa-4a03af2a-12c4-4f27-881b-bc15f459b13c.parquet failed: Incorrect file size (1187) for file (end of stream not reached): gs://buket/dir/data/id_trunc_2=3e/20250305_141051_06175_fjupa-4a03af2a-12c4-4f27-881b-bc15f459b13c.parquet", query_id=20250317_151116_30308_u8j29)
Seems like this happens because the file 20250305_141051_06175_fjupa-4a03af2a-12c4-4f27-881b-bc15f459b13c.parquet was rewritten (is a delete file) but the info on the manifest was not
"file_sequence_number": null,
"data_file": {
"content": 1,
"file_path": "gs://buket/dir/data/id_trunc_2=3e/20250305_141051_06175_fjupa-4a03af2a-12c4-4f27-881b-bc15f459b13c.parquet",
"file_format": "PARQUET",
"partition": {
"id_trunc_2": {
"string": "3e"
}
},
"record_count": 1,
"file_size_in_bytes": 1187,
"column_sizes": {
"array": [
{
"key": 2147483546,
"value": 199
},
{
"key": 2147483545,
"value": 27
}
]
},
1187 was the size of the parquet file before rewritten, the new size is 1799, but that value is not updated
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity
jotarada commentedon Mar 17, 2025
@szehon-ho Can you have a look?
dramaticlly commentedon Mar 18, 2025
I can help take a look
dramaticlly commentedon Mar 28, 2025
I think the problem might be where the FileSizeInBytes is not known for newly rewritten deleteManifests https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java#L399-L403 and the fact that we write delete manifests before the position delete file
szehon-ho commentedon Mar 28, 2025
Yea I see, I guess its because the filePath will be different size in the rewritten delete file than the original delete file, so delete file size is wrong..
Originally I had it so we write the delete file first and then delete manifest, but we changed it to make the writing delete file parallel. I already fixed the upper and lower bounds for delete manifest (https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java#L396) , but the size is something I missed to fix (and is harder). We might need a large refactor to fix this (write the delete file first in one Spark job, and then delete manifest in another Spark job). @flyrain any thoughts?
suisenkotoba commentedon Apr 17, 2025
Hi I am planning to use this procedure rewrite_table_path for iceberg table migration to other location. I am wondering if this bug is also happening in other iceberg versions
adrians commentedon Jul 24, 2025
I found a similar problem with a table migrated using
rewrite_table_pathwhen querying using Impala.Basically Impala reads the manifest-list, gets the list of manifest-files and their sizes, and since the manifest-files were rewritten for the new path (don't have the same exact size as in the original table, as the file-contant has changed) and the manifest-list does not contain the updated sizes, Impala throws an error (the actual manifest-file sizes doesn't match the expected manifest-file size as declared in the manifest-list).
Hive and Spark query-engines don't check the manifest-file length, but Impala does.
Impala error-message:
It seems that the logical flow should be
github-actions commentedon Jan 21, 2026
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
github-actions commentedon Feb 4, 2026
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
benjibuiltit commentedon Feb 10, 2026
I'm encountering this same issue executing
rewrite_table_pathprocedure via Google Dataproc running Iceberg 1.10.1When I go to query the migrated data via Trino, I encounter the same issue @adrians reported regarding varying file lengths.
rewrite_table_pathwritermbutrovich commentedon Feb 26, 2026
DataFusion Comet is now hitting this as well after a recent improvement to iceberg-rust to avoid a
stat()call on delete files by using thefile_size_in_bytesfrom the manifest (apache/iceberg-rust#2175). After a table rewrite, thefile_size_in_bytesis wrong which causes the file reader to fail. I can work around it by having Comet issue astat()but it's unfortunate to give up this optimization.steveloughran commentedon Apr 10, 2026
a lot of object store client skip that HEAD request, trusting the metadata. Looks like it doesn't hold here.
fix(reader): rely on Parquet's footer check for delete-file sizes
fix(reader): rely on Parquet's footer check for delete-file sizes