You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e238121
Browse filesBrowse the repository at this point in the historyBrowse files
fix: release an Iceberg file's reservation once its row groups leave memory
A data file's share was its written size capped at one row group, so every fanout file that had
flushed stayed charged a whole row group until it closed. A local file keeps nothing it has
flushed, so a pool of eight 128 KiB row groups failed a twelve-partition write whose files held
almost nothing.
Each file's OutputFile is now wrapped so that its writer counts the bytes that leave memory on
their way to storage, and the file reports what it has written less those. A local file releases
a row group as it is handed on. S3 and GCS do not: OpenDAL collects what a file flushes into parts
of at least 5 MiB and holds each part until the next one is complete or the file closes, so a file
there keeps what it has flushed since its last upload charged.
0 commit comments