Skip to content

Commit 099d4f3

Browse files
TRO-Wolfclaude
andcommitted
fix(safety): honest join-failure propagation + close the DeleteFileIndex lost-wakeup hang (SAF-006/SAF-007)
Two hang/panic classes from the 2026-07-23 external audit, both on the shipping read path. SAF-006 - runtime/mod.rs `JoinHandle` ------------------------------------- `Future::poll` collapsed EVERY join failure into `.expect("tokio spawned task failed")`, so a panicking task's payload was destroyed and a *cancelled* task (nothing failed) was reported with the same misleading string. There is exactly one runtime arm in this crate (tokio) - no wasm/async-std/blocking alternative - so this is the whole join-failure policy. * Panic -> `std::panic::resume_unwind(join_error.into_panic())`. The waiter now observes the ORIGINAL payload, which is also the Java contract: `Future.get()` rethrows the task's own exception at the caller (`ExecutionException.getCause` is the original throwable; Iceberg's `Tasks`/`ExceptionUtil.castAndThrow` rethrows it unwrapped). The old `expect` was neither Java parity nor debuggable. * Cancellation -> `Output = T` has no error channel, so a panic is unavoidable; it now says what actually happened instead of blaming the task. * New `JoinHandle::try_join() -> Result<T>` maps BOTH cases to a typed `ErrorKind::Unexpected` (panic payload rendered into the message and logged via `tracing`; the `JoinError` kept as `source()` for a cancellation). Converted the two await sites whose signature already carries an error - the Avro and ORC data-file readers, where the panic source is third-party decode over untrusted file bytes. A malformed file now fails that read instead of unwinding into the awaiting scan. This deliberately does NOT convert `Future::Output` from `T` to `Result<T, _>`; that rewrite across every awaited AND detached spawn site stays deferred (as #166 recorded), and `try_join` reaches the call sites that matter without it. SAF-007 - delete_file_index.rs ------------------------------ The audit's stated mechanism (a full `channel(10)` hanging waiters) is STALE: the populate task's `collect()` drains continuously, so a full buffer only backpressures the senders, and the scan awaits the delete-manifest block, which closes the channel. Two OTHER paths to a permanent hang were real, and are fixed: * Lost wakeup. `get_deletes_for_data_file` observed `Populating`, released the read lock, and only then created its `Notified`. `notify_waiters()` stores no permit and `notified_owned()` snapshots the counter at CALL time, so a populate task that published in that window left the scan awaiting forever. The notifier is now armed while the read lock is still held (`lookup_or_arm`): the populate task cannot notify until it takes the write lock, so any notification follows the arming and is delivered. Same fix as upstream apache#2696 for this site (apache#2859 is the same class on the positional-delete path). * Populate task death. The task is detached; if it unwinds or the runtime is torn down while it is parked on `collect()`, the state stayed `Populating` forever and no notification could ever be sent. A `PopulateGuard` armed before the task's first await now publishes a terminal `DeleteFileIndexState::Failed` on drop, so every waiter gets a typed error. Mirrors `EqDelState::Failed` in `arrow/delete_filter.rs` (#160). Tests (+8, `-p iceberg --lib` 2855 -> 2863), each mutation-proved RED: `test_await_re_raises_the_original_panic_payload`, `test_try_join_maps_a_task_panic_to_a_typed_error`, `test_try_join_maps_a_cancelled_task_to_a_typed_error`, `test_try_join_describes_a_non_string_panic_payload`, `test_try_join_returns_the_task_value`, `test_waiter_is_armed_before_the_publisher_can_notify`, `test_dead_populate_task_yields_a_typed_error_not_a_hang`, `test_failed_index_state_is_terminal_for_later_callers`. The two hang probes carry their own 5s timeout so a regression fails instead of wedging CI. The panic-classification assertions read `Error::message()`, not `to_string()`: `Display for Error` appends the source chain and a `JoinError` renders the payload itself, which made a `to_string().contains(payload)` check pass even with the conversion removed. Known follow-up, NOT in this change: `arrow/delete_filter.rs` has the same lost-wakeup pattern at three sites (`PosDelLoadAction::WaitFor(Arc<Notify>)` :112 awaited in `caching_delete_file_loader.rs` :276/:362, and the equality-delete waits :232/:270). Upstream apache#2859 fixes the positional one and is a clean cherry-pick; the equality-delete pair is not covered by it. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MRhAQnro5233xoVDLfib1X
1 parent c88888c commit 099d4f3

4 files changed

Lines changed: 478 additions & 45 deletions

File tree

‎crates/iceberg/src/arrow/avro_reader.rs‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -139,12 +139,16 @@ pub(crate) async fn read_avro_data_file(
139139
batch_size: usize,
140140
) -> Result<Vec<RecordBatch>> {
141141
let bytes = input.read().await?;
142-
// Decode off the async executor: Avro datum decode is CPU-bound. The crate's `JoinHandle`
143-
// yields the closure's `Result` directly (it propagates a join panic internally).
142+
// Decode off the async executor: Avro datum decode is CPU-bound. `try_join` (not a bare
143+
// `.await`) because the decoder is third-party code running on bytes we do not control: a
144+
// panic inside `apache_avro`/`arrow` on a malformed or hostile file must fail THIS read with a
145+
// typed error, not unwind into whoever is awaiting the scan. The outer `?` unwraps the join
146+
// result; the inner `Result` is the decode's own and is returned as-is.
144147
crate::runtime::spawn_blocking(move || {
145148
read_avro_data_bytes(&bytes, expected.as_ref(), batch_size)
146149
})
147-
.await
150+
.try_join()
151+
.await?
148152
}
149153

150154
/// Decode Avro OCF `bytes` into Arrow [`RecordBatch`]es against the `expected` projection schema.

‎crates/iceberg/src/arrow/orc_reader.rs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -128,10 +128,15 @@ pub(crate) async fn read_orc_data_file(
128128
batch_size: usize,
129129
) -> Result<Vec<RecordBatch>> {
130130
let bytes = input.read().await?;
131+
// `try_join` (not a bare `.await`) for the same reason as the Avro reader: an `orc-rust`
132+
// or `arrow` panic on a malformed file must fail this read with a typed error rather than
133+
// unwind into the awaiting scan. The outer `?` unwraps the join result; the inner `Result`
134+
// is the decode's own.
131135
crate::runtime::spawn_blocking(move || {
132136
read_orc_data_bytes(&bytes, expected.as_ref(), batch_size)
133137
})
134-
.await
138+
.try_join()
139+
.await?
135140
}
136141

137142
/// Decode ORC `bytes` into Arrow [`RecordBatch`]es against the `expected` projection schema.

‎crates/iceberg/src/delete_file_index.rs‎

Lines changed: 255 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,12 @@
1616
// under the License.
1717

1818
use std::collections::HashMap;
19-
use std::ops::Deref;
2019
use std::sync::{Arc, RwLock};
2120

2221
use futures::StreamExt;
2322
use futures::channel::mpsc::{Sender, channel};
2423
use tokio::sync::Notify;
24+
use tokio::sync::futures::OwnedNotified;
2525

2626
use crate::runtime::spawn;
2727
use crate::scan::{DeleteFileContext, FileScanTaskDeleteFile};
@@ -38,6 +38,80 @@ pub(crate) struct DeleteFileIndex {
3838
enum DeleteFileIndexState {
3939
Populating(Arc<Notify>),
4040
Populated(PopulatedDeleteFileIndex),
41+
/// The populate task terminated WITHOUT publishing an index — it unwound, or the runtime was
42+
/// torn down and the task's future was dropped at its `collect()` await.
43+
///
44+
/// Terminal, exactly like [`DeleteFileIndexState::Populated`]: the populate task is the sole
45+
/// writer and runs once, so if it dies the state can never advance on its own. Without this
46+
/// variant the state would stay `Populating` forever and every scan parked in
47+
/// [`DeleteFileIndex::get_deletes_for_data_file`] would wait on a notification that can no
48+
/// longer be sent. The `String` is the reason, rendered into each waiter's typed error. This
49+
/// mirrors `EqDelState::Failed` in [`crate::arrow::delete_filter`].
50+
Failed(String),
51+
}
52+
53+
/// Publishes the populate task's TERMINAL state and wakes the waiters, on every exit path.
54+
///
55+
/// Constructed at the top of the populate task, before its first await, and disarmed by
56+
/// [`PopulateGuard::publish`] on the success path. If the task never reaches that call — it
57+
/// unwound (tokio drops the task's future as the panic propagates), or the runtime shut down and
58+
/// dropped the parked future — `Drop` publishes [`DeleteFileIndexState::Failed`] instead, so every
59+
/// waiter reaches a terminal state and gets a typed error rather than hanging forever.
60+
///
61+
/// Both paths write the state under the write lock and fire the notifier only AFTER releasing it,
62+
/// so a woken waiter always observes the terminal state (see [`DeleteFileIndex::lookup_or_arm`]
63+
/// for the other half of that handshake).
64+
struct PopulateGuard {
65+
state: Arc<RwLock<DeleteFileIndexState>>,
66+
notify: Arc<Notify>,
67+
armed: bool,
68+
}
69+
70+
impl PopulateGuard {
71+
fn new(state: Arc<RwLock<DeleteFileIndexState>>, notify: Arc<Notify>) -> Self {
72+
Self {
73+
state,
74+
notify,
75+
armed: true,
76+
}
77+
}
78+
79+
/// Publish `terminal` and wake every waiter, disarming the guard.
80+
fn publish(&mut self, terminal: DeleteFileIndexState) {
81+
{
82+
// Recover a poisoned guard rather than cascading the panic: this is the sole writer
83+
// and it runs once, so recovering and completing the transition is always the right
84+
// move (a stranded `Populating` state would hang every waiting scan on the notifier
85+
// below).
86+
let mut guard = self
87+
.state
88+
.write()
89+
.unwrap_or_else(|poisoned| poisoned.into_inner());
90+
*guard = terminal;
91+
}
92+
self.armed = false;
93+
self.notify.notify_waiters();
94+
}
95+
}
96+
97+
impl Drop for PopulateGuard {
98+
fn drop(&mut self) {
99+
if self.armed {
100+
self.publish(DeleteFileIndexState::Failed(
101+
"the delete file index populate task terminated before publishing an index \
102+
(it panicked, or the runtime was shut down)"
103+
.to_string(),
104+
));
105+
}
106+
}
107+
}
108+
109+
/// The outcome of consulting the index state once — see [`DeleteFileIndex::lookup_or_arm`].
110+
enum IndexLookup {
111+
/// The index had reached a terminal state; this is the answer.
112+
Ready(Vec<FileScanTaskDeleteFile>),
113+
/// The index was still populating. Await this future, then read the state again.
114+
Wait(OwnedNotified),
41115
}
42116

43117
#[derive(Debug)]
@@ -80,22 +154,17 @@ impl DeleteFileIndex {
80154
spawn({
81155
let state = state.clone();
82156
async move {
157+
// Armed BEFORE the first await, so every way out of this task — including the
158+
// runtime dropping the future while it is parked on `collect()` below — publishes
159+
// a terminal state and wakes the waiters.
160+
let mut guard = PopulateGuard::new(state, notify);
161+
83162
let delete_files: Vec<DeleteFileContext> =
84163
delete_file_stream.collect::<Vec<_>>().await;
85164

86165
let populated_delete_file_index = PopulatedDeleteFileIndex::new(delete_files);
87166

88-
{
89-
// Recover a poisoned guard rather than cascading the panic: this is the sole
90-
// writer and it runs once, so recovering and completing the Populated
91-
// transition is always the right move (a stranded Populating state would hang
92-
// every waiting scan on the notifier below).
93-
let mut guard = state
94-
.write()
95-
.unwrap_or_else(|poisoned| poisoned.into_inner());
96-
*guard = DeleteFileIndexState::Populated(populated_delete_file_index);
97-
}
98-
notify.notify_waiters();
167+
guard.publish(DeleteFileIndexState::Populated(populated_delete_file_index));
99168
}
100169
});
101170

@@ -113,40 +182,58 @@ impl DeleteFileIndex {
113182
data_file: &DataFile,
114183
seq_num: Option<i64>,
115184
) -> Result<Vec<FileScanTaskDeleteFile>> {
116-
let notifier = {
117-
let guard = self.state.read().map_err(|_| {
118-
Error::new(
119-
ErrorKind::Unexpected,
120-
"delete file index RwLock was poisoned",
121-
)
122-
})?;
123-
match *guard {
124-
DeleteFileIndexState::Populating(ref notifier) => notifier.clone(),
125-
DeleteFileIndexState::Populated(ref index) => {
126-
return index.get_deletes_for_data_file(data_file, seq_num);
127-
}
128-
}
129-
};
185+
match self.lookup_or_arm(data_file, seq_num)? {
186+
IndexLookup::Ready(deletes) => return Ok(deletes),
187+
IndexLookup::Wait(notified) => notified.await,
188+
}
130189

131-
notifier.notified().await;
190+
// The populate task publishes a TERMINAL state under the write lock and only then fires
191+
// the notifier, so a woken waiter always observes `Populated` or `Failed`. (This second
192+
// call arms a notifier it will not use — one `Arc` clone — which keeps the lock/arm
193+
// handshake in exactly one place.)
194+
match self.lookup_or_arm(data_file, seq_num)? {
195+
IndexLookup::Ready(deletes) => Ok(deletes),
196+
IndexLookup::Wait(_) => Err(Error::new(
197+
ErrorKind::Unexpected,
198+
"delete file index notifier fired but the index is not populated",
199+
)),
200+
}
201+
}
132202

203+
/// Read the index state once: answer outright if it is terminal, otherwise ARM the notifier.
204+
///
205+
/// The arming MUST happen here, while the read lock is still held.
206+
/// [`Notify::notify_waiters`] stores no permit and only wakes `Notified` futures that already
207+
/// EXIST when it fires, and `Notify::notified_owned` snapshots the notifier's
208+
/// `notify_waiters` counter at CALL time — so a `Notified` created after the populate task
209+
/// fired is never woken. Returning a bare `Arc<Notify>` and calling `.notified()` at the await
210+
/// site left exactly that window open between releasing the read lock and creating the future:
211+
/// if the populate task published and notified in it, the wakeup was dropped and the scan
212+
/// awaited forever (upstream apache/iceberg-rust#2696, the same class as #2859 on the
213+
/// positional-delete wait path).
214+
///
215+
/// Creating the future under the read lock closes the window: the populate task cannot fire
216+
/// the notifier until it has taken the WRITE lock, which cannot be granted while this read
217+
/// lock is held, so any notification necessarily follows this arming and is delivered.
218+
fn lookup_or_arm(&self, data_file: &DataFile, seq_num: Option<i64>) -> Result<IndexLookup> {
133219
let guard = self.state.read().map_err(|_| {
134220
Error::new(
135221
ErrorKind::Unexpected,
136222
"delete file index RwLock was poisoned",
137223
)
138224
})?;
139-
match guard.deref() {
140-
DeleteFileIndexState::Populated(index) => {
141-
index.get_deletes_for_data_file(data_file, seq_num)
142-
}
143-
// The populate task sets `Populated` and only then fires the notifier, so after the
144-
// await the state is populated. A non-populated state here is an invariant violation
145-
// (or a spurious wakeup) — surface a typed error rather than panicking the scan.
146-
_ => Err(Error::new(
225+
226+
match &*guard {
227+
DeleteFileIndexState::Populated(index) => Ok(IndexLookup::Ready(
228+
index.get_deletes_for_data_file(data_file, seq_num)?,
229+
)),
230+
DeleteFileIndexState::Failed(reason) => Err(Error::new(
147231
ErrorKind::Unexpected,
148-
"delete file index notifier fired but the index is not populated",
232+
format!("delete file index is unavailable: {reason}"),
149233
)),
234+
DeleteFileIndexState::Populating(notifier) => {
235+
Ok(IndexLookup::Wait(notifier.clone().notified_owned()))
236+
}
150237
}
151238
}
152239
}
@@ -329,6 +416,8 @@ impl PopulatedDeleteFileIndex {
329416

330417
#[cfg(test)]
331418
mod tests {
419+
use std::time::Duration;
420+
332421
use uuid::Uuid;
333422

334423
use super::*;
@@ -877,6 +966,136 @@ mod tests {
877966
);
878967
}
879968

969+
/// Risk pinned (audit SAF-007 / upstream apache/iceberg-rust#2696): the waiter must ARM its
970+
/// notifier while it still holds the read lock, so a `notify_waiters()` that fires before the
971+
/// waiter awaits still wakes it. `notify_waiters()` stores no permit, so this test's ordering
972+
/// — publish FIRST, await SECOND — only completes when the `Notified` already existed at
973+
/// publish time.
974+
///
975+
/// MUTATION (semantic revert to the base contract: `IndexLookup::Wait` carries a raw
976+
/// `Arc<Notify>` and the waiter calls `.notified()` at the await site): the future is created
977+
/// after `publish`, the wakeup is lost, and the timeout below fires (RED).
978+
#[tokio::test]
979+
async fn test_waiter_is_armed_before_the_publisher_can_notify() {
980+
// The sender stays alive for the whole test, so the real populate task stays parked on
981+
// its `collect()` and this test is the only publisher.
982+
let (index, _tx) = DeleteFileIndex::new();
983+
let data_file = build_unpartitioned_data_file();
984+
985+
let notified = match index
986+
.lookup_or_arm(&data_file, Some(0))
987+
.expect("arming a populating index must not error")
988+
{
989+
IndexLookup::Wait(notified) => notified,
990+
IndexLookup::Ready(_) => {
991+
panic!("the index must still be populating while the sender is alive")
992+
}
993+
};
994+
995+
// Publish + `notify_waiters()` through the production publisher, BEFORE the waiter awaits.
996+
let notifier = {
997+
let guard = index.state.read().expect("read the index state");
998+
match &*guard {
999+
DeleteFileIndexState::Populating(notifier) => notifier.clone(),
1000+
other => panic!("expected a populating index, got {other:?}"),
1001+
}
1002+
};
1003+
PopulateGuard::new(index.state.clone(), notifier).publish(DeleteFileIndexState::Populated(
1004+
PopulatedDeleteFileIndex::new(vec![]),
1005+
));
1006+
1007+
tokio::time::timeout(Duration::from_secs(5), notified)
1008+
.await
1009+
.expect("a notification fired after arming must wake the waiter, not be lost");
1010+
1011+
let deletes = index
1012+
.get_deletes_for_data_file(&data_file, Some(0))
1013+
.await
1014+
.expect("the woken waiter must read the published index");
1015+
assert!(
1016+
deletes.is_empty(),
1017+
"the published index was empty, so no deletes may apply"
1018+
);
1019+
}
1020+
1021+
/// Risk pinned (audit SAF-007): if the populate task dies WITHOUT publishing — here by tearing
1022+
/// down the runtime that hosts it while it is parked on `collect()` — the index must reach the
1023+
/// terminal `Failed` state so every waiter gets a typed error. Nothing can advance the state
1024+
/// afterwards, so without that transition the waiter parks on a notification that can never be
1025+
/// sent.
1026+
///
1027+
/// MUTATION: reverting `delete_file_index.rs` (no `PopulateGuard`, no `Failed`) leaves the
1028+
/// state at `Populating`; the waiter never wakes and the timeout below fires (RED).
1029+
#[tokio::test]
1030+
async fn test_dead_populate_task_yields_a_typed_error_not_a_hang() {
1031+
let data_file = build_unpartitioned_data_file();
1032+
1033+
// Build the index on a runtime that is then DESTROYED, cancelling the populate task and
1034+
// dropping its future. Done on a separate thread because dropping a runtime from inside
1035+
// an async context panics.
1036+
let (index, _tx) = std::thread::spawn(|| {
1037+
let runtime = tokio::runtime::Builder::new_current_thread()
1038+
.enable_all()
1039+
.build()
1040+
.expect("build the throwaway runtime that hosts the populate task");
1041+
let (index, tx) = runtime.block_on(async {
1042+
let (index, tx) = DeleteFileIndex::new();
1043+
// Let the populate task reach its `collect()` await, past the point where its
1044+
// drop guard is armed.
1045+
for _ in 0..4 {
1046+
tokio::task::yield_now().await;
1047+
}
1048+
(index, tx)
1049+
});
1050+
drop(runtime);
1051+
(index, tx)
1052+
})
1053+
.join()
1054+
.expect("the runtime-teardown thread must not panic");
1055+
1056+
let error = tokio::time::timeout(
1057+
Duration::from_secs(5),
1058+
index.get_deletes_for_data_file(&data_file, Some(0)),
1059+
)
1060+
.await
1061+
.expect("a dead populate task must not hang the waiter")
1062+
.expect_err("a dead populate task must surface a typed error");
1063+
1064+
assert_eq!(error.kind(), ErrorKind::Unexpected);
1065+
assert!(
1066+
error.to_string().contains("populate task"),
1067+
"the error must name the dead populate task, got: {error}"
1068+
);
1069+
}
1070+
1071+
/// The terminal `Failed` state must be reported to EVERY subsequent caller, not just the one
1072+
/// that was parked when the populate task died — the state can never advance again.
1073+
#[tokio::test]
1074+
async fn test_failed_index_state_is_terminal_for_later_callers() {
1075+
let (index, _tx) = DeleteFileIndex::new();
1076+
let data_file = build_unpartitioned_data_file();
1077+
1078+
{
1079+
let mut guard = index.state.write().expect("write the index state");
1080+
*guard = DeleteFileIndexState::Failed("populate task was cancelled".to_string());
1081+
}
1082+
1083+
for _ in 0..2 {
1084+
let error = tokio::time::timeout(
1085+
Duration::from_secs(5),
1086+
index.get_deletes_for_data_file(&data_file, Some(0)),
1087+
)
1088+
.await
1089+
.expect("a failed index must answer immediately")
1090+
.expect_err("a failed index must surface a typed error");
1091+
assert_eq!(error.kind(), ErrorKind::Unexpected);
1092+
assert!(
1093+
error.to_string().contains("populate task was cancelled"),
1094+
"the error must carry the recorded reason, got: {error}"
1095+
);
1096+
}
1097+
}
1098+
8801099
fn build_unpartitioned_data_file() -> DataFile {
8811100
DataFileBuilder::default()
8821101
.file_path(format!("{}-data.parquet", Uuid::new_v4()))

0 commit comments

Comments
 (0)