Repository navigation
Commit 099d4f3
fix(safety): honest join-failure propagation + close the DeleteFileIndex lost-wakeup hang (SAF-006/SAF-007)
Two hang/panic classes from the 2026-07-23 external audit, both on the shipping
read path.
SAF-006 - runtime/mod.rs `JoinHandle`
-------------------------------------
`Future::poll` collapsed EVERY join failure into `.expect("tokio spawned task
failed")`, so a panicking task's payload was destroyed and a *cancelled* task
(nothing failed) was reported with the same misleading string. There is exactly
one runtime arm in this crate (tokio) - no wasm/async-std/blocking alternative -
so this is the whole join-failure policy.
* Panic -> `std::panic::resume_unwind(join_error.into_panic())`. The waiter now
observes the ORIGINAL payload, which is also the Java contract: `Future.get()`
rethrows the task's own exception at the caller (`ExecutionException.getCause`
is the original throwable; Iceberg's `Tasks`/`ExceptionUtil.castAndThrow`
rethrows it unwrapped). The old `expect` was neither Java parity nor
debuggable.
* Cancellation -> `Output = T` has no error channel, so a panic is unavoidable;
it now says what actually happened instead of blaming the task.
* New `JoinHandle::try_join() -> Result<T>` maps BOTH cases to a typed
`ErrorKind::Unexpected` (panic payload rendered into the message and logged via
`tracing`; the `JoinError` kept as `source()` for a cancellation). Converted
the two await sites whose signature already carries an error - the Avro and ORC
data-file readers, where the panic source is third-party decode over untrusted
file bytes. A malformed file now fails that read instead of unwinding into the
awaiting scan.
This deliberately does NOT convert `Future::Output` from `T` to `Result<T, _>`;
that rewrite across every awaited AND detached spawn site stays deferred (as
#166 recorded), and `try_join` reaches the call sites that matter without it.
SAF-007 - delete_file_index.rs
------------------------------
The audit's stated mechanism (a full `channel(10)` hanging waiters) is STALE:
the populate task's `collect()` drains continuously, so a full buffer only
backpressures the senders, and the scan awaits the delete-manifest block, which
closes the channel. Two OTHER paths to a permanent hang were real, and are fixed:
* Lost wakeup. `get_deletes_for_data_file` observed `Populating`, released the
read lock, and only then created its `Notified`. `notify_waiters()` stores no
permit and `notified_owned()` snapshots the counter at CALL time, so a populate
task that published in that window left the scan awaiting forever. The notifier
is now armed while the read lock is still held (`lookup_or_arm`): the populate
task cannot notify until it takes the write lock, so any notification follows
the arming and is delivered. Same fix as upstream apache#2696 for
this site (apache#2859 is the same class on the positional-delete path).
* Populate task death. The task is detached; if it unwinds or the runtime is torn
down while it is parked on `collect()`, the state stayed `Populating` forever
and no notification could ever be sent. A `PopulateGuard` armed before the
task's first await now publishes a terminal `DeleteFileIndexState::Failed` on
drop, so every waiter gets a typed error. Mirrors `EqDelState::Failed` in
`arrow/delete_filter.rs` (#160).
Tests (+8, `-p iceberg --lib` 2855 -> 2863), each mutation-proved RED:
`test_await_re_raises_the_original_panic_payload`,
`test_try_join_maps_a_task_panic_to_a_typed_error`,
`test_try_join_maps_a_cancelled_task_to_a_typed_error`,
`test_try_join_describes_a_non_string_panic_payload`,
`test_try_join_returns_the_task_value`,
`test_waiter_is_armed_before_the_publisher_can_notify`,
`test_dead_populate_task_yields_a_typed_error_not_a_hang`,
`test_failed_index_state_is_terminal_for_later_callers`.
The two hang probes carry their own 5s timeout so a regression fails instead of
wedging CI. The panic-classification assertions read `Error::message()`, not
`to_string()`: `Display for Error` appends the source chain and a `JoinError`
renders the payload itself, which made a `to_string().contains(payload)` check
pass even with the conversion removed.
Known follow-up, NOT in this change: `arrow/delete_filter.rs` has the same
lost-wakeup pattern at three sites (`PosDelLoadAction::WaitFor(Arc<Notify>)` :112
awaited in `caching_delete_file_loader.rs` :276/:362, and the equality-delete
waits :232/:270). Upstream apache#2859 fixes the positional one and is a clean
cherry-pick; the equality-delete pair is not covered by it.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MRhAQnro5233xoVDLfib1X1 parent c88888c commit 099d4f3
4 files changed
Lines changed: 478 additions & 45 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
139 | 139 | | |
140 | 140 | | |
141 | 141 | | |
142 | | - | |
143 | | - | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
144 | 147 | | |
145 | 148 | | |
146 | 149 | | |
147 | | - | |
| 150 | + | |
| 151 | + | |
148 | 152 | | |
149 | 153 | | |
150 | 154 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
131 | 135 | | |
132 | 136 | | |
133 | 137 | | |
134 | | - | |
| 138 | + | |
| 139 | + | |
135 | 140 | | |
136 | 141 | | |
137 | 142 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
20 | 19 | | |
21 | 20 | | |
22 | 21 | | |
23 | 22 | | |
24 | 23 | | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
41 | 115 | | |
42 | 116 | | |
43 | 117 | | |
| |||
80 | 154 | | |
81 | 155 | | |
82 | 156 | | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
83 | 162 | | |
84 | 163 | | |
85 | 164 | | |
86 | 165 | | |
87 | 166 | | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
| 167 | + | |
99 | 168 | | |
100 | 169 | | |
101 | 170 | | |
| |||
113 | 182 | | |
114 | 183 | | |
115 | 184 | | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
130 | 189 | | |
131 | | - | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
132 | 202 | | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
133 | 219 | | |
134 | 220 | | |
135 | 221 | | |
136 | 222 | | |
137 | 223 | | |
138 | 224 | | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
147 | 231 | | |
148 | | - | |
| 232 | + | |
149 | 233 | | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
150 | 237 | | |
151 | 238 | | |
152 | 239 | | |
| |||
329 | 416 | | |
330 | 417 | | |
331 | 418 | | |
| 419 | + | |
| 420 | + | |
332 | 421 | | |
333 | 422 | | |
334 | 423 | | |
| |||
877 | 966 | | |
878 | 967 | | |
879 | 968 | | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
880 | 1099 | | |
881 | 1100 | | |
882 | 1101 | | |
| |||
0 commit comments