Repository navigation
Commit 76e54f7
fix(watcher): back off re-forking a hard-failing index worker
A hard index failure (index_fn < 0) left the retry cadence untouched, so a
persistently failing project re-forked an index worker every poll for as
long as the daemon lived. Observed on 0.10.8/macOS: 2233 consecutive failed
workers over 4h43m, ~8/min, every one dying at the same worker-side
coordination seal. Read queries stayed healthy throughout, so nothing
surfaced to the user while no project could be indexed at all.
#937 deliberately leaves the baseline uncommitted on a failed reindex so the
change is retried rather than lost. That guarantee is kept; what changes is
the cadence. Each consecutive hard failure doubles the delay, capped at five
minutes, taking a permanently failing project from ~480 attempts/hour to
~12 while still recovering on its own once the cause clears. Busy-skip
(rc > 0) and success paths are untouched.
The delay arithmetic is a pure exported helper so it is unit-testable
without a clock. watcher.index.err now carries rc and the consecutive-
failure count, and a distinct watcher.index.sustained_failure names a
project whose failures are clearly not transient.
Note: itoa_buf returns a single shared per-thread buffer, so the two-value
log line uses local buffers rather than two itoa_buf calls.
Refs #2015
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01FbrM52pmB1m2SR1vRRoNnf
Signed-off-by: Shane McCarron <[email protected]>1 parent 4d8ac9e commit 76e54f7
3 files changed
Lines changed: 115 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
75 | 75 | | |
76 | 76 | | |
77 | 77 | | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
78 | 86 | | |
79 | 87 | | |
80 | 88 | | |
| |||
114 | 122 | | |
115 | 123 | | |
116 | 124 | | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
117 | 135 | | |
118 | 136 | | |
119 | 137 | | |
| |||
143 | 161 | | |
144 | 162 | | |
145 | 163 | | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
146 | 177 | | |
147 | 178 | | |
148 | 179 | | |
| |||
1429 | 1460 | | |
1430 | 1461 | | |
1431 | 1462 | | |
| 1463 | + | |
1432 | 1464 | | |
1433 | 1465 | | |
1434 | 1466 | | |
| |||
1447 | 1479 | | |
1448 | 1480 | | |
1449 | 1481 | | |
1450 | | - | |
| 1482 | + | |
| 1483 | + | |
| 1484 | + | |
| 1485 | + | |
| 1486 | + | |
| 1487 | + | |
| 1488 | + | |
| 1489 | + | |
| 1490 | + | |
| 1491 | + | |
| 1492 | + | |
| 1493 | + | |
| 1494 | + | |
| 1495 | + | |
| 1496 | + | |
1451 | 1497 | | |
1452 | 1498 | | |
1453 | 1499 | | |
1454 | | - | |
| 1500 | + | |
| 1501 | + | |
| 1502 | + | |
| 1503 | + | |
1455 | 1504 | | |
1456 | 1505 | | |
1457 | 1506 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
97 | 104 | | |
98 | 105 | | |
99 | 106 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
| |||
85 | 86 | | |
86 | 87 | | |
87 | 88 | | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
88 | 140 | | |
89 | 141 | | |
90 | 142 | | |
| |||
3143 | 3195 | | |
3144 | 3196 | | |
3145 | 3197 | | |
| 3198 | + | |
| 3199 | + | |
| 3200 | + | |
| 3201 | + | |
| 3202 | + | |
3146 | 3203 | | |
3147 | 3204 | | |
3148 | 3205 | | |
| |||
0 commit comments