Repository navigation
fix(daemon): stop the lifetime-lock probe from dropping a held lock - #2207
Conversation
|
Thanks for opening this — it has been seen, and it is queued. This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence. Current review status: working through a backlog. What that means for this PR, concretely:
Things that will genuinely speed it up whenever review does happen:
If this fixes a bug, a reproduction we can run is worth more than a description of the symptom. Thanks for contributing, and sorry in advance for the wait. |
DeusData
added
bug
|
Thank you for isolating the claim-lock descriptor path and providing a separate-process regression. This affects coordination between processes, so we need more time for the lifecycle review before making an integration decision. It should not be treated as proof that every cold-start race in #2162 is resolved. |
DeusData
left a comment
DeusData
left a comment
There was a problem hiding this comment.
Thank you, @AmirF194, for this one. It is a genuinely subtle catch, and the separate-process regression makes it easy to verify. Here is the lifecycle review we promised on the 19th.
The diagnosis is right. The lifetime lock is our only fcntl record lock, and POSIX releases those when any descriptor to the file is closed. The holder's own probe therefore opened a second descriptor, closed it, and quietly gave the lock away. Your early return fixes the probe. Windows is not affected: LockFileEx locks belong to a handle, and that path is not compiled there.
Two small asks before we merge:
- The sibling path in
posix_lifetime_lock_try_acquirehas the same pattern. When this process already holds the claim (process_result == 0), it callsprivate_regular_file_at_is_safe(endpoint->dir_fd, lock_name, 1)(src/daemon/ipc.c:1261on currentmain). That helper opens and closes the lock file (ipc.c:848-857), so it drops the held lock exactly the way the probe did. Could you give it the same no-descriptor treatment, for exampleprocess_result == 0 && endpoint_runtime_still_valid(endpoint) ? 0 : -1? Please add a sibling test too: acquire, calltry_acquireagain in the same process (expect 0), and check that the child still cannot take the lock. - Harden the child's verdict in the test. Report "held" only when
open()succeeded andF_SETLKfailed withEACCES/EAGAIN, and use a distinct value for anything else. At the moment any failure in the child, even a wrong path, looks like a pass.
Optional: a one-line comment at posix_record_lock_set stating the rule: never open and close the lifetime lock file in a process that may hold it, because fcntl locks are per process. The lock is fcntl rather than flock on purpose, so that forked children do not inherit it, which is exactly why the rule has to hold.
For the record: this hardens every process that holds the lifetime lock. The production daemon never probes its own lock, though, so this does not resolve the #2162 cold-start reports or the #2057 cold-storm flake. We will keep those open separately. Thanks again for tracking this down. Findings like this are what make the coordination layer trustworthy.
posix_lifetime_lock_probe opened a throwaway fd on the lock file to confirm an already-claimed lock, then closed it. fcntl(2) record locks are scoped to (process, inode), so that close silently released the real lock too, even though the reservation's own fd stayed open. Check the in-process claim before opening anything and return early; that branch only ever needed to trust the registry. Signed-off-by: Amir Fathi <[email protected]>
45c94be to
6313175
Compare
|
Pushed both. Also hardened the child verdict as asked: it's Ran the full |
|
Thank you, @AmirF194. All three points are in, and done well. The One tiny thing, and it's on us: the comment we suggested trips our security audit. |
posix_lifetime_lock_try_acquire opened a throwaway fd on the lock file to confirm an already-claimed lock, then closed it, in both the probe path and the try-acquire-again path. fcntl(2) record locks are scoped to (process, inode), so that close silently released the real lock too, even though the reservation's own fd stayed open. Check the in-process claim before opening anything and return early in both places; neither branch ever needed more than the registry. Also hardens the child-process verdict in the regression test: it now reports held only when open() succeeded and F_SETLK failed with EACCES or EAGAIN, so a wrong path or an unrelated errno can no longer read as a pass. Signed-off-by: Amir Fathi <[email protected]>
6313175 to
8bd9d37
Compare
|
Pushed both. |
pr-smoke (macos-14) was cancelled mid-run (operation was canceled) at an unrelated UI-server check, with all 30+ other legs green; ci-ok failed only as a consequence. Empty commit to get a fresh run. Signed-off-by: Amir Fathi <[email protected]>
343cc94 to
1c840df
Compare
|
Thanks for applying both the comment correction and the named child verdict. I have read the latest update; the remaining recheck and landing decision are maintainer-side work. The separate cold-start issue remains outside this fix, as already documented. |
|
Thank you for completing the comment correction and named child verdict. The remaining recheck of the lifetime-lock fix is on us, with the separate cold-start issue still outside this change. Sorry for the delay—I’ll follow up soon. |
Signed-off-by: Martin Vogel <[email protected]>
|
Thanks again, @AmirF194. The earlier red was not your change: a published language-count check, since repaired on main, stopped the test legs before the suites ran (your two new lock tests did run and pass on the TSan legs). We have merged current main into the branch with a plain signed-off merge commit (9c48545); your commits are unchanged. Next on our side is the recheck we promised: proving the two tests fail with the old probe and pass with yours. If you push again, please pull first so the merge commit is kept. |
Signed-off-by: Martin Vogel <[email protected]>
|
Thank you for your patience, @AmirF194. We did the recheck we promised: with the old probe and try_acquire code put back, both of your new tests fail exactly as intended (the separate process takes the lock), and with your change they pass, together with the rest of the daemon suites (264 passed) on your branch merged with current main. Main has moved in src/daemon/ipc.c and tests/test_daemon_ipc.c since your last CI run (the heartbeat change in #2289, which refreshes the held lock through the held descriptor and does not interact with your fix), so we merged current main into your branch (no conflicts, your change is unchanged) for a fresh CI run. Please pull before pushing again. We will merge as soon as the required checks are green. |
|
Merged in c80a256. Thank you, @AmirF194, for finding and fixing this one. It is a genuinely subtle bug: fcntl record locks belong to the process, not to the descriptor, so a process that already held the lifetime lock was quietly giving it away every time it opened and closed a second descriptor on the lock file. You spotted it while reading code for a different issue, kept it separate from that issue, and proved it with a test that takes the lock from a truly independent process instead of going through our own APIs. That is exactly the kind of evidence that makes a fix like this easy to trust. You also took every round of review in stride: the same no-descriptor treatment for the try_acquire-again path with its own sibling test, the three-way child verdict so a wrong path can never pass as "held", the free-after-release check as a positive control, and the comment rewording our own security scan needed. Before merging we did the recheck we promised: with the old probe and try_acquire code put back, both of your new tests fail exactly as intended (the separate process takes the lock), and with your change they pass, along with the rest of the daemon suites. As discussed, this hardens every process that holds the lifetime lock, and it is separate from the #2162 cold-start reports, which stay open on their own. Thanks again for the care you put into this. Findings like this are what make the coordination layer trustworthy. |
While chasing #2162 I found a real, separate bug in
posix_lifetime_lock_probe(src/daemon/ipc.c): when the in-process registry says this process already holds the lifetime lock, it still opens a throwaway fd on the lock file just to close it again and return 1.fcntl(2)record locks are scoped to (process, inode), not (fd, inode), so that close silently drops the real lock too, even though the reservation's own fd stays open.Two probes back to back are enough to lose it. I wrote a test that acquires the reservation, probes it twice, then forks a genuinely separate process that tries to take the same lock directly. On main it succeeds, the lock is gone. With the fix it fails as expected.
Fix: check the in-process claim before opening anything and return early. That branch only ever needed to trust the registry, not touch the file.
scripts/test.sh --suites daemon_ipc: the new test fails on main (child_result == 1), 50 passed / 1 skipped with the fix.Not a fix for #2162 itself, I ran into this while reading that code, it's a separate bug.