Repository navigation
fix(solidlsp): fail the pending request when the stdin write fails - #2030
Conversation
| @@ -662,8 +662,11 @@ def _send_payload(self, payload: StringDict) -> None: | |||
| self._process.stdin.writelines(msg) | |||
There was a problem hiding this comment.
Ran this against 6e9d8dc in a clean python:3.11-slim container, mocking a real closed pipe instead of a MagicMock so writelines hits actual closed-file semantics.
Line 662: writing to a stdin that _stop's _safely_close_pipe (line 565) already closed raises ValueError: I/O operation on closed file, not one of the caught (BrokenPipeError, ConnectionResetError, OSError). _send_payload's guard at line 654 only checks for None, not .closed, so in the window between _safely_close_pipe(self._process.stdin) and _stop's finally: self._process = None (lines 554-561), a concurrent send_request hits this. The exception propagates out of _send_payload uncaught, so the just-registered request is never passed to _cancel_pending_requests and stays in _pending_requests until its full timeout, the exact failure mode #2004 and this PR are otherwise fixing.
raised: ValueError('I/O operation on closed file.')
pending_requests after call: {1: Request[id=..., status='pending', request_id=1, method='textDocument/hover']}
Adding ValueError to the except tuple, or checking self._process.stdin.closed in the guard, would close this the same way the PR already closes the broken-pipe case.
There was a problem hiding this comment.
Confirmed on 6e9d8dc with a real pipe rather than a stand-in: _safely_close_pipe(self._process.stdin), then _send_payload raises ValueError('I/O operation on closed file.') out of the call, and the entry is still in _pending_requests afterwards. Through send_request on another thread the caller gets that ValueError and the request is never cancelled, so both halves of your description reproduce.
Addressed in 900a774, which takes the first option: ValueError is now in the caught tuple, so a closed stdin is routed to _cancel_pending_requests exactly like a broken pipe. I left the guard alone because a guard that only checks .closed and returns is the thing that strands the request here; failing the request is the point of the change.
There is a second window next to yours in the same three lines. _stop waits inside terminate(timeout=...) and only nulls _process in its finally, so _process stays set for the whole termination, and the function read self._process.stdin twice (guard, then write). A thread switch between those reads turns this into AttributeError instead. The process and its stdin are bound to locals once now, which is what TCPLanguageServer._send_payload already does with its socket.
The new test uses a real subprocess whose stdin is closed by _stop's own _safely_close_pipe. It fails with the ValueError on 6e9d8dc and passes on 900a774, and taking ValueError back out of the tuple fails only that test. Restoring the double read changes no test result, so the local bindings stay untested rather than pinned to the shape of the code.
One thing your report sent me back through the repo's own gates: this PR had no CHANGELOG.md entry, though the PR template's checklist asks for one, and the "up to 235 s" figure I had written in the summary does not exist anywhere in the code (the default is DEFAULT_LS_REQUEST_TIMEOUT = 300.0). Both corrected in 900a774 and in the description.
900a774 to
c514eda
Compare
|
Rebased onto main ( Head is now c514eda. Re-measured on the rebased tree: the four tests pass, dropping |
c514eda to
7e5f463
Compare
|
Rebased onto current Nothing semantic moved.
The only content change is the Re-measured on the rebased tree, fresh venv on Python 3.11 (project deps installed from Mutation, to confirm the suite still bites here: removing One thing I should say rather than let it pass: the part you called "the local bindings close the second window" is not covered by a test. I checked — reverting the single-read of CI on the rebased head is still in flight; I will not treat it as green until the runs report. |
|
Follow-up commit After the rebase the They were latent in my test code before and surfaced once upstream made Two changes, both inside the test:
Re-measured after the change, in this tree: No product code moved, so the |
Fixes oraios#2004. Signed-off-by: fei <[email protected]>
A stdin which `_stop` closed raises ValueError, not OSError, so the write failure escaped the handler and the request stayed pending. Read the process and its stdin once so the same race cannot surface as an attribute error. Adds the CHANGELOG.md entry required by CONTRIBUTING.md. Reported by @AmirF194 in review. Signed-off-by: fei <[email protected]>
901ac17 to
d98a59a
Compare
|
Rebased onto current
Nothing semantic moved. Both code blobs are byte-identical to the
The only content change is the changelog entry, which collided with the Re-measured on the rebased tree, fresh venv (
CI is running on The description's base/head hashes and the CI attribution have been updated to match. |
|
Follow-up on the CI state of
What I can establish from here: the same job is green on Everything else that has finished is green: Codespell, Docs Build, Python Versions, CodeQL, |
|
CI rollup for
So one red job, in a language-server group this change does not touch, on the one OS where I cannot run it. |
|
Supplementing my 09-20 comment above, because I have since found the evidence that turns "probably not mine" into "demonstrably not mine", and it also answers the re-run question. Heads that do not contain this change fail the same leg with the same signature. Their Conversely, the blob this PR introduces, Two clarifications to what I wrote on 09-20, both of which make this easier to read:
I still cannot run Windows here, so I am not naming a root cause — I am only reporting that the evidence does not implicate this diff. Local macOS controls, which I offer as controls and not as Windows evidence: |
|
Checking in before this goes stale. Is there anything blocking review, or is this simply lower in the queue? I will not ping again. |
|
@feiiiiii5 this is resolving an issue which was closed and marked as invalid. |
|
I read the thread. @tagawa0525's point stands, and I have reproduced it on current Your reading covers the case you were describing — a request already pending when the server dies is failed by the reader thread. What is left is the request registered after that thread has exited, and nothing on Driving the real Same script, only Two precisions, since #2003 is the natural place to push this back to. This is not the #2003 claim — that was about a timeout reaching the restart path, and you and @tagawa0525 agreed it should not: a response that times out does not imply the server is broken. This PR never turns a timeout into a restart; it only converts a detected pipe failure into Reproduction and what I did not run
I did not exercise the whole tool layer (
The abandoned If you would rather this not be a PR — it is a behaviour change to an error path, and #2004 is closed as invalid, so the bar for reopening it is yours, not mine — say so and I will close it. What I would ask is that it is not closed as "the reader thread already handles it", because that is the part I can now show is true for the pending case and false for this one. |
|
Thanks for your contribution. My point was that this issue cannot really occur in the context of Serena, but since we should treat SolidLSP as a library (which could be used independently), the change you proposed makes sense. |
Checklist
CONTRIBUTING.mdregarding the scope of PRs.CHANGELOG.md, which concisely describes the change.Summary
Fixes #2004:
StdioLanguageServer._send_payloadswallowed a failed stdin write and returned, leaving the request registered moments earlier in_pending_requestsuntil its timeout (300 s viaDEFAULT_LS_REQUEST_TIMEOUT, shorter for servers that callset_request_timeout, unbounded when the timeout isNone). A failed write means the server is gone, so the pending requests are now failed immediately withLanguageServerTerminatedException, the signal the restart path acts on.Changes
src/solidlsp/ls_process.py,StdioLanguageServer._send_payload:_cancel_pending_requests(LanguageServerTerminatedException("Stdio send error", self.ls_id, cause=e))rather than a bare return. Two precedents in this file already do exactly this:TCPLanguageServer._send_payloadon a send error, and the stdout-reader death path.ValueErrorjoins the caught exceptions. A stdin that_stop's_safely_close_pipe(line 555) already closed raisesValueError: I/O operation on closed file, which is not anOSError, so that write failure still escaped and stranded the request. Addressed in response to @AmirF194's report and reproduction.self._process.stdinafter the guard._stopcloses stdin, waits for the process insideterminate(timeout=...), and only then nulls_process, so a send from another thread can run the guard and the write against two different states;TCPLanguageServer._send_payloadbinds its socket to a local for the same reason.The method still returns normally when the write fails, so nothing new is raised into callers, and the success path is unchanged.
test/solidlsp/test_stdio_send_failure.py, four tests:LanguageServerTerminatedException, preserves the cause, and clears the table._stop's own_safely_close_pipedoes the same with aValueErrorcause. This one uses a real subprocess pipe rather than a stand-in, since whether a write hits closed-file semantics is the interpreter's decision._send_payload.CHANGELOG.md: entry under "General" in the Unreleased section.The branch sits on
mainc4dc91a7. The repository relicensed by component while this PR was open;src/solidlspandtest/solidlspstay MIT, and the new test file carries# SPDX-License-Identifier: MITas CONTRIBUTING.md requires for new files.Testing
All four tests against the base
c4dc91a7with only this test file copied in: 3 failed, 1 passed. That run pinsPYTHONPATHto the base tree and printssolidlsp.__file__as/private/tmp/r24-serena-base/src/solidlsp/__init__.py, because the editable install otherwise resolvessolidlspto the branch checkout and the base run proves nothing. The failing three aretest_broken_pipe_fails_pending_request_with_terminated,test_closed_stdin_fails_pending_request_with_terminatedandtest_failed_write_does_not_raise_from_send_payload; the closed-stdin failure isValueError: I/O operation on closed fileand the two that assert the request is failed reachTimeoutError: Request timed out (timeout=10), which is the stranding from #2004. Against the current headd98a59ad: 4 passed, re-measured in a fresh Python 3.11.15 venv created from this branch (uv pip install -e ".[dev]"), withsolidlsp.__file__printed as/private/tmp/r24-serena-2030/src/solidlsp/__init__.pyso the run is known to have exercised this tree. The other unmarked tests that drive this module also pass on the changed tree:test_content_modified_retry.py,test_pdeathsig.pyandtest_process_group_cleanup.pytogether report 20 passed, 2 skipped, where the skips arePR_SET_PDEATHSIG is Linux-specific.Mutation: taking
ValueErrorback out of the except tuple fails exactly the closed-stdin test and leaves the other three passing, so that test is pinned to the claim. Restoring the secondself._process.stdinread changes no test result; the local bindings are deliberately untested, because a test for them would be implementation-coupled, which.serena/memories/critical_info.mdrules out.ls_process.pywas restored to byte-identical after each mutation (sha256de0bc405ef4d7564on the current head, re-checked after the rebase).Gates on the current head:
ruff checkandruff format --checkat the pinned 0.12.5 andty checkat the pinned 0.0.24, all clean on both changed Python files (ty check --python <venv python> test/solidlsp/test_stdio_send_failure.py src/solidlsp/ls_process.py-> "All checks passed!"). Project-wide,ty checkreports the same 49 diagnostics with identical text onmainc4dc91a7and on this branch, andruff check .the same 20, so nothing here adds to whatmainalready carries.Corrections to this description: an earlier version said "up to 235 s with Serena's defaults", a figure that does not exist (the default is
DEFAULT_LS_REQUEST_TIMEOUT = 300.0), and referred to_subscribe(None), which is not part of this codebase. Both are gone.Not PR-related failures
niche (ubuntu-latest)was red on the pre-rebase head, and it is red for a reason that predates the PR: the failing step runsuv run poe test -m "julia or r or perl or lean4 or nix or ocaml"and reports2 failed, 55 passed, the two failures beingtest/solidlsp/r/test_r_basic.py::TestRLanguageServer::test_find_definition_across_files[r]and...::test_find_references_across_files[r](the R server returns empty locations/references). Main reports the same two failures with the same counts: run 34983624289 on704e8c3dand run 34887825086 on18fa47bf. The last green on that job is 09-08. What this PR does run in CI: thecatch-allbatch takes everything outside the named language groups, and on the pre-rebase head itsubuntu-latestjob reported1354 passed, 13 skippedwith all four oftest/solidlsp/test_stdio_send_failure.pyPASSED, andcatch-all(all three OSes) is also whereruffandtyrun, all green. Ond98a59adthat evidence has not been re-produced by CI, since these jobs are still in progress; the local re-measurement above is what covers the new head.jvm (windows-latest)has since gone red ond98a59adwithtest/solidlsp/nextflow/test_nextflow_basic.py::TestNextflowLanguageServer::test_document_symbols_workflows[nextflow]and::test_full_symbol_tree[nextflow], both reporting an empty Nextflow symbol tree. I am reporting rather than explaining it: the same job is green onc4dc91a7, was green on the pre-rebase head carrying this exactls_process.pyblob, and is green on the two other open PRs in this repo; the Nextflow server on a Windows runner is not something I can reproduce here. A failed write on this code path surfaces asLanguageServerTerminatedExceptionor the caller's own timeout rather than as a returned-but-empty tree, which is the shape these two failures have, but I have not established what did cause them. A re-run of that job would settle whether it is transient.Signed-off-by: fei [email protected]