Repository navigation
fix: retain concurrency permits until timed-out model work finishes - #3977
maziyarpanahi merged 7 commits into
Conversation
kokokoXUY
left a comment
There was a problem hiding this comment.
Reviewed at head b7b9c0296791e3a09abd640391ed2a41595039ca against current master (252806aa946c200f3857cc0f6fda37fd83a5cc85).
"Timing out must not free the slot while the model work it started is still running" is a claim that either reproduces or it does not, so I ran both sides locally instead of reading the diff only.
Before the change (current master sources)
I took this PR's own test_timeout_retains_capacity_until_model_work_finishes unchanged, except that the two assertions on the new openmed_service_orphaned_work gauge and on app.state.inflight are neutralised — those signals do not exist on master:
python -m pytest tests/unit/service/test_r211_master_repro.py -q -p no:randomly -k timeout_retains
6 failed, 15 deselected in 3.84s
tests/unit/service/test_r211_master_repro.py:387: _assert_error_payload(busy, 503, "service_busy")
E assert 200 == 503
+ where 200 = <Response [200 OK]>.status_code
All six parametrisations (rest / batch / graphql × worker_fails False / True) fail the same way: once the first request has returned 504, the next request is served 200 instead of 503 service_busy, i.e. the request whose model work still occupies the only slot no longer holds it. That is the #3902 behaviour, reproduced from this PR's own regression test.
After the change (this PR's sources on the same tree)
python -m pytest tests/unit/service/test_r211_pr_repro.py -q -p no:randomly
21 passed in 4.60s
python -m pytest tests/unit/service/test_r211_pr_repro.py -q -p no:randomly -k "timeout_retains or cancelled_http_wait"
9 passed, 12 deselected in 3.72s
The whole file passes with the new tests in it, so the change fixes the reproduction without disturbing the existing throttle tests.
Additional check: what happens when no limit is configured
Since dispatch no longer short-circuits on not self.enabled (openmed/service/throttle.py:298), I wanted to know whether the retained tokens behave sanely on a server with no concurrency limit and no rate limit at all. A direct TestClient probe (with ModelLoader and analyze_text stubbed the way the new tests do) gives:
first POST -> 504 {"error": {"code": "timeout", "details": {"timeout_seconds": 0.1}}}
right after -> app.state.inflight == 1, openmed_service_orphaned_work == 1
work ends -> app.state.inflight == 0, openmed_service_orphaned_work == 0
next POST -> 200 (model calls: 2, peak concurrent: 1)
So the accounting is not confined to deployments that set OPENMED_SERVICE_RATE_LIMIT_MAX_CONCURRENCY: the readiness/shutdown drain token and the new gauge cover unfinished model work on unthrottled servers too, and they do return to zero. Shutdown cannot hang on this — app.py:867 still bounds the drain wait by shutdown_drain_seconds. This follows from ConcurrencyGate.acquire() returning None when disabled (throttle.py:133-134) and RequestWork.acquire_model_slot() returning None for a request that holds no permit (throttle.py:206-207), so no ServiceBusyError can be raised there.
Two non-blocking notes
-
Docs wording. Both
docs/rest-service.mdand the CHANGELOG read as if retaining permits and drain tokens only matters under a configured concurrency bound. Given the probe above, a half-sentence saying that this also applies when no limit is configured would keep readers from assuming unthrottled servers are unaffected. Not a blocker. -
Documented invariant.
RequestWork._finish_if_idle()callsself._permit.release()(throttle.py:246-247) even though a worker'srelease_slotmay have released the same permit already. That is safe only because_ConcurrencyPermit.release()is idempotent (throttle.py:101-106). One comment onRequestWork(or an assert) recording that dependency would prevent a later refactor of the permit from turning the second release into an extra semaphore credit, which would silently raise the effective concurrency bound. Not a blocker.
Things I checked that look right
- Admission ownership has no leak I could find.
work_doneis settled only inside_release_admission_locked(openmed/service/batcher.py:585-591, guarded byjob.admission_released), and that is reached from the normal path, the cancelled-wait path (batcher.py:282-287),_remove_pending_job(batcher.py:562-580),close()(batcher.py:338-363) and from_run_batch'sfinallyfor jobs already dispatched into a batch.submitalso settles it on anyBaseExceptionbefore the job is queued (batcher.py:278-280). I could not construct a path where an admitted request keeps its permit after its work ends, nor one where the permit is released while its work still runs. - Timeouts still do not cancel started work.
cancel_on_timeout=Trueis used only for the batched-submit path, where cancellation concerns a job that has not started; REST and GraphQL model work is wrapped withasyncio.shield, matching the docstring and the new docs paragraph. - The gauge stays value-free. It is unlabelled and counts admitted requests with outstanding work; nothing attaches request content or identifiers to it.
- The REST
504envelope is untouched by the diff, and the probe above shows the sameerror.code=timeout/details.timeout_secondsshape as before. - The overlap note matches the diff.
_should_throttle(throttle.py:367-370) is unchanged; this only removes thenot self.enabledshort-circuit in front of it, so the predicate #3960 touches is not modified here.
I did not run the full tests/unit/service suite or the full 23k-test run myself — the pass counts in the description are the author's; the runs quoted above are the ones I executed.
|
Re-checked at Disclosure: this note was prepared with an AI coding assistant under my direction. |
|
Updated in This follow-up changes only documentation and a code comment; the throttle runtime AST is unchanged. Fresh checks passed for formatting, lint and type checking, the service suite (575 passed, 5 skipped), and the full suite (23,145 passed, 230 skipped). |
# Conflicts: # openmed/service/throttle.py
|
Thank you @snowyukitty for your first OpenMed contribution and the thorough timeout and cancellation coverage. I kept your commits intact and added maintainer follow-ups to integrate current I also corrected the synthetic logging subprocess test to capture output in a temporary file while retaining every privacy assertion and request timeout. The same launch timeout appeared in another PR's full run; output-file capture now passes the complete service suite. Validation:
Hosted workflows are awaiting approval; readiness is based on the completed local validation above. Issue #3902's concurrency, cleanup, timeout-envelope and privacy criteria are covered. Ready for squash merge. Thank you for helping improve the service runtime! |
Description
With concurrency limited to one, a timed-out model call could keep running after its request released the permit, allowing a second model call to start. The synthetic reproduction changes from two
504responses and peak concurrency two to504followed by503 service_busy, with peak concurrency one.Type of change
Changes made
openmed_service_orphaned_workgauge and consume late failures without logging input. Preserve the REST504envelope and existing GraphQL resolver error masking.Testing
Fresh checks on Python 3.12.14 used the frozen development lockfile, offline synthetic fixtures and no model weights. Repository-policy checks used a disposable index of the tracked source manifest and original file modes.
make format,make lint,make type-check,make format-check: passed.python -m pytest tests/unit/service -q -o faulthandler_timeout=120: 575 passed, 5 skipped.python -m pytest tests/ -q -o faulthandler_timeout=120: 23,453 passed, 188 skipped.9678cf6; this current-master merge does not change their runtime or test code. GitHub Actions results remain a separate check after this update.Documentation and dependencies
Updated
docs/rest-service.md,CHANGELOG.md, and lifecycle documentation. No new dependencies.Overlapping work
#3960 changes the admission predicate in
ServiceThrottle.dispatch, and #3614 changes the GraphQL mount. Their behavior is separate from timeout ownership. If either merges first, this branch needs to preserve its admission/mount behavior when updating; the throttle predicate must also retain worker tracking when limits are unset.Checklist
Fixes #3902.