Skip to content

fix: retain concurrency permits until timed-out model work finishes - #3977

Merged
maziyarpanahi merged 7 commits into
maziyarpanahi:masterfrom
snowyukitty:fix/issue-3902-timeout-permit-accounting
Oct 11, 2026
Merged

maziyarpanahi merged 7 commits into
maziyarpanahi:masterfrom
snowyukitty:fix/issue-3902-timeout-permit-accounting

Conversation

@snowyukitty

@snowyukitty snowyukitty commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

With concurrency limited to one, a timed-out model call could keep running after its request released the permit, allowing a second model call to start. The synthetic reproduction changes from two 504 responses and peak concurrency two to 504 followed by 503 service_busy, with peak concurrency one.

Type of change

  • Bug fix
  • Documentation and test updates

Changes made

  • Keep concurrency permits and existing inflight drain accounting attached to actual threadpool or batch completion, including cancellation and parallel GraphQL resolvers.
  • Return completed workers' capacity so waiting GraphQL fields can progress before the HTTP response finishes.
  • Continue cancelling abandoned batch jobs before dispatch; dispatched jobs retain capacity until completion or failure.
  • Expose the unlabelled openmed_service_orphaned_work gauge and consume late failures without logging input. Preserve the REST 504 envelope and existing GraphQL resolver error masking.
  • Following review, clarify that drain and orphaned-work accounting also apply without concurrency or rate limits, and document that the worker and request cleanup may release the same idempotent permit.

Testing

Fresh checks on Python 3.12.14 used the frozen development lockfile, offline synthetic fixtures and no model weights. Repository-policy checks used a disposable index of the tracked source manifest and original file modes.

  • make format, make lint, make type-check, make format-check: passed.
  • python -m pytest tests/unit/service -q -o faulthandler_timeout=120: 575 passed, 5 skipped.
  • python -m pytest tests/ -q -o faulthandler_timeout=120: 23,453 passed, 188 skipped.
  • This merge-conflict follow-up preserves all production and regression-test ASTs from the published revision and all upstream changelog entries. The throttle runtime AST is unchanged. Regression cases cover timeout, HTTP cancellation, late worker failure, queued batch removal, capacity recovery, privacy-safe errors, and GraphQL resolver concurrency limits of one and two.
  • Earlier baseline differential evidence and delayed-dispatch counterfactuals remain recorded at 9678cf6; this current-master merge does not change their runtime or test code. GitHub Actions results remain a separate check after this update.

Documentation and dependencies

Updated docs/rest-service.md, CHANGELOG.md, and lifecycle documentation. No new dependencies.

Overlapping work

#3960 changes the admission predicate in ServiceThrottle.dispatch, and #3614 changes the GraphQL mount. Their behavior is separate from timeout ownership. If either merges first, this branch needs to preserve its admission/mount behavior when updating; the throttle predicate must also retain worker tracking when limits are unset.

Checklist

  • Read the contributing, code of conduct and maintainer guides.
  • Performed a self-review and kept changes within timeout/permit accounting.
  • Added native regressions and reran the repository checks.

Fixes #3902.

@kokokoXUY kokokoXUY left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head b7b9c0296791e3a09abd640391ed2a41595039ca against current master (252806aa946c200f3857cc0f6fda37fd83a5cc85).

"Timing out must not free the slot while the model work it started is still running" is a claim that either reproduces or it does not, so I ran both sides locally instead of reading the diff only.

Before the change (current master sources)

I took this PR's own test_timeout_retains_capacity_until_model_work_finishes unchanged, except that the two assertions on the new openmed_service_orphaned_work gauge and on app.state.inflight are neutralised — those signals do not exist on master:

python -m pytest tests/unit/service/test_r211_master_repro.py -q -p no:randomly -k timeout_retains

6 failed, 15 deselected in 3.84s
tests/unit/service/test_r211_master_repro.py:387: _assert_error_payload(busy, 503, "service_busy")
E   assert 200 == 503
+   where 200 = <Response [200 OK]>.status_code

All six parametrisations (rest / batch / graphql × worker_fails False / True) fail the same way: once the first request has returned 504, the next request is served 200 instead of 503 service_busy, i.e. the request whose model work still occupies the only slot no longer holds it. That is the #3902 behaviour, reproduced from this PR's own regression test.

After the change (this PR's sources on the same tree)

python -m pytest tests/unit/service/test_r211_pr_repro.py -q -p no:randomly
21 passed in 4.60s

python -m pytest tests/unit/service/test_r211_pr_repro.py -q -p no:randomly -k "timeout_retains or cancelled_http_wait"
9 passed, 12 deselected in 3.72s

The whole file passes with the new tests in it, so the change fixes the reproduction without disturbing the existing throttle tests.

Additional check: what happens when no limit is configured

Since dispatch no longer short-circuits on not self.enabled (openmed/service/throttle.py:298), I wanted to know whether the retained tokens behave sanely on a server with no concurrency limit and no rate limit at all. A direct TestClient probe (with ModelLoader and analyze_text stubbed the way the new tests do) gives:

first POST  -> 504 {"error": {"code": "timeout", "details": {"timeout_seconds": 0.1}}}
right after -> app.state.inflight == 1, openmed_service_orphaned_work == 1
work ends   -> app.state.inflight == 0, openmed_service_orphaned_work == 0
next POST   -> 200      (model calls: 2, peak concurrent: 1)

So the accounting is not confined to deployments that set OPENMED_SERVICE_RATE_LIMIT_MAX_CONCURRENCY: the readiness/shutdown drain token and the new gauge cover unfinished model work on unthrottled servers too, and they do return to zero. Shutdown cannot hang on this — app.py:867 still bounds the drain wait by shutdown_drain_seconds. This follows from ConcurrencyGate.acquire() returning None when disabled (throttle.py:133-134) and RequestWork.acquire_model_slot() returning None for a request that holds no permit (throttle.py:206-207), so no ServiceBusyError can be raised there.

Two non-blocking notes

  1. Docs wording. Both docs/rest-service.md and the CHANGELOG read as if retaining permits and drain tokens only matters under a configured concurrency bound. Given the probe above, a half-sentence saying that this also applies when no limit is configured would keep readers from assuming unthrottled servers are unaffected. Not a blocker.

  2. Documented invariant. RequestWork._finish_if_idle() calls self._permit.release() (throttle.py:246-247) even though a worker's release_slot may have released the same permit already. That is safe only because _ConcurrencyPermit.release() is idempotent (throttle.py:101-106). One comment on RequestWork (or an assert) recording that dependency would prevent a later refactor of the permit from turning the second release into an extra semaphore credit, which would silently raise the effective concurrency bound. Not a blocker.

Things I checked that look right

  • Admission ownership has no leak I could find. work_done is settled only inside _release_admission_locked (openmed/service/batcher.py:585-591, guarded by job.admission_released), and that is reached from the normal path, the cancelled-wait path (batcher.py:282-287), _remove_pending_job (batcher.py:562-580), close() (batcher.py:338-363) and from _run_batch's finally for jobs already dispatched into a batch. submit also settles it on any BaseException before the job is queued (batcher.py:278-280). I could not construct a path where an admitted request keeps its permit after its work ends, nor one where the permit is released while its work still runs.
  • Timeouts still do not cancel started work. cancel_on_timeout=True is used only for the batched-submit path, where cancellation concerns a job that has not started; REST and GraphQL model work is wrapped with asyncio.shield, matching the docstring and the new docs paragraph.
  • The gauge stays value-free. It is unlabelled and counts admitted requests with outstanding work; nothing attaches request content or identifiers to it.
  • The REST 504 envelope is untouched by the diff, and the probe above shows the same error.code=timeout / details.timeout_seconds shape as before.
  • The overlap note matches the diff. _should_throttle (throttle.py:367-370) is unchanged; this only removes the not self.enabled short-circuit in front of it, so the predicate #3960 touches is not modified here.

I did not run the full tests/unit/service suite or the full 23k-test run myself — the pass counts in the description are the author's; the runs quoted above are the ones I executed.

@kokokoXUY

Copy link
Copy Markdown
Contributor

Re-checked at 9678cf68eb65: the new commit only adds synchronisation to the parallel GraphQL checks
(tests/unit/service/test_throttle.py, +19/-9, 4 hunks) and touches no implementation file, so the
reproduction in my review above still applies unchanged to this head.

Disclosure: this note was prepared with an AI coding assistant under my direction.

@snowyukitty

Copy link
Copy Markdown
Contributor Author

Updated in 8bb62fb to cover both notes: the documentation and changelog now explicitly say that drain-token and orphaned-work accounting also cover deployments without concurrency or rate limits, and _finish_if_idle explains why releasing a permit already released by a worker is safe.

This follow-up changes only documentation and a code comment; the throttle runtime AST is unchanged. Fresh checks passed for formatting, lint and type checking, the service suite (575 passed, 5 skipped), and the full suite (23,145 passed, 230 skipped).

@maziyarpanahi maziyarpanahi added bug Something isn't working P1 High roadmap-v3 OpenMed V3 roadmap backlog labels Oct 11, 2026
@maziyarpanahi

Copy link
Copy Markdown
Owner

Thank you @snowyukitty for your first OpenMed contribution and the thorough timeout and cancellation coverage. I kept your commits intact and added maintainer follow-ups to integrate current master, preserve the newer route-admission rules, and test unfinished-worker accounting with no throttle limits configured.

I also corrected the synthetic logging subprocess test to capture output in a temporary file while retaining every privacy assertion and request timeout. The same launch timeout appeared in another PR's full run; output-file capture now passes the complete service suite.

Validation:

  • make format, make lint, make format-check, and make type-check: passed.
  • .venv/bin/python -m pytest tests/unit/service -q -o faulthandler_timeout=120: 933 passed, 5 skipped.
  • make docs-build: passed against current master.
  • OPENMED_SKIP_HF_TESTS=1 .venv/bin/python -m pytest tests/ -q -o faulthandler_timeout=120: 29,213 passed, 178 skipped.

Hosted workflows are awaiting approval; readiness is based on the completed local validation above. Issue #3902's concurrency, cleanup, timeout-envelope and privacy criteria are covered. Ready for squash merge. Thank you for helping improve the service runtime!

@maziyarpanahi
maziyarpanahi merged commit 75e3a14 into maziyarpanahi:master Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working P1 High roadmap-v3 OpenMed V3 roadmap backlog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hold concurrency permits until timed-out model work actually stops

3 participants