fix: hop scanner JSON parse/normalize off the event loop (CON-4/H5) - #67
Merged
Conversation
Each scanner's `_execute` parsed and normalized the full scanner report inline on the event loop. A large report (stdout is capped at 512 MiB; the archive's own run produced 7,072 findings) is seconds of pure CPU, freezing every coroutine during the parse — including the /healthz poll the container healthcheck restarts on. Run the parse in a worker thread via `anyio.to_thread.run_sync`, reusing the same primitive CON-5 used for blocking DB work. Both scanners' `parse_output` call the shared `load_json_output`, so hopping `parse_output` moves the json.loads and the normalization loop off the loop in one place; the parsing logic is untouched. Adds a regression test proving a slow (large-report) parse no longer starves the loop — a heartbeat coroutine keeps ticking while a blocking stand-in parse runs — plus thread-identity assertions for both engines. Records the fix (and the previously-absent record of this finding) in docs/ARCHIVE.md §14.
The token-bearing publish workflows already drop the persisted checkout token, but ci.yml's four checkout steps were missed. CI only needs contents: read and no step pushes or authenticates to the remote, so clear the token rather than leave it in .git/config for later steps. Closes the L24/SC-11 residual.
tyler-rich
added a commit
that referenced
this pull request
Jul 20, 2026
…w-remediation batch (#74) Re-verifies every finding across all six original code-review reports (including findings never carried into 00-summary.md) against the current state of dev, after the H5/CON-4 follow-up (#67) and the dev->main promotion (#70). Records the current STILL-OPEN backlog (frontend Priority-3 batch, SC-12/SC-14, D5b, test debt), deferred-by-decision items with their tracking refs, a resolved index, and the ARCHIVE.md section-14 entry gaps (#53 H1/SEC-1, #57 H9+H10, #65 D1/D2/R1-R6, #59).
tyler-rich
added a commit
that referenced
this pull request
Jul 31, 2026
The [0.2.0] section carried only what had been written into [Unreleased] since roughly 2026-07-24. Everything promoted in #70 (2026-07-13, i.e. #53-#67) and the #77-#88 batch that followed had never been changelogged at all — v0.1.0 was tagged 2026-07-09 and #70 landed four days later — so about two dozen PRs of security and correctness work that ships in 0.2.0 was absent. Backfilled from the #70 commit range and the §14 entries for that batch, merged into the existing Added/Fixed/Changed/Security sections rather than added as a separate block: 0.2.0 is one release, and a changelog-within-a-changelog would make a reader track which half applies to them. Three of the release's upgrade-affecting items live here and were invisible before: the SSRF egress guard (SCRYE_ALLOW_INTERNAL_EGRESS, default off), the remote-clone-URL requirement for repository targets, and the master-key entropy floor — which refuses to start a v0.1.0 deployment whose key file holds a raw passphrase, and whose remedy is the boot-and-rotate escape hatch plus a backup/restore cycle, not a fresh key. Also records three contract-visible API changes narrower than the envelope: timestamps serialize with an explicit Z, /api/audit renamed entries -> items, and scan list rows dropped options/error in favour of has_error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes CON-4 / H5 — the one genuine miss from the review-remediation batch (
docs/reviews/fix-verification.md§1.A): scanner JSON parse/normalize was still running synchronously on the event loop, and the finding had nodocs/ARCHIVE.md§14 entry at all. Also folds in the L24/SC-11 residual as a separate commit.What changed
CON-4 (commit 1) — Each scanner's
_executenow runs its parse+normalize in a worker thread viaanyio.to_thread.run_syncinstead of inline on the loop:backend/app/scanners/trivy.py—await anyio.to_thread.run_sync(parse_output, result.stdout)backend/app/scanners/grype.py— same, returning(findings, version)Both
parse_outputfunctions call the sharedload_json_output(scanners/base.py) internally, so hoppingparse_outputmoves thejson.loadsand the per-finding normalization loop off the loop in one place — no separate change atbase.pywas needed. This reuses the same primitive CON-5 used for blocking DB work; the parsing logic itself is untouched.Why it matters: scanner stdout is capped at
SCRYE_SCANNER_MAX_OUTPUT_BYTES(512 MiB) and a large report (the archive's own run produced 7,072 findings) is seconds of pure CPU — on the loop that froze every request, including the/healthzpoll the container healthcheck restarts on.SC-11 (commit 2) — Added
persist-credentials: falseto.github/workflows/ci.yml's four checkout steps (the publish workflows already had it; ci.yml was missed). CI iscontents: readonly, so this drops the token rather than leaving it in.git/config.Tests
backend/tests/test_scanners.py:test_large_scan_parse_does_not_starve_the_event_loop— a deliberately blocking stand-in parse runs while a heartbeat coroutine keeps ticking on the loop; asserts the loop stayed responsive (verified to fail when the parse is reverted to run on-loop).test_trivy_parse_runs_off_the_event_loop/test_grype_parse_runs_off_the_event_loop— thread-identity assertions that the parse executes off the loop thread for both engines.Full backend suite green (553 passed, 3 pre-existing skips);
ruff+blackclean.See
docs/ARCHIVE.md§ Deviations for the §14 entry recording this fix and closing the record gap.