You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e5e5a21
Browse filesBrowse the repository at this point in the historyBrowse files
fix(input-handler): bound URL, zip, and git ingest paths
ClosesNVIDIA#21. ClosesNVIDIA#131.
The per-file analysis cap (MAX_FILE_BYTES, 1 MB) sits downstream of
InputHandler.resolve(), which pulls scan targets from URLs, zips, or
git clones with no size budget of its own. As a result, the per-file
cap can be defeated upstream: a multi-GB URL is a memory DoS, a zip
bomb fills the disk before extraction is gated, and a large clone
lands on disk regardless of the per-file analysis budget. PR NVIDIA#19
explicitly deferred this; this PR addresses the deferred work.
Adds two ingest budgets enforced at every remote/archive ingest path:
- INGEST_MAX_BYTES (100 MiB): caps streamed URL downloads, total
uncompressed size of zip archives, and post-clone disk usage of
Git repos.
- INGEST_MAX_ZIP_MEMBERS (10,000): caps the number of entries in a
single zip (defends against the "many tiny files" zip-bomb variant).
Per ingest path:
- _download_file streams the body in 64 KiB chunks directly to a temp
file inside the existing session temp dir, with a running byte
counter. The cap check fires before each chunk is written, so the
response body is never accumulated in memory. Content-Length is
checked up-front when the server provides it (aborts before reading
any body bytes); the streamed counter is authoritative if the
header is missing or malformed. A breach mid-stream removes the
partial file before the exception propagates, so an attacker who
ships exactly INGEST_MAX_BYTES + 1 bytes cannot fill the temp dir.
- _extract_zip sums ZipInfo.file_size across all members and checks
the member count *before* calling extractall, so classic zip bombs
(small archive, huge declared uncompressed size) are rejected
without materialising any of the bomb on disk.
- _clone_git measures the cloned tree's on-disk size after the
existing 60s timeout completes and rejects + cleans up if it
exceeds the cap. Symlinks are skipped to avoid runaway counts on
malicious /dev/zero-style links.
All limit breaches raise IngestLimitExceededError (subclass of
ValueError so existing callers that catch ValueError keep working)
with a clear message including the limit and the observed size.
Adds 10 unit tests covering: under-cap success paths for URL/zip/clone;
oversized Content-Length rejected before body read; chunked overflow
caught by the streamed counter; *the partial download file is removed
when a breach fires mid-stream*; *streaming to disk produces a file of
the expected size with no intermediate in-memory concatenation*;
declared-uncompressed-oversize zip rejected without extraction;
member-count zip-bomb rejected; oversize clone rejected and cleaned
up. Full suite: 730 passed, 12 skipped.
README documents both caps and their relationship to the per-file
analysis cap.
Signed-off-by: Rohan Isawe <[email protected]>
SkillSpector enforces two independent caps on remote and archive inputs to bound the impact of oversized downloads and zip bombs:
143
+
144
+
-**Per-ingest cap**: `INGEST_MAX_BYTES` (100 MiB) — applied to streamed URL downloads, total uncompressed size of zip archives, and post-clone disk usage of Git repos.
145
+
-**Zip member cap**: `INGEST_MAX_ZIP_MEMBERS` (10,000) — caps the number of entries in a single zip.
146
+
147
+
Note that the per-file 1 MB analysis cap (`MAX_FILE_BYTES`) is a separate, downstream limit: it bounds what individual analyzers will read out of an already-ingested directory. The ingest caps above bound how much content can land on disk in the first place. A breach of either ingest cap fails closed with an `IngestLimitExceededError`.
0 commit comments