Skip to content

Commit e5e5a21

Browse files
committed
fix(input-handler): bound URL, zip, and git ingest paths
Closes NVIDIA#21. Closes NVIDIA#131. The per-file analysis cap (MAX_FILE_BYTES, 1 MB) sits downstream of InputHandler.resolve(), which pulls scan targets from URLs, zips, or git clones with no size budget of its own. As a result, the per-file cap can be defeated upstream: a multi-GB URL is a memory DoS, a zip bomb fills the disk before extraction is gated, and a large clone lands on disk regardless of the per-file analysis budget. PR NVIDIA#19 explicitly deferred this; this PR addresses the deferred work. Adds two ingest budgets enforced at every remote/archive ingest path: - INGEST_MAX_BYTES (100 MiB): caps streamed URL downloads, total uncompressed size of zip archives, and post-clone disk usage of Git repos. - INGEST_MAX_ZIP_MEMBERS (10,000): caps the number of entries in a single zip (defends against the "many tiny files" zip-bomb variant). Per ingest path: - _download_file streams the body in 64 KiB chunks directly to a temp file inside the existing session temp dir, with a running byte counter. The cap check fires before each chunk is written, so the response body is never accumulated in memory. Content-Length is checked up-front when the server provides it (aborts before reading any body bytes); the streamed counter is authoritative if the header is missing or malformed. A breach mid-stream removes the partial file before the exception propagates, so an attacker who ships exactly INGEST_MAX_BYTES + 1 bytes cannot fill the temp dir. - _extract_zip sums ZipInfo.file_size across all members and checks the member count *before* calling extractall, so classic zip bombs (small archive, huge declared uncompressed size) are rejected without materialising any of the bomb on disk. - _clone_git measures the cloned tree's on-disk size after the existing 60s timeout completes and rejects + cleans up if it exceeds the cap. Symlinks are skipped to avoid runaway counts on malicious /dev/zero-style links. All limit breaches raise IngestLimitExceededError (subclass of ValueError so existing callers that catch ValueError keep working) with a clear message including the limit and the observed size. Adds 10 unit tests covering: under-cap success paths for URL/zip/clone; oversized Content-Length rejected before body read; chunked overflow caught by the streamed counter; *the partial download file is removed when a breach fires mid-stream*; *streaming to disk produces a file of the expected size with no intermediate in-memory concatenation*; declared-uncompressed-oversize zip rejected without extraction; member-count zip-bomb rejected; oversize clone rejected and cleaned up. Full suite: 730 passed, 12 skipped. README documents both caps and their relationship to the per-file analysis cap. Signed-off-by: Rohan Isawe <[email protected]>
1 parent 36cb67d commit e5e5a21

4 files changed

Lines changed: 575 additions & 20 deletions

File tree

‎README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,15 @@ skillspector scan https://github.com/user/my-skill
137137
skillspector scan ./my-skill.zip
138138
```
139139

140+
#### Size limits
141+
142+
SkillSpector enforces two independent caps on remote and archive inputs to bound the impact of oversized downloads and zip bombs:
143+
144+
- **Per-ingest cap**: `INGEST_MAX_BYTES` (100 MiB) — applied to streamed URL downloads, total uncompressed size of zip archives, and post-clone disk usage of Git repos.
145+
- **Zip member cap**: `INGEST_MAX_ZIP_MEMBERS` (10,000) — caps the number of entries in a single zip.
146+
147+
Note that the per-file 1 MB analysis cap (`MAX_FILE_BYTES`) is a separate, downstream limit: it bounds what individual analyzers will read out of an already-ingested directory. The ingest caps above bound how much content can land on disk in the first place. A breach of either ingest cap fails closed with an `IngestLimitExceededError`.
148+
140149
### Output Formats
141150

142151
```bash

‎src/skillspector/input_handler.py‎

Lines changed: 165 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,14 @@
2323
- Single markdown files
2424
- Local directories
2525
26-
Ported from legacy implementation.
26+
Each remote/archive ingest path is bounded by ``INGEST_MAX_BYTES`` and
27+
``INGEST_MAX_ZIP_MEMBERS`` so that the per-file analysis caps downstream
28+
of ``InputHandler.resolve()`` are not defeated by an oversized download,
29+
a zip bomb, or a too-large git clone. This file fails closed on any
30+
ingest budget breach (closes #21 / #131).
31+
32+
URL-based ingest is additionally gated by an SSRF host allowlist plus a
33+
private-IP check, and zip extraction is guarded against zip-slip.
2734
"""
2835

2936
from __future__ import annotations
@@ -62,6 +69,30 @@
6269
}
6370
)
6471

72+
# Hard ceiling on what any single ingest path can pull into the temp dir.
73+
# Sized above the per-file analysis cap (``MAX_FILE_BYTES`` = 1 MB) so a
74+
# legitimate multi-file skill is not blocked at ingest, but tight enough
75+
# to bound memory / disk DoS from a malicious source.
76+
INGEST_MAX_BYTES = 100 * 1024 * 1024 # 100 MiB
77+
78+
# Hard ceiling on the number of members in a zip we are willing to
79+
# extract. Catches the "many tiny files" zip-bomb variant where each
80+
# entry is small but the entry count itself exhausts the filesystem.
81+
INGEST_MAX_ZIP_MEMBERS = 10_000
82+
83+
# Chunk size for streaming HTTP downloads. Small enough that the
84+
# byte-count breach check fires promptly; large enough to keep syscall
85+
# overhead reasonable on legitimate inputs.
86+
_DOWNLOAD_CHUNK_BYTES = 64 * 1024
87+
88+
89+
class IngestLimitExceededError(ValueError):
90+
"""Raised when an ingest path exceeds an ``INGEST_MAX_*`` budget.
91+
92+
Subclass of ``ValueError`` so existing callers that catch
93+
``ValueError`` from ``InputHandler.resolve()`` continue to work.
94+
"""
95+
6596

6697
def _is_private_ip(host: str) -> bool:
6798
"""Return True if host resolves to a private/reserved IP address."""
@@ -103,8 +134,10 @@ def resolve(self, input_path: str) -> tuple[Path, str]:
103134
source_type is one of: "git", "url", "zip", "file", "directory"
104135
105136
Raises:
106-
ValueError: If input type cannot be determined
107-
FileNotFoundError: If local path doesn't exist
137+
ValueError: If input type cannot be determined, or if an
138+
ingest path exceeds ``INGEST_MAX_BYTES`` /
139+
``INGEST_MAX_ZIP_MEMBERS`` (``IngestLimitExceededError``).
140+
FileNotFoundError: If local path doesn't exist.
108141
"""
109142
input_path = input_path.strip()
110143

@@ -191,7 +224,7 @@ def _validate_url_host(self, url: str, allowed_hosts: frozenset[str]) -> str:
191224
return host
192225

193226
def _clone_git(self, url: str) -> Path:
194-
"""Clone a Git repository to a temporary directory."""
227+
"""Clone a Git repository to a temporary directory, bounded by ``INGEST_MAX_BYTES``."""
195228
self._validate_url_host(url, ALLOWED_GIT_HOSTS)
196229
temp_dir = self._get_temp_dir()
197230
clone_dir = temp_dir / "repo"
@@ -214,44 +247,139 @@ def _clone_git(self, url: str) -> Path:
214247
raise ValueError(
215248
"Git is not installed. Please install git to scan repositories."
216249
) from None
250+
251+
# Post-clone size check: a successful --depth 1 clone may still
252+
# land an arbitrarily large tree on disk before we can measure
253+
# it, so this is a fail-closed cap rather than a hard prefilter.
254+
# Residual window: within the 60s clone timeout an attacker can
255+
# transiently consume up to whatever the network + disk let
256+
# through before this check runs; bounded by the timeout, but
257+
# not zero. ``.git/`` objects are counted toward the cap, so a
258+
# legitimate repo with a working tree just under ``INGEST_MAX_BYTES``
259+
# can still be rejected once packfiles are added.
260+
total = _directory_size_bytes(clone_dir)
261+
if total > INGEST_MAX_BYTES:
262+
shutil.rmtree(clone_dir, ignore_errors=True)
263+
logger.warning(
264+
"Git clone of %s exceeded ingest cap: %d > %d bytes",
265+
url,
266+
total,
267+
INGEST_MAX_BYTES,
268+
)
269+
raise IngestLimitExceededError(
270+
f"Git clone exceeded ingest cap: {total} bytes > "
271+
f"INGEST_MAX_BYTES ({INGEST_MAX_BYTES})"
272+
)
217273
return clone_dir
218274

219275
def _download_file(self, url: str) -> Path:
220-
"""Download a file from URL to a temporary directory."""
276+
"""Download a file from URL to a temporary directory.
277+
278+
Streams the body to disk in chunks while running a byte counter.
279+
The cap check fires before each chunk is written, so a breach
280+
aborts immediately without accumulating the body in memory. A
281+
partial file produced by a mid-stream breach is removed before
282+
the exception propagates.
283+
"""
221284
self._validate_url_host(url, ALLOWED_DOWNLOAD_HOSTS)
222285
temp_dir = self._get_temp_dir()
223286
parsed = urlparse(url)
224287
filename = Path(parsed.path).name or "SKILL.md"
288+
# Write to a stable target path inside the temp dir so we can
289+
# rename / move it after the download succeeds without ever
290+
# holding the body in memory. Use a sentinel name for the
291+
# download itself; we rename / replace at the end.
292+
download_path = temp_dir / "_download.partial"
293+
content_type = ""
225294
try:
226295
with httpx.Client(follow_redirects=False, timeout=30) as client:
227-
response = client.get(url)
228-
response.raise_for_status()
229-
content = response.content
296+
with client.stream("GET", url) as response:
297+
response.raise_for_status()
298+
content_type = response.headers.get("content-type", "")
299+
# Cheap up-front check: trust Content-Length when the
300+
# server provides it, so we abort before reading any
301+
# body bytes. Streaming check below covers the case
302+
# where the header is missing or wrong.
303+
declared = response.headers.get("content-length")
304+
if declared is not None:
305+
try:
306+
declared_bytes = int(declared)
307+
except ValueError:
308+
# Malformed header — fall through to the
309+
# streamed byte counter, which is authoritative.
310+
declared_bytes = None
311+
if declared_bytes is not None and declared_bytes > INGEST_MAX_BYTES:
312+
raise IngestLimitExceededError(
313+
f"Download exceeded ingest cap: "
314+
f"Content-Length {declared} bytes > "
315+
f"INGEST_MAX_BYTES ({INGEST_MAX_BYTES})"
316+
)
317+
318+
received = 0
319+
with download_path.open("wb") as out:
320+
for chunk in response.iter_bytes(_DOWNLOAD_CHUNK_BYTES):
321+
received += len(chunk)
322+
if received > INGEST_MAX_BYTES:
323+
raise IngestLimitExceededError(
324+
f"Download exceeded ingest cap: streamed "
325+
f"{received} bytes > INGEST_MAX_BYTES "
326+
f"({INGEST_MAX_BYTES})"
327+
)
328+
out.write(chunk)
230329
except httpx.HTTPError as e:
330+
# Best-effort cleanup of any partial download.
331+
download_path.unlink(missing_ok=True)
231332
logger.warning("Download failed for %s: %s", url, e)
232333
raise ValueError(f"Failed to download file: {e}") from e
233-
if filename.endswith(".zip") or (
234-
response.headers.get("content-type", "").startswith("application/zip")
235-
):
334+
except IngestLimitExceededError:
335+
# Don't leave the partial bomb on disk.
336+
download_path.unlink(missing_ok=True)
337+
raise
338+
339+
is_zip = filename.endswith(".zip") or content_type.startswith("application/zip")
340+
if is_zip:
236341
zip_path = temp_dir / "download.zip"
237-
zip_path.write_bytes(content)
342+
download_path.replace(zip_path)
238343
return self._extract_zip(zip_path)
239344
file_path = temp_dir / filename
240-
file_path.write_bytes(content)
345+
download_path.replace(file_path)
241346
return temp_dir
242347

243348
def _extract_zip(self, zip_path: Path) -> Path:
244-
"""Extract a zip file to a temporary directory with path traversal protection."""
349+
"""Extract a zip file, bounded by ``INGEST_MAX_BYTES`` and ``INGEST_MAX_ZIP_MEMBERS``.
350+
351+
Sums ``ZipInfo.file_size`` (uncompressed size) across all members
352+
before extracting and refuses to extract if either the total or
353+
the member count exceeds the cap. This rejects classic zip
354+
bombs (small archive, huge declared uncompressed size) without
355+
materialising any of the bomb on disk. A zip-slip check on each
356+
member name is applied before extraction to reject entries whose
357+
resolved path escapes the extraction directory.
358+
"""
245359
if not zip_path.exists():
246360
raise FileNotFoundError(f"Zip file not found: {zip_path}") from None
247361
temp_dir = self._get_temp_dir()
248362
extract_dir = temp_dir / "extracted"
249363
extract_dir.mkdir(exist_ok=True)
250364
try:
251365
with zipfile.ZipFile(zip_path, "r") as zf:
366+
infos = zf.infolist()
367+
if len(infos) > INGEST_MAX_ZIP_MEMBERS:
368+
raise IngestLimitExceededError(
369+
f"Zip exceeded ingest cap: {len(infos)} members > "
370+
f"INGEST_MAX_ZIP_MEMBERS ({INGEST_MAX_ZIP_MEMBERS})"
371+
)
372+
total_uncompressed = sum(info.file_size for info in infos)
373+
if total_uncompressed > INGEST_MAX_BYTES:
374+
raise IngestLimitExceededError(
375+
f"Zip exceeded ingest cap: uncompressed "
376+
f"{total_uncompressed} bytes > INGEST_MAX_BYTES "
377+
f"({INGEST_MAX_BYTES})"
378+
)
379+
extract_root = extract_dir.resolve()
252380
for member in zf.namelist():
253381
member_path = (extract_dir / member).resolve()
254-
if not str(member_path).startswith(str(extract_dir.resolve())):
382+
if not str(member_path).startswith(str(extract_root)):
255383
raise ValueError(
256384
f"Zip entry '{member}' would escape extraction directory (zip-slip). "
257385
"Archive is potentially malicious."
@@ -273,3 +401,25 @@ def _wrap_single_file(self, file_path: Path) -> Path:
273401
dest = temp_dir / file_path.name
274402
shutil.copy2(file_path, dest)
275403
return temp_dir
404+
405+
406+
def _directory_size_bytes(path: Path) -> int:
407+
"""Return the total size of all regular files under *path*, in bytes.
408+
409+
Symlinks are explicitly skipped via ``Path.is_symlink()`` — note that
410+
``Path.is_file()`` follows symlinks and would otherwise return
411+
``True`` for a symlink pointing at a regular file, so the
412+
``not p.is_symlink()`` guard is load-bearing and must not be removed.
413+
This is what prevents a malicious symlink to ``/dev/zero`` (or any
414+
large file outside the walked tree) from inflating the count.
415+
"""
416+
total = 0
417+
for p in path.rglob("*"):
418+
if p.is_file() and not p.is_symlink():
419+
try:
420+
total += p.stat().st_size
421+
except OSError:
422+
# File disappeared mid-walk (race with concurrent fs ops).
423+
# Skip rather than fail the whole ingest.
424+
continue
425+
return total

0 commit comments

Comments
 (0)