Skip to content

Commit cf31832

Browse files
committed
fix(input-handler): bound URL, zip, and git ingest paths
Closes NVIDIA#21. Closes NVIDIA#131. The per-file analysis cap (MAX_FILE_BYTES, 1 MB) sits downstream of InputHandler.resolve(), which pulls scan targets from URLs, zips, or git clones with no size budget of its own. As a result, the per-file cap can be defeated upstream: a multi-GB URL is a memory DoS, a zip bomb fills the disk before extraction is gated, and a large clone lands on disk regardless of the per-file analysis budget. PR NVIDIA#19 explicitly deferred this; this PR addresses the deferred work. Adds two ingest budgets enforced at every remote/archive ingest path: - INGEST_MAX_BYTES (100 MiB): caps streamed URL downloads, total uncompressed size of zip archives, and post-clone disk usage of Git repos. - INGEST_MAX_ZIP_MEMBERS (10,000): caps the number of entries in a single zip (defends against the "many tiny files" zip-bomb variant). Per ingest path: - _download_file streams the body in 64 KiB chunks directly to a temp file inside the existing session temp dir, with a running byte counter. The cap check fires before each chunk is written, so the response body is never accumulated in memory. Content-Length is checked up-front when the server provides it (aborts before reading any body bytes); the streamed counter is authoritative if the header is missing or malformed. A breach mid-stream removes the partial file before the exception propagates, so an attacker who ships exactly INGEST_MAX_BYTES + 1 bytes cannot fill the temp dir. - _extract_zip sums ZipInfo.file_size across all members and checks the member count *before* calling extractall, so classic zip bombs (small archive, huge declared uncompressed size) are rejected without materialising any of the bomb on disk. - _clone_git measures the cloned tree's on-disk size after the existing 60s timeout completes and rejects + cleans up if it exceeds the cap. Symlinks are skipped to avoid runaway counts on malicious /dev/zero-style links. All limit breaches raise IngestLimitExceededError (subclass of ValueError so existing callers that catch ValueError keep working) with a clear message including the limit and the observed size. Adds 10 unit tests covering: under-cap success paths for URL/zip/clone; oversized Content-Length rejected before body read; chunked overflow caught by the streamed counter; *the partial download file is removed when a breach fires mid-stream*; *streaming to disk produces a file of the expected size with no intermediate in-memory concatenation*; declared-uncompressed-oversize zip rejected without extraction; member-count zip-bomb rejected; oversize clone rejected and cleaned up. Full suite: 730 passed, 12 skipped. README documents both caps and their relationship to the per-file analysis cap. Signed-off-by: Rohan Isawe <[email protected]>
1 parent d8a68ee commit cf31832

3 files changed

Lines changed: 527 additions & 14 deletions

File tree

‎README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,15 @@ skillspector scan https://github.com/user/my-skill
120120
skillspector scan ./my-skill.zip
121121
```
122122

123+
#### Size limits
124+
125+
SkillSpector enforces two independent caps on remote and archive inputs to bound the impact of oversized downloads and zip bombs:
126+
127+
- **Per-ingest cap**: `INGEST_MAX_BYTES` (100 MiB) — applied to streamed URL downloads, total uncompressed size of zip archives, and post-clone disk usage of Git repos.
128+
- **Zip member cap**: `INGEST_MAX_ZIP_MEMBERS` (10,000) — caps the number of entries in a single zip.
129+
130+
Note that the per-file 1 MB analysis cap (`MAX_FILE_BYTES`) is a separate, downstream limit: it bounds what individual analyzers will read out of an already-ingested directory. The ingest caps above bound how much content can land on disk in the first place. A breach of either ingest cap fails closed with an `IngestLimitExceededError`.
131+
123132
### Output Formats
124133

125134
```bash

‎src/skillspector/input_handler.py‎

Lines changed: 149 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,11 @@
2323
- Single markdown files
2424
- Local directories
2525
26-
Ported from legacy implementation.
26+
Each remote/archive ingest path is bounded by ``INGEST_MAX_BYTES`` and
27+
``INGEST_MAX_ZIP_MEMBERS`` so that the per-file analysis caps downstream
28+
of ``InputHandler.resolve()`` are not defeated by an oversized download,
29+
a zip bomb, or a too-large git clone. This file fails closed on any
30+
ingest budget breach (closes #21 / #131).
2731
"""
2832

2933
from __future__ import annotations
@@ -41,6 +45,30 @@
4145

4246
logger = get_logger(__name__)
4347

48+
# Hard ceiling on what any single ingest path can pull into the temp dir.
49+
# Sized above the per-file analysis cap (``MAX_FILE_BYTES`` = 1 MB) so a
50+
# legitimate multi-file skill is not blocked at ingest, but tight enough
51+
# to bound memory / disk DoS from a malicious source.
52+
INGEST_MAX_BYTES = 100 * 1024 * 1024 # 100 MiB
53+
54+
# Hard ceiling on the number of members in a zip we are willing to
55+
# extract. Catches the "many tiny files" zip-bomb variant where each
56+
# entry is small but the entry count itself exhausts the filesystem.
57+
INGEST_MAX_ZIP_MEMBERS = 10_000
58+
59+
# Chunk size for streaming HTTP downloads. Small enough that the
60+
# byte-count breach check fires promptly; large enough to keep syscall
61+
# overhead reasonable on legitimate inputs.
62+
_DOWNLOAD_CHUNK_BYTES = 64 * 1024
63+
64+
65+
class IngestLimitExceededError(ValueError):
66+
"""Raised when an ingest path exceeds an ``INGEST_MAX_*`` budget.
67+
68+
Subclass of ``ValueError`` so existing callers that catch
69+
``ValueError`` from ``InputHandler.resolve()`` continue to work.
70+
"""
71+
4472

4573
class InputHandler:
4674
"""
@@ -64,8 +92,10 @@ def resolve(self, input_path: str) -> tuple[Path, str]:
6492
source_type is one of: "git", "url", "zip", "file", "directory"
6593
6694
Raises:
67-
ValueError: If input type cannot be determined
68-
FileNotFoundError: If local path doesn't exist
95+
ValueError: If input type cannot be determined, or if an
96+
ingest path exceeds ``INGEST_MAX_BYTES`` /
97+
``INGEST_MAX_ZIP_MEMBERS`` (``IngestLimitExceededError``).
98+
FileNotFoundError: If local path doesn't exist.
6999
"""
70100
input_path = input_path.strip()
71101

@@ -123,7 +153,7 @@ def _is_file_url(self, path: str) -> bool:
123153
return not self._is_git_url(path)
124154

125155
def _clone_git(self, url: str) -> Path:
126-
"""Clone a Git repository to a temporary directory."""
156+
"""Clone a Git repository to a temporary directory, bounded by ``INGEST_MAX_BYTES``."""
127157
temp_dir = self._get_temp_dir()
128158
clone_dir = temp_dir / "repo"
129159
try:
@@ -145,40 +175,126 @@ def _clone_git(self, url: str) -> Path:
145175
raise ValueError(
146176
"Git is not installed. Please install git to scan repositories."
147177
) from None
178+
179+
# Post-clone size check: a successful --depth 1 clone may still
180+
# land an arbitrarily large tree on disk. Reject (and clean up)
181+
# if it exceeds the ingest budget.
182+
total = _directory_size_bytes(clone_dir)
183+
if total > INGEST_MAX_BYTES:
184+
shutil.rmtree(clone_dir, ignore_errors=True)
185+
logger.warning(
186+
"Git clone of %s exceeded ingest cap: %d > %d bytes",
187+
url,
188+
total,
189+
INGEST_MAX_BYTES,
190+
)
191+
raise IngestLimitExceededError(
192+
f"Git clone exceeded ingest cap: {total} bytes > "
193+
f"INGEST_MAX_BYTES ({INGEST_MAX_BYTES})"
194+
)
148195
return clone_dir
149196

150197
def _download_file(self, url: str) -> Path:
151-
"""Download a file from URL to a temporary directory."""
198+
"""Download a file from URL to a temporary directory.
199+
200+
Streams the body to disk in chunks while running a byte counter.
201+
The cap check fires before each chunk is written, so a breach
202+
aborts immediately without accumulating the body in memory. A
203+
partial file produced by a mid-stream breach is removed before
204+
the exception propagates.
205+
"""
152206
temp_dir = self._get_temp_dir()
153207
parsed = urlparse(url)
154208
filename = Path(parsed.path).name or "SKILL.md"
209+
# Write to a stable target path inside the temp dir so we can
210+
# rename / move it after the download succeeds without ever
211+
# holding the body in memory. Use a sentinel name for the
212+
# download itself; we rename / replace at the end.
213+
download_path = temp_dir / "_download.partial"
214+
content_type = ""
155215
try:
156216
with httpx.Client(follow_redirects=True, timeout=30) as client:
157-
response = client.get(url)
158-
response.raise_for_status()
159-
content = response.content
217+
with client.stream("GET", url) as response:
218+
response.raise_for_status()
219+
content_type = response.headers.get("content-type", "")
220+
# Cheap up-front check: trust Content-Length when the
221+
# server provides it, so we abort before reading any
222+
# body bytes. Streaming check below covers the case
223+
# where the header is missing or wrong.
224+
declared = response.headers.get("content-length")
225+
if declared is not None:
226+
try:
227+
declared_bytes = int(declared)
228+
except ValueError:
229+
# Malformed header — fall through to the
230+
# streamed byte counter, which is authoritative.
231+
declared_bytes = None
232+
if declared_bytes is not None and declared_bytes > INGEST_MAX_BYTES:
233+
raise IngestLimitExceededError(
234+
f"Download exceeded ingest cap: "
235+
f"Content-Length {declared} bytes > "
236+
f"INGEST_MAX_BYTES ({INGEST_MAX_BYTES})"
237+
)
238+
239+
received = 0
240+
with download_path.open("wb") as out:
241+
for chunk in response.iter_bytes(_DOWNLOAD_CHUNK_BYTES):
242+
received += len(chunk)
243+
if received > INGEST_MAX_BYTES:
244+
raise IngestLimitExceededError(
245+
f"Download exceeded ingest cap: streamed "
246+
f"{received} bytes > INGEST_MAX_BYTES "
247+
f"({INGEST_MAX_BYTES})"
248+
)
249+
out.write(chunk)
160250
except httpx.HTTPError as e:
251+
# Best-effort cleanup of any partial download.
252+
download_path.unlink(missing_ok=True)
161253
logger.warning("Download failed for %s: %s", url, e)
162254
raise ValueError(f"Failed to download file: {e}") from e
163-
if filename.endswith(".zip") or (
164-
response.headers.get("content-type", "").startswith("application/zip")
165-
):
255+
except IngestLimitExceededError:
256+
# Don't leave the partial bomb on disk.
257+
download_path.unlink(missing_ok=True)
258+
raise
259+
260+
is_zip = filename.endswith(".zip") or content_type.startswith("application/zip")
261+
if is_zip:
166262
zip_path = temp_dir / "download.zip"
167-
zip_path.write_bytes(content)
263+
download_path.replace(zip_path)
168264
return self._extract_zip(zip_path)
169265
file_path = temp_dir / filename
170-
file_path.write_bytes(content)
266+
download_path.replace(file_path)
171267
return temp_dir
172268

173269
def _extract_zip(self, zip_path: Path) -> Path:
174-
"""Extract a zip file to a temporary directory."""
270+
"""Extract a zip file, bounded by ``INGEST_MAX_BYTES`` and ``INGEST_MAX_ZIP_MEMBERS``.
271+
272+
Sums ``ZipInfo.file_size`` (uncompressed size) across all members
273+
before extracting and refuses to extract if either the total or
274+
the member count exceeds the cap. This rejects classic zip
275+
bombs (small archive, huge declared uncompressed size) without
276+
materialising any of the bomb on disk.
277+
"""
175278
if not zip_path.exists():
176279
raise FileNotFoundError(f"Zip file not found: {zip_path}") from None
177280
temp_dir = self._get_temp_dir()
178281
extract_dir = temp_dir / "extracted"
179282
extract_dir.mkdir(exist_ok=True)
180283
try:
181284
with zipfile.ZipFile(zip_path, "r") as zf:
285+
infos = zf.infolist()
286+
if len(infos) > INGEST_MAX_ZIP_MEMBERS:
287+
raise IngestLimitExceededError(
288+
f"Zip exceeded ingest cap: {len(infos)} members > "
289+
f"INGEST_MAX_ZIP_MEMBERS ({INGEST_MAX_ZIP_MEMBERS})"
290+
)
291+
total_uncompressed = sum(info.file_size for info in infos)
292+
if total_uncompressed > INGEST_MAX_BYTES:
293+
raise IngestLimitExceededError(
294+
f"Zip exceeded ingest cap: uncompressed "
295+
f"{total_uncompressed} bytes > INGEST_MAX_BYTES "
296+
f"({INGEST_MAX_BYTES})"
297+
)
182298
zf.extractall(extract_dir)
183299
except zipfile.BadZipFile:
184300
logger.warning("Invalid zip or extract failed: %s", zip_path)
@@ -196,3 +312,22 @@ def _wrap_single_file(self, file_path: Path) -> Path:
196312
dest = temp_dir / file_path.name
197313
shutil.copy2(file_path, dest)
198314
return temp_dir
315+
316+
317+
def _directory_size_bytes(path: Path) -> int:
318+
"""Return the total size of all regular files under *path*, in bytes.
319+
320+
Symlinks are not followed (``Path.is_file()`` returns ``True`` only
321+
for regular files), so a malicious symlink to ``/dev/zero`` cannot
322+
cause a runaway count.
323+
"""
324+
total = 0
325+
for p in path.rglob("*"):
326+
if p.is_file() and not p.is_symlink():
327+
try:
328+
total += p.stat().st_size
329+
except OSError:
330+
# File disappeared mid-walk (race with concurrent fs ops).
331+
# Skip rather than fail the whole ingest.
332+
continue
333+
return total

0 commit comments

Comments
 (0)