Repository navigation
Commit cf31832
committed
fix(input-handler): bound URL, zip, and git ingest paths
Closes NVIDIA#21. Closes NVIDIA#131.
The per-file analysis cap (MAX_FILE_BYTES, 1 MB) sits downstream of
InputHandler.resolve(), which pulls scan targets from URLs, zips, or
git clones with no size budget of its own. As a result, the per-file
cap can be defeated upstream: a multi-GB URL is a memory DoS, a zip
bomb fills the disk before extraction is gated, and a large clone
lands on disk regardless of the per-file analysis budget. PR NVIDIA#19
explicitly deferred this; this PR addresses the deferred work.
Adds two ingest budgets enforced at every remote/archive ingest path:
- INGEST_MAX_BYTES (100 MiB): caps streamed URL downloads, total
uncompressed size of zip archives, and post-clone disk usage of
Git repos.
- INGEST_MAX_ZIP_MEMBERS (10,000): caps the number of entries in a
single zip (defends against the "many tiny files" zip-bomb variant).
Per ingest path:
- _download_file streams the body in 64 KiB chunks directly to a temp
file inside the existing session temp dir, with a running byte
counter. The cap check fires before each chunk is written, so the
response body is never accumulated in memory. Content-Length is
checked up-front when the server provides it (aborts before reading
any body bytes); the streamed counter is authoritative if the
header is missing or malformed. A breach mid-stream removes the
partial file before the exception propagates, so an attacker who
ships exactly INGEST_MAX_BYTES + 1 bytes cannot fill the temp dir.
- _extract_zip sums ZipInfo.file_size across all members and checks
the member count *before* calling extractall, so classic zip bombs
(small archive, huge declared uncompressed size) are rejected
without materialising any of the bomb on disk.
- _clone_git measures the cloned tree's on-disk size after the
existing 60s timeout completes and rejects + cleans up if it
exceeds the cap. Symlinks are skipped to avoid runaway counts on
malicious /dev/zero-style links.
All limit breaches raise IngestLimitExceededError (subclass of
ValueError so existing callers that catch ValueError keep working)
with a clear message including the limit and the observed size.
Adds 10 unit tests covering: under-cap success paths for URL/zip/clone;
oversized Content-Length rejected before body read; chunked overflow
caught by the streamed counter; *the partial download file is removed
when a breach fires mid-stream*; *streaming to disk produces a file of
the expected size with no intermediate in-memory concatenation*;
declared-uncompressed-oversize zip rejected without extraction;
member-count zip-bomb rejected; oversize clone rejected and cleaned
up. Full suite: 730 passed, 12 skipped.
README documents both caps and their relationship to the per-file
analysis cap.
Signed-off-by: Rohan Isawe <[email protected]>1 parent d8a68ee commit cf31832
3 files changed
Lines changed: 527 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
123 | 132 | | |
124 | 133 | | |
125 | 134 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
27 | 31 | | |
28 | 32 | | |
29 | 33 | | |
| |||
41 | 45 | | |
42 | 46 | | |
43 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
44 | 72 | | |
45 | 73 | | |
46 | 74 | | |
| |||
64 | 92 | | |
65 | 93 | | |
66 | 94 | | |
67 | | - | |
68 | | - | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
69 | 99 | | |
70 | 100 | | |
71 | 101 | | |
| |||
123 | 153 | | |
124 | 154 | | |
125 | 155 | | |
126 | | - | |
| 156 | + | |
127 | 157 | | |
128 | 158 | | |
129 | 159 | | |
| |||
145 | 175 | | |
146 | 176 | | |
147 | 177 | | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
148 | 195 | | |
149 | 196 | | |
150 | 197 | | |
151 | | - | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
152 | 206 | | |
153 | 207 | | |
154 | 208 | | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
155 | 215 | | |
156 | 216 | | |
157 | | - | |
158 | | - | |
159 | | - | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
160 | 250 | | |
| 251 | + | |
| 252 | + | |
161 | 253 | | |
162 | 254 | | |
163 | | - | |
164 | | - | |
165 | | - | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
166 | 262 | | |
167 | | - | |
| 263 | + | |
168 | 264 | | |
169 | 265 | | |
170 | | - | |
| 266 | + | |
171 | 267 | | |
172 | 268 | | |
173 | 269 | | |
174 | | - | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
175 | 278 | | |
176 | 279 | | |
177 | 280 | | |
178 | 281 | | |
179 | 282 | | |
180 | 283 | | |
181 | 284 | | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
182 | 298 | | |
183 | 299 | | |
184 | 300 | | |
| |||
196 | 312 | | |
197 | 313 | | |
198 | 314 | | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
0 commit comments