Repository navigation
Commit 6e07493
committed
fix(static_yara): bind skip metadata to its rules and name rejected files
Addresses the three review findings on NVIDIA#557. All three share one shape: the
dropped-rule total was reported through a channel not tied to the scan that
produced it.
1. Skip count raced across concurrent scans (rng1995, P1)
`node()` called `_load_rules()` and then read `rules_skipped_count()` as a
separate step. Two concurrent MCP/graph scans can interleave between those:
scan B loads its own rule set and overwrites `_rules_skipped_count` before
scan A reads it, so A runs rules A while reporting B's total. If B skipped
nothing, A reports `completed` even though one of A's own rules was dropped --
the false-clean result NVIDIA#554 exists to prevent.
Adds `load_rules_with_skips()`, which returns the rules and their own skip
count from one transaction guarded by a reentrant `_RULES_LOCK`, and switches
`node()` to it. `_load_rules()` keeps its single-value signature, and
`load_rules_with_skips` calls it through the module global, so every existing
`monkeypatch.setattr(static_yara, "_load_rules", ...)` double still applies.
`rules_skipped_count()` is retained for single-threaded callers and now reads
under the lock. The three cache globals are documented as one logical value
that must only be written or read as a set.
The lock serializes rule compilation across concurrent scans. That is a
deliberate trade: compilation is cached and already deadline-bounded, and a
scanner reporting a false clean is worse than one loading rules serially.
2. Rule-load event collided with a component of the same name (yashrajp22)
`ledger_event` derives the work identity as
`analyzer_id or f"{record_type}:{phase}"`, and the synthetic `yara_rules/`
scope normalizes to `yara_rules`. Passing `analyzer_id=ANALYZER_ID` therefore
produced the same work ID as the planned work item for a scanned component
literally named `yara_rules`: both planned targets resolved to two matching
events, and reconciliation raised a fatal `unaccounted_work` with
`execution_successful=false` and CLI exit 2, instead of the nonfatal partial
scan this event is meant to record.
Omits `analyzer_id` on that one event so the identity falls back to
`system:static`, which is disjoint from every analyzer work item by
construction. As the review noted, renaming the synthetic path alone would
only move the collision to the next unlucky filename.
3. Rejected rules were invisible at default log level (yashrajp22, NVIDIA#554)
Both rejection handlers logged at DEBUG, so a malformed `acme.yar`, a BOM
rule, or a non-UTF-8 `.yar` produced no default-level warning, and the public
ledger event is scoped to the rule set rather than the file. The operator
could see that a detector was dropped but not which one to repair.
Both handlers now log at WARNING, naming the file and a bounded reason.
`_build_namespace_map` optionally fills a `{namespace: filename}` map -- passed
in rather than returned, to keep its two-value signature -- so the compile path
can name `acme.yar` instead of the extension-stripped namespace `acme`.
`_bounded_rejection_reason` collapses newlines and caps the echoed text at 200
characters, because rule sources are attacker-influenced when
`--yara-rules-dir` points at untrusted content and YARA errors can quote the
offending source line.
Tests
New `TestRuleSkipAccounting` (9 tests): a deterministic pairing test, a
serialization test that asserts the lock is genuinely held for the whole
load-and-read transaction rather than racing and hoping, a contended
two-thread test over 50 observations, the `yara_rules` work-ID collision case
asserting both event and planned-work IDs stay distinct, three parametrized
rejection-diagnostic cases (malformed, BOM, non-UTF-8), and two bounding
tests. The contended test surfaces worker-thread exceptions and asserts an
observation count, so it cannot pass vacuously when the scans never ran.
The autouse cache fixture now also resets `_rules_skipped_count`, which is
part of that cache and would otherwise leak between tests.
Verification
- Negative control: all 9 new tests fail with the source change reverted and
the tests kept; 9/9 pass with it.
- `tests/nodes/analyzers/test_static_yara.py`: 96 passed.
- Full suite: 18 pre-existing failures, byte-identical to the same run on
unmodified `4e753fe` (build_context, compare_scan_accuracy,
create_github_release, input_handler, json_container_ownership,
security_end_to_end -- all environmental, none in the touched files).
- `ruff check`, `ruff format --check`, and `mypy` clean on both files.
- Windows / Python 3.13 only; the pre-existing failures above are consistent
with that environment rather than with this change.
Signed-off-by: Souptik Chakraborty <[email protected]>1 parent 4e753fe commit 6e07493
2 files changed
Lines changed: 387 additions & 38 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
174 | 175 | | |
175 | 176 | | |
176 | 177 | | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
177 | 183 | | |
178 | 184 | | |
179 | 185 | | |
180 | 186 | | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
181 | 191 | | |
182 | 192 | | |
183 | 193 | | |
| |||
336 | 346 | | |
337 | 347 | | |
338 | 348 | | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
339 | 364 | | |
340 | 365 | | |
341 | 366 | | |
342 | 367 | | |
343 | 368 | | |
| 369 | + | |
344 | 370 | | |
345 | | - | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
346 | 379 | | |
347 | 380 | | |
348 | 381 | | |
| |||
353 | 386 | | |
354 | 387 | | |
355 | 388 | | |
| 389 | + | |
| 390 | + | |
356 | 391 | | |
357 | 392 | | |
358 | 393 | | |
359 | 394 | | |
360 | | - | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
361 | 405 | | |
362 | 406 | | |
363 | 407 | | |
364 | | - | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
365 | 413 | | |
366 | 414 | | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
367 | 418 | | |
368 | 419 | | |
369 | 420 | | |
| |||
384 | 435 | | |
385 | 436 | | |
386 | 437 | | |
387 | | - | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
388 | 446 | | |
389 | 447 | | |
390 | 448 | | |
| |||
406 | 464 | | |
407 | 465 | | |
408 | 466 | | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
409 | 471 | | |
410 | 472 | | |
411 | 473 | | |
412 | | - | |
413 | | - | |
414 | | - | |
415 | | - | |
416 | | - | |
417 | | - | |
418 | | - | |
419 | | - | |
420 | | - | |
421 | | - | |
422 | | - | |
423 | | - | |
424 | | - | |
425 | | - | |
426 | | - | |
427 | | - | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
428 | 480 | | |
429 | | - | |
430 | | - | |
431 | | - | |
432 | | - | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
433 | 486 | | |
434 | | - | |
435 | | - | |
436 | | - | |
437 | | - | |
438 | | - | |
439 | | - | |
440 | | - | |
441 | | - | |
442 | | - | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
443 | 530 | | |
444 | 531 | | |
445 | 532 | | |
| |||
448 | 535 | | |
449 | 536 | | |
450 | 537 | | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
451 | 542 | | |
452 | | - | |
| 543 | + | |
| 544 | + | |
453 | 545 | | |
454 | 546 | | |
455 | 547 | | |
| |||
940 | 1032 | | |
941 | 1033 | | |
942 | 1034 | | |
943 | | - | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
944 | 1038 | | |
945 | 1039 | | |
946 | 1040 | | |
947 | 1041 | | |
948 | | - | |
949 | 1042 | | |
950 | 1043 | | |
951 | 1044 | | |
| |||
1106 | 1199 | | |
1107 | 1200 | | |
1108 | 1201 | | |
1109 | | - | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
| 1206 | + | |
| 1207 | + | |
| 1208 | + | |
| 1209 | + | |
| 1210 | + | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
| 1214 | + | |
1110 | 1215 | | |
1111 | 1216 | | |
1112 | 1217 | | |
| |||
0 commit comments