Skip to content

Commit 20ed0fa

Browse files
Yoseph-ZuskinOpenCode DeepSeek V4.1 Flash Free (1M context)
andcommitted
fix(analyzers): budget GIF walks and close image-cache gaps
- Bound GIF sub-block walks by count (2,000,000) instead of absolute bytes so benign multi-megabyte frames stay COMPLETED; flag every pre-trailer stop. - Fill and publish image text caches for every input shape and drop the runner's raw extraction fallback, so analyzers never re-extract outside the shared deadline. - Decode inline data: URIs per RFC 2397 and record inline_image_decode_error for malformed payloads. - Document the 512-character markdown destination bound; run the compiled-graph image test from tests/nodes so CI executes it. Signed-off-by: Yoseph Zuskin <[email protected]> Co-Authored-By: OpenCode DeepSeek V4.1 Flash Free (1M context) <[email protected]>
1 parent 0167d86 commit 20ed0fa

9 files changed

Lines changed: 274 additions & 149 deletions

File tree

‎docs/scan-completeness.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,7 @@ JSON and MCP expose `analysis_completeness.status`, `is_complete`, `ledger_excep
136136

137137
`image_inventory` lists per-skill `local_images` (resolved markdown targets plus loose image files, no verdicts), `remote_images` (remote citations as scheme-plus-host only, never fetched), and `inline_images` (`data:` image URIs as MIME-plus-size summaries, never content). Exactly one inventory truncation shape exists: entry, line, deadline, or inline-size pressure appends a single `PARTIAL` row naming the output limit, and the public projection carries the same bound information.
138138

139-
Embedded text is extracted stdlib-only (PNG `tEXt`/`iTXt`/`zTXt`/`eXIf`, JPEG `COM` plus EXIF `UserComment`/`ImageDescription`/`XPComment`, GIF comments) once per image under the shared deadline and routed through the static patterns; findings carry an `image-text-layer` tag whose line numbers point into the extracted text, not file bytes. Text past a bound yields its bounded prefix and the image's static row reads `PARTIAL` with a size reason, never `COMPLETED`. Deliberate gaps: pixel decoding (no OCR), remote fetching, GIF plain-text extensions, WebP/BMP text layers, COM segments past JPEG scan data (progressive scans), JPEG XMP, and rendered/QR/steganographic content (needs the separate vision design).
139+
Embedded text is extracted stdlib-only (PNG `tEXt`/`iTXt`/`zTXt`/`eXIf`, JPEG `COM` plus EXIF `UserComment`/`ImageDescription`/`XPComment`, GIF comments) once per image under the shared deadline and routed through the static patterns; findings carry an `image-text-layer` tag whose line numbers point into the extracted text, not file bytes. Text past a bound yields its bounded prefix and the image's static row reads `PARTIAL` with a size reason, never `COMPLETED`. Inline `data:` images are inventoried only from markdown destinations of at most 512 characters; the reference extractor drops longer destinations, so the 8-image and 1 MB inline caps rarely bind. GIF walks are bounded by sub-block count (not bytes), so a benign multi-megabyte frame stays `COMPLETED`. A PNG chunk length that overruns the file ends the chunk walk silently by design: the chunks already read were complete, and the remaining bytes are not a well-formed chunk stream (corrupt container, not a lost text layer). Deliberate gaps: pixel decoding (no OCR), remote fetching, GIF plain-text extensions, WebP/BMP text layers, COM segments past JPEG scan data (progressive scans), JPEG XMP, and rendered/QR/steganographic content (needs the separate vision design).
140140

141141
A subtlety is that component coverage can still read **100%** for AE6: analyzer work completed on each file, while a separate system event records unresolved interpretation. Consumers must use `is_complete` and the ledger, not the coverage percentage alone. No new telemetry platform is needed to explain these decisions; the existing public fields distinguish the causes.
142142

‎src/skillspector/image_text.py‎

Lines changed: 27 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,12 @@
1111
1212
Bounds are truncation-honest, never silent: every cap reports whether
1313
text may have been lost, so callers record ``PARTIAL`` instead of
14-
claiming ``COMPLETED``. Caps apply to text bytes (per-chunk and
15-
global), not to chunk counts, so padding with dummy chunks cannot hide
16-
a later payload. Oversized text yields its bounded prefix.
14+
claiming ``COMPLETED``. Text caps apply to text bytes (per-chunk and
15+
global), GIF walks to a sub-block budget, so padding cannot hide a
16+
later payload. Oversized text yields its bounded prefix. One
17+
deliberate carve-out remains: a PNG chunk length that overruns the
18+
file ends the walk silently (see ``_png_text``), because the chunks
19+
already read were complete.
1720
"""
1821

1922
from __future__ import annotations
@@ -40,10 +43,10 @@
4043
# ~2 s. A 1 MiB comment needs ~4,120 full-size blocks, so 16,384 blocks
4144
# keeps any legitimate comment while stopping a crafted walk in ~2 ms.
4245
_MAX_SUB_BLOCKS: Final = 16_384
43-
# GIF walks skip pixel data by offset arithmetic (cheap), so the per-file
44-
# bound only needs to stop pathological walks: 4 MB covers multi-megabyte
45-
# benign frames while a crafted 16 MiB walk trips it loudly.
46-
_MAX_GIF_WALK_BYTES: Final = 4_000_000
46+
# Skipped GIF data is bounded by sub-block count, not absolute bytes: a
47+
# benign 16 MiB frame needs ~66k full-size sub-blocks, while a crafted
48+
# walk of one-byte blocks stops at 2,000,000 (~0.4 s) and reports it.
49+
_MAX_GIF_SUB_BLOCKS: Final = 2_000_000
4750

4851
_IFD_TYPE_SIZES = {1: 1, 2: 1, 3: 2, 4: 4, 7: 1}
4952

@@ -354,29 +357,30 @@ def _read_sub_blocks(data: bytes, offset: int) -> tuple[bytes, int, bool]:
354357
return bytes(collected), offset, capped
355358

356359

357-
def _skip_sub_blocks(data: bytes, offset: int) -> tuple[int, bool]:
358-
"""Advance past GIF data sub-blocks without collecting or capping.
360+
def _skip_sub_blocks(data: bytes, offset: int, budget: int) -> tuple[int, bool, int]:
361+
"""Advance past GIF data sub-blocks within the shared walk budget.
359362
360363
Pixel data and non-comment extensions are skipped by offset
361364
arithmetic only: collecting them would let padding-sized frames
362-
hide a later comment or force a false PARTIAL. Reports whether
363-
the walk tripped the per-skip byte bound or a truncated tail.
365+
hide a later comment or force a false PARTIAL. Reports whether the
366+
walk tripped the sub-block budget or a truncated tail, plus the
367+
budget left for the rest of the walk.
364368
"""
365-
start = offset
366369
capped = False
367370
while offset < len(data):
368-
if offset - start > _MAX_GIF_WALK_BYTES:
371+
if budget <= 0:
369372
capped = True
370373
break
371374
size = data[offset]
372375
offset += 1
376+
budget -= 1
373377
if size == 0:
374378
break
375379
if offset + size > len(data):
376380
capped = True
377381
break
378382
offset += size
379-
return offset, capped
383+
return offset, capped, budget
380384

381385

382386
def _gif_text(data: bytes) -> tuple[str, bool]:
@@ -389,12 +393,10 @@ def _gif_text(data: bytes) -> tuple[str, bool]:
389393
packed = data[10]
390394
if packed & 0x80:
391395
offset += 3 * (2 ** ((packed & 0x07) + 1))
396+
budget = _MAX_GIF_SUB_BLOCKS
392397
while True:
393398
if offset >= len(data):
394399
break
395-
if offset > _MAX_GIF_WALK_BYTES:
396-
truncated = True
397-
break
398400
if total >= MAX_IMAGE_TEXT_CHARS:
399401
truncated = True
400402
break
@@ -403,6 +405,7 @@ def _gif_text(data: bytes) -> tuple[str, bool]:
403405
break
404406
if separator == 0x21:
405407
if offset + 2 > len(data):
408+
truncated = True
406409
break
407410
label = data[offset + 1]
408411
offset += 2
@@ -412,21 +415,26 @@ def _gif_text(data: bytes) -> tuple[str, bool]:
412415
total, lost = _collect(parts, total, _decode(comment).strip("\x00").strip())
413416
truncated = truncated or lost
414417
else:
415-
offset, capped = _skip_sub_blocks(data, offset)
418+
offset, capped, budget = _skip_sub_blocks(data, offset, budget)
416419
truncated = truncated or capped
417420
elif separator == 0x2C:
418421
if offset + 10 > len(data):
422+
truncated = True
419423
break
420424
packed = data[offset + 9]
421425
offset += 10
422426
if packed & 0x80:
423427
offset += 3 * (2 ** ((packed & 0x07) + 1))
424428
if offset >= len(data):
429+
truncated = True
425430
break
426431
offset += 1
427-
offset, capped = _skip_sub_blocks(data, offset)
432+
offset, capped, budget = _skip_sub_blocks(data, offset, budget)
428433
truncated = truncated or capped
429434
else:
435+
# Unknown block byte before the trailer: the rest of the
436+
# container is unread, so say so rather than reporting clean.
437+
truncated = True
430438
break
431439
return "\n".join(part for part in parts if part), truncated
432440

‎src/skillspector/inspection_ledger.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,7 @@ class LedgerReason(StrEnum):
114114
STATIC_PARSE_LIMIT = "static_parse_limit"
115115
PYTHON_SOURCE_AMBIGUOUS = "python_source_ambiguous"
116116
PYTHON_SOURCE_DECODE_ERROR = "python_source_decode_error"
117+
INLINE_IMAGE_DECODE_ERROR = "inline_image_decode_error"
117118
OBFUSCATED_INSTRUCTION_TEXT = "obfuscated_instruction_text"
118119

119120

@@ -246,6 +247,9 @@ class LedgerReason(StrEnum):
246247
LedgerReason.PYTHON_SOURCE_DECODE_ERROR: (
247248
"Python source bytes could not be decoded under their declared encoding."
248249
),
250+
LedgerReason.INLINE_IMAGE_DECODE_ERROR: (
251+
"An inline data: image URI could not be decoded under its declared encoding."
252+
),
249253
LedgerReason.OBFUSCATED_INSTRUCTION_TEXT: (
250254
"Obfuscated instruction text could not be fully evaluated by the deterministic layer."
251255
),

‎src/skillspector/nodes/analyzers/static_runner.py‎

Lines changed: 42 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -41,10 +41,8 @@
4141
is_default_ignorable,
4242
security_text_views,
4343
)
44-
from skillspector.constants import MAX_ANALYZABLE_FILE_BYTES
4544
from skillspector.image_text import (
4645
MAX_IMAGE_TEXT_CHARS,
47-
extract_image_text_detailed,
4846
has_image_magic,
4947
)
5048
from skillspector.inspection_ledger import (
@@ -572,44 +570,40 @@ def _infer_file_type(path: str) -> str:
572570

573571

574572
def _image_extracted_text(state: Mapping[str, object], path: str) -> tuple[str, bool]:
575-
"""Return ``(text, truncated)`` for one image path from cache or raw bytes.
576-
577-
Prefers the build-time extract-once cache; falls back to direct raw
578-
extraction only for states without it (unit-called runners predate the
579-
cache). When the cache exists but lacks the path, build ran and skipped
580-
it under the shared deadline — or it is not an image — so there is no
581-
fallback: per-analyzer re-extraction would pay the crafted-input walk
582-
again outside the deadline. The caller records PARTIAL runtime_limit
583-
for magic-byte misses. Identified by magic bytes, never
584-
suffix-gated, so a renamed image is still covered. Images without a text layer
585-
yield ``("", False)`` and keep their existing skip accounting.
573+
"""Return ``(text, truncated)`` for one image path from the build cache.
574+
575+
The build fills the extract-once cache for every input shape, so a
576+
missing entry means the build saw the path and skipped it (deadline) or
577+
it is not an image. There is no raw fallback: per-analyzer
578+
re-extraction would pay the crafted-input walk again outside the
579+
deadline. The caller records PARTIAL runtime_limit for magic-byte
580+
misses. Identified by magic bytes, never suffix-gated, so a renamed
581+
image is still covered. Images without a text layer yield
582+
``("", False)`` and keep their existing skip accounting.
586583
"""
587584
text_cache = state.get("image_text_cache")
588585
entry = _coerce_image_text_entry(text_cache.get(path)) if isinstance(text_cache, dict) else None
589586
if entry is not None:
590-
return entry
591-
if isinstance(text_cache, dict):
592-
return "", False
593-
raw_cache = state.get("raw_file_cache")
594-
raw = raw_cache.get(path) if isinstance(raw_cache, dict) else None
595-
if not isinstance(raw, bytes) or not raw or len(raw) > MAX_ANALYZABLE_FILE_BYTES:
596-
# Oversized raws are unreadable here, which is itself a possible
597-
# loss: flag it rather than claiming textless-complete.
598-
return "", True
599-
if not has_image_magic(raw):
600-
return "", False
601-
return extract_image_text_detailed(raw)
602-
603-
604-
def _coerce_image_text_entry(value: object) -> tuple[str, bool] | None:
605-
"""Return a ``(text, truncated)`` pair from cache state, else None.
606-
607-
Tolerates lists: checkpoints serialize stored tuples as lists.
587+
return entry[0], entry[1]
588+
return "", False
589+
590+
591+
def _coerce_image_text_entry(value: object) -> tuple[str, bool, LedgerReason | None] | None:
592+
"""Return a ``(text, truncated, reason)`` triple from cache state, else None.
593+
594+
Tolerates lists (checkpoints serialize stored tuples as lists) and
595+
legacy two-tuples written before the reason field existed.
608596
"""
609-
if isinstance(value, (tuple, list)) and len(value) == 2:
610-
text, truncated = value
597+
if isinstance(value, (tuple, list)) and len(value) in (2, 3):
598+
text, truncated = value[0], value[1]
599+
reason: LedgerReason | None = None
600+
if len(value) == 3 and isinstance(value[2], str):
601+
try:
602+
reason = LedgerReason(value[2])
603+
except ValueError:
604+
reason = None
611605
if isinstance(text, str) and isinstance(truncated, bool):
612-
return text, truncated
606+
return text, truncated, reason
613607
return None
614608

615609

@@ -3438,14 +3432,15 @@ def run_static_patterns_with_ledger(
34383432
)
34393433
inline_scan_content: str | None = None
34403434
inline_scan_truncated = False
3435+
inline_scan_reason: LedgerReason | None = None
34413436
inline_cache = state.get("image_inline_text_cache")
34423437
inline_entry = (
34433438
_coerce_image_text_entry(inline_cache.get(path))
34443439
if isinstance(inline_cache, dict)
34453440
else None
34463441
)
34473442
if inline_entry is not None:
3448-
inline_scan_content, inline_scan_truncated = inline_entry
3443+
inline_scan_content, inline_scan_truncated, inline_scan_reason = inline_entry
34493444
if not inline_scan_content and not inline_scan_truncated:
34503445
inline_scan_content = None
34513446
if path not in container_paths and path in source_classification_limitations:
@@ -3718,11 +3713,13 @@ def run_static_patterns_with_ledger(
37183713
# partial, emitted ids from both) and no inline row is added.
37193714
size_content = content
37203715
inline_error_class: str | None = None
3716+
# A decode failure has no text to scan but still loses a
3717+
# layer, so its reason must fold into this row too.
3718+
inline_reason: LedgerReason | None = inline_scan_reason
3719+
inline_metrics: dict[str, object] = {}
37213720
if inline_scan_content is not None:
37223721
inline_budget = MAX_FINDINGS_PER_ANALYZER - len(findings)
37233722
inline_remaining = transitive_remaining_seconds(cast(SkillspectorState, state))
3724-
inline_reason: LedgerReason | None = None
3725-
inline_metrics: dict[str, object] = {}
37263723
try:
37273724
if inline_budget <= 0 or (
37283725
inline_remaining is not None and inline_remaining <= 0
@@ -3751,8 +3748,10 @@ def run_static_patterns_with_ledger(
37513748
_tag_image_text_findings(inline_findings)
37523749
findings.extend(inline_findings)
37533750
path_findings.extend(inline_findings)
3754-
inline_reason = inline_limit or (
3755-
LedgerReason.SIZE_LIMIT if inline_scan_truncated else None
3751+
inline_reason = (
3752+
inline_limit
3753+
or inline_scan_reason
3754+
or (LedgerReason.SIZE_LIMIT if inline_scan_truncated else None)
37563755
)
37573756
except Exception as exc:
37583757
logger.warning(
@@ -3762,9 +3761,10 @@ def run_static_patterns_with_ledger(
37623761
exc,
37633762
)
37643763
inline_error_class = type(exc).__name__
3765-
if inline_reason is not None and resource_limit is None:
3766-
resource_limit = inline_reason
3767-
resource_metrics = inline_metrics
3764+
if inline_reason is not None and resource_limit is None:
3765+
resource_limit = inline_reason
3766+
resource_metrics = inline_metrics
3767+
if inline_scan_content is not None:
37683768
size_content = inline_scan_content
37693769
# An inline-scan error folds as PARTIAL, never FAILED: a
37703770
# FAILED row cannot carry emitted ids (ledger_event raises),

0 commit comments

Comments
 (0)