Repository navigation
fix(router): preserve unmarked HTML tool output - #3778
JerrettDavis merged 5 commits into
Conversation
PR governanceThis PR follows the template and is marked ready for human review. |
|
Reviewed exact draft head ed38fb4. The two guard-set additions correctly preserve unmarked lossy HTML tool ground truth while leaving direct HTML extraction unchanged. Current main is already contained; all 137 affected tests passed locally with the real html extra, and Ruff passed. No code/security blocker found. I am holding approval because the PR is still draft and only governance checks exist—once it is marked ready and the hosted suite runs green, this head is otherwise ready for approval. |
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
Hosted review update for exact head ed38fb4: the HTML-extra job, security, merge-conflict, workflow-validation, evaluation, and all code-related/full-suite coverage passed except one unrelated baseline regression in shard 1: ests/test_cli/test_install_cli.py::test_install_restart_waits_for_old_process_before_starting fails because its fake manifest no longer satisfies the current Manifest schema. That same fixture/schema issue is already fixed in release PR #3713. No contributor change requested here; approval remains held only because the required exact-head suite is not green. Once the baseline fix reaches main, this head should be updated/rerun. |
|
Thanks for reviewing Does that existing base failure block your review, or would you prefer a separate fixture fix? |
JerrettDavis
left a comment
There was a problem hiding this comment.
Re-reviewed exact head fd8e435 after the current-main refresh. The focused HTML/router/content-detection suite (105 tests), Ruff, and diff checks passed locally. The maintainer-gated hosted matrix is now green after the single unrelated SIGHUP proxy-reaping flake passed on rerun, including native Docker coverage.
|
I merged current The new hosted runs for this fork are awaiting maintainer approval. Could a maintainer approve them so the exact-head suite can run? I'll address any failures it reports. |
|
I synced current The new exact-head CI run and other hosted workflows are awaiting fork approval. Could a maintainer approve them so this head can be verified? |
JerrettDavis
left a comment
There was a problem hiding this comment.
Re-reviewed exact head 1b10e12 after the current-main merge. The HTML strategy is correctly classified as lossy without a retrieval marker on both router and provider-shell paths; 32 focused HTML/reversibility tests and Ruff passed locally, and all hosted checks are terminal green.
…positives `is_mixed_content` keys off regexes: an inline `<script type="application/ld+json">` block trips `has_json_blocks` and ordinary body copy trips `has_prose`, so a page with a clear DOCTYPE and `<html>` root is routed to MIXED. Splitting it section by section skips whole-document extraction, and the sections lose the parent HTML type: on a 1244-char product page the mixed path keeps 1211 chars (ratio 0.97) while the HTML strategy keeps 571 (ratio 0.46). Trust the detector's HTML verdict the same way the existing branch trusts SOURCE_CODE for code blobs. No second confidence gate here: detect_content_type only returns HTML above 0.7 and the native backend hardcodes 1.0, so a threshold at this call site never filtered anything. Tool output stays safe because HTML is in LOSSY_UNMARKED_STRATEGIES (added separately in headroomlabs-ai#3778): unmarked extraction cannot replace tool ground truth, so the JSON-LD facts survive. The tests cover that apply()-level path, the assistant path that still compresses, and the no-trafilatura passthrough.
…positives `is_mixed_content` keys off regexes: an inline `<script type="application/ld+json">` block trips `has_json_blocks` and ordinary body copy trips `has_prose`, so a page with a clear DOCTYPE and `<html>` root is routed to MIXED. Splitting it section by section skips whole-document extraction, and the sections lose the parent HTML type: on a 1244-char product page the mixed path keeps 1211 chars (ratio 0.97) while the HTML strategy keeps 571 (ratio 0.46). Trust the detector's HTML verdict the same way the existing branch trusts SOURCE_CODE for code blobs. No second confidence gate here: detect_content_type only returns HTML above 0.7 and the native backend hardcodes 1.0, so a threshold at this call site never filtered anything. Tool output stays safe because HTML is in LOSSY_UNMARKED_STRATEGIES (added separately in headroomlabs-ai#3778): unmarked extraction cannot replace tool ground truth, so the JSON-LD facts survive. The tests cover that apply()-level path, the assistant path that still compresses, and the no-trafilatura passthrough.
…positives `is_mixed_content` keys off regexes: an inline `<script type="application/ld+json">` block trips `has_json_blocks` and ordinary body copy trips `has_prose`, so a page with a clear DOCTYPE and `<html>` root is routed to MIXED. Splitting it section by section skips whole-document extraction, and the sections lose the parent HTML type: on a 1244-char product page the mixed path keeps 1211 chars (ratio 0.97) while the HTML strategy keeps 571 (ratio 0.46). Trust the detector's HTML verdict the same way the existing branch trusts SOURCE_CODE for code blobs. No second confidence gate here: detect_content_type only returns HTML above 0.7 and the native backend hardcodes 1.0, so a threshold at this call site never filtered anything. Only a complete top-level document (DOCTYPE or `<html>` root, after envelope stripping) is rerouted. A fenced HTML example inside a prose answer is also detected as HTML, and extracting the whole answer would drop the surrounding instructions and the fence lines, so that shape stays on MIXED. Tool output stays safe because HTML is in LOSSY_UNMARKED_STRATEGIES (added separately in headroomlabs-ai#3778): unmarked extraction cannot replace tool ground truth, so the JSON-LD facts survive. The tests cover that apply()-level path, the assistant path that still compresses, the fenced-example case, and the no-trafilatura passthrough.
…t heuristics (#3627) ## Description A complete HTML document (clear DOCTYPE/`<html>` root) that also contains embedded JSON (e.g. a `<script type="application/ld+json">` block) or enough body prose trips `is_mixed_content`'s regex heuristics (`has_json_blocks`, `has_prose`), so it gets routed through the generic MIXED path (split section-by-section, each section compressed independently) instead of the dedicated HTML extractor. The split sections lose the parent HTML type, so the document never reaches whole-document extraction. On the 1244-char product page used in the tests, `compress()` keeps 1211 chars through MIXED (ratio 0.97) and 571 chars through the HTML strategy (ratio 0.46). `_determine_strategy` already has this exact override for `SOURCE_CODE` (a Python file with dict literals and docstrings trips the same heuristics, and the code trusts the detector's SOURCE_CODE verdict over them). HTML never got the same override. Closes #3608 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/content_router.py`: `_determine_strategy` now trusts a `ContentType.HTML` detection over the `is_mixed_content` regex heuristics, same pattern as the `SOURCE_CODE` override immediately above it. Only a complete top-level document qualifies: the payload (after envelope stripping) must start with a DOCTYPE or an `<html>` root. The detector scores tags anywhere in the sample, so a fenced HTML example inside a prose answer also reads as HTML, and extracting the whole answer would drop the surrounding instructions and the fence lines. There is deliberately no confidence check at this call site. `detect_content_type` only returns HTML above 0.7 and the native backend hardcodes 1.0, so the previous `>= 0.7` re-check could never be false and only read as a guard that was not one. - `tests/test_transforms_content_router.py`: six tests. A complete product page with inline JSON-LD is shown to trip the real `is_mixed_content` and the real pure-Python detector, then route to HTML. The override is pinned as a single gate. A prose answer with a fenced `<!DOCTYPE html>` example is shown to be mixed and detected HTML, and is pinned to stay MIXED. An `apply()`-level test drives a `role=tool` message through the real router with a deterministic extractor that drops the JSON-LD, and asserts the SKU, price and rating survive. Two supporting tests keep the guard scoped to tool output and cover the no-`trafilatura` passthrough. Tool output is protected by `CompressionStrategy.HTML` being in `LOSSY_UNMARKED_STRATEGIES`. That entry landed independently on `main` in #3778, so this PR does not add it; the `apply()` test above pins the interaction between the routing change and that guard. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom/transforms/content_router.py` is clean on Linux; the local Windows run reports only pre-existing POSIX-only attribute errors, see below) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Windows 11, Python 3.13.3 $ python -m pytest tests/test_transforms_content_router.py tests/test_html_tool_reversibility.py \ tests/test_content_router_tool_role_reversibility.py tests/test_compression_units.py \ tests/test_compression_batches.py tests/test_no_ccr_lossy.py tests/test_kompress_marker_gate.py \ tests/test_router_registry_dispatch.py tests/test_builtin_compressor_adapters.py \ tests/test_canonical_pipeline.py tests/test_cross_turn_dedup.py -q 193 passed, 1 skipped in 26.05s # Ubuntu (WSL2), Python 3.13.16, trafilatura 2.3.1, onnxruntime 1.31.1, same 11 files 193 passed, 1 skipped in 26.77s $ python -m pytest tests/test_transforms_content_router.py -k html -q 8 passed, 85 deselected in 3.71s $ ruff check headroom/transforms/content_router.py tests/test_transforms_content_router.py All checks passed! $ ruff format --check headroom/transforms/content_router.py tests/test_transforms_content_router.py 2 files already formatted $ python -m mypy headroom/transforms/content_router.py headroom\fileperms.py:92: error: Module has no attribute "fchmod"; maybe "chmod"? [attr-defined] headroom\proxy\interceptors\astgrep.py:220,229,242: error: Module has no attribute "O_DIRECTORY" / "O_NOFOLLOW" / "O_NONBLOCK" [attr-defined] Found 7 errors in 2 files (checked 1 source file) # Ubuntu (WSL2), same command $ python -m mypy headroom/transforms/content_router.py Success: no issues found in 1 source file ``` The mypy errors are all POSIX-only `os` attributes missing on Windows, in files this PR does not touch. Fail-without/pass-with: removing `CompressionStrategy.HTML` from `LOSSY_UNMARKED_STRATEGIES` makes `test_tool_html_with_inline_jsonld_kept_verbatim` fail with the 571-char extraction replacing the page. Reverting just the routing branch makes `test_html_override_has_no_second_confidence_gate` fail (`MIXED` returned instead of `HTML`). ## Real Behavior Proof - Environment: two. (1) Windows 11, Python 3.13.3, `trafilatura==2.3.1`, source checkout at `main` `17d38e56` plus this patch, `PYTHONPATH` pointed at the checkout. Windows selects the pure-Python detection backend by default. (2) Ubuntu on WSL2 (kernel 6.18), Python 3.13.16, `trafilatura==2.3.1`, `onnxruntime==1.31.1`, same checkout and patch, run with both `HEADROOM_DETECT_BACKEND=rust` and `=python`. On Linux the default is the native Magika/ONNX detector, which is the backend CI uses. The fixture is a synthetic product page: DOCTYPE, `<html>`/`<head>`/`<body>`, an `application/ld+json` block with SKU, price and `aggregateRating`, six paragraphs of article copy, a nav and a footer. - Exact command / steps: a script builds that page, prints `is_mixed_content`, `detect_content_type`, what the real extractor keeps, then runs `ContentRouter().compress(doc)` and `ContentRouter().apply([{"role": "tool", "tool_call_id": "call_bash_1", "content": doc}], tokenizer, protect_recent=0, protect_analysis_context=False)`. It was run once with `content_router.py` taken from `main` and once with this patch. - Observed result, `compress()`: before, `is_mixed_content` is `True`, detection is `ContentType.HTML` at confidence 1.0, strategy is `mixed`, 1244 -> 1211 chars (ratio 0.967); after, same detection, strategy is `html`, 1244 -> 571 chars (ratio 0.457). - Observed result, raw extractor and `apply()`: the real extractor returns 571 chars and drops all three facts. With this patch, `apply()` keeps the tool message verbatim at 1244 chars and the SKU, price and rating are all present. The unchanged result comes from the `LOSSY_UNMARKED_STRATEGIES` guard (#3778) now that routing reaches HTML. Removing that one entry reproduces the reviewer's replay on this branch: `apply()` replaces the message with the 571-char extraction and the SKU, price and rating are gone. - Observed result, native detector on Linux: `HEADROOM_DETECT_BACKEND=rust` detects the same page as `ContentType.HTML` at confidence 1.0, routes it to `strategy=html` (1244 -> 571), `apply()` keeps all 1244 chars and all three facts, and the mutation above still makes the regression test fail. The `=python` run matches. This is the backend Linux CI takes by default, so the routing branch is not relying on the Windows regex detector's verdict. - Observed result, fenced example: a prose answer with a fenced complete HTML page is `is_mixed_content: True` and detected `ContentType.HTML` at 0.71. With the unconditional override it routed to `strategy=html` and shrank 842 -> 450 chars, dropping the closing instructions and the fence lines. With the top-level check it stays `mixed` and the surrounding text survives. `test_fenced_html_example_stays_mixed` fails when the check is removed (it sees `HTML` instead of `MIXED`). - Not tested: the reporters' actual pages, a live provider request, or the full pytest suite with the `html` extra installed. The targeted set above is what ran locally. The native Magika chain in the WSL run came from the `headroom-ai==0.41.0` manylinux abi3 wheel's `_core.abi3.so`, not from a local Rust build of this checkout; only the Python routing code under test came from the patch. The `_core` ABI is unchanged by this PR. ## Runtime Rollout Safety - Rollout-managed feature(s): none. This is a routing decision inside `ContentRouter`, not behind a new flag. - Minimum rollout channel: N/A - Stable/default behavior changed: yes, in two ways. A complete HTML document that trips the mixed heuristics now routes to the HTML strategy, which compresses it much further (0.97 -> 0.46 on the fixture) and, for non-tool content, keeps only extracted prose. For `role=tool` content whose extraction is unmarked, the guard keeps the original verbatim, so mixed HTML tool pages that used to shrink slightly through MIXED (about 3% on the fixture) now keep all their bytes. Same trade-off #3778 already made for HTML pages that route to HTML directly. - Kill switch / disable path: `enable_html_extractor=False` stops extraction and the HTML branch falls through to passthrough. `ccr_inject_marker=False` remains the existing intentional opt-out of the tool-output guard. - Unsafe override required: none. - Qualification impact: none. CI's `test-extras` job already installs the `html` extra and runs the HTML reversibility tests (#3778). - Rollback path: revert this commit. One additive branch plus tests, no state, schema or migration. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: no public API or config change) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective and that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`, it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes The branch was resynchronized with current `main` (`17d38e56`, 578 commits since the original base) and squashed into a single commit on top of it, so the merge conflict with #3778's guard changes is resolved by taking `main`'s version of the strategy set as-is. The delta is now the routing branch plus the tests, with no changes to `LOSSY_UNMARKED_STRATEGIES`. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3627" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> ### Maintainer current-main requalification Exact head `82e1ef5ab82c9553b0104ba934f7231213c480b8`, upstream main `2d3ee51846435ef5b320eea4806af14e602b0d8c`. Re-reviewed latest integration head 89da6eb and author discussion, including the earlier JSON-LD/tool-output preservation and fenced-example fixes. Clean integration with upstream/main 2d3ee51. The router trusts HTML detection only for a top-level DOCTYPE/html document; prose containing fenced HTML remains mixed. Existing tool-output preservation guard is retained. No new PR-specific source blocker found in reviewed scope. Combined router/release/version/MCP run: 181 passed, three failed. All three failing cases were rerun on a separate clean current-main worktree with the same existing native extension and Python detection backend, and failed identically: JSON-in-HTML unrecoverable_sections and two minification token-shrink assertions. They are inherited/local-environment qualification failures, not PR regressions and not grounds for withholding approval under the user directive. Ruff check/format and whitespace pass. No fresh native build/live provider claim. Maintainers handle fresh exact-head qualification; nothing merged. Exact-head hosted checks are being handled by maintainers. No new finding in the reviewed scope; formal approval awaits exact-head hosted qualification; nothing merged. Co-authored-by: Ayush <[email protected]> Co-authored-by: JD Davis <[email protected]>
Description
HTMLExtractorcan reduce a tool result to article prose without a CCR retrieval marker. The existing tool ground-truth guard omittedCompressionStrategy.HTML, so the deleted script or structured data could not be recovered.This is independent of #3627's HTML-vs-MIXED routing change and #3774's test isolation change. Closes #3775.
Type of Change
Changes Made
HTMLto the lossy/unmarked strategy sets used byContentRouter.applyand the provider-neutral shell-output unit path. Unmarked HTML extraction now leaves tool output unchanged; a future extraction with a real CCR marker can still be accepted.trafilaturaregression cases for Anthropictool_result, OpenAIrole=tool, and a structuredlocal_shell_call_output. The tests failed on the original code and passed with the two set entries.htmlextra in CI'stest-extrasjob and run the HTML extractor and new reversibility tests there. The main pytest shards currently install[dev]withouthtml; the[sandbox]smoke job installs it but does not run pytest.Testing
pytest) — the full suite was not run; the focused run below passed.ruff check .)mypy headroom) — local Windows run has the unrelated errors listed below.Test Output
Real Behavior Proof
df8ebdf97plus this patch,uv sync --frozen --extra dev --extra htmlusing that interpreter,trafilatura==2.0.0.ContentRouterConfig(enable_kompress=False, min_section_tokens=10), default CCR marker setting, localgpt-4otoken counter. Windows selected the pure-Python content detector; no live provider request was made.A second unmocked probe with the same fixture yielded
role=tool: 10294 -> 10294, same=Trueandlocal_shell_call_output: html, modified=False, same=True, reason=lossy_unrecoverable_tool_output.htmlinstalled.Runtime Rollout Safety
ContentRouter.compress()extraction is unchanged.enable_html_extractor=Falseprevents HTML extraction.ccr_inject_marker=Falseremains the existing intentional opt-out for theContentRouter.applyguard.test-extrasjob now installs[html]and runs real HTML tests; main shards are untouched. The test: stop three sources of cross-test pollution on main #3774 test isolation change is in current main, outside this PR diff.Review Readiness
Current head
d361388emerges latestmain, including #3790's fix for the unrelated SQLite test that failed in the previous CI run. On this head, that test, the 137 focused HTML/router/unit tests, and all 9 dense-line-elider tests passed locally; Ruff andgit diff --checkpassed. Exact-head hosted checks are pending.Checklist
CHANGELOG.md, it is generated by release-please from my Conventional Commit PR titleAdditional Notes
This keeps the existing reversible-tool-output contract: an unmarked lossy transform cannot replace tool ground truth. It deliberately does not add a retrieval store or change HTML routing.