Skip to content

fix: decode JSON-LD escapes before normalizing metadata values - #999

Merged
adbar merged 2 commits into
adbar:masterfrom
Enesuygurs:fix/jsonld-escaped-characters
Oct 23, 2026
Merged

adbar merged 2 commits into
adbar:masterfrom
Enesuygurs:fix/jsonld-escaped-characters

Conversation

@Enesuygurs

Copy link
Copy Markdown
Contributor

Problem

extract_meta_json() ran normalize_json() on the raw JSON-LD text before json.loads. That decodes \uXXXX escapes and strips <...> as HTML tags on the serialized JSON instead of on the values:

  • "headline": "A "quoted" title": the decoded quote ends the string, so parsing fails and the regex fallback returns the title A.
  • "headline": "Why 3 < 5", ..., "name": "<b>Acme</b> > News": the decoded < and > are stripped as a single tag that spans both fields. The title becomes Why 3 Acme > News and the publisher name is lost.
  • Emoji escaped as surrogate pairs (😀) are dropped instead of decoded.

These escapes are common. Go's encoding/json, Django's json_script and Rails escape <, > and & this way by default, and PHP's json_encode escapes all non-ASCII characters.

Changes

  • The minified JSON is parsed first, and then normalize_json is applied to each string value through a new normalize_json_values helper. Markup in values is still stripped and entities are still decoded.
  • If the raw text does not parse, the previous path runs unchanged: normalize the text, parse again, then fall back to extract_json_parse_error.
  • Added the regression test test_extract_meta_json_escaped_characters in tests/metadata_tests.py.

pytest, mypy, ruff check ., ruff format --check and python tests/eval_gate.py pass, and the F1 scores are unchanged.

@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.95%. Comparing base (8d87c74) to head (dbffee0).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #999   +/-   ##
=======================================
  Coverage   99.95%   99.95%           
=======================================
  Files          21       21           
  Lines        4260     4278   +18     
=======================================
+ Hits         4258     4276   +18     
  Misses          2        2           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@adbar

adbar commented Oct 23, 2026

Copy link
Copy Markdown
Owner

Parsing before normalizing is the right order indeed.

One side effect: the entity decoding and surrogate removal in normalize_json only run when the string contains a backslash. &amp; now stays in categories, e.g. Food &amp; Drink. A lone \ud800 in articleSection also survives, and the JSON output then fails to encode as UTF-8.

Could normalize_json_values always unescape and strip surrogates on the parsed strings? A test with &amp; in articleSection would cover it.

@Enesuygurs

Copy link
Copy Markdown
Contributor Author

Thanks, good catch. normalize_json_values now always strips lone surrogates and unescapes HTML entities on the parsed strings, then strips tags and trims as before. The \uXXXX decoding stays in normalize_json for the raw-text fallback only, so nothing is decoded twice. I added a test with Food &amp; Drink and a lone \ud800 in articleSection, which also checks that the JSON output encodes as UTF-8.

@adbar
adbar merged commit 59e6078 into adbar:master Oct 23, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants