Repository navigation
fix: decode JSON-LD escapes before normalizing metadata values - #999
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #999 +/- ##
=======================================
Coverage 99.95% 99.95%
=======================================
Files 21 21
Lines 4260 4278 +18
=======================================
+ Hits 4258 4276 +18
Misses 2 2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Parsing before normalizing is the right order indeed. One side effect: the entity decoding and surrogate removal in Could |
|
Thanks, good catch. |
Problem
extract_meta_json()rannormalize_json()on the raw JSON-LD text beforejson.loads. That decodes\uXXXXescapes and strips<...>as HTML tags on the serialized JSON instead of on the values:"headline": "A "quoted" title": the decoded quote ends the string, so parsing fails and the regex fallback returns the titleA."headline": "Why 3 < 5", ..., "name": "<b>Acme</b> > News": the decoded<and>are stripped as a single tag that spans both fields. The title becomesWhy 3 Acme > Newsand the publisher name is lost.😀) are dropped instead of decoded.These escapes are common. Go's
encoding/json, Django'sjson_scriptand Rails escape<,>and&this way by default, and PHP'sjson_encodeescapes all non-ASCII characters.Changes
normalize_jsonis applied to each string value through a newnormalize_json_valueshelper. Markup in values is still stripped and entities are still decoded.extract_json_parse_error.test_extract_meta_json_escaped_charactersintests/metadata_tests.py.pytest,mypy,ruff check .,ruff format --checkandpython tests/eval_gate.pypass, and the F1 scores are unchanged.