Repository navigation
Reject hosts with Unicode default-ignorable code points - #1801
Conversation
pyba-chronographer
Bot
added
the
bot:chronographer:provided
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Coverage Report for CI Build 29702735471Warning No base build found for commit Coverage: 98.75%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsRequires a base build to compare against. How to fix this → Coverage Stats
💛 - Coveralls |
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1801 +/- ##
========================================
Coverage 99.86% 99.86%
========================================
Files 21 21
Lines 4328 4462 +134
Branches 252 254 +2
========================================
+ Hits 4322 4456 +134
+ Misses 4 3 -1
- Partials 2 3 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
259e84e to
f5b9eac
Compare
There was a problem hiding this comment.
Pull request overview
This PR hardens yarl host handling by rejecting Unicode Default_Ignorable_Code_Point characters before IDNA/UTS-46 processing, preventing hosts from being silently normalized into a different value than the caller supplied.
Changes:
- Added a pre-IDNA validation step in
_encode_host()to reject hosts containing default-ignorable Unicode code points. - Added pytest coverage for the
URL(...)constructor and builder APIs (URL.build(),URL.with_host()). - Updated docs spelling wordlist and added a towncrier news fragment for the behavior change.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
yarl/_url.py |
Adds _DEFAULT_IGNORABLE_RE and rejects default-ignorables before IDNA encoding in _encode_host(). |
tests/test_url.py |
Adds constructor-level test coverage for rejecting default-ignorable code points in hosts. |
tests/test_url_update_netloc.py |
Adds with_host() test coverage for rejecting default-ignorable code points. |
tests/test_url_build.py |
Adds URL.build() test coverage for rejecting default-ignorable code points. |
docs/spelling_wordlist.txt |
Adds ignorable to satisfy doc spelling. |
CHANGES/1801.bugfix.rst |
Adds release note fragment describing the behavior change. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Coverage Report for CI Build 29704148905Coverage increased (+0.01%) to 98.779%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
f5b9eac to
8f20056
Compare
|
|
IDNA/UTS-46 mapping silently deletes default-ignorable code points (soft hyphen, zero-width space, word joiner, bidi controls, variation selectors, and so on), so a host like 'e<ZWSP>vil.com' encoded to 'evil.com', a different host than the input string. _encode_host now rejects any host containing such a code point before IDNA encoding, on every path including the plain URL(str) constructor (validate_host=False) and the builder APIs, so the parsed host cannot diverge from the input. This matches idna, httpx, and urllib3.
8f20056 to
211abb6
Compare
PR Review — Reject hosts with Unicode default-ignorable code pointsMerge-ready. Solid, well-pinned defense against invisible-host confusion; only a non-blocking note on test runtime.
🟢 Suggestions
1. Sweep test brute-forces ~135k code points through IDNA
|
What do these changes do?
Invisible default-ignorable and format code points (soft hyphen, zero-width space, word joiner, bidi controls, variation selectors, and so on) let the parsed host diverge from the string the caller validated. Depending on the code point, IDNA either silently deletes it (so
e<ZWSP>vil.comencoded toevil.com) or folds it into a different punycode host; in both cases.hostandstr(url)no longer matched the input._encode_hostnow rejects any host containing such a code point before IDNA encoding. This runs on every path, including the plainURL(str)constructor and the builder APIs, so the parsed host cannot diverge from the input. This matchesidna,httpxandurllib3, which all reject these hosts.The rejected set is the union of two authoritative sources, matching the two encoders
_idna_encodedispatches to: the UnicodeDefault_Ignorable_Code_Pointproperty (for theidnauts46 path) and RFC 3454 Stringprep Table B.1 "commonly mapped to nothing" (for the stdlibstr.encode("idna")fallback). The latter is the source ofU+1806, which is not default-ignorable but is still stripped by the fallback.Are there changes in behavior for the user?
Yes; a host containing a default-ignorable code point is now rejected with
ValueErrorinstead of being silently collapsed to a different host.Is it a substantial burden for the maintainers to support this?
No; the check is a single regex search on the already-slow non-ascii branch of
_encode_host, and the ascii hot path is untouched. A sweep test brute-forces every code point through_idna_encodeand fails if any point it silently deletes is not covered by the character class, so the hand-written ranges stay pinned to the installedidna/Unicode data rather than drifting.Related issue number
Reported by @bilguunbicktivism; thanks for the report.
Checklist
CONTRIBUTORS.txtCHANGES/folder