Repository navigation
fix: keep non-Latin path text in single-link scrape titles - #7025
Merged
timothycarambat merged 2 commits intoNov 6, 2026
Merged
Conversation
The single-link scraper titled a page slugify(host + path) + ".html". slugify deletes every character it has no transliteration for, so pages under Chinese, Japanese, Korean, Thai, Hebrew or Hindi paths all got the same title, e.g. "zh.wikipedia.org_wiki_.html" for every zh.wikipedia article. Query strings were dropped too. Build the default title the way the bulk link scraper already does: the decoded host and path plus the query string, unslugified. The url and the file name are unchanged.
b84330808
force-pushed
the
fix-link-title-non-latin
branch
from
November 5, 2026 08:18
d5244e9 to
105b9b6
Compare
shatfield4
approved these changes
Nov 5, 2026
shatfield4
left a comment
Collaborator
There was a problem hiding this comment.
LGTM
Matches the same fix made in #6650 for a similar issue.
3 of 9 tasks
This was referenced Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Type
Relevant Issues
resolves #7024
Description
For a single-link scrape, the default document title in
collector/processLink/convert/generic.jswasslugify(filename) + ".html".slugifydeletes characters it cannot transliterate, so pages under non-Latin paths (Chinese, Japanese, Korean, Thai, Hebrew, Hindi, ...) all collapsed to the same title — every zh.wikipedia article becamezh.wikipedia.org_wiki_.html— and the query string was dropped, so URLs differing only in?id=collided too.The default title now follows the bulk link scraper's format (since #6650):
${filename}${url.search}.html, without slugifying. Titles for Latin paths are unchanged,metadata.titlestill takes precedence, and the on-disk filename anddata.urlremain slugs as before.Visuals (if applicable)
N/A
Additional Information
A regression test is added in
collector/__tests__/processLink/convert/generic.test.js(keeps non-Latin path text and the query string in the default title). I ran that suite on Linux: the new test fails without this change (1 failed, 26 passed) and all 27 pass with it.Full disclosure on the two unchecked boxes below: a complete dependency install does not finish in my environment (a native package in the tree fails to install there), so I could not honestly run
yarn lintor the Docker build. The collector test suite for the touched file was run with the real sources and dependencies for that module. Happy to re-run lint/Docker if a maintainer wants, or for CI to cover it.Developer Validations
yarn lintfrom the root of the repo & committed changes