Skip to content

fix: keep non-Latin path text in single-link scrape titles - #7025

Merged
timothycarambat merged 2 commits into
Mintplex-Labs:masterfrom
b84330808:fix-link-title-non-latin
Nov 6, 2026
Merged

timothycarambat merged 2 commits into
Mintplex-Labs:masterfrom
b84330808:fix-link-title-non-latin

Conversation

@b84330808

Copy link
Copy Markdown
Contributor

Pull Request Type

  • ✨ feat (New feature)
  • 🐛 fix (Bug fix)
  • ♻️ refactor (Code refactoring without changing behavior)
  • 💄 style (UI style changes)
  • 🔨 chore (Build, CI, maintenance)
  • 📝 docs (Documentation updates)

Relevant Issues

resolves #7024

Description

For a single-link scrape, the default document title in collector/processLink/convert/generic.js was slugify(filename) + ".html". slugify deletes characters it cannot transliterate, so pages under non-Latin paths (Chinese, Japanese, Korean, Thai, Hebrew, Hindi, ...) all collapsed to the same title — every zh.wikipedia article became zh.wikipedia.org_wiki_.html — and the query string was dropped, so URLs differing only in ?id= collided too.

The default title now follows the bulk link scraper's format (since #6650): ${filename}${url.search}.html, without slugifying. Titles for Latin paths are unchanged, metadata.title still takes precedence, and the on-disk filename and data.url remain slugs as before.

Visuals (if applicable)

N/A

Additional Information

A regression test is added in collector/__tests__/processLink/convert/generic.test.js (keeps non-Latin path text and the query string in the default title). I ran that suite on Linux: the new test fails without this change (1 failed, 26 passed) and all 27 pass with it.

Full disclosure on the two unchecked boxes below: a complete dependency install does not finish in my environment (a native package in the tree fails to install there), so I could not honestly run yarn lint or the Docker build. The collector test suite for the touched file was run with the real sources and dependencies for that module. Happy to re-run lint/Docker if a maintainer wants, or for CI to cover it.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated (if applicable)
  • I have tested my code functionality
  • Docker build succeeds locally

The single-link scraper titled a page slugify(host + path) + ".html".
slugify deletes every character it has no transliteration for, so pages
under Chinese, Japanese, Korean, Thai, Hebrew or Hindi paths all got the
same title, e.g. "zh.wikipedia.org_wiki_.html" for every zh.wikipedia
article. Query strings were dropped too.

Build the default title the way the bulk link scraper already does:
the decoded host and path plus the query string, unslugified. The url
and the file name are unchanged.
@b84330808
b84330808 force-pushed the fix-link-title-non-latin branch from d5244e9 to 105b9b6 Compare November 5, 2026 08:18

@shatfield4 shatfield4 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Matches the same fix made in #6650 for a similar issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

PR:Ready-to-merge PR has been reviewed by core team and is ready to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Single-link scrape titles collapse for non-Latin URLs and drop the query string

3 participants