Skip to content

markdown: predicted heading slugs diverge from GitHub's actual anchors (emoji VS16 dropped, No-category characters kept), so internal links to those headings break #190

Open

Description

@rahul05ranjan

resolve_anchors binds a link to a heading-coincident anchor to the fragment the heading "will have in the rendered Markdown" (src/render/markdown/anchors.rs:1-5). For that binding to work, gfm_slug has to reproduce the anchor GitHub actually generates. It doesn't for two classes of characters, in opposite directions, and every internal link to such a heading lands on a fragment GitHub never assigned.

Divergence 1: variation selectors — anydoc drops them, GitHub keeps them

gfm_slug (src/render/markdown/anchors.rs:180) keeps only is_alphanumeric, the hardcoded mark ranges in is_combining_mark (anchors.rs:199), connector punctuation, and hyphens. U+FE0F (VARIATION SELECTOR-16) is in none of those, so it is filtered out. GitHub's anchor algorithm (github-slugger, which GitHub renders with) removes a fixed punctuation set; VS16 is not in it, so the anchor keeps the invisible selector.

Divergence 2: No-category characters — anydoc keeps them, GitHub drops them

² (U+00B2) and ½ (U+00BD) pass Rust's char::is_alphanumeric (category No is numeric), so gfm_slug keeps them. github-slugger removes both, so the GitHub anchor does not contain them.

Repro

Minimal .docx (styles.xml maps Heading1, bookmark inside each heading, w:hyperlink w:anchor=... after it), converted with 0.2.4 (commit 261fc25, identical to v0.2.4):

# ❤️ Love      → [go to heading](#-love)      GitHub anchor: #️-love   (fe0f kept)
# 2️⃣ Second   → [go to second](#2⃣-second)   GitHub anchor: #2️⃣-second (fe0f kept)
# x² formula   → [go to x](#x²-formula)       GitHub anchor: #x-formula (00b2 dropped)

Exact fragments from the conversion (hex-escaped):

anydoc:        2d 6c 6f 76 65            = "-love"
github-slugger: fe0f 2d 6c 6f 76 65      = "️-love"

anydoc:        32 20e3 2d 73 65 63 6f 6e 64      = "2⃣-second"
github-slugger: 32 fe0f 20e3 2d 73 65 63 6f 6e 64 = "2️⃣-second"

anydoc:        78 b2 2d 66 6f 72 6d 75 6c 61   = "x²-formula"
github-slugger: 78 2d 66 6f 72 6d 75 6c 61      = "x-formula"

vs github-slugger 2.0.0, which reproduces GitHub's anchors. Heading text such as ## ⚠️ Deprecation notice or ## ✨ What's new is common in real documents, so the wrong fragment is emitted for exactly the headings people decorate.

Verified against the npm build @firecrawl/anydoc 0.2.4 (source on main is unchanged: only README commits since v0.2.4).

Direction

gfm_slug's keep-list is close but not equal to github-slugger's remove-list. Matching github-slugger exactly for these two classes — keep U+FE0F (it is a mark GitHub's slugger preserves), and drop the No/Nl characters github-slugger removes (or, more robustly, derive the keep predicate from the same character set github-slugger removes) — would fix both directions. A differential test over a corpus of tricky headings against github-slugger would keep this property from regressing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions