fix(context): measure visible link text when skipping tables of contents - #190

Problem

isTableOfContents skips a section when more than half of its content is links. It counted each link's whole Markdown, URL included, as link text:

const links = content.match(linkPattern) || [];
const linkTextLength = links.reduce((sum, link) => sum + link.length, 0);
if (content.length > 0 && linkTextLength / content.length > TOC_LINK_RATIO) {

A prose paragraph with two or three long URLs therefore reads as "mostly links" and is left out of the index. tRPC's AWS Lambda adapter page is one example. Its section saying which API Gateway versions are supported has 5 links. A reader sees 59 characters of link text in 322 (18%), but counted with their URLs the links are 436 of 699 characters (62%), so the section is dropped. It opens:

The AWS Lambda adapter is supported for API Gateway [REST API(v1)](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-rest-api.html) and [HTTP API(v2)](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api.html), and [Lambda Function URL](https://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html) use cases.

HTML is hit hardest, because turndown writes Sphinx's cross-references with relative paths and title attributes ([`PyObject`](structures.html#c.PyObject "PyObject")) and every definition has a [¶](#c.PyDict_Check "Link to this definition") permalink. Most Python C API pages are one long section, so the whole page goes.

I parsed every file with parseDocument on main and with this change. The Python run uses the registry definition's exclude_paths:

Python pages that produce no sections on main include the C API pages for dict (10,403 tokens), list, set, tuple, datetime, sequence and monitoring, plus library/errno.html, library/token.html, library/asyncio-exceptions.html and library/keyword.html. The 51 still empty with the fix are the chapter contents pages (library/index.html, contents.html, library/datatypes.html, ...) and Sphinx's generated index pages (genindex*.html, py-modindex.html).

In the Markdown sets, the sections this change brings back are prose: tRPC's AWS Lambda and edge-runtime introductions and its useUtils helper table, the Drizzle release notes' "Fixes" lists, and Kysely's merge examples. I read all of them.

End to end, building a package from c-api/dict.html, library/errno.html and tRPC's aws-lambda.md with buildPackage and then calling search:

The package has 3 sections (1,493 tokens) on main and 25 sections (16,943 tokens) with the fix.

Fix

Measure what a reader sees. Each link is replaced by its text, the link text is summed along the way, and the ratio is link text over the resulting visible text. The 50% threshold is unchanged. Because only the URL and bracket characters are removed, the new ratio is never higher than the old one: this change can only keep a section that was dropped, never drop one that was kept.

Sphinx's generated index pages (genindex*.html, py-modindex.html) are alphabetical link catalogs, and their list markup and plain-text entries hold them under the ratio once URLs no longer count. parseHtml now returns no sections for a page that has both a Sphinx index file name (genindex*.html, or a domain index such as py-modindex.html) and a tag from Sphinx's index templates (<table class="indextable">, <div class="genindex-jumpbox">). A page with only one of the two is still indexed. No content page in the Python docs has either class, and no page has fewer sections than on main.

Not changed

• The 50% threshold and the title rules (table of contents, toc, contents, index).
• Images are still matched as links, as before (![alt](src) counts its alt text as link text). Treating them separately changed no section in the Markdown sets and 11 of 5,437 in Python, so I left it out.

Test

html.test.ts: a genindex landing page, a letter page and py-modindex.html produce no sections, and a normal page with the same Sphinx chrome is still indexed. On the previous commit the three index cases fail. More pages stay indexed: a guide with its own indextable table, a guide that shows Sphinx's index markup in a code sample and a comment, a page named genindex.html that shows the markup in a code sample, a page named genindex.html without Sphinx's markup, and an api-index.html guide with an indextable. Dropping either half of the check fails one of them.

build.test.ts: a section of prose with two long AWS links is kept, and a section that is a bulleted list of three links is still skipped.

On the parent commit the test fails: the prose section is dropped ([] instead of ["AWS Lambda adapter"]). With the fix it passes.

I broke each part of the fix in turn, and each time the test went red:

Validation

• vitest run in packages/context: 18 files, 361 tests pass.
• biome ci --error-on-warnings: clean.
• tsc --noEmit -p tsconfig.build.json: clean.
• Changeset: .changeset/quiet-owls-wander.md (patch).