docs: mark Doxygen source listings noindex and keep them out of the sitemap - #398

…itemap

The API documentation site draws most of its search impressions to raw
C++ source listings (Doxygen's per-file `*_8<ext>_source.html` pages),
while the API reference pages get few.

docs/seo_postprocess.py now adds one `<meta name="robots"
content="noindex">` in the head of a source listing, replacing any robots
meta it already had, and drops every page that carries a noindex or none
directive from sitemap.xml. A page is a source listing only when its file
name matches that pattern and its title ends with "Source File", which is
how Doxygen titles a listing: a hand-written page can choose a colliding
name with an explicit page id and must stay indexable. File reference pages (`*_8hpp.html`), class,
struct, namespace, member, group and directory pages, the hand-written
pages, the landing page and the 404 page are unchanged. robots.txt keeps
allowing everything: a crawler that may not fetch a page never sees its
noindex. A second pass over the output is byte-identical.

The `--self-test-seo` self-test gains a second phase with six source
fixtures (including one with a conflicting robots meta, one with an
uppercase tag and one whose title has trailing whitespace), twelve API and
narrative fixtures, among them a class whose name ends in `source`, and a
hand-written page whose explicit id collides with the listing name pattern
and must stay indexable and in the sitemap. It asserts which pages get the
tag, the sitemap URL set, the position of the tag, untouched assets and a
byte-identical second pass. The footer already links to pineforge.dev on
every page, source listings included; docs/ci.md says so.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>