docs: one canonical URL per page, robots.txt, a sitemap and an honest version label (#380, part b) - #394

Part B of #380, on main a871161 (after #390, part A, and #393). It carries the docs build's SEO post-processing and the docs-only CI check: docs/build.sh, docs/seo_postprocess.py (new), docs/_theme/robots.txt (new), the robots line in docs/Doxyfile, docs/ci.md, scripts/ci_docs_only.py with its test, and a new CTest row test_docs_seo in tests/CMakeLists.txt. Because it changes build and CI scripts and adds a CTest row, it is not a documentation-only change and is verified with the full set of steps. The description below covers the whole of #380.

cdocs.pineforge.dev is indexed twice per page (/page and /page.html), has no robots.txt or sitemap, carries one meta description on every page, shows a git-describe string as its version (v1.5.0-rc.1-7-g42fa1d4b), and links nowhere to the product site. This fixes what the repository can fix (EN-94, items C1-C5).

What changes

• C1 docs/_theme/footer.html: every page footer links https://pineforge.dev (both footer branches).
• C2 new docs/seo_postprocess.py (standard library only; called by docs/build.sh right after Doxygen): adds one rel="canonical" per page (https://cdocs.pineforge.dev/<page>, the landing page as the site root) and rewrites relative page links from X.html to X (index.html to ./, anchors kept). Absolute, mailto:, protocol-relative, root-relative, nested and non-page links, scripts, comments and every non-page file are left alone. Running it twice changes nothing. All pages are validated before any is written; an empty or missing site fails the build.
• C3 docs/_theme/robots.txt (allow all, names the sitemap) and docs/_theme/404.html are copied to the site root through HTML_EXTRA_FILES (docs/Doxyfile); sitemap.xml is generated from the built pages (sorted, canonical URLs, no 404, no clock data). 404.html is a plain static page (absolute stylesheet links, no scripts) and is left untouched by the post-processing.
• C4 docs/_theme/header.html: the meta description is built from each page's title; the landing page gets one sentence taken from docs/pages/index.md (the C++17 engine in two layers: a kernel that knows nothing about PineScript or TradingView, and the Pine adapter).
• C5 docs/build.sh: the version label is the tag name only for a clean checkout of an exact tag, and the word unreleased otherwise.

Please read: C5 (stated)

main is already past the v1.5.0 tag, so the deployed site will read unreleased from the first deploy until the next tag is cut. A build of a tag shows that tag.

Please read: C3 needs a host-side check (not part of this PR; for the site owners)

Today every unknown path on cdocs.pineforge.dev answers 200 with the home page (measured 10-12: /robots.txt, /sitemap.xml, /nonexistent-xyz all 200 text/html). Nothing in this repository produces that; it is host-side configuration of the Cloudflare Pages project pineforge-docs (or whatever fronts the domain), and this PR does not touch Cloudflare. Real files are served correctly (/custom.css 200 text/css), so robots.txt and sitemap.xml will be served as real files after the deploy. For /nonexistent-xyz the decider is
curl -s -o /dev/null -w '%{http_code}\n' https://cdocs.pineforge.dev/nonexistent-xyz:
200 means the host still rewrites unknown paths to /: the change is then to turn off the single-page-app / not-found fallback on the Pages project (in the site owner's Cloudflare account) so the root 404.html is served with status 404. 404 means done.

Notes for the reviewer

• The sitemap lists every generated top-level page: the docs step built 1143 pages in the lab run, so sitemap.xml has 1142 URLs (well under the 50,000 limit). The ~70 URLs seen in Google's index are a subset.
• docs/build.sh is a proof-read file, so this PR takes the full CI path (not the documentation-only path). scripts/ci_docs_only.py, scripts/test_ci_docs_only.py and docs/ci.md list docs/seo_postprocess.py next to it. docs/ci.md words changed: "four files a proof job reads" -> "five files" (two places) and the docs/build.sh sentence now names test_docs_seo besides test_docs_doxygen_retry. If this PR is squash-merged, build the squash body from the two commit messages so those namings stay.
• Why two commits: 0e7a69f7 is the change as reviewed; 9827eaad moves the test_docs_seo registration to the END of tests/CMakeLists.txt, because tests/private_fixture_reads.evidence pins CMake lines by number and a registration in the middle moved a pinned line (the first lab run failed exactly test_private_fixture_reads and test_private_fixture_reads_controls).
• The self-test (bash docs/build.sh --self-test-seo, CTest row test_docs_seo) runs the real script on a synthetic site and compares the output byte for byte: canonical per page, rewritten and untouched links, escaped descriptions, sitemap URL set, untouched assets, a second pass identical, failure on empty input, the version rule.
• Order: lands after the corpus gitlink PR corpus: move the gitlink to 9245f28 (probe directories removed, headers state settings only) #376; built on main 2dbd72f; rebased onto Use the shared PineForge favicon in API docs #355 (shared favicon, same HTML_EXTRA_FILES lines) if that lands first.

Verification (lab remote, no tests on the Mac)

• Job rj-20261012t195837-b0760d (commit 0e7a69f): preflight passed (49 stages, 0 advisory); docs passed (exit 0, Doxygen warnings 0, ==> SEO post-processing: 1143 pages and sitemap.xml). Release, debug, sanitizers, kernel: 2 failures each, test_private_fixture_reads*, fixed by 9827eaa as above.
• Job rj-20261012t205443-8ec24a (commit 9827eaa, from a worktree of the primary checkout so abi-prior-fetch has its remote): ci_verify release passed, 911 of 911 tests, 38 stages, 0 failures, including test_docs_seo, test_docs_doxygen_retry and test_private_fixture_reads.
• Job rj-20261012t204300-5f005c (commit 9827eaa): debug exit 0, kernel exit 0; sanitizers: see the PR conversation for the final line.

🤖 Generated with Claude Code