Skip to content

Commit dc77b26

Browse files
luisleo526claude
andcommitted
docs: one canonical URL per page, robots.txt, a sitemap and an honest version label, with their self-test (#380, part b)
Part b of #380 (EN-94), split by ENGINE-TOP and L6; part a (the theme: the 404 page, per-page descriptions and the product link) lands first. cdocs.pineforge.dev is indexed twice per page (/page and /page.html), has no robots.txt or sitemap, and shows a git-describe string as its version. This part is the build, its output and its test: - docs/seo_postprocess.py runs after Doxygen and adds one rel=canonical per page (https://cdocs.pineforge.dev/<page>, the landing page as the site root) and rewrites relative page links from X.html to X (index.html to ./, anchors kept). Absolute, mailto, protocol-relative, root-relative, nested and non-page links, scripts, comments and every non-page file are left alone; a second pass changes nothing. It sets one meta description per page, built from the page title (the landing page gets one sentence condensed from docs/pages/index.md), and generates sitemap.xml from the built pages (sorted, canonical URLs, no 404, no clock data). - docs/_theme/robots.txt (allow all, names the sitemap) is copied to the site root through HTML_EXTRA_FILES in docs/Doxyfile; it lands here because the sitemap it names is generated by this part. - docs/build.sh calls the post-process after Doxygen and sets the version label to the tag name only for a clean checkout of an exact tag and to the word "unreleased" otherwise, so a git-describe string never reaches the site again. `docs/build.sh --self-test-seo` runs the real script on a synthetic site and asserts the output byte for byte, the sitemap URL set, untouched assets, idempotence, failure on empty input, and the version-label rule. - tests/CMakeLists.txt registers it as the CTest row test_docs_seo at the end of the file, so no earlier line moves (the private fixture read evidence pins CMake lines by number). - docs/build.sh stays a proof-read file and docs/seo_postprocess.py joins it, so scripts/ci_docs_only.py (PROOF_READS) and scripts/test_ci_docs_only.py list both, and docs/ci.md says so: five files a proof job reads. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent a871161 commit dc77b26

8 files changed

Lines changed: 385 additions & 14 deletions

File tree

‎docs/Doxyfile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,7 @@ HTML_EXTRA_FILES = _theme/doxygen-awesome/doxygen-awesome-darkmode-toggle.
202202
_theme/doxygen-awesome/doxygen-awesome-paragraph-link.js \
203203
_theme/doxygen-awesome/doxygen-awesome-interactive-toc.js \
204204
_theme/doxygen-awesome/doxygen-awesome-tabs.js \
205+
_theme/robots.txt \
205206
_theme/404.html
206207
HTML_COLORSTYLE = LIGHT
207208
HTML_COLORSTYLE_HUE = 209

‎docs/_theme/robots.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
User-agent: *
2+
Allow: /
3+
Sitemap: https://cdocs.pineforge.dev/sitemap.xml

‎docs/build.sh‎

Lines changed: 195 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# Build the PineForge API documentation site.
33
#
44
# Outputs: docs/site/html/ (static HTML, ready to deploy)
5-
# Requires: doxygen, dot (graphviz), curl, awk
5+
# Requires: doxygen, dot (graphviz), curl, awk, python3
66
#
77
# CI usage: bash docs/build.sh
88
# Local: bash docs/build.sh # docs/site/html/index.html
@@ -89,12 +89,198 @@ self_test_doxygen_retry() {
8989
return "$result"
9090
}
9191

92+
docs_version_label() {
93+
local exact_tag="$1" worktree_status="$2"
94+
if [[ -n "$exact_tag" && -z "$worktree_status" ]]; then
95+
printf '%s\n' "$exact_tag"
96+
else
97+
printf '%s\n' unreleased
98+
fi
99+
}
100+
101+
self_test_seo() {
102+
if [[ "$(docs_version_label "v1.5.0" "")" != "v1.5.0" \
103+
|| "$(docs_version_label "" "")" != "unreleased" \
104+
|| "$(docs_version_label "v1.5.0" " M docs/build.sh")" != "unreleased" \
105+
|| "$(docs_version_label "v1.5.0" "?? new-page.md")" != "unreleased" ]]; then
106+
echo "==> FAIL: docs version must be an exact clean tag or unreleased" >&2
107+
return 1
108+
fi
109+
local fixture script result
110+
fixture="$(mktemp -d "${TMPDIR:-/tmp}/pineforge-seo.XXXXXX")"
111+
script="$(cd "$(dirname "$0")" && pwd)/seo_postprocess.py"
112+
cat > "$fixture/check_seo.py" <<'PY'
113+
from html.parser import HTMLParser
114+
from pathlib import Path
115+
import subprocess
116+
import sys
117+
import xml.etree.ElementTree as ET
118+
119+
script = Path(sys.argv[1])
120+
fixture = Path(sys.argv[2])
121+
site = fixture / "html"
122+
site.mkdir()
123+
old_meta = '<meta name="description" content="Old shared description"/>\n'
124+
old_canonical = '<link href="https://example.com/old.html" rel="canonical"/>\n'
125+
duplicate_canonical = '<link rel="canonical" href="https://cdocs.pineforge.dev/other"/>\n'
126+
titles = {
127+
"index.html": "PineForge: Landing",
128+
"x.html": "PineForge: Orders &amp; &quot;fills&quot; &lt;T&gt;",
129+
"other.html": "PineForge: Other reference",
130+
"404.html": "Page not found",
131+
}
132+
bodies = {
133+
"index.html": '''<a href="x.html">X</a>
134+
<a href="index.html">Home</a>
135+
<a href="index.html#start">Start</a>''',
136+
"x.html": '''<a href="other.html#a">Other</a>
137+
<a href="./other.html">Relative</a>
138+
<a title='href="other.html"' data-href="x.html" href="other.html">Tooltip</a>
139+
<a href="https://example.com/x.html">External</a>
140+
<a href="mailto:[email protected]">Mail</a>
141+
<a href="#frag">Fragment</a>
142+
<a href="other.html#https://example.com">Embedded URL</a>
143+
<a href="//example.com/x.html">Protocol relative</a>
144+
<a href="/other.html">Root path</a>
145+
<a href="../other.html">Parent path</a>
146+
<a href="search/help.html">Search path</a>
147+
<a href="icon.png">Image</a>
148+
<link href="style.css" rel="stylesheet"/>
149+
<script src="links.js"></script>
150+
<script>const text = 'href="other.html"';</script>
151+
<!-- href="other.html" -->''',
152+
"other.html": '<a href="x.html">X</a>',
153+
"404.html": '<p>Page missing.</p><a href="/">Home</a><a href="https://pineforge.dev">pineforge.dev</a>',
154+
}
155+
descriptions = {
156+
"index.html": (
157+
"PineForge&#x27;s C++17 engine has two layers documented in this API reference: "
158+
"a kernel that knows nothing about PineScript or TradingView and a Pine "
159+
"adapter that reproduces TradingView&#x27;s execution semantics."
160+
),
161+
"x.html": "Orders &amp; &quot;fills&quot; &lt;T&gt; — PineForge C++17 engine API documentation.",
162+
"other.html": "Other reference — PineForge C++17 engine API documentation.",
163+
}
164+
165+
def page(title, body, metadata):
166+
return (f"<!doctype html>\n<html><head>\n<title>{title}</title>\n"
167+
f"{metadata}</head>\n<body>\n{body}\n</body></html>\n")
168+
169+
for name, title in titles.items():
170+
metadata = "" if name == "404.html" else old_meta
171+
if name == "other.html":
172+
metadata += old_canonical + duplicate_canonical
173+
(site / name).write_bytes(page(title, bodies[name], metadata).encode("utf-8"))
174+
175+
untouched = {
176+
"links.js": b'const page = "foo.html"; const link = \'href="x.html"\';\r\n',
177+
"style.css": b'/* foo.html */\n',
178+
"search/searchdata.js": b'const page = "foo.html";\n',
179+
"search/help.html": b'<a href="x.html">Nested input stays untouched</a>\n',
180+
}
181+
for name, contents in untouched.items():
182+
(site / name).parent.mkdir(parents=True, exist_ok=True)
183+
(site / name).write_bytes(contents)
184+
185+
expected_bodies = dict(bodies)
186+
expected_bodies["index.html"] = (bodies["index.html"]
187+
.replace('href="x.html"', 'href="x"')
188+
.replace('href="index.html"', 'href="./"')
189+
.replace('href="index.html#start"', 'href="./#start"'))
190+
expected_bodies["x.html"] = (bodies["x.html"]
191+
.replace('<a href="other.html#a">', '<a href="other#a">')
192+
.replace('<a href="./other.html">', '<a href="./other">')
193+
.replace('data-href="x.html" href="other.html">', 'data-href="x.html" href="other">'))
194+
expected_bodies["other.html"] = '<a href="x">X</a>'
195+
196+
class Metadata(HTMLParser):
197+
def __init__(self):
198+
super().__init__(convert_charrefs=True)
199+
self.canonicals = []
200+
self.descriptions = []
201+
202+
def handle_starttag(self, tag, attrs):
203+
attrs = dict(attrs)
204+
if tag == "link" and "canonical" in attrs.get("rel", "").split():
205+
self.canonicals.append(attrs.get("href"))
206+
if tag == "meta" and attrs.get("name") == "description":
207+
self.descriptions.append(attrs.get("content"))
208+
209+
command = [sys.executable, "-I", "-B", str(script), str(site)]
210+
subprocess.run(command, check=True)
211+
parsed_descriptions = []
212+
canonical_urls = {}
213+
for name, title in titles.items():
214+
url = "https://cdocs.pineforge.dev/" + ("" if name == "index.html" else name[:-5])
215+
canonical_urls[name] = url
216+
canonical = f'<link rel="canonical" href="{url}"/>\n'
217+
actual = (site / name).read_bytes().decode("utf-8")
218+
if name == "404.html":
219+
# The error page is left exactly as written: no canonical, no description, no link rewrite.
220+
assert actual == page(title, bodies[name], ""), "404.html must be left untouched"
221+
continue
222+
meta = f'<meta name="description" content="{descriptions[name]}"/>\n'
223+
assert canonical + "</head>" in actual, f"{name}: missing canonical before </head>"
224+
assert actual == page(title, expected_bodies[name], meta + canonical), f"{name}: wrong HTML output"
225+
parsed = Metadata()
226+
parsed.feed(actual)
227+
assert parsed.canonicals == [url], f"{name}: canonical must occur exactly once"
228+
assert len(parsed.descriptions) == 1, f"{name}: description must occur exactly once"
229+
parsed_descriptions.extend(parsed.descriptions)
230+
assert len(set(parsed_descriptions)) == len(titles) - 1, "page descriptions must differ (404 has none)"
231+
assert '<a href="x">X</a>' in (site / "index.html").read_text(), "x.html was not rewritten"
232+
for name, contents in untouched.items():
233+
assert (site / name).read_bytes() == contents, f"{name}: asset or nested page changed"
234+
235+
namespace = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
236+
root = ET.parse(site / "sitemap.xml").getroot()
237+
assert root.tag == namespace + "urlset" and root.attrib == {}, "invalid sitemap urlset"
238+
urls = []
239+
for entry in root:
240+
assert entry.tag == namespace + "url" and entry.attrib == {}, "invalid sitemap entry"
241+
assert len(entry) == 1 and entry[0].tag == namespace + "loc", "sitemap must contain only locations"
242+
assert entry[0].attrib == {}, "sitemap location has unexpected attributes"
243+
urls.append(entry[0].text)
244+
expected_urls = sorted(canonical_urls[path.name] for path in site.glob("*.html") if path.name != "404.html")
245+
assert urls == expected_urls, "sitemap differs from the sorted top-level page set minus 404"
246+
247+
first_pass = {path.relative_to(site): path.read_bytes() for path in site.rglob("*") if path.is_file()}
248+
subprocess.run(command, check=True)
249+
second_pass = {path.relative_to(site): path.read_bytes() for path in site.rglob("*") if path.is_file()}
250+
assert second_pass == first_pass, "a second pass must be byte-identical, including canonical and sitemap"
251+
252+
empty = fixture / "empty"
253+
empty.mkdir()
254+
only_404 = fixture / "only-404"
255+
only_404.mkdir()
256+
(only_404 / "404.html").write_bytes((site / "404.html").read_bytes())
257+
for invalid_site in (empty, only_404, fixture / "missing"):
258+
result = subprocess.run(command[:-1] + [str(invalid_site)], capture_output=True)
259+
assert result.returncode != 0, "missing or empty documentation must fail"
260+
assert not (invalid_site / "sitemap.xml").exists(), "invalid input produced a sitemap"
261+
PY
262+
if python3 -I -B "$fixture/check_seo.py" "$script" "$fixture"; then
263+
result=0
264+
else
265+
result=$?
266+
fi
267+
rm -rf "$fixture"
268+
if (( result == 0 )); then
269+
echo "==> Docs SEO self-test: canonical, links, descriptions, sitemap, assets, version ... OK"
270+
fi
271+
return "$result"
272+
}
273+
92274
if [[ "${1:-}" == "--self-test-retry" && "$#" == 1 ]]; then
93275
self_test_doxygen_retry
94276
exit $?
95277
fi
278+
if [[ "${1:-}" == "--self-test-seo" && "$#" == 1 ]]; then
279+
self_test_seo
280+
exit $?
281+
fi
96282
if (( $# != 0 )); then
97-
echo "usage: docs/build.sh [--self-test-retry]" >&2
283+
echo "usage: docs/build.sh [--self-test-retry|--self-test-seo]" >&2
98284
exit 2
99285
fi
100286

@@ -120,11 +306,12 @@ if [[ ! -f "$THEME_DIR/.version" ]] \
120306
echo "$DOXYGEN_AWESOME_VERSION" > "$THEME_DIR/.version"
121307
fi
122308

123-
# 2. Resolve project version from VERSION file (or git tag if available).
124-
VERSION="$(cat ../VERSION 2>/dev/null || echo unknown)"
125-
if command -v git >/dev/null 2>&1 && git -C .. rev-parse --git-dir >/dev/null 2>&1; then
126-
GIT_VERSION="$(git -C .. describe --tags --dirty --always 2>/dev/null || true)"
127-
[[ -n "$GIT_VERSION" ]] && VERSION="$GIT_VERSION"
309+
# 2. Label only a clean, exactly tagged checkout as a release.
310+
VERSION="unreleased"
311+
if command -v git >/dev/null 2>&1 \
312+
&& GIT_VERSION="$(git -C .. describe --tags --exact-match 2>/dev/null)" \
313+
&& GIT_STATUS="$(git -C .. status --porcelain 2>/dev/null)"; then
314+
VERSION="$(docs_version_label "$GIT_VERSION" "$GIT_STATUS")"
128315
fi
129316
echo "==> Building docs for PineForge $VERSION"
130317

@@ -136,6 +323,7 @@ CONFIG="$(mktemp "${TMPDIR:-/tmp}/pineforge-doxyfile.XXXXXX")"
136323
trap 'rm -f "$CONFIG"' EXIT
137324
{ cat Doxyfile; echo "PROJECT_NUMBER = $VERSION"; } > "$CONFIG"
138325
run_doxygen_with_retry
326+
python3 -I -B seo_postprocess.py site/html
139327

140328
# 3b. The scoped warning gate (R5 lane L14-A).
141329
#

‎docs/ci.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -779,14 +779,16 @@ The first job of `ci.yml`, `changes`, decides with
779779
documentation when it is a Markdown file (`*.md`) or lies under `docs/`,
780780
unless it lies under `.github/`, `benchmarks/`, `scripts/` or `tests/`, is a
781781
CMake file or a git control file such as `.gitattributes`, or is one of the
782-
four files a proof job reads:
782+
five files a proof job reads:
783783
`docs/pages/pine-to-native.md` and `docs/pages/native-engine.md`, whose code
784784
blocks `test_pine_to_native_worked` and `test_native_engine_complete_host`
785785
compile and run; `docs/adr/0001-kernel-adapter-boundary.md`, which
786786
`test_kernel_residuals`, `test_native_feature_rulings` and the kernel
787-
profile's `kernel-residuals` stage hold the kernel against; and
788-
`docs/build.sh`, whose retry self-test is `test_docs_doxygen_retry`. A change
789-
to one of those four runs every job. A change is documentation only when every
787+
profile's `kernel-residuals` stage hold the kernel against;
788+
`docs/build.sh`, whose self-tests are `test_docs_doxygen_retry` and
789+
`test_docs_seo`; and `docs/seo_postprocess.py`, which `test_docs_seo` executes
790+
on synthetic pages. A change
791+
to one of those five runs every job. A change is documentation only when every
790792
path it changes, both sides of a rename, is documentation. The change is the
791793
checked-out commit against its first parent: for a pull request, the merge
792794
GitHub tests against the base it merged into; for a push, the pushed commit
@@ -823,7 +825,7 @@ full run on `main` holds the code's verdict.
823825
holds the rule against real git histories, pull request merges and pushes
824826
alike. It also fails when a CMake file names, through `${PROJECT_SOURCE_DIR}`
825827
or a sibling source-directory variable, a path the rule would call
826-
documentation. The four files above were found by reading every CMake file,
828+
documentation. The five files above were found by reading every CMake file,
827829
every `ci_verify.py` stage and every script a CTest row runs; a reader added
828830
outside CMake must list its file in `PROOF_READS` by hand. The
829831
`ci-workflow-contract` stage pins the wiring, the classification step

0 commit comments

Comments
 (0)