Skip to content

Commit 0e7a69f

Browse files
luisleo526claude
andcommitted
docs: one canonical URL per page, a product link, robots.txt, sitemap.xml, a 404 page, page descriptions, an honest version label
cdocs.pineforge.dev is indexed twice per page (/page and /page.html), has no robots.txt or sitemap, carries one meta description on every page, shows a git-describe string as its version, and links nowhere to the product site. This fixes what the repository can fix (EN-94, items C1-C5): - C1: every page footer links https://pineforge.dev (both footer branches). - C2: docs/seo_postprocess.py runs after Doxygen and adds one rel=canonical per page (https://cdocs.pineforge.dev/<page>, the landing page as the site root) and rewrites relative page links from X.html to X (index.html to ./, anchors kept). Absolute, mailto, protocol-relative, root-relative, nested and non-page links, scripts, comments and every non-page file are left alone; a second pass changes nothing. - C3: robots.txt (allow all, names the sitemap) and 404.html are copied to the site root through HTML_EXTRA_FILES; sitemap.xml is generated from the built pages (sorted, canonical URLs, no 404, no clock data). The 404.html is a plain static page; whether the host serves it for unknown paths is a host setting that this change does not touch. - C4: the meta description is built from each page's title; the landing page gets one sentence from docs/pages/index.md. - C5: the version label is the tag name only for a clean checkout of an exact tag, and the word "unreleased" otherwise, so a git-describe string never reaches the site again. Tests: docs/build.sh --self-test-seo (CTest row test_docs_seo) runs the real script on a synthetic site and asserts the output byte for byte, the sitemap URL set, untouched assets, idempotence, failure on empty input, and the version-label rule. docs/build.sh stays a proof-read file, so scripts/ci_docs_only.py, scripts/test_ci_docs_only.py and docs/ci.md list docs/seo_postprocess.py beside it. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
1 parent 2dbd72f commit 0e7a69f

11 files changed

Lines changed: 407 additions & 16 deletions

File tree

‎docs/Doxyfile‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -200,7 +200,9 @@ HTML_EXTRA_FILES = _theme/doxygen-awesome/doxygen-awesome-darkmode-toggle.
200200
_theme/doxygen-awesome/doxygen-awesome-fragment-copy-button.js \
201201
_theme/doxygen-awesome/doxygen-awesome-paragraph-link.js \
202202
_theme/doxygen-awesome/doxygen-awesome-interactive-toc.js \
203-
_theme/doxygen-awesome/doxygen-awesome-tabs.js
203+
_theme/doxygen-awesome/doxygen-awesome-tabs.js \
204+
_theme/robots.txt \
205+
_theme/404.html
204206
HTML_COLORSTYLE = LIGHT
205207
HTML_COLORSTYLE_HUE = 209
206208
HTML_COLORSTYLE_SAT = 255

‎docs/_theme/404.html‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
<!DOCTYPE html>
2+
<html lang="en" class="light-mode">
3+
<head>
4+
<meta charset="utf-8"/>
5+
<meta name="viewport" content="width=device-width, initial-scale=1"/>
6+
<title>Page not found</title>
7+
<link rel="stylesheet" href="/doxygen-awesome.css"/>
8+
<link rel="stylesheet" href="/custom.css"/>
9+
<style>
10+
body { margin: 0; background: var(--page-background-color, #fff); color: var(--page-foreground-color, #2c2c2c); font-family: var(--font-family, sans-serif); line-height: 1.65; }
11+
main { max-width: 42rem; margin: 5rem auto; padding: 0 1.5rem; }
12+
a { color: var(--primary-color, #1565c0); }
13+
</style>
14+
</head>
15+
<body>
16+
<main>
17+
<h1>Page not found</h1>
18+
<p>The requested documentation page could not be found.</p>
19+
<p><a href="/">API reference</a> &mdash; <a href="https://pineforge.dev">pineforge.dev</a></p>
20+
</main>
21+
</body>
22+
</html>

‎docs/_theme/footer.html‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
PineForge $projectnumber
99
&mdash; Apache-2.0
1010
&mdash; <a href="https://github.com/pineforge-4pass/pineforge-engine">github.com/pineforge-4pass/pineforge-engine</a>
11+
&mdash; <a href="https://pineforge.dev">pineforge.dev</a>
1112
</li>
1213
</ul>
1314
</div>
@@ -16,6 +17,7 @@
1617
<hr class="footer"/><address class="footer"><small>
1718
PineForge $projectnumber &mdash; Apache-2.0 &mdash;
1819
<a href="https://github.com/pineforge-4pass/pineforge-engine">source on GitHub</a>
20+
&mdash; <a href="https://pineforge.dev">pineforge.dev</a>
1921
</small></address>
2022
</div><!-- doc-content -->
2123
<!--END !GENERATE_TREEVIEW-->

‎docs/_theme/header.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
<meta http-equiv="X-UA-Compatible" content="IE=11"/>
2424
<meta name="generator" content="Doxygen $doxygenversion"/>
2525
<meta name="viewport" content="width=device-width, initial-scale=1"/>
26-
<meta name="description" content="PineForge runtime — public C ABI reference. Deterministic PineScript v6 backtest engine, validated trade-for-trade against TradingView."/>
26+
<meta name="description" content="$title — PineForge C++17 engine API documentation."/>
2727
<!--BEGIN PROJECT_NAME--><title>$projectname: $title</title><!--END PROJECT_NAME-->
2828
<!--BEGIN !PROJECT_NAME--><title>$title</title><!--END !PROJECT_NAME-->
2929
<!--BEGIN PROJECT_ICON-->

‎docs/_theme/robots.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
User-agent: *
2+
Allow: /
3+
Sitemap: https://cdocs.pineforge.dev/sitemap.xml

‎docs/build.sh‎

Lines changed: 195 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# Build the PineForge API documentation site.
33
#
44
# Outputs: docs/site/html/ (static HTML, ready to deploy)
5-
# Requires: doxygen, dot (graphviz), curl, awk
5+
# Requires: doxygen, dot (graphviz), curl, awk, python3
66
#
77
# CI usage: bash docs/build.sh
88
# Local: bash docs/build.sh # docs/site/html/index.html
@@ -89,12 +89,198 @@ self_test_doxygen_retry() {
8989
return "$result"
9090
}
9191

92+
docs_version_label() {
93+
local exact_tag="$1" worktree_status="$2"
94+
if [[ -n "$exact_tag" && -z "$worktree_status" ]]; then
95+
printf '%s\n' "$exact_tag"
96+
else
97+
printf '%s\n' unreleased
98+
fi
99+
}
100+
101+
self_test_seo() {
102+
if [[ "$(docs_version_label "v1.5.0" "")" != "v1.5.0" \
103+
|| "$(docs_version_label "" "")" != "unreleased" \
104+
|| "$(docs_version_label "v1.5.0" " M docs/build.sh")" != "unreleased" \
105+
|| "$(docs_version_label "v1.5.0" "?? new-page.md")" != "unreleased" ]]; then
106+
echo "==> FAIL: docs version must be an exact clean tag or unreleased" >&2
107+
return 1
108+
fi
109+
local fixture script result
110+
fixture="$(mktemp -d "${TMPDIR:-/tmp}/pineforge-seo.XXXXXX")"
111+
script="$(cd "$(dirname "$0")" && pwd)/seo_postprocess.py"
112+
cat > "$fixture/check_seo.py" <<'PY'
113+
from html.parser import HTMLParser
114+
from pathlib import Path
115+
import subprocess
116+
import sys
117+
import xml.etree.ElementTree as ET
118+
119+
script = Path(sys.argv[1])
120+
fixture = Path(sys.argv[2])
121+
site = fixture / "html"
122+
site.mkdir()
123+
old_meta = '<meta name="description" content="Old shared description"/>\n'
124+
old_canonical = '<link href="https://example.com/old.html" rel="canonical"/>\n'
125+
duplicate_canonical = '<link rel="canonical" href="https://cdocs.pineforge.dev/other"/>\n'
126+
titles = {
127+
"index.html": "PineForge: Landing",
128+
"x.html": "PineForge: Orders &amp; &quot;fills&quot; &lt;T&gt;",
129+
"other.html": "PineForge: Other reference",
130+
"404.html": "Page not found",
131+
}
132+
bodies = {
133+
"index.html": '''<a href="x.html">X</a>
134+
<a href="index.html">Home</a>
135+
<a href="index.html#start">Start</a>''',
136+
"x.html": '''<a href="other.html#a">Other</a>
137+
<a href="./other.html">Relative</a>
138+
<a title='href="other.html"' data-href="x.html" href="other.html">Tooltip</a>
139+
<a href="https://example.com/x.html">External</a>
140+
<a href="mailto:[email protected]">Mail</a>
141+
<a href="#frag">Fragment</a>
142+
<a href="other.html#https://example.com">Embedded URL</a>
143+
<a href="//example.com/x.html">Protocol relative</a>
144+
<a href="/other.html">Root path</a>
145+
<a href="../other.html">Parent path</a>
146+
<a href="search/help.html">Search path</a>
147+
<a href="icon.png">Image</a>
148+
<link href="style.css" rel="stylesheet"/>
149+
<script src="links.js"></script>
150+
<script>const text = 'href="other.html"';</script>
151+
<!-- href="other.html" -->''',
152+
"other.html": '<a href="x.html">X</a>',
153+
"404.html": '<p>Page missing.</p><a href="/">Home</a><a href="https://pineforge.dev">pineforge.dev</a>',
154+
}
155+
descriptions = {
156+
"index.html": (
157+
"PineForge&#x27;s C++17 engine has two layers documented in this API reference: "
158+
"a kernel that knows nothing about PineScript or TradingView and a Pine "
159+
"adapter that reproduces TradingView&#x27;s execution semantics."
160+
),
161+
"x.html": "Orders &amp; &quot;fills&quot; &lt;T&gt; — PineForge C++17 engine API documentation.",
162+
"other.html": "Other reference — PineForge C++17 engine API documentation.",
163+
}
164+
165+
def page(title, body, metadata):
166+
return (f"<!doctype html>\n<html><head>\n<title>{title}</title>\n"
167+
f"{metadata}</head>\n<body>\n{body}\n</body></html>\n")
168+
169+
for name, title in titles.items():
170+
metadata = "" if name == "404.html" else old_meta
171+
if name == "other.html":
172+
metadata += old_canonical + duplicate_canonical
173+
(site / name).write_bytes(page(title, bodies[name], metadata).encode("utf-8"))
174+
175+
untouched = {
176+
"links.js": b'const page = "foo.html"; const link = \'href="x.html"\';\r\n',
177+
"style.css": b'/* foo.html */\n',
178+
"search/searchdata.js": b'const page = "foo.html";\n',
179+
"search/help.html": b'<a href="x.html">Nested input stays untouched</a>\n',
180+
}
181+
for name, contents in untouched.items():
182+
(site / name).parent.mkdir(parents=True, exist_ok=True)
183+
(site / name).write_bytes(contents)
184+
185+
expected_bodies = dict(bodies)
186+
expected_bodies["index.html"] = (bodies["index.html"]
187+
.replace('href="x.html"', 'href="x"')
188+
.replace('href="index.html"', 'href="./"')
189+
.replace('href="index.html#start"', 'href="./#start"'))
190+
expected_bodies["x.html"] = (bodies["x.html"]
191+
.replace('<a href="other.html#a">', '<a href="other#a">')
192+
.replace('<a href="./other.html">', '<a href="./other">')
193+
.replace('data-href="x.html" href="other.html">', 'data-href="x.html" href="other">'))
194+
expected_bodies["other.html"] = '<a href="x">X</a>'
195+
196+
class Metadata(HTMLParser):
197+
def __init__(self):
198+
super().__init__(convert_charrefs=True)
199+
self.canonicals = []
200+
self.descriptions = []
201+
202+
def handle_starttag(self, tag, attrs):
203+
attrs = dict(attrs)
204+
if tag == "link" and "canonical" in attrs.get("rel", "").split():
205+
self.canonicals.append(attrs.get("href"))
206+
if tag == "meta" and attrs.get("name") == "description":
207+
self.descriptions.append(attrs.get("content"))
208+
209+
command = [sys.executable, "-I", "-B", str(script), str(site)]
210+
subprocess.run(command, check=True)
211+
parsed_descriptions = []
212+
canonical_urls = {}
213+
for name, title in titles.items():
214+
url = "https://cdocs.pineforge.dev/" + ("" if name == "index.html" else name[:-5])
215+
canonical_urls[name] = url
216+
canonical = f'<link rel="canonical" href="{url}"/>\n'
217+
actual = (site / name).read_bytes().decode("utf-8")
218+
if name == "404.html":
219+
# The error page is left exactly as written: no canonical, no description, no link rewrite.
220+
assert actual == page(title, bodies[name], ""), "404.html must be left untouched"
221+
continue
222+
meta = f'<meta name="description" content="{descriptions[name]}"/>\n'
223+
assert canonical + "</head>" in actual, f"{name}: missing canonical before </head>"
224+
assert actual == page(title, expected_bodies[name], meta + canonical), f"{name}: wrong HTML output"
225+
parsed = Metadata()
226+
parsed.feed(actual)
227+
assert parsed.canonicals == [url], f"{name}: canonical must occur exactly once"
228+
assert len(parsed.descriptions) == 1, f"{name}: description must occur exactly once"
229+
parsed_descriptions.extend(parsed.descriptions)
230+
assert len(set(parsed_descriptions)) == len(titles) - 1, "page descriptions must differ (404 has none)"
231+
assert '<a href="x">X</a>' in (site / "index.html").read_text(), "x.html was not rewritten"
232+
for name, contents in untouched.items():
233+
assert (site / name).read_bytes() == contents, f"{name}: asset or nested page changed"
234+
235+
namespace = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
236+
root = ET.parse(site / "sitemap.xml").getroot()
237+
assert root.tag == namespace + "urlset" and root.attrib == {}, "invalid sitemap urlset"
238+
urls = []
239+
for entry in root:
240+
assert entry.tag == namespace + "url" and entry.attrib == {}, "invalid sitemap entry"
241+
assert len(entry) == 1 and entry[0].tag == namespace + "loc", "sitemap must contain only locations"
242+
assert entry[0].attrib == {}, "sitemap location has unexpected attributes"
243+
urls.append(entry[0].text)
244+
expected_urls = sorted(canonical_urls[path.name] for path in site.glob("*.html") if path.name != "404.html")
245+
assert urls == expected_urls, "sitemap differs from the sorted top-level page set minus 404"
246+
247+
first_pass = {path.relative_to(site): path.read_bytes() for path in site.rglob("*") if path.is_file()}
248+
subprocess.run(command, check=True)
249+
second_pass = {path.relative_to(site): path.read_bytes() for path in site.rglob("*") if path.is_file()}
250+
assert second_pass == first_pass, "a second pass must be byte-identical, including canonical and sitemap"
251+
252+
empty = fixture / "empty"
253+
empty.mkdir()
254+
only_404 = fixture / "only-404"
255+
only_404.mkdir()
256+
(only_404 / "404.html").write_bytes((site / "404.html").read_bytes())
257+
for invalid_site in (empty, only_404, fixture / "missing"):
258+
result = subprocess.run(command[:-1] + [str(invalid_site)], capture_output=True)
259+
assert result.returncode != 0, "missing or empty documentation must fail"
260+
assert not (invalid_site / "sitemap.xml").exists(), "invalid input produced a sitemap"
261+
PY
262+
if python3 -I -B "$fixture/check_seo.py" "$script" "$fixture"; then
263+
result=0
264+
else
265+
result=$?
266+
fi
267+
rm -rf "$fixture"
268+
if (( result == 0 )); then
269+
echo "==> Docs SEO self-test: canonical, links, descriptions, sitemap, assets, version ... OK"
270+
fi
271+
return "$result"
272+
}
273+
92274
if [[ "${1:-}" == "--self-test-retry" && "$#" == 1 ]]; then
93275
self_test_doxygen_retry
94276
exit $?
95277
fi
278+
if [[ "${1:-}" == "--self-test-seo" && "$#" == 1 ]]; then
279+
self_test_seo
280+
exit $?
281+
fi
96282
if (( $# != 0 )); then
97-
echo "usage: docs/build.sh [--self-test-retry]" >&2
283+
echo "usage: docs/build.sh [--self-test-retry|--self-test-seo]" >&2
98284
exit 2
99285
fi
100286

@@ -120,11 +306,12 @@ if [[ ! -f "$THEME_DIR/.version" ]] \
120306
echo "$DOXYGEN_AWESOME_VERSION" > "$THEME_DIR/.version"
121307
fi
122308

123-
# 2. Resolve project version from VERSION file (or git tag if available).
124-
VERSION="$(cat ../VERSION 2>/dev/null || echo unknown)"
125-
if command -v git >/dev/null 2>&1 && git -C .. rev-parse --git-dir >/dev/null 2>&1; then
126-
GIT_VERSION="$(git -C .. describe --tags --dirty --always 2>/dev/null || true)"
127-
[[ -n "$GIT_VERSION" ]] && VERSION="$GIT_VERSION"
309+
# 2. Label only a clean, exactly tagged checkout as a release.
310+
VERSION="unreleased"
311+
if command -v git >/dev/null 2>&1 \
312+
&& GIT_VERSION="$(git -C .. describe --tags --exact-match 2>/dev/null)" \
313+
&& GIT_STATUS="$(git -C .. status --porcelain 2>/dev/null)"; then
314+
VERSION="$(docs_version_label "$GIT_VERSION" "$GIT_STATUS")"
128315
fi
129316
echo "==> Building docs for PineForge $VERSION"
130317

@@ -136,6 +323,7 @@ CONFIG="$(mktemp "${TMPDIR:-/tmp}/pineforge-doxyfile.XXXXXX")"
136323
trap 'rm -f "$CONFIG"' EXIT
137324
{ cat Doxyfile; echo "PROJECT_NUMBER = $VERSION"; } > "$CONFIG"
138325
run_doxygen_with_retry
326+
python3 -I -B seo_postprocess.py site/html
139327

140328
# 3b. The scoped warning gate (R5 lane L14-A).
141329
#

‎docs/ci.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -787,14 +787,16 @@ The first job of `ci.yml`, `changes`, decides with
787787
documentation when it is a Markdown file (`*.md`) or lies under `docs/`,
788788
unless it lies under `.github/`, `benchmarks/`, `scripts/` or `tests/`, is a
789789
CMake file or a git control file such as `.gitattributes`, or is one of the
790-
four files a proof job reads:
790+
five files a proof job reads:
791791
`docs/pages/pine-to-native.md` and `docs/pages/native-engine.md`, whose code
792792
blocks `test_pine_to_native_worked` and `test_native_engine_complete_host`
793793
compile and run; `docs/adr/0001-kernel-adapter-boundary.md`, which
794794
`test_kernel_residuals`, `test_native_feature_rulings` and the kernel
795-
profile's `kernel-residuals` stage hold the kernel against; and
796-
`docs/build.sh`, whose retry self-test is `test_docs_doxygen_retry`. A change
797-
to one of those four runs every job. A change is documentation only when every
795+
profile's `kernel-residuals` stage hold the kernel against;
796+
`docs/build.sh`, whose self-tests are `test_docs_doxygen_retry` and
797+
`test_docs_seo`; and `docs/seo_postprocess.py`, which `test_docs_seo` executes
798+
on synthetic pages. A change
799+
to one of those five runs every job. A change is documentation only when every
798800
path it changes, both sides of a rename, is documentation. The change is the
799801
checked-out commit against its first parent: for a pull request, the merge
800802
GitHub tests against the base it merged into; for a push, the pushed commit
@@ -831,7 +833,7 @@ full run on `main` holds the code's verdict.
831833
holds the rule against real git histories, pull request merges and pushes
832834
alike. It also fails when a CMake file names, through `${PROJECT_SOURCE_DIR}`
833835
or a sibling source-directory variable, a path the rule would call
834-
documentation. The four files above were found by reading every CMake file,
836+
documentation. The five files above were found by reading every CMake file,
835837
every `ci_verify.py` stage and every script a CTest row runs; a reader added
836838
outside CMake must list its file in `PROOF_READS` by hand. The
837839
`ci-workflow-contract` stage pins the wiring, the classification step

0 commit comments

Comments
 (0)