Repository navigation
fix: recognize mermaid fences with info strings or trailing space - #240
Conversation
The website, EPUB, and Pele validator only matched a fence that was exactly ```mermaid followed by a newline. CommonMark allows a trailing space or an info string (```mermaid title=Overview), so those diagrams were skipped and never rendered or checked.
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2fa6e95618
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| IGNORE_DIRS = {".venv", "node_modules", ".git", "blog-posts", ".agents"} | ||
| MERMAID_BLOCK_RE = re.compile(r"```mermaid\n(.*?)```", re.DOTALL) | ||
| MERMAID_BLOCK_RE = re.compile(r"```[ \t]*mermaid[^\n]*\n(.*?)```", re.DOTALL) |
There was a problem hiding this comment.
Anchor Mermaid matches to actual fence lines
The unanchored regex also matches the inline fence-syntax example in CONTRIBUTING.md:108, then consumes the following hook table and headings up to that file's standalone fence at line 114. extract_blocks() therefore sends non-Mermaid Markdown to Pele; the docs-check.yml pull-request workflow runs python scripts/check_mermaid.py --strict when scripts/check_*.py changes, so this existing false block can fail the check. Restrict matches to actual fence-opening lines.
Useful? React with 👍 / 👎.
The unanchored pattern let [^\n]* swallow backticks, so the inline code span ` ```mermaid ` in the CONTRIBUTING.md hook tables was picked up as a diagram opener and rejected by check_mermaid. Anchor the fence to line start (optional indent) and only accept whitespace-separated, backtick-free info strings, which also stops ```mermaidjs matching. Repo-wide match count is back to 377.
|
Thanks @kevin9327, nice catch! 🙌 Fences like |
|
Thanks @kevin9327, this is a nice robustness fix, and it's now merged! 🙏 CommonMark (§4.5) allows an info string and trailing spaces after the language. Before this, a fence like Two small follow-ups came out of the review. We'll track them in a separate issue, so there's nothing more you need to do:
Thanks again! |
The fence regex in `build_website.py`, `build_epub.py` and `check_mermaid.py` matched the leading whitespace as part of the fence but did not keep it. `replace_mermaid_blocks` and `process_mermaid_blocks` then wrote the diagram at column 0, so a fence nested in a list item ended the item.
## What changed
- The leading indentation is captured as a named `indent` group and the diagram source as `code`. Both builds write `indent` again before `<pre class="mermaid">` (website) and the `` image (EPUB).
- `build_epub.py` used to repeat the pattern as a literal in two functions. It now compiles it once as `MERMAID_BLOCK_RE`, like the other two scripts. A comment above each copy points to the other two.
- `check_mermaid.py` reads `group("code")`. Line numbers are unchanged because the match still starts at the beginning of the fence line.
- **Website, 4+ columns:** at this indent Python-Markdown no longer reads the `<pre>` as raw HTML. A blank line in the diagram would end the block, and `*`, `_`, `` ` ``, `[…]` and `\` would be formatted as Markdown before the page script reads `pre.textContent`. At 4+ columns only, tabs, newlines and those characters are now written as numeric character references. These decode back to the original characters in the parsed HTML and in `textContent`, so Pele gets the exact fence source. A block at 0–3 columns is raw HTML and is written exactly as before.
- **Website:** the `<p>` that Python-Markdown wraps around a nested mermaid `<pre>` is unwrapped, because `<p><pre>` is invalid HTML. Only a `<p>` whose sole content is that `<pre>` is unwrapped.
- **`check_mermaid.py`** prints a non-fatal `⚠` warning for a fence indented 4+ columns (a tab advances to the next multiple of 4). Such a fence renders as a diagram only inside a list item, and as an indented code block anywhere else. The warning does not change the exit code or `--strict` failures.
## Deviation from the issue: indentation is `[ \t]*`, not 0–3 spaces
The issue suggested CommonMark's 0–3 spaces. Both builds render with **Python-Markdown**, which continues a list item only at **four** spaces. I checked rendered output for both limits:
| Nested fence | `{0,3}` group | `[ \t]*` group (this PR) |
|---|---|---|
| 2–3 spaces, blank line before the fence | Diagram renders outside the list, same as main | Same as main (Python-Markdown limit) |
| 1–3 spaces, directly under the item text | List ends; diagram at top level | Website keeps the diagram inside the `<li>`, within the item's paragraph; the EPUB ends the list as on main. Diagram text is exact in both |
| 4 spaces or a tab, in a list item | **Not matched**: becomes inline code, and `check_mermaid` stops validating it | Diagram inside the `<li>` on the website and in the EPUB. The website `<pre>` text is byte-identical to the fence source, including blank lines and Markdown characters |
| 4+ columns, not in a list | Not matched | Indented code block on the website and in the EPUB (as on GitHub). `check_mermaid` validates it and prints a warning |
A 0–3 limit would bring no rendering improvement, and it would be a regression for 4-space fences, which would stop matching. Keeping `[ \t]*` leaves the set of matched fences exactly as it is on main. If you still want strict CommonMark, it is a one-token change in three places, and the parity test keeps the copies in step.
## Tests
- `process_mermaid_blocks` with an info-string fence. Replacement has to match the same fences extraction did, or the build raises `MermaidRenderError`.
- A nested fence keeps its indentation: website `replace_mermaid_blocks` (3 spaces) and EPUB `process_mermaid_blocks` (3 and 4 spaces).
- A 4-space nested diagram renders inside its `<li>`: website `render_markdown` and EPUB `md_to_html`.
- Website `pre.textContent` equals the fence source at indent 0, 3 spaces, 4 spaces and a tab, for a diagram with a blank line, `` *x* _y_ `z` ``, `[l](u)` and `\\`. No `<em>`, `<code>` or `<a>` appears inside the `<pre>`.
- A top-level diagram renders to exactly the HTML main produces. Below 4 columns the `<pre>` is not entity-encoded.
- A nested mermaid `<pre>` is a direct child of its `<li>`, with no `<p>` around it.
- `check_mermaid` warns for 4 spaces, a tab, 2 spaces plus a tab, and 6 spaces, and not for 0, 1 or 3 spaces. The warning leaves a Pele rejection and a strict "Pele unavailable" failure as failures.
- ```` ``` mermaid ```` (a space before the language) is matched: website, EPUB and `check_mermaid`.
- `mermaid-foo` is ignored by the website and EPUB builds (`check_mermaid` already had this test).
- `check_mermaid` reports the right line for 3- and 4-space nested fences.
- The three `MERMAID_BLOCK_RE` patterns and flags are identical.
All 234 tests pass.
## No change to current output
- Using `git ls-files '*.md'` (679 files) with each module's regex, the count is **416** on main and **416** for each of the three patterns. Every match has the same span and source as on main. All 416 fences are at column 0, so `check_mermaid` prints no indentation warning.
- `render_markdown` gives byte-identical HTML to main's for all 679 tracked pages.
- `check_mermaid.py --strict` (Pele 1.6.0) renders all 416 blocks.
## Gates
`pytest scripts/tests/` (234 passed), `ruff check`, `ruff format --check`, `mypy` and `bandit` all pass.
Closes #244
Follow-up to #240 by @kevin9327
Summary
The website builder, EPUB builder, and Pele mermaid validator only matched a fence that was exactly
\``mermaidfollowed by a newline. CommonMark allows a trailing space on the info string, and authors often write```mermaid title=Overview`. Those diagrams were skipped, so they were not rendered and were not syntax-checked.The fence regex now allows optional whitespace after the backticks and any remainder of the info-string line.
Tests
test_extracts_fences_with_info_string_or_trailing_spaceintest_check_mermaid.pytest_replaces_fence_with_info_string_or_trailing_spaceintest_build_website.pytest_extract_mermaid_blocks_with_info_stringintest_build_epub.py