Skip to content

fix: recognize mermaid fences with info strings or trailing space - #240

Merged
luongnv89 merged 3 commits into
luongnv89:mainfrom
kevin9327:fix-mermaid-info-string-fences
Nov 12, 2026
Merged

luongnv89 merged 3 commits into
luongnv89:mainfrom
kevin9327:fix-mermaid-info-string-fences

Conversation

@kevin9327

Copy link
Copy Markdown
Contributor

Summary

The website builder, EPUB builder, and Pele mermaid validator only matched a fence that was exactly \``mermaidfollowed by a newline. CommonMark allows a trailing space on the info string, and authors often write```mermaid title=Overview`. Those diagrams were skipped, so they were not rendered and were not syntax-checked.

The fence regex now allows optional whitespace after the backticks and any remainder of the info-string line.

Tests

  • test_extracts_fences_with_info_string_or_trailing_space in test_check_mermaid.py
  • test_replaces_fence_with_info_string_or_trailing_space in test_build_website.py
  • test_extract_mermaid_blocks_with_info_string in test_build_epub.py

The website, EPUB, and Pele validator only matched a fence that was
exactly ```mermaid followed by a newline. CommonMark allows a trailing
space or an info string (```mermaid title=Overview), so those diagrams
were skipped and never rendered or checked.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T05:52:40.913952Z 2fa6e95 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2fa6e95618

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/check_mermaid.py Outdated

IGNORE_DIRS = {".venv", "node_modules", ".git", "blog-posts", ".agents"}
MERMAID_BLOCK_RE = re.compile(r"```mermaid\n(.*?)```", re.DOTALL)
MERMAID_BLOCK_RE = re.compile(r"```[ \t]*mermaid[^\n]*\n(.*?)```", re.DOTALL)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Anchor Mermaid matches to actual fence lines

The unanchored regex also matches the inline fence-syntax example in CONTRIBUTING.md:108, then consumes the following hook table and headings up to that file's standalone fence at line 114. extract_blocks() therefore sends non-Mermaid Markdown to Pele; the docs-check.yml pull-request workflow runs python scripts/check_mermaid.py --strict when scripts/check_*.py changes, so this existing false block can fail the check. Restrict matches to actual fence-opening lines.

Useful? React with 👍 / 👎.

The unanchored pattern let [^\n]* swallow backticks, so the inline code
span ` ```mermaid ` in the CONTRIBUTING.md hook tables was picked up
as a diagram opener and rejected by check_mermaid. Anchor the fence to
line start (optional indent) and only accept whitespace-separated,
backtick-free info strings, which also stops ```mermaidjs matching.
Repo-wide match count is back to 377.
@luongnv89
luongnv89 merged commit 7013577 into luongnv89:main Nov 12, 2026
20 checks passed
@luongnv89

Copy link
Copy Markdown
Owner

Thanks @kevin9327, nice catch! 🙌 Fences like ```mermaid title=Overview really were being skipped by the validator and both builders. I pushed one follow-up before merging: the pattern is now anchored to the start of a line and only accepts a space-separated info string. Without that, the inline ```mermaid in the CONTRIBUTING.md hook table was being read as a diagram, and ```mermaidjs matched too. I also added a regression test for each case. The block count across the repo is the same as before, and all 418 diagrams render with Pele. Merged!

@luongnv89

Copy link
Copy Markdown
Owner

Thanks @kevin9327, this is a nice robustness fix, and it's now merged! 🙏

CommonMark (§4.5) allows an info string and trailing spaces after the language. Before this, a fence like ```mermaid title=Overview was silently skipped by both rendering and the strict Pele check. CI was green across tests, lint and all six EPUB builds. The new regex matches exactly the same fences in the repo as before, so nothing that passed before started failing.

Two small follow-ups came out of the review. We'll track them in a separate issue, so there's nothing more you need to do:

  1. Preserve indentation in the replacements. ^[ \t]* is now part of the match, so for a fence nested in a list, replace_mermaid_blocks (scripts/build_website.py) and process_mermaid_blocks (scripts/build_epub.py) emit their output at column 0, which can end the list item. No current content is affected.
  2. Test process_mermaid_blocks with an info-string fence (scripts/tests/test_build_epub.py). Extraction and replacement need to match the same blocks.

Thanks again!

luongnv89 added a commit that referenced this pull request Nov 12, 2026
The fence regex in `build_website.py`, `build_epub.py` and `check_mermaid.py` matched the leading whitespace as part of the fence but did not keep it. `replace_mermaid_blocks` and `process_mermaid_blocks` then wrote the diagram at column 0, so a fence nested in a list item ended the item.

## What changed
- The leading indentation is captured as a named `indent` group and the diagram source as `code`. Both builds write `indent` again before `<pre class="mermaid">` (website) and the `![Diagram](…)` image (EPUB).
- `build_epub.py` used to repeat the pattern as a literal in two functions. It now compiles it once as `MERMAID_BLOCK_RE`, like the other two scripts. A comment above each copy points to the other two.
- `check_mermaid.py` reads `group("code")`. Line numbers are unchanged because the match still starts at the beginning of the fence line.
- **Website, 4+ columns:** at this indent Python-Markdown no longer reads the `<pre>` as raw HTML. A blank line in the diagram would end the block, and `*`, `_`, `` ` ``, `[…]` and `\` would be formatted as Markdown before the page script reads `pre.textContent`. At 4+ columns only, tabs, newlines and those characters are now written as numeric character references. These decode back to the original characters in the parsed HTML and in `textContent`, so Pele gets the exact fence source. A block at 0–3 columns is raw HTML and is written exactly as before.
- **Website:** the `<p>` that Python-Markdown wraps around a nested mermaid `<pre>` is unwrapped, because `<p><pre>` is invalid HTML. Only a `<p>` whose sole content is that `<pre>` is unwrapped.
- **`check_mermaid.py`** prints a non-fatal `⚠` warning for a fence indented 4+ columns (a tab advances to the next multiple of 4). Such a fence renders as a diagram only inside a list item, and as an indented code block anywhere else. The warning does not change the exit code or `--strict` failures.

## Deviation from the issue: indentation is `[ \t]*`, not 0–3 spaces
The issue suggested CommonMark's 0–3 spaces. Both builds render with **Python-Markdown**, which continues a list item only at **four** spaces. I checked rendered output for both limits:

| Nested fence | `{0,3}` group | `[ \t]*` group (this PR) |
|---|---|---|
| 2–3 spaces, blank line before the fence | Diagram renders outside the list, same as main | Same as main (Python-Markdown limit) |
| 1–3 spaces, directly under the item text | List ends; diagram at top level | Website keeps the diagram inside the `<li>`, within the item's paragraph; the EPUB ends the list as on main. Diagram text is exact in both |
| 4 spaces or a tab, in a list item | **Not matched**: becomes inline code, and `check_mermaid` stops validating it | Diagram inside the `<li>` on the website and in the EPUB. The website `<pre>` text is byte-identical to the fence source, including blank lines and Markdown characters |
| 4+ columns, not in a list | Not matched | Indented code block on the website and in the EPUB (as on GitHub). `check_mermaid` validates it and prints a warning |

A 0–3 limit would bring no rendering improvement, and it would be a regression for 4-space fences, which would stop matching. Keeping `[ \t]*` leaves the set of matched fences exactly as it is on main. If you still want strict CommonMark, it is a one-token change in three places, and the parity test keeps the copies in step.

## Tests
- `process_mermaid_blocks` with an info-string fence. Replacement has to match the same fences extraction did, or the build raises `MermaidRenderError`.
- A nested fence keeps its indentation: website `replace_mermaid_blocks` (3 spaces) and EPUB `process_mermaid_blocks` (3 and 4 spaces).
- A 4-space nested diagram renders inside its `<li>`: website `render_markdown` and EPUB `md_to_html`.
- Website `pre.textContent` equals the fence source at indent 0, 3 spaces, 4 spaces and a tab, for a diagram with a blank line, `` *x* _y_ `z` ``, `[l](u)` and `\\`. No `<em>`, `<code>` or `<a>` appears inside the `<pre>`.
- A top-level diagram renders to exactly the HTML main produces. Below 4 columns the `<pre>` is not entity-encoded.
- A nested mermaid `<pre>` is a direct child of its `<li>`, with no `<p>` around it.
- `check_mermaid` warns for 4 spaces, a tab, 2 spaces plus a tab, and 6 spaces, and not for 0, 1 or 3 spaces. The warning leaves a Pele rejection and a strict "Pele unavailable" failure as failures.
- ```` ``` mermaid ```` (a space before the language) is matched: website, EPUB and `check_mermaid`.
- `mermaid-foo` is ignored by the website and EPUB builds (`check_mermaid` already had this test).
- `check_mermaid` reports the right line for 3- and 4-space nested fences.
- The three `MERMAID_BLOCK_RE` patterns and flags are identical.

All 234 tests pass.

## No change to current output
- Using `git ls-files '*.md'` (679 files) with each module's regex, the count is **416** on main and **416** for each of the three patterns. Every match has the same span and source as on main. All 416 fences are at column 0, so `check_mermaid` prints no indentation warning.
- `render_markdown` gives byte-identical HTML to main's for all 679 tracked pages.
- `check_mermaid.py --strict` (Pele 1.6.0) renders all 416 blocks.

## Gates
`pytest scripts/tests/` (234 passed), `ruff check`, `ruff format --check`, `mypy` and `bandit` all pass.

Closes #244

Follow-up to #240 by @kevin9327
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants