feat(core): expose Markdown tokenizer and inline renderer hooks #1464

Activity

1. simonklee commented on Sep 4, 2026

   

   simonklee

   on Sep 4, 2026

   Member

   More actions

   I just ran into this the other day myself. started prototyping something.
2. FasterPHP commented on Oct 11, 2026

   

   FasterPHP

   on Oct 11, 2026

   More actions

   Version and environment: @opentui/core 0.5.12 (unchanged on 0.5.17 and main), Bun 1.4.0.

   A second consumer for this, with different extensions: we are building an Agent Client Protocol (ACP) terminal client that renders agent Markdown with MarkdownRenderable, and it needs footnotes ([^1] and [^1]: ...), maths ($a_1$, \(...\), $$...$$) and GitHub alerts (> [!NOTE]). Maths is the case that cannot be done after lexing: marked reads $a_1$ and $b_2$ as emphasis and \( as an escape, so by the time renderNode runs the information is gone. We currently work around it with a pre-pass over the source that swaps maths spans and footnote references for private-use placeholders outside code, which is fragile (it must escape any private-use characters in the agent's text, track fences itself, and cope with streaming).

   Minimal reproduction (markdown-extension-hook.ts, about 55 lines): a block extension registered with marked.use() yields a callout token when marked is used directly, but never reaches MarkdownRenderable, whether through renderNode or at all:

   marked with the extension : paragraph, callout, paragraph
   MarkdownRenderable.renderNode: paragraph, paragraph, paragraph
   
   rendered frame:
   Before
   :::note
   A callout that a marked extension should recognise.
   :::
   After
   

   Why marked.use() cannot work (packages/core/src at v0.5.12): renderables/markdown-parser.ts lines 20, 65 and 73 call Lexer.lex(src, { gfm: true }) with its own options object, and marked uses a passed options object instead of the defaults that marked.use() changes (the point made in PR #1345's description). In the published bundle marked is also inlined (index.bun.js:9453, [email protected]), so the application's own marked is a different instance anyway. MarkdownOptions (renderables/Markdown.ts:87 to 126) has renderNode for top-level blocks only and no tokenizer option.

   What we would need, in line with #1464's list: a per-renderable extensions option (marked's MarkedExtension[] shape: extensions with level, start and tokenizer, applied to the same Lexer.lex calls so the incremental parser and the unstable trailing block keep working), and a way to render an inline custom token (today renderInlineToken is private, Markdown.ts:575, and renderNode is block level). Block tokens could go through renderNode unchanged. Scoped to the renderable, not a global marked.use().

   If a prototype exists, we would gladly test it against a footnotes and maths extension and our streaming test corpus.

   Reproduction: markdown-extension-hook.ts

   // Repro for report 5: MarkdownRenderable has no way to use marked tokenizer extensions.
   // Run from the repository root: bun docs/mvp-notes/opentui-repro/markdown-extension-hook.ts
   //
   // The script registers a small block extension with the `marked` package (the same major
   // version OpenTUI depends on, 17.0.1), then renders the same source through
   //   (a) marked directly: the extension runs and yields a `callout` token;
   //   (b) OpenTUI's MarkdownRenderable: `renderNode` is the only hook, and it never sees a
   //       `callout` token, because OpenTUI lexes with `Lexer.lex(src, { gfm: true })` on its own
   //       options object (renderables/markdown-parser.ts) and (in the published bundle) with its
   //       own inlined copy of marked, so nothing registered by the application reaches it.
   import { MarkdownRenderable, SyntaxStyle, type Renderable } from "@opentui/core";
   import { createTestRenderer } from "@opentui/core/testing";
   import { Lexer, marked, type Token } from "marked";
   
   const callout = {
     name: "callout",
     level: "block" as const,
     start: (src: string) => src.match(/^:::/m)?.index,
     tokenizer(src: string) {
       const match = /^:::(\w+)\n([\s\S]*?)\n:::(?:\n|$)/.exec(src);
       if (!match) return undefined;
       return { type: "callout", raw: match[0], kind: match[1], text: match[2], tokens: [] };
     },
   };
   marked.use({ extensions: [callout] });
   
   const source = "Before\n\n:::note\nA callout that a marked extension should recognise.\n:::\n\nAfter\n";
   
   // (a) marked on its own
   const direct = Lexer.lex(source, { ...marked.defaults, gfm: true }) as Token[];
   console.log("marked with the extension :", direct.filter((t) => t.type !== "space").map((t) => t.type).join(", "));
   
   // (b) MarkdownRenderable, observing every token that reaches renderNode
   const { renderer, renderOnce, captureCharFrame } = await createTestRenderer({ width: 70, height: 12 });
   const seen: string[] = [];
   const syntaxStyle = SyntaxStyle.fromStyles({ default: { fg: "#ffffff" } });
   const markdown = new MarkdownRenderable(renderer, {
     content: source,
     syntaxStyle,
     internalBlockMode: "top-level",
     renderNode: (token): Renderable | undefined => {
       seen.push(token.type);
       return undefined; // keep the default rendering
     },
   });
   renderer.root.add(markdown);
   await renderOnce();
   await Bun.sleep(100);
   await renderOnce();
   
   console.log("MarkdownRenderable.renderNode:", seen.filter((t) => t !== "space").join(", "));
   console.log("\nrendered frame:\n" + captureCharFrame().split("\n").filter((l) => l.trim()).join("\n"));
   console.log(seen.includes("callout") ? "\nOK: the extension reached OpenTUI." : "\nPROBLEM: the `callout` token never reached renderNode; the source is drawn as paragraph text.");
   renderer.destroy();
3. viralcodex commented on Oct 23, 2026

   

   viralcodex

   on Oct 23, 2026

   Contributor

   More actions

   @simonklee is this still relevant or can be closed?
4. simonklee

   added 2 commits that reference this issue on Nov 11, 2026

   core: let a Markdown hook rewrite prose spans (#2023)

   ...

   8bb0c6f

   rust: parse Markdown math with an extensions option (#2072)

   ...

   68dbd5d