Skip to content

✨ Install structural Markdown syntax from a package (#806) - #807

Closed
taras wants to merge 2 commits into
mainfrom
agent/issue-806-structural-expansion
Closed

taras wants to merge 2 commits into
mainfrom
agent/issue-806-structural-expansion

Conversation

@taras

@taras taras commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Why

Adding a structural Markdown form meant editing core: its reserved-name table,
its structural rules, document validation, the syntax catalog and expand.ts
dispatch. That couples the engine to every extension. The terminal-grid stack is
the worked example — a provider-neutral terminal package still needed core to
recognize and execute terminal syntax.

Closes #806.

What changes

A trusted host can now install a coordinated structural parent and child form,
and the installation that declares them supplies the implementation that expands
them.

Before:

<Terminal.Grid> and <Terminal> were engine syntax. Core reserved both names,
carried their grammar, their placement rules and their layout, and dispatched
them from expand.ts. Every execution had them, and no package could add
anything like them.

After:

ExecutionInstallation.declarations is one discriminated catalog:
kind: "markdown" is the existing declared-Markdown tier, and
kind: "structural" is an installed form. @executablemd/terminal/xmd declares
the grid and its panes and expands them; core contains no terminal name, rule or
dispatch branch. An ordinary run installs the package on every runtime, and the
xmd test root and workflow executions install none — where the names then
resolve to nothing at all.

How it works

ExecutionInstallation.declarations
  → capture (copied, frozen, owner kept)
  → intrinsic catalog validation      (before any install() hook)
  → admission                          (schemas compiled, pairing checked)
  → selection / xmd syntax / xmd plan / document validation
  → one occurrence: source placement → props → freeze → Execution.expand
  → captured owner's expand(request, regions)
  → region.expand() → backpressured {text, exact} stream

Core decides that an occurrence is wrong; the declaration decides only how the
sentence reads, through inert templates with three substitutions.

Review guide

Start with: packages/core/src/execution-declarations.ts

Then review:

  1. packages/core/src/expansion-request.ts — the public request, and the
    canonical issuance that makes a forged, stale, sibling or twice-delegated
    request a protocol refusal.
  2. packages/core/src/expansion-region.ts — the per-chunk rendezvous.
  3. packages/core/src/expand.ts — expandInstalledStructural and
    expandRegionScoped, replacing the terminal dispatch.
  4. packages/core/src/execute.ts — capture, the intrinsic pass before hooks,
    and the prepared-execution conflicts.
  5. packages/terminal/ — the neutral root and the /xmd integration.
  6. packages/cli/src/cli.ts — the explicit "run" | "test" | "workflow"
    profile and the installation factory threaded to all four entrypoints.

Look carefully at:

  • The ordering of the seven steps in expandInstalledStructural. A failure
    anywhere before the last publishes no request, invokes no middleware, reaches
    no region producer and starts nothing the installing package owns.
  • regionStream teardown: the producer is spawned inside the resource, so
    leaving the handler's scope halts and joins it.

What must stay true

  • One installation owns a structural form and its implementation. Enforced
    by admitStructuralDeclarations; checked by the
    structural-without-expand / expand-without-structural rows in
    execution-declarations.test.ts.
  • The catalog is captured by value. Enforced by detachStructural +
    freezeThrough; checked by the mutation-during-install() rows.
  • Public middleware cannot expand a child or substitute an implementation.
    Enforced by issueExpansion; checked by the EX1 rows in
    structural-expansion.test.ts.
  • A region's producer never outlives its consumer. Enforced by the resource
    in regionStream; checked by the four RS3 halt-and-join latches.
  • Core stays generic. Enforced by nothing in code, so it is checked:
    xmd-package-boundary.test.ts walks core's module graph and the rg scans in
    the verification section.

How to verify it

  • RS2: the producer is suspended on the chunk the consumer holds proves the
    rendezvous and fails if delivery stops waiting for acknowledgement.
  • The four RS3: … halts and joins … rows prove teardown and all four fail if
    the producer is spawned outside the resource. The
    RS3: the control — work the region does not own row exists so their absence
    assertions are known to be capable of failing.
  • DC1: engine constructs come first, then installed forms in capture order
    uses a Mid, Zeta, Alpha fixture, so capture order, alphabetical order and
    parents-first order are three different lists; it fails against either wrong
    ordering.
  • TG1/TG2/TG4 in packages/terminal/tests/terminal-grid-structure.test.ts
    are the pre-existing core suite, moved and driven through the installation —
    the grammar, placement, one-to-five row-major layouts, duplicate-title
    ordinals and the { layout } cause are unchanged.
  • PF1/PF2 in packages/cli/tests/structural-profile.test.ts run under Deno,
    Node and Bun and prove a run has the syntax while a profile that installs none
    reports Cannot resolve component: Terminal.Grid.

Static boundary evidence, run as part of the focused set:

! rg -n '@executablemd/(terminal|grid)|Terminal\.Grid|"(Terminal|Grid|Pane)"' packages/core/src
! rg -n '@executablemd/core|/xmd' packages/terminal/mod.ts packages/terminal/src/layout.ts
rg -n '@executablemd/core' packages/terminal/xmd.ts packages/terminal/src/xmd.ts

Scope

Included

  • The discriminated declaration catalog, its capture, and its admission.
  • Public Execution.expand and private owner dispatch.
  • Resource-scoped, backpressured region streams.
  • Extraction of the current providerless terminal-grid behavior to
    @executablemd/terminal and its /xmd subpath.
  • Explicit "run" | "test" | "workflow" profile assembly across all four
    runtime entrypoints, xmd syntax, xmd plan and the nested-run host.
  • architecture.md, specs/executable-mdx-spec.md (new §6.1.1) and
    specs/release-process-spec.md.

Intentionally unchanged

  • The terminal grid's live lifecycle, retained records, replay, provider
    failure, reader close, cancellation and teardown. Those are Execute concurrent terminal panes through a replaceable provider #730's, and this
    PR routes only the behavior already on main through the new boundary.
  • Ordinary function-component registration, and the declared-Markdown tier's
    digest, schema, form, private-closure and selection semantics.
  • Core's own control-flow constructs, which remain protected engine syntax an
    installation cannot claim.
  • No general plugin interface, family, group or registrar; no AST or macro
    system; no contextual region-output routing; no duplicate-use enforcement.

New abstractions

  • ExecutionDeclaration / StructuralDeclaration exist because a profile must
    be able to describe a form to xmd syntax, xmd plan and validation without
    running it. Consumers: core selection, inspection, validation, execution, and
    @executablemd/terminal/xmd.
  • ExpansionRequest / ExpansionRegion exist because a structural parent needs
    its children as things it expands, which no function component can receive.
  • regionStream exists because the alternative — a channel — does not wait for
    its consumer, which is the one property a pane needs.
  • Each new abstraction has multiple concrete uses or a clear justification.
  • No speculative functionality is included.

New dependencies

  • Package: @executablemd/terminal (new workspace member, 0.12.1)
  • Used for: the provider-neutral grid layout and its executable-Markdown
    integration.
  • Why existing dependencies are insufficient: it is the extraction itself — core
    must not depend on it, and the CLI must.

Generated or mechanical changes

  • .github/workflows/publish-packages.yml comes from
    deno task gen:publish-workflow; it adds the terminal job and puts it
    between core and cli.
  • deno.lock, pnpm-lock.yaml and bun.lock come from
    deno install --frozen=false, deno task setup and bun install --lockfile-only for the new workspace member.
  • packages/terminal/src/layout.ts and
    packages/terminal/tests/terminal-grid-structure.test.ts are moves from
    packages/core (git records them as renames at 75% and 61%); the test's
    harness is adapted to drive the installation, and its assertions are unchanged.

Risks and limitations

  • @executablemd/terminal is not on npm — the name currently returns 404.
    The documented bootstrap (components/BootstrapNpmPackage.md) must be
    completed before a tagged release publishes it or a CLI that depends on it.
    This PR mutates no registry.
  • Delivery verification has not run. deno task lint, check, check:jsr,
    the compiled build with
    scripts/tests/structural-profile-compiled.test.ts (CP1, never executed),
    verify:clean, and test-weight measurement are all outstanding. The corpus
    and the dependency graph both moved, so Measure Test Weights is required
    on the final head before merge.
  • Stack position. This is the layer beneath the remaining Quest: Add interactive grids to executable documents #717 grid stories:
    main → #806 → rebased #738/#730 → #731 → #732 → #781. It is based on main
    and depends on none of them.

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • Generated or mechanical changes are clearly identified.
  • The description matches the final diff and test results.

An `ExecutionInstallation` may now declare a coordinated structural parent and
child form and supply the implementation that expands them. `declarations`
becomes one discriminated catalog carrying that arm beside the existing declared
Markdown tier, and canonical core captures and validates the whole catalog
before any installation hook, middleware or document code runs — so selection,
`xmd syntax`, `xmd plan`, document validation and execution all read the same
definitions and cannot describe different environments.

`Execution.expand` is the public policy surface for one occurrence. A handler
receives the authored form, the source position and the evaluated, frozen props,
and may observe, refuse or delegate; it receives no child regions and no
implementation, so only canonical core hands an occurrence to the installation
that declared it. Each accepted child is a region whose `expand()` returns a
resource-scoped stream of `{ text, exact }` chunks with per-chunk rendezvous
backpressure: the producer does not advance past a delivered chunk until the
consumer advances, region output never travels through `DocumentOutput`, and
leaving the handler's scope halts and joins the producer.

The terminal grid becomes the first consumer. Its syntax, validation, layout and
providerless refusal move to the new `@executablemd/terminal` package — a
provider-neutral root that loads no engine, and an `/xmd` subpath that depends on
core — and core keeps no terminal name, rule or dispatch branch. An ordinary run
installs it on every runtime; the `xmd test` root and workflow executions do not.
Review feedback on 8c2b9b1, without changing the settled architecture.

The capture now reads the discriminant rather than trusting it: an entry whose
kind is neither "markdown" nor "structural" refuses the catalog instead of being
treated as the other arm. Structural entries are copied at capture and frozen
through — `structuredClone` detached the schema but left every nested object
writable — and the admitted copies are what selection, execution, `<Syntax>`,
inspection and validation all read, so a host that mutates what it handed over
from inside its own `install()` changes nothing.

Whole-catalog validation that depends on nothing but the declarations moved
ahead of the first `install()`: an unknown discriminant, a malformed Markdown
entry, and a name claimed by both arms or by a private closure. The conflicts
that genuinely need a prepared execution — a reserved registration, a workflow
bundle — stay at that boundary. Declared Markdown keeps its own admission, so
its digest, schema, form and selection semantics are unchanged.

Syntax ordering is restored to the frozen contract: protected engine constructs
first, then installed declarations in captured installation and declaration
order. Admission emits in declaration order rather than grouping parents ahead
of their children.

New regressions: mutation during `install()` reaching neither execution,
`<Syntax>`, nor the captured catalog; an unknown discriminant and an intrinsic
cross-arm collision refusing before `install()`; a three-way ordering fixture
that separates capture order from alphabetical and from parents-first; and RS3
producer-child and finalizer latches proving early stop, scope exit, handler
failure and cancellation each halt *and* join the producer, with a detached-work
control that shows the absence assertions can fail.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 8 redundant comments. Inline suggestions to remove them below.

Comment thread packages/cli/src/cli.ts
// above is the entrypoint's: a `host="run"` child is an ordinary run
// whatever command is hosting it, so it installs the run profile's
// structural syntax even under `xmd test`, which installs none for its own
// document.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// document.

// Kept per installation rather than flattened, because a structural
// declaration and the `expand` that implements it are owned together: a
// flattened catalog could not say which installation core must dispatch an
// occurrence to.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// occurrence to.

// Emitted in the order the installation declared them, which is the order
// `xmd syntax` lists them in. Grouping parents ahead of their children here
// would make what a reader is shown a fact about this loop rather than
// about the catalog.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// about the catalog.

// `as` is not stripped the way an ordinary component's is. An installed form
// produces no value, so there is nothing for a binding to capture, and the
// declared schema refuses it as the unknown prop it is — in the declaration's
// own wording rather than in a second sentence about bindings.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// own wording rather than in a second sentence about bindings.

*#next(): Operation<IteratorResult<ExpansionChunk, void>> {
// Releasing here rather than where the chunk was taken is the whole
// contract: the producer stays suspended across the consumer's use of the
// chunk it delivered, and resumes only when the consumer asks for another.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// chunk it delivered, and resumes only when the consumer asks for another.

// The engine's own constructs, and the installed declarations, are kept apart
// while the walk runs: the category's order is the frozen contract — protected
// engine syntax first, then what a host installed, in the order it was
// captured — and a single name-sorted list cannot express the second half.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// captured — and a single name-sorted list cannot express the second half.


// Only when nothing else in the body was already reported. Content that is
// not an accepted child has its own violation above, and saying the parent is
// also short of children repeats that complaint in a second sentence.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// also short of children repeats that complaint in a second sentence.

properties: {
// A positive integer, spelled as a number with a whole-number constraint
// rather than as `integer`: the two report a fractional count under
// different keywords, and the sentence a reader gets is decided by which.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant comment — restates what the code does.

Suggested change
// different keywords, and the sentence a reader gets is decided by which.

@github-actions

Copy link
Copy Markdown

PR #807: ✨ Install structural Markdown syntax from a package (#806)

61 files, +5690 / -757

Scope

🔴 PR has 6447 lines changed. Split into focused PRs.

🟡 6447 lines changed. PRs under 400 receive more thorough review.

🟡 61 files changed. Are all changes related?

🟡 Changes span 9 directories.

🟡 PR mixes config and source changes.

Structural

🟡 Type declarations with no consumers: DeclaredMarkdownComponent.
Symbol Declared at Refs in diff Why flagged
DeclaredMarkdownComponent packages/core/src/components/declared-markdown.ts:84 1 referenced ≤1× within the added diff (pre-existing usages not counted)

Oxlint structural signals:

  • no-unused-vars ×17: packages/cli/tests/support/plan-harness.ts, packages/cli/src/plan-component.ts, packages/cli/src/plan.ts (+4)
  • no-unnecessary-type-assertion ×6: packages/core/src/expand.ts, packages/core/src/execute.ts
  • no-empty-function ×2: packages/cli/src/testing-host.ts, packages/cli/src/cli.ts

Slop

  • packages/cli/src/cli.ts:1177 — // document.
  • packages/core/src/execute.ts:3445 — // occurrence to.
  • packages/core/src/execution-declarations.ts:561 — // about the catalog.
  • packages/core/src/expand.ts:2173 — // own wording rather than in a second sentence about bindings.
  • packages/core/src/expansion-region.ts:78 — // chunk it delivered, and resumes only when the consumer asks for another.
  • packages/core/src/inspect.ts:583 — // captured — and a single name-sorted list cannot express the second half.
  • packages/core/src/structural-rules.ts:1145 — // also short of children repeats that complaint in a second sentence.
  • packages/terminal/src/xmd.ts:44 — // different keywords, and the sentence a reader gets is decided by which.

Oxlint slop signals:

  • no-console ×3: packages/cli/src/cli.ts
  • no-inferrable-types ×2: packages/core/src/expand.ts

Static Analysis

Oxlint: 95 diagnostics across 14 files (20 rules)
Density: 0.017 violations/added-line

no-unused-vars (17): packages/cli/tests/support/plan-harness.ts, packages/cli/src/plan-component.ts, packages/cli/src/plan.ts (+4)
unbound-method (13): packages/cli/src/plan-component.ts, packages/cli/tests/support/plan-harness.ts, packages/core/src/expand.ts (+4)
no-unsafe-type-assertion (12): packages/cli/src/deno.ts, packages/cli/src/plan-component.ts, packages/core/src/expand.ts (+2)
no-base-to-string (10): packages/cli/src/plan-component.ts, packages/core/src/expand.ts
no-unnecessary-type-assertion (6): packages/core/src/expand.ts, packages/core/src/execute.ts
no-array-sort (5): packages/core/src/syntax-reference.ts, packages/core/src/document-validation.ts, packages/core/src/inspect.ts (+1)
consistent-return (5): packages/core/src/document-validation.ts, packages/core/src/execute.ts, packages/core/src/inspect.ts (+1)
no-shadow (4): packages/core/src/execute.ts, packages/core/src/expand.ts
no-unnecessary-type-conversion (4): packages/core/src/syntax-reference.ts
no-console (3): packages/cli/src/cli.ts
no-floating-promises (3): packages/core/src/expansion-region.ts, packages/cli/src/cli.ts, packages/core/src/execute.ts
no-this-alias (2): packages/core/src/components/import-authority.ts
no-empty-function (2): packages/cli/src/testing-host.ts, packages/cli/src/cli.ts
no-inferrable-types (2): packages/core/src/expand.ts
no-misused-spread (2): packages/core/src/inspect.ts
consistent-function-scoping (1): packages/cli/src/plan-component.ts
no-unmodified-loop-condition (1): packages/cli/src/cli.ts
restrict-template-expressions (1): packages/core/src/expand.ts
no-implied-eval (1): packages/core/src/expand.ts
no-duplicate-type-constituents (1): packages/core/src/execute.ts

Correctness

No extraneous code patterns detected.

@taras

taras commented Sep 12, 2026

Copy link
Copy Markdown
Owner Author

Closing this implementation and restarting #806 from the merged main. The branch is intentionally retained as forensic evidence only: its unplanned diagnostic mini-language, request-authority protocol, and combined core/package/profile migration are not inputs to the replacement implementation.

@taras taras closed this Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make structural Markdown expansion extensible by installed packages

1 participant