[P2] Publish a small stable Semaprax embedding API with explicit host capabilities

Summary

Let editors, build systems, services, and applications embed parsing, checking, interpretation, semantic query, candidate validation, and selected execution without spawning the CLI or receiving ambient host authority.

Why this matters competitively

Mog’s strongest advantage is safe embeddability. Semaprax’s internal libraries are rich but not yet an intentionally small, supported embedding surface.

Current state at the audit baseline

Rust crates and persistent semantic service kernels exist.
Some host libraries are private or developer-preview.
No small stable public embedding ABI/API is supported across versions.
Host authority is already modeled explicitly in many runtime providers.

Audit baseline: ae25c6a49dc09ec4613c7a6f52a27daa69dcd3f3 (2026-09-11). Recheck main, this issue, and the owning specifications before changing code; later commits may have superseded individual details.

Required outcome

Define a minimal versioned embedding interface with explicit caller-provided source, limits, allocators/storage where needed, cancellation, diagnostics, and capability/provider objects. Offer Rust first and a C ABI or Wasm component later, with strict lifecycle and no ambient fallback.

In scope

Compiler/session creation.
Source/Project load and authenticated refresh.
Check, format, graph/query/context, candidate validate/replay.
Deterministic interpreter execution for admitted profiles.
Explicit provider/capability injection.
Memory/resource ownership and cancellation.
Version/feature negotiation.
Thread-safety and reentrancy contract.

Explicitly out of scope

Exposing all internal Rust structs as stable API.
Reading filesystem/environment/network implicitly.
Embedding publication or Git authority in the default interface.
Promising ABI stability before versioning/lifecycle rules exist.
Allowing callers to construct validated HIR directly.

Starting points in the repository

persistent semantic service core
public/private crate boundaries
interpreter and provider interfaces
Project revision store
generated C/Rust SDK patterns
docs/UNIFIED-CLI-V1.md

Use rg to find the current owners of the named schema, diagnostic, operation, or test before editing. Do not assume a path is still authoritative merely because it is listed here.

Implementation sequence

Define the smallest use cases and a versioned opaque-handle API. Keep validated compiler objects private and expose canonical reports/results.
Specify ownership, thread safety, reentrancy, cancellation, memory limits, panic/error normalization, and destruction for every handle.
Require callers to provide source bytes/closures or an explicit source provider. No default filesystem crawling.
Require effectful execution providers/capabilities to be passed explicitly per session/invocation. Missing provider yields authority denied.
Implement Rust API first with a compatibility facade over existing kernels. Add C ABI only after stable owned string/record/result conventions are selected.
Add feature negotiation and reject unsupported schema/profile combinations.
Create examples for editor analysis, server-side context service, and deterministic embedded execution.
Run leak, panic, concurrency, cancellation, malformed input, and version mismatch tests.

Failure and security cases that must be considered

Public Rust types can freeze unstable internals.
Opaque handles can leak or be used after free across FFI.
Callbacks can outlive borrowed caller memory.
A convenience source provider can reintroduce ambient filesystem authority.
Panics crossing FFI are undefined/unsafe.

Required tests and evidence

Create/use/destroy lifecycle and leak checks.
Malformed source, capacity, cancellation, panic normalization, and unsupported version.
Thread/reentrancy behavior according to the contract.
No filesystem/network/process access without explicit provider.
Stale refresh and candidate replay rejection.
Rust example compatibility and later C ABI sanitizers where implemented.

Acceptance criteria

A host can embed core Semaprax analysis through a small documented API.
Validated internals remain compiler-owned.
All authority and source access are explicit.
Resource lifecycle and errors are stable and testable.
The API has a clear compatibility/version policy.

Dependencies and sequencing

[P1] Consolidate CLI, LSP, MCP, and generated SDKs on one versioned semantic service API

Rules for the implementing agent

Read AGENTS.md, docs/COMPLETION-MATRIX.md, docs/ROADMAP.md, docs/QUALITY-GATES.md, and every owning versioned specification named above before editing.
Prefer extending the existing semantic object, operation table, diagnostic catalogue, profile dispatcher, or evidence harness. Do not create a parallel source of truth.
Preserve canonical source, persistent identities, revision binding, fail-closed replay, deterministic rendering, and the separation between evidence and authority.
A local test is local evidence. A hosted run is hosted evidence. A generated artifact is not a support or publication decision. Record exactly what ran and do not widen claims.
Any new wire format, ABI, descriptor, carrier, evidence document, or compatibility surface requires an explicit versioned schema, independent decoder/replay, hostile-input tests, bounds, and nonclaims.
Never grant ambient filesystem, process, network, model, signing, payment, commit, publication, or registry authority merely because a declaration, receipt, test, or proof exists.
Keep changes reviewable. Split implementation into commits that each preserve the existing test suite, and explain any known-answer update field by field.
Do not weaken or delete a failing regression to obtain green CI. Fix the cause or narrow the admitted feature with a stable diagnostic.

Definition of done

The implementation, tests, documentation, status wording, diagnostics, and evidence agree on one exact scope. Targeted tests pass, the applicable repository quality gate passes, hostile cases fail with the documented stable reason, and no unrelated public surface or authority is widened.

feat(semantic-embedding): add a capability-gated embedding boundary +…

feat(embedding-api): add a compiler-embedding check surface for #203

style: apply cargo fmt to the embedding API tests

First real slice of the embedding surface landed (`e30c4605`) — and a naming collision that would have caused a false closure

Read this before crediting `semantic_embedding` to this issue

main carries cb72738e/29c84258, adding src/semantic_embedding/ and
docs/SEMANTIC-EMBEDDING-V1.md. That module computes a vector embedding
(bytes → Vec<f32> for retrieval — the machine-learning sense of the word)
behind a capability/provider seam.

Issue #203 is about embedding the compiler as a library — "editors, build
systems, services, and applications embed parsing, checking, interpretation
without spawning the CLI". Different meaning of "embedding" entirely. That
module's own doc says so: "Issue #203 asks for a much larger surface... None of
that lives in this module."

None of #203's acceptance criteria — compiler/session creation, source load,
check/format/graph/query, candidate validate/replay, capability-gated execution,
version/lifecycle contract — were touched by that tranche. Recorded here because
a reader scanning commit titles for "#203" would reasonably conclude otherwise.

What actually landed

src/embedding_api.rs — check_source(unit_name, source) -> CheckOutcome.

criterion	test	negative control
Explicit source, no filesystem crawling	`unit_name_is_never_read_from_disk`	succeeds even when `unit_name` names a path guaranteed not to exist
Validated internals stay compiler-owned	all tests touch only `CheckOutcome`	`Program`/`Analysis`/`ResolvedProgram` never exposed; the doc names what is hidden
Diagnostics never silently dropped	`a_declaration_missing_id_still_checks_ok_but_keeps_its_warning`	asserts `SPX-S103` present with `ok: true` — `crate::check` discards warnings on success, this does not
Panic normalization	`embedding_boundary_normalizes_a_panic_into_a_diagnostic_never_propagating_the_unwind`	a test-only `SourceChecker` panics deliberately; asserts the message never contains `SPX-P101`
Distinct refusal reasons	`malformed_source_fails_with_the_specific_parser_diagnostic`	asserts neither diagnostic's code or message contains the other's code, so a genuine parse failure and an internal defect cannot be confused
Version negotiation	`version_negotiation_accepts_matching_major_and_refuses_a_different_one`	refuses at major 0 and 2, not merely accepting at 1

That last control is the kind this backlog usually lacks: testing only the
accepting case would leave a is_compatible_with that returns true
unconditionally looking perfectly healthy.

cargo test --locked -p semaprax --lib embedding_api::tests → 6 passed,
non-zero selected count confirmed. Coordinator re-ran fmt (applied) and
clippy --all-targets -D warnings (exit 0) centrally.

Deliberately not exposed — stated so the surface is not over-read

No opaque session or compiler handle and no open/close lifecycle
(check_source is stateless); no multi-file Project load; no execution
capability — no interpreter is ever run, and a future execution slice needs its
own explicit non-Default capability type mirroring
ModelInvokeCapability/EmbeddingCapability rather than one invented
speculatively here; no cancellation token; no C ABI (this issue sequences that
after the Rust surface stabilizes); no candidate validate/replay (off-lease).
format, graph::to_json and graph::context_json remain unwrapped and are
named as remaining work rather than claimed done.

Issue stays open: this is one slice of a much larger surface.
Evidence class: local, re-runnable. Not hosted.

feat(embedding-api): wrap canonical formatting in the embedding facade

Audited against main; one gap closed (merge cce695a2). Not closing — roughly four of six in-scope bullets remain.

Caller check first, since it changes what "delivered" means here: embedding_api's only caller is src/lib.rs's pub mod embedding_api;. check_source had zero callers outside embedding_api::tests. It is not wired into the CLI, the LSP, or any other module — it is a facade nothing yet embeds.

Criterion	Status	Evidence
A host can embed core analysis through a small documented API	partial	`check_source` and now `format_source` are stateless, documented and tested — two of ~6 in-scope bullets. No session/handle, no graph/context, no execution.
Validated internals stay compiler-owned	met	`CheckOutcome`/`FormatOutcome` never expose `Program`/`Analysis`/`ResolvedProgram`
All authority and source access explicit	met for what exists	both take only `source`; `unit_name` is never read as a path — `unit_name_is_never_read_from_disk`, `format_unit_name_is_never_read_from_disk`
Resource lifecycle and errors stable and testable	met, narrowly	stateless functions have no lifecycle to get wrong; panic normalization tested for both
Clear compatibility/version policy	met	`EMBEDDING_API_VERSION`, major-only compatibility, tested at 0/1/2

Added: format_source, plus FormatOutcome and a SourceFormatter/StandardFormatter seam, mirroring check_source's conventions exactly — no ambient authority, panic normalization to SPX-EMB001 via a test-only panicking double. EMBEDDING_API_VERSION bumped 1.0.0 → 1.1.0, additive-only per the doc's own policy. docs/EMBEDDING-API-V1.md updated: in-scope row, a FormatOutcome contract section, compatibility policy, nonclaims, evidence.

This was the one gap the doc itself named as real, remaining and bounded-worker-implementable.

Deliberately left open rather than guessed at: graph::context_json was considered and rejected as a wrapping target. The CLI actually uses graph::agent_context_json/agent_context_v2_json with AgentContextOptions (filters, byte and node budgets, direction), not the simpler two-argument form. Choosing which surface the embedding facade exposes is a design decision, not a mechanical extension, and inventing it here would have been the wrong call.

Evidence class: local re-runnable. cargo test --locked -p semaprax --lib embedding_api::tests → 11 passed, 0 failed (6 pre-existing + 5 new).

Remaining for closure: session/handle lifecycle, graph and context re-exposure, candidate validate/replay, execution capability, a C ABI, thread/reentrancy stress, multi-file Project load — and at least one real embedder calling any of it.

feat(embedding-api): re-expose graph::to_json through the embedding f…

One more cell closed in eec9b187. Not closing — most of the surface is still open, and the facade still has no embedder.

Added: graph_source(unit_name, source) -> GraphOutcome plus pub struct GraphOutcome { unit_name, ok, diagnostics, graph_json }, in src/embedding_api.rs. Both build only on already-public types (Diagnostic, Program, graph::to_json) — no internal type was promoted or dragged into the public surface. The private SourceGrapher/StandardGrapher seam mirrors the existing SourceChecker/SourceFormatter pattern and exists solely so a double can force the panic path.

Deliberately left out, with reasons rather than silence:

graph::context_json — it takes a symbol and depth and filters through AgentContextOptions with byte and node budgets, while the CLI's real entry points are agent_context_json/agent_context_v2_json. Choosing which subset to expose is a design decision, not a mechanical wrap.
Session/handle lifecycle, execution capability, C ABI, candidate validate/replay, cancellation — unchanged nonclaims. Adding them now would invent surface ahead of a real need, and every exported item becomes a compatibility obligation.

Capability audit. graph_source takes no capability because it is pure and effect-free by construction — parse, resolve, serialize, no I/O — so there is nothing for a capability to gate. unit_name is proven never read from disk (graph_unit_name_is_never_read_from_disk).

Worth recording about the one genuinely capability-gated object nearby: semantic_embedding::EmbeddingCapability is a required, non-optional parameter with no Default on embed. There is no way to compile a call without constructing one via EmbeddingCapability::grant(reason). That makes "fails without the capability" a compile-time fact rather than a runtime path — which is why no runtime both-directions test exists for it, and why that is the stronger arrangement rather than a gap.

Non-vacuity proved on the load-bearing test: valid_source_graph_matches_graph_to_json_exactly asserts byte-identity with calling graph::to_json directly. Stubbing StandardGrapher::graph to return "{}" made it fail (left: "{}", right: the full graph JSON); restored, 16/16 pass.

One test locks in a real difference on both sides at once: check_source keeps the SPX-S103 warning while graph_source does not, because hir::resolve's success path drops diagnostics. Asserting both halves makes that a deliberate, documented difference rather than an accident nobody noticed.

Evidence: --lib embedding_api::tests → 16 passed (11 pre-existing + 5 new). Local, offline.

The honest limitation: grep -rn "embedding_api" src/ outside the module returns exactly one hit — src/lib.rs:76: pub mod embedding_api;. No caller outside its own tests. This is a facade nothing embeds. An embedding API with no embedder cannot be said to work; "at least one real embedder" remains this issue's most load-bearing open criterion, along with session/handle lifecycle, execution capability, C ABI, thread/reentrancy stress and multi-file Project load.

Version bumped 1.1.0 → 1.2.0, additive only, tested at major 0/1/2.

feat(embedding-api): re-expose graph::to_json through the embedding f…

feat(embedding): add graph_source to the embedding facade (#203)

fix: complete reviewed workspace-limit and live-contract handoff

Implementation batch landed on main: e0ed786.

The compiler embedding facade is now version 1.5.0. Added bounded context v1 and bidirectional v2 calls with facade-owned option/filter/direction types, canonical graph validation/output, pathless caller source, no-match outcomes, and panic normalization. Added execute_entry_source in a child module: explicit execution capability, fuel limit, atomic cancellation, closed outcomes, and the existing prepared zero-argument i64 resolved-HIR interpreter on its prescribed fixed stack. There are no host-effect providers or implicit path reads. Existing check/format/graph API behavior is retained.

Focused evidence: all selected embedding tests passed in the assembled receipt/embedding library run, including direct context parity, hostile options/source, no-match/panic handling, explicit-capability execution, pre-cancel and fuel exhaustion. Module-size and formatting checks passed. No full quality suite or separate dependency rebuild was run.

Issue remains open for a small opaque session/Project facade over the existing source/refresh/candidate kernels, with lifecycle/resource/authority contracts and the corresponding host examples/evidence. semantic_embedding is the separate vector-embedding module, not a compiler session implementation. C ABI/Wasm support is explicitly later in this issue and is not treated as a current blocker.

Landed opaque Project embedding sessions on main at 852aa90; facade version 1.6.0.

ProjectSessionInput accepts only owned manifest/source bytes and preflights source count, aggregate bytes/overflow and duplicate labels before admission copies. ProjectSession wraps the existing incremental service for open, atomic refresh, canonical semantic query and v2 candidate validate/replay. Compiler/cache/candidate internals remain private; outputs are canonical reports with no publication capability. Stateful panic normalization poisons the session and refuses reuse. Normal refresh failures retain the old generation.

Focused facade tests cover pathless source labels, bounds/duplicates, successful changed refresh, stale/invalid retention, query/candidate replay, rejection of old evidence after refresh, and a deliberate boundary panic proving poisoning. Standard Rust ownership/drop governs lifetime.

Remaining: document and exercise the full host-facing lifecycle/resource and thread/reentrancy acceptance contract and examples, and decide/implement cancellation for long-running session analysis (the existing service lacks a cancellation seam; only the bounded interpreter facade currently has one). No C ABI/Wasm promotion or effect-provider/session support is inferred from this additive Rust facade.

Local focused evidence: 56 selected library tests passed (one serial build), including the new integration paths, model-policy preservation tests, and receipt projection. Module-size, format and diff checks passed. No full quality suite, hosted CI, live provider or billing claim.

Implemented and pushed in b35139d. Local macOS arm64 verification: 56 selected library tests passed, module-size and format/diff checks passed. The external Rust embedding consumer compiled against the freshly built library and ran successfully. No full quality run, hosted CI, or network provider evidence is claimed.

Extended the facade to v1.7: stable major/minimum-minor/feature refusal, bounded exact feature-name negotiation, monotonic cancellation on bounded context v1/v2, and pre-cancellation for opaque Project open/refresh. Cancelled refresh preserves the generation and reusable handle. The external facade-only consumer now exercises context, explicit deterministic execution, Project session creation, and version/profile compatibility.

Remains open for the final resource/lifecycle contract audit, including explicit source-analysis input limits and thread/lifecycle evidence. Cancellation is cooperative: pure context can discard its completed report, Project refresh samples before staging, and no mid-kernel interruption is claimed. C ABI/Wasm remains a later interface.

Completed the issue's bounded Rust embedding interface and pushed 0e1ff3e to main. The current contract is docs/EMBEDDING-API-V2.md; the v1 reference remains historical.

Opaque compiler-owned Project sessions open from explicit manifest/source bytes, refresh with exact revision checks and staged adoption, and expose canonical query/candidate validation/replay reports.
Stateless check/format/graph/context v1/v2 accept reusable borrowed AnalysisRequest values with caller-tightened bounds. Fixed caps cover source (16 MiB), diagnostic labels and context symbols (4 KiB), including bounded UTF-8 refusal echoes. Execution shares the hard input caps and retains explicit capability, fuel and cancellation.
Cancellation, malformed input, capacity errors and panic normalization remain closed outcomes. Stateful panic poisoning and failed-refresh retention are tested. Dropping a session releases its actual active ProjectRevision (Weak reference regression). Concurrent analysis requires two successful results with identical revisions.
Exact major/minimum-minor/feature negotiation refuses incompatible requirements. Mandatory input caps narrow old accepted inputs, so this is API 2.0.0, not an allegedly compatible 1.x update. Major-1 hosts are explicitly refused.
The external Rust facade-only consumer compiled against the rebuilt library and ran, exercising bounded requests, context, explicit deterministic execution, Project session lifetime, and v1 refusal/v2 acceptance.

Local evidence: 47 embedding tests passed; the two version/feature tests were rerun after the major-version correction and passed. The compiled external API-v2 consumer passed. Module-size, formatting and diff checks passed. Tests and sources are committed; no full/hosted quality run is claimed.

The C ABI/Wasm interface is explicitly later in this issue, and no registry publication, general host-effect provider, FFI sanitizer, or multi-platform support is claimed. The Rust in-scope API, lifecycle, explicit authority, resource/error and compatibility acceptance criteria are complete; closing this issue.

Navigation Menu

Repository navigation

[P2] Publish a small stable Semaprax embedding API with explicit host capabilities #203

Summary

Why this matters competitively

Current state at the audit baseline

Required outcome

In scope

Explicitly out of scope

Starting points in the repository

Implementation sequence

Failure and security cases that must be considered

Required tests and evidence

Acceptance criteria

Dependencies and sequencing

Rules for the implementing agent

Definition of done

First real slice of the embedding surface landed (`e30c4605`) — and a naming collision that would have caused a false closure

Read this before crediting `semantic_embedding` to this issue

What actually landed

Deliberately not exposed — stated so the surface is not over-read

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[P2] Publish a small stable Semaprax embedding API with explicit host capabilities #203

Description

Summary

Why this matters competitively

Current state at the audit baseline

Required outcome

In scope

Explicitly out of scope

Starting points in the repository

Implementation sequence

Failure and security cases that must be considered

Required tests and evidence

Acceptance criteria

Dependencies and sequencing

Rules for the implementing agent

Definition of done

Activity

wsdt commented on Sep 12, 2026

First real slice of the embedding surface landed (e30c4605) — and a naming collision that would have caused a false closure

Read this before crediting semantic_embedding to this issue

What actually landed

Deliberately not exposed — stated so the surface is not over-read

wsdt commented on Sep 12, 2026

wsdt commented on Sep 12, 2026

wsdt commented on Sep 13, 2026

wsdt commented on Sep 13, 2026

wsdt commented on Sep 13, 2026

wsdt commented on Sep 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

First real slice of the embedding surface landed (`e30c4605`) — and a naming collision that would have caused a false closure

Read this before crediting `semantic_embedding` to this issue