Repository navigation
[P2] Add a compact lossless semantic projection optimized for model context and transport #201
Description
Summary
Provide a compact representation of the canonical semantic program for agents, caches, and protocols while retaining .spx as the authoritative human/Git source and ProgramRoot/graph as the full semantic model.
Why this matters competitively
B-IR, Tacit, LLMLang, and Sever explore compact machine-oriented representations. Semaprax should gain their token/transport advantages without sacrificing readable source or introducing a second authority.
Current state at the audit baseline
- Semaprax has canonical source, stable-ID HIR, versioned semantic graph, ProgramRoot, bounded context, and JSON schemas.
- These representations can be verbose for model context and repeated transport.
- No explicitly optimized compact projection with lossless round-trip to selected semantic facts is supported.
- The project correctly keeps source as the Git authority.
Audit baseline: ae25c6a49dc09ec4613c7a6f52a27daa69dcd3f3 (2026-09-11). Recheck main, this issue, and the owning specifications before changing code; later commits may have superseded individual details.
Required outcome
Define semaprax.compact-semantic-projection.v1, a deterministic dictionary/index-based encoding of selected graph/HIR facts with explicit schema/version/root binding, suitable for token-efficient textual use and binary transport, independently replayable to the same selected semantic view.
In scope
- Stable-ID dictionary and indexed references.
- Types, signatures, contracts, effects, ownership, calls, tests, requirements, diagnostics, and selected bodies as supported.
- Canonical textual and optional binary encoding.
- Projection profiles for task context, API surface, candidate diff, and Agent definition.
- Lossless replay to the selected full semantic view.
- Token/byte benchmarks.
- Version negotiation.
Explicitly out of scope
- Making the compact projection the source of truth.
- Allowing edits directly against anonymous numeric indexes without revision/ID checks.
- Compressing by omitting semantic facts while calling the result lossless.
- Using an undocumented binary format.
- Replacing goal-aware context selection.
Starting points in the repository
- semantic graph and ProgramRoot schemas
- bounded context and universal query
- canonical render/decoder utilities
- agent skill and semantic service
- benchmark harness
Use rg to find the current owners of the named schema, diagnostic, operation, or test before editing. Do not assume a path is still authoritative merely because it is listed here.
Implementation sequence
- Define projection profiles and exactly which full semantic object each profile can reconstruct.
- Build a deterministic stable-ID/type/string dictionary ordered by canonical bytes. Numeric indexes are local compression only and never persistent identity.
- Encode edges and repeated structures compactly while preserving explicit schema/version/root/profile digests and bounds.
- Provide a readable compact text form for model prompts and a length-framed binary form for transport/cache if useful.
- Implement independent decode/replay to the selected ordinary semantic graph/context object and compare canonical digests.
- Bind any semantic operation referring to compact entries back to stable IDs and exact revision; never accept a naked index.
- Benchmark bytes and model tokens across representative Projects and task contexts.
- Add migration/version refusal rather than heuristic decoding of unknown versions.
Failure and security cases that must be considered
- Dictionary indexes can be mistaken for stable identities.
- Compression can obscure ownership/effect details and invite incomplete model edits.
- Binary decoders are vulnerable to hostile lengths/counts.
- A projection profile can drift from the full graph.
- Token savings may be negligible or model-dependent.
Required tests and evidence
- Canonical encode/decode/re-encode known answers.
- Replayed selected view matches the full semantic object exactly.
- Stable-ID/display rename and revision-change behavior.
- Hostile length/count/index/duplicate/order/version/root cases.
- No semantic operation accepts an index without stable-ID/revision resolution.
- Byte and token measurements across multiple tokenizer versions and Projects.
Acceptance criteria
- Models/tools can consume a materially smaller semantic representation where measured.
- The representation is deterministic, bounded, versioned, and lossless for its stated profile.
- Readable
.spxremains the canonical source. - Stable IDs, not local indexes, remain operation identities.
- Benefits are demonstrated empirically rather than assumed.
Dependencies and sequencing
- [P1] Add goal-aware, model-token-budgeted semantic context compilation
Rules for the implementing agent
- Read
AGENTS.md,docs/COMPLETION-MATRIX.md,docs/ROADMAP.md,docs/QUALITY-GATES.md, and every owning versioned specification named above before editing. - Prefer extending the existing semantic object, operation table, diagnostic catalogue, profile dispatcher, or evidence harness. Do not create a parallel source of truth.
- Preserve canonical source, persistent identities, revision binding, fail-closed replay, deterministic rendering, and the separation between evidence and authority.
- A local test is local evidence. A hosted run is hosted evidence. A generated artifact is not a support or publication decision. Record exactly what ran and do not widen claims.
- Any new wire format, ABI, descriptor, carrier, evidence document, or compatibility surface requires an explicit versioned schema, independent decoder/replay, hostile-input tests, bounds, and nonclaims.
- Never grant ambient filesystem, process, network, model, signing, payment, commit, publication, or registry authority merely because a declaration, receipt, test, or proof exists.
- Keep changes reviewable. Split implementation into commits that each preserve the existing test suite, and explain any known-answer update field by field.
- Do not weaken or delete a failing regression to obtain green CI. Fix the cause or narrow the admitted feature with a stable diagnostic.
Definition of done
The implementation, tests, documentation, status wording, diagnostics, and evidence agree on one exact scope. Targeted tests pass, the applicable repository quality gate passes, hostile cases fail with the documented stable reason, and no unrelated public surface or authority is widened.
Activity
feat(compact-semantic-projection): add lossless dictionary-encoded wi…
wsdt commentedon Sep 12, 2026
Landed on
main(merge2ed47537):src/compact_semantic_projection.rs+tests.rs,docs/COMPACT-SEMANTIC-PROJECTION-V1.md. Not closing — four in-scope items remain.No prior compact-projection or dictionary-index work existed. This reuses
graph::to_jsonandgraph::agent_context_v2_json(selection, closure and budget engines) rather than duplicating them;src/semantic_task_context.rswas read-only throughout.Measured compaction — real numbers on committed examples, full-graph profile:
.spxgraph::to_jsonhttp_app_routing.spxbanking_ledger.spxRecorded honestly in the spec doc, including that a first per-token-framed text design measured larger than the original on
http_app_routing.spx— which is why the shipped text form uses inline~index~markers with no per-token framing.Lossless means detected, not asserted.
tampered_wire_content_fails_digest_verificationandtampered_text_body_content_fails_digest_verificationflip one byte, preserving length and counts, and the decode is refused withSPX-Z907specifically — a genuine negative control, not a self-comparison. Both round-trip tests compare against a second, independent call to the real engine and assert the wire form differs from the original bytes.All six
MAX_*bounds are driven one-under, at, and one-over, each isolated so only that bound fires. Hostile coverage asserts a specific code per case (SPX-Z901…SPX-Z909): declared-count-versus-buffer refused before allocating, out-of-range index, duplicate and reordered dictionary entries, unrecognized tag, truncation, trailing bytes, and a field-injected root that plain decode accepts butdecode_binary_and_verifyrefuses — which is the test that proves the root binding is doing work rather than decorating the header. An unknown format version is refused, never heuristically decoded.Evidence class: local re-runnable.
cargo test --locked -p semaprax --lib compact_semantic_projection→ 26 passed, 0 failed.Remaining in scope for closure:
wsdt commentedon Sep 13, 2026
Implemented and pushed in b35139d. Local macOS arm64 verification: 56 selected library tests passed, module-size and format/diff checks passed. The external Rust embedding consumer compiled against the freshly built library and ran successfully. No full quality run, hosted CI, or network provider evidence is claimed.
Added authoritative compact profiles for goal-aware task context, the explicitly scoped Project v8 public API descriptor, candidate semantic-delta catalog, and compiled Agent graph. Text/binary selected replay regenerates the owning kernel and compares exact reconstructed bytes; a self-rehashed alternate payload with matching profile/root/revision is rejected (SPX-Z910). Tests cover all four producers, stale revision, and same-binding forgery.
Remains open: format/encoding negotiation, CLI/MCP exposure, and empirical measurements across real model tokenizer versions. Byte and lexical counters are not model tokenizer evidence.
wsdt commentedon Sep 13, 2026
Pushed 0e1ff3e: all six existing compact profiles now have public CLI encode/replay paths; the same producers are exposed through revision-bound workspace/compact-projection and MCP forwarding. Added bounded deterministic format/encoding/profile negotiation. Project/candidate selection remains authenticated/retained, source labels cannot open host paths, and CLI replay compares to freshly generated selected content. Added real offline tokenizer measurement tooling and committed its exact-byte/vocabulary-fingerprinted report.
Not closing: the measurements reveal the remaining performance defect. Current text saves bytes but increases cl100k_base/o200k_base tokens by 21–52% across banking_ledger, http_app_routing, calculator Project and frame-payload Project. Binary is also larger than JSON on both measured Projects. Full counts/hashes/fingerprints are in benchmarks/compact-semantic-projection-v1/local-token-measurements.json and docs/COMPACT-SEMANTIC-PROJECTION-V1.md. Byte savings are not model-token savings.
Next work must change the representation based on these measurements while preserving exact replay, stable IDs and the frozen v1 contract; simply relabeling byte compression as a context optimization would not finish this issue.
Focused local evidence: compact negotiation tests, CLI parser/actual graph reconstruction tests, stdio/MCP and retained Project projection tests passed. The benchmark executed both wire forms and replayed them against ordinary graph output on all four cases, with cached tiktoken 0.12.0 and network connections disabled during tokenizer loading. No hosted, provider billing, or universal savings claim.
wsdt commentedon Sep 13, 2026
Implemented on main in c9205ad, completing the compact-projection scope.
All six real producer profiles now have deterministic lossless compact replay, CLI selection/replay, retained service/MCP selection, and explicit version/profile/encoding negotiation. Frozen v1 text/binary remains compatible. Additive model-text v2 uses a selective dictionary with bounded independent expansion, digest/binding verification, canonical re-encode checks and a frozen known answer. No semantic operation accepts anonymous numeric references as authority.
The committed offline benchmark uses cached tiktoken 0.12.0 with cl100k_base and o200k_base, recording exact bytes/hashes and vocabulary fingerprints. Complete v2 envelopes reduce tokens by 14–18% on banking/HTTP graphs and 11% on HTTP task context. Two small Project graphs and banking task context grow 5–7%; that limitation and v1's earlier token regression remain explicit. Every measured wire replays to exact selected bytes; graph cases also match ordinary graph output.
Evidence: 81 focused checks passed across codec/streaming/adapter library tests, CLI, retained service and typed runtime. The additive MCP encoding catalogue entry received static review after those checks; no repeat build for that literal enum addition. Full/hosted CI was deliberately not rerun under the requested speed policy. New specification and report: docs/COMPACT-SEMANTIC-PROJECTION-V2.md and benchmarks/compact-semantic-projection-v2/local-token-measurements.json. This is local measurement evidence, not provider billing or universal token savings.