@@ -95,15 +95,30 @@ revision, retained Project revision, candidate revision plus expected candidate
9595digest, and AgentGraph digest; they do not accept a self-rehashed envelope as
9696replay evidence.
9797
98- ## Deliberately out of scope
98+ ## Negotiation and transport
9999
100- Issue #201 additionally asks for byte/token benchmarks across multiple * model*
101- tokenizer versions, version * negotiation* (as opposed to version * refusal* ,
102- which this module does implement), and CLI/MCP exposure. Those are not in this
103- module. The selected profiles reuse existing authoritative producers and the
104- unchanged bounded dictionary/text/binary codec. Selected replay adds exact
105- regeneration equality (` SPX-Z910 ` on disagreement) to envelope integrity and
106- profile/root/revision checks. It grants no candidate commit or runtime authority.
100+ ` compact_semantic_projection::negotiation ` selects only existing v1 text or
101+ binary encodings and the six implemented profile names. Both parties provide
102+ at most 16 explicit offers; an unknown version or profile refuses
103+ the exchange rather than falling back. The encoding is a closed Rust enum. Selection is deterministic across
104+ offer ordering: it chooses the lexicographically earliest common profile,
105+ then binary before text. Negotiation changes no codec or wire format.
106+
107+ The CLI and retained semantic service expose these projections through the
108+ existing compiler producers. ` compact graph <project> ` also wraps the full
109+ retained Project graph with profile ` full-graph ` , root ` * ` , and Project revision;
110+ its reconstructed graph schema distinguishes it from a single-source graph.
111+ ` semaprax compact ... --replay <wire> ` reconstructs only after comparison with
112+ freshly selected content. MCP ` workspace__compact_projection ` uses the same
113+ ` workspace/compact-projection ` dispatcher and requires the exact active
114+ workspace revision. Caller source labels select retained Project sources;
115+ they cannot open host paths. See [ Unified CLI v1] ( UNIFIED-CLI-V1.md ) and
116+ [ Semantic Service Transport v1] ( PERSISTENT-SEMANTIC-SERVICE-TRANSPORT-V1.md ) .
117+
118+ The library exposes explicit format/encoding/profile negotiation, and the
119+ host-side ` scripts/benchmark_compact_projection.py ` measures actual tokenizer
120+ counts using cached, versioned encodings. Measurement evidence and its
121+ limitations are recorded below; byte compaction alone is not a token claim.
107122
108123## Honesty bar: what "lossless" means here, exactly
109124
@@ -123,9 +138,8 @@ dropped, summarized, or approximated.
123138This module claims a materially smaller wire size ** only where measured**
124139below, not as a universal property, and it does not claim:
125140
126- - a real model tokenizer's token count (only wire bytes are measured here;
127- ` crate::semantic_task_context ` owns this repository's honestly-labeled
128- token-accounting units);
141+ - a universal token reduction or provider billing equivalence; the host-side
142+ measurements below name exact encoding versions and vocabulary fingerprints;
129143- that every profile or content shape compacts (a selected view with few
130144 repeated strings could compact only slightly, or -- for the binary
131145 form specifically -- not at all, see the measurements below);
@@ -224,3 +238,33 @@ network.
224238` crate::graph::to_json ` /` agent_context_v2_json ` . [ ` decode_text ` ] and
225239[ ` decode_binary ` ] take only byte/string slices. Nothing in this module
226240opens a file, spawns a process, or contacts a network.
241+
242+ ## Offline model-token measurements (2026-09-13)
243+
244+ The reproducible helper runs the actual CLI encoder and replay for both wire
245+ forms, compares reconstructed bytes with the ordinary graph command (allowing
246+ its one display newline), then measures UTF-8 text with cached ` tiktoken 0.12.0 `
247+ encodings. Socket connections are disabled while tokenizer assets load. The
248+ [ complete local report] ( ../benchmarks/compact-semantic-projection-v1/local-token-measurements.json )
249+ records exact content hashes and tokenizer vocabulary/pattern fingerprints.
250+ Binary bytes are measured as transport bytes, never decoded as model text.
251+
252+ | Input | Full bytes | Text bytes | Binary bytes | cl100k full → text tokens | o200k full → text tokens |
253+ | --- | ---: | ---: | ---: | ---: | ---: |
254+ | ` examples/banking_ledger.spx ` | 139,770 | 88,586 | 139,129 | 37,846 → 46,781 | 38,779 → 47,124 |
255+ | ` examples/http_app_routing.spx ` | 610,864 | 399,030 | 566,677 | 161,861 → 220,491 | 165,329 → 221,565 |
256+ | ` examples/calculator-project/semaprax.toml ` | 9,893 | 7,474 | 11,513 | 2,500 → 3,606 | 2,521 → 3,626 |
257+ | ` examples/frame-payload-project/semaprax.toml ` | 6,615 | 5,471 | 8,203 | 1,766 → 2,679 | 1,784 → 2,694 |
258+
259+ Text saves bytes on all four examples but ** increases model tokens by 21–52%** .
260+ Binary is larger than ordinary JSON on both measured Projects. These results
261+ are a negative control against assuming byte savings imply token savings.
262+ Issue #201 remains open for a measured token-efficient representation; the
263+ current text form is a byte-compaction option, not a recommended model-token
264+ optimization. Neither measurement implies provider billing or model quality.
265+
266+ Reproduce with an already installed ` tiktoken ` and cached encoding assets:
267+
268+ ``` sh
269+ python3 scripts/benchmark_compact_projection.py --semaprax target/debug/semaprax --root . --output compact-measurements.json
270+ ```
0 commit comments