[P1] Add goal-aware, model-token-budgeted semantic context compilation #197
Description
Summary
Extend bounded semantic context from a declaration-centered byte/node slice into a goal-aware compiler product that selects the smallest sufficient semantic closure for an agent task under an explicit model-token budget.
Why this matters competitively
Lume treats context as a budgeted product, while Axis and Zero optimize what agents need to generate or modify. Semaprax already has the semantic graph, stable identities, impact edges, contracts, tests, and history needed to outperform simple file selection.
Current state at the audit baseline
semaprax contextsupports a selected ID, depth, maximum bytes, and maximum nodes.- Exact Program Context and persistent semantic service projections exist.
- Context is not yet selected from a natural-language/structured goal, model tokenizer, task history, or assurance impact.
- Representative token/task economics remain open and benchmark methodology needs repair.
Audit baseline: ae25c6a49dc09ec4613c7a6f52a27daa69dcd3f3 (2026-09-11). Recheck main, this issue, and the owning specifications before changing code; later commits may have superseded individual details.
Required outcome
Add a deterministic context compile operation that accepts a structured goal, target operation, candidate/task facts, token budget, tokenizer identity, and policy; returns an ordered semantic context bundle; and explains included, summarized, and omitted material.
In scope
- Structured goal representation with optional natural-language text as data.
- Seed detection using explicit IDs, diagnostics, paths, requirements, or query results.
- Semantic closure over callers/callees, types, effects, ownership, contracts, tests, requirements, packages, and recent changes.
- Token counting through explicitly selected tokenizer adapters.
- Prioritization and summarization policy.
- Omission report and completeness risks.
- Incremental/cache behavior and deterministic replay.
- CLI/MCP/SDK exposure.
Explicitly out of scope
- Letting an LLM decide authoritative context membership without compiler constraints.
- Guaranteeing that any finite context is sufficient for every task.
- Using approximate token counts while claiming exact budget adherence.
- Including secrets or inaccessible source because it is semantically relevant.
- Treating context data as source or execution authority.
Starting points in the repository
- context and semantic graph modules
- Exact Program Context v1/v2
- universal semantic query
- persistent semantic service
- impact/review/test facts
- future requirement/intent objects
- benchmark methodology
Use rg to find the current owners of the named schema, diagnostic, operation, or test before editing. Do not assume a path is still authoritative merely because it is listed here.
Implementation sequence
- Specify
SemanticTaskContext v1: goal fields, seeds, revision, tokenizer, budget, inclusion policy, selected items, summaries, omissions, uncertainty, and digest. - Support explicit seed IDs first. Add deterministic lexical matching over names/docs only as a seed suggestion, never as a replacement for semantic resolution.
- Define ranked edge classes and mandatory closure rules. For example, a function body may require its signature/types/contracts/effects/ownership facts, while distant callers may be summarized.
- Integrate requirements, tests, diagnostics, candidate diff, and recent semantic history where available. Bind all items to stable IDs and exact revisions.
- Implement tokenizer adapters with explicit version/digest and exact byte-to-token accounting. If the tokenizer is unavailable, refuse exact-token mode or use clearly labeled byte mode.
- Select/summarize deterministically within budget. Never truncate canonical JSON or source fragments mid-unit; omit whole typed units with reasons.
- Return an omission/risk report identifying excluded callers, tests, effects, packages, or unsupported external facts.
- Add cache keys over revision, goal, policy, tokenizer, and budget; verify stale source or policy invalidates the cache.
Failure and security cases that must be considered
- Goal text can be prompt-injected and should remain untrusted data.
- Aggressive summarization can omit behavior-changing contracts or effects.
- Tokenizer version drift changes the budget.
- Context caching can leak source across authorization boundaries.
- Semantic closure can explode through widely used types/functions.
Required tests and evidence
- Deterministic selection for fixed goal/revision/policy/tokenizer/budget.
- Exact budget boundary and maximum-plus-one cases with multiple tokenizers.
- Mandatory contract/effect/ownership/test facts are never truncated or silently omitted.
- Stale revision, tokenizer, policy, and access-scope cache entries are rejected.
- Secrets/inaccessible modules are omitted with explicit blind-spot facts.
- Representative tasks compare full repository, current context, and goal-aware context for outcome and token usage after issue [P2] Criterion benches: prepared-evaluator is inverted and rebuild-unchanged shows no speedup #85 methodology is fixed.
Acceptance criteria
- Agents can request task-specific semantic context under a real tokenizer budget.
- Every inclusion, summary, and omission is explainable and revision-bound.
- The compiler, not the model, owns semantic closure rules.
- Context generation grants no additional source or runtime authority.
- Task-level benchmark evidence shows when the context product helps and where it does not.
Dependencies and sequencing
- Implementation plan for existing issue [P2] Criterion benches: prepared-evaluator is inverted and rebuild-unchanged shows no speedup #85: repair benchmark comparability and measure actual semantic-service gains
Rules for the implementing agent
- Read
AGENTS.md,docs/COMPLETION-MATRIX.md,docs/ROADMAP.md,docs/QUALITY-GATES.md, and every owning versioned specification named above before editing. - Prefer extending the existing semantic object, operation table, diagnostic catalogue, profile dispatcher, or evidence harness. Do not create a parallel source of truth.
- Preserve canonical source, persistent identities, revision binding, fail-closed replay, deterministic rendering, and the separation between evidence and authority.
- A local test is local evidence. A hosted run is hosted evidence. A generated artifact is not a support or publication decision. Record exactly what ran and do not widen claims.
- Any new wire format, ABI, descriptor, carrier, evidence document, or compatibility surface requires an explicit versioned schema, independent decoder/replay, hostile-input tests, bounds, and nonclaims.
- Never grant ambient filesystem, process, network, model, signing, payment, commit, publication, or registry authority merely because a declaration, receipt, test, or proof exists.
- Keep changes reviewable. Split implementation into commits that each preserve the existing test suite, and explain any known-answer update field by field.
- Do not weaken or delete a failing regression to obtain green CI. Fix the cause or narrow the admitted feature with a stable diagnostic.
Definition of done
The implementation, tests, documentation, status wording, diagnostics, and evidence agree on one exact scope. Targeted tests pass, the applicable repository quality gate passes, hostile cases fail with the documented stable reason, and no unrelated public surface or authority is widened.
Activity
style(semantic-discovery): apply cargo fmt to the new module
feat(context): add goal-aware, token-budgeted semantic task context (#…
docs(semantic-task-context): separate the tokenizer-refusal paragraph…
test(semantic-task-context): prove the exact budget boundary holds un…
wsdt commentedon Sep 12, 2026
Audited against
main; one required-test gap closed (mergee84d5bad, commit7f8b5ba0). Not closing — and the issue's own title overstates what exists.The title claim, checked directly
This issue is called "model-token-budgeted semantic context compilation".
src/semantic_task_context.rsbudgets in exactly two units:byte-v1—text.len(), UTF-8 bytes, reported"exactness":"exact", documented in code as "not a model token".lexical-v1—agent_economics::lexical_tokens, itself documented atsrc/agent_economics.rs:3as "deliberately not a model tokenizer", always reported"exactness":"approximate".No model-tokenizer adapter exists or is invoked anywhere, and
CompilationBudget::newrefuses any other tokenizer name withSPX-Z803rather than approximating one. That refusal is the correct behaviour — this issue's own out-of-scope list forbids "using approximate token counts while claiming exact budget adherence" — but it means a model-token budget is not delivered, and no comment on this issue should say otherwise. The neighbouringcompact_semantic_projectionwork took the same care today: it measured wire bytes and explicitly declined to call that a token claim.HUMAN_BLOCKED: needs a tokenizer dependency decision.A real BPE/tiktoken-style counter needs a crate not inCargo.toml, which a worker may not edit. The honest refusal already in the code is not a gap to paper over — it is the right behaviour pending that decision.Caller check
grep -rl "semantic_task_context" src→src/lib.rs(thepub modline) andsrc/compact_semantic_projection.rs, where all three mentions are doc comments, not calls. NoCompilationGoal/CompilationBudget/CompilationSeed/compile(usage outside the module and its tests. No CLI subcommand reaches it —src/cli/context.rshandles only the pre-existing Project-manifest byte-bounded path. There is no MCP or SDK module in this repository at all.Scaffolding, by the same measure as five other modules audited today: its only callers are its own 17 tests and the crate's
pub modline.Criterion summary
Met: deterministic selection (
identical_goal_revision_and_budget_produce_byte_identical_output); structured goal with natural-language text treated as data (seed_reason_text_never_influences_selection_or_budget); whole-unit omission with no mid-unit truncation; per-seedomitted_budget_exhaustedwith exact cost; compiler-owned closure rules (delegated toagent_context_v2_json); no additional authority (takes only an already-parsed&Program, no I/O).Added this pass:
multi_seed_budget_is_enforced_at_its_exact_boundary_under_lexical_tokenizer. The issue's required tests ask for exact-boundary and max+1 coverage "with multiple tokenizers"; onlybyte-v1was covered.Not met: summarization — the concept does not exist, selection is binary include/omit, so every "summary" criterion is vacuous. No target-operation or policy parameter. Seeds come only from explicit stable IDs — no diagnostics, paths, requirements, or query results.
goal_digestis a real cache key, proven sensitive to revision/tokenizer/budget/goal, but no cache store exists, so "stale entries are rejected" cannot be tested — there is nothing to reject from. No CLI/MCP/SDK exposure. No benchmark against #85, which is itself unresolved.Evidence class: local re-runnable.
cargo test --locked -p semaprax --lib semantic_task_context→ 17 passed, 0 failed.wsdt commentedon Sep 13, 2026
Pushed to main in
4103e23f1471fd0689c9dca01d147e42d3a53e85.Extended the existing
compact task-contextroute with ordered multi-seed goals, per-seed priority/reason, exact revision binding, and explicit byte-v1/lexical-v1 accounting. Goal text stays untrusted explanatory data. Parser resource caps are enforced before source loading; the legacy one-seed route retains byte-identical output.Byte accounting is exact and lexical accounting is explicitly approximate; neither is represented as real model-token accounting. Real tokenizer integration, richer semantic facts, cache/service integration, and task-level evidence remain open. #85 was not worked on. Keeping this issue open.
Local evidence: 8 compact CLI tests and 10 help/catalog tests, including legacy byte-identical output. The combined library/binary compile passed. No full quality profile or hosted CI run was performed for this batch, per the requested implementation-first workflow.
feat(task-context): add seed suggestion, a real cache key, and tokeni…
wsdt commentedon Sep 18, 2026
02e84428— steps 2, 5 and 8 advanced; issue stays openAudited the implementation sequence against current
main(superseding theissue's
ae25c6a4baseline) and closed three gaps with executable tests.32/32
semantic_task_context::tests pass.policy(depth/max_bytes/max_nodes/direction) andbudget.tokenizer_digest. Still nouncertaintyfield; requirement/test facets depend on step 4suggest_seedsranks by word overlap overcrate::doc::documentname/description, andsuggest_seeds_never_influences_a_goal_that_does_not_explicitly_include_itproves a suggestion never reaches a compiled bundle unless explicitly listedgraph::agent_context_v2_jsonowns contracts/ownership/effects/types facets and the resumable-frontier omission reasonsmulti_seed_budget_is_enforced_at_its_exact_boundary[_under_lexical_tokenizer]). Summarization is ABSENT: units are only ever included or omitted, never shortenedcache_key,TaskContextCache,compile_cached, with no-stale-reuse tests across revision, tokenizer, policy, budget, seed priority/reason, andaccess_scopeThe security dimension worth calling out
The issue lists "context caching can leak source across authorization
boundaries" as a failure case.
different_access_scopes_never_share_a_cache_entrycloses it by test:
access_scopeis part of the cache key, so two callers withdifferent authorization can never be served each other's bundle.
cache_key_changes_with_every_declared_dimensionguards the key itself againsta future field being forgotten.
Goal text remains untrusted data. Hostile query text was verified inert beyond
contributing plain words to the overlap ranking, and never becomes source or
execution authority.
Nonclaim — read this before treating token budgets as model-exact
No real model tokenizer (cl100k or similar) exists in this crate. Adding one
requires a new Cargo dependency. "Exact" mode today means an exact UTF-8 byte
count under
byte-v1, explicitly labeled as not a model token;lexical-v1isexplicitly labeled approximate. Nothing here presents an approximate count as
exact model-token budget adherence — the issue forbids exactly that, and the
labeling is the reason this was not quietly closed.
TokenizerId::algorithm_digest()was added to defend the remaining drift risk:a SHA-256 of each unit's versioned identity, distinct from its short name, folded
into the goal digest and cache key. An algorithm change that forgets to bump its
name now produces a key mismatch instead of silently reusing a
differently-counted bundle.
Residual scope — what remains
Cargo.tomldecision).step 7's deeper omission reporting that depends on it — both need
src/graph.rs.src/cli/compact.rs.wsdt commentedon Sep 18, 2026
Closure audit @
e9ab9242— NOT CLOSEABLE.Current implementation is byte/lexical accounting only. A real model tokenizer needs a
Cargo.tomldependency decision (maintainer call). Also absent: summarization, CLI/MCP exposure, and seed integration across diagnostics/tests/requirements.wsdt commentedon Sep 18, 2026
Audit update — the "current state" section above is stale.
This issue's "Current state at the audit baseline" (baseline
ae25c6a4, 2026-09-11) says "Context is not yet selected from a natural-language/structured goal, model tokenizer, task history, or assurance impact." The goal/tokenizer-budget half of that claim is no longer accurate.src/semantic_task_context.rsimplements a multi-seed goal (explicit stable IDs plus priority; an untrustedreasonstring proven never to affect selection or budget, viatests::seed_reason_text_never_influences_selection_or_budget), an honestly labeled token budget (byte-v1exact;lexical-v1marked"exactness":"approximate"; unknown tokenizer refused withSPX-Z803), and deterministic whole-seed inclusion/omission under that budget (order-independent, pertests::seed_list_order_does_not_affect_output). It is exposed viasemaprax compact task-context <file> <stable-id> [--goal text]. Gate:cargo test --locked -p semaprax --lib semantic_task_context::(32 tests). This is local evidence, not a hosted CI run.What genuinely remains: seed detection is stable-ID-only — no diagnostic/path/requirement/query-result seeding. No real content summarization (the module only ever includes or omits a whole typed unit, never a compressed substitute). No cross-seed deduplication (two seeds with an overlapping closure each pay the full cost independently, by design, to avoid re-deriving the single-seed engine's facet rules). Requirement/test/candidate-diff facets are not integrated into the closure itself. None of this is externally blocked; it is unstarted scope.
fix(diagnostics): keep a fictional test code out of the installed cat…