[P2] [SPX-AI-025] Implement catalog-normalizer in Semaprax and pass the independent application oracle

SPX-AI-025 — Implement catalog-normalizer in Semaprax and pass the independent application oracle

Priority: P2 | Workstream: application | Import action: New implementation issue
Planning baseline: wavect/semaprax@ae25c6a49dc09ec4613c7a6f52a27daa69dcd3f3 (11 September 2026).
Implementation prerequisites: SPX-AI-018, SPX-AI-020, SPX-AI-022, SPX-AI-023, SPX-AI-024
Important: prerequisites mean the required implementation/evidence is accepted, not merely that its GitHub issue exists.

Outcome

Build the frozen acceptance application with Semaprax-owned business logic and realistic cross-module composition. It must pass the independently frozen oracle, not merely its own generated tests.

Current evidence and distinction from existing work

SPX-AI-018 defines this proposed fixture. The other application issues supply specific composition gaps. Existing ordinary package, JSON, effect and collection implementations are the starting point; this is not a request to reimplement those libraries.

Read these owners first

Directory pointers are navigation roots. Resolve current files with rg --files; proposed new files must follow the current module ownership. The referenced snapshot may be older than the implementing branch.

Bounded scope and non-goals

Use synthetic input only, bounded profile limits and a documented set of supported backends. Keep normalization, duplicate detection, checked aggregation and error decisions in .spx modules. Rust/JS providers may implement host I/O and transport, not perform the business transformation invisibly.

Implementation steps

Create the application under a new clearly labeled example/product fixture path following existing multi-module layouts. Read the frozen oracle contract first and do not modify it to match implementation output.
Split source by genuine responsibilities: bounded record parsing, validation/normalization, collection processing, optional enrichment, output formatting and entry/test composition. Avoid one package per helper merely to bypass limits.
Import current standard-library operations by stable identity. Pin the selected dependency inputs and profile. Document any real unsupported requirement as a blocker with a minimum source fixture; do not silently change the application contract.
Implement exact ASCII trimming, duplicate-id rejection, checked quantity totals, input-order preservation and canonical output. A bounded simple duplicate search is acceptable initially; do not add a map implementation unless measured needs justify it.
Use the typed provider/I/O seam for enrichment and output. Default operation is offline. For a live network smoke, use only an independently approved endpoint and state clearly which physical target is exercised.
Run the same corpus through interpreter, C11 O0/O2 and Core Wasm when the profile claims those lanes. Compare actual output bytes, status, error positions, effect calls and allocation settlement with the oracle.
Add a feature-change exercise and an intentionally introduced bug for later agent trials. Keep their solutions outside candidate-visible files. Record exact reproduction commands and installed-toolchain prerequisites.

Required tests and failure cases

Every frozen valid fixture matches canonical output and totals exactly.
Every malformed, duplicate, oversized, overflow or provider-failure fixture matches the specified category/position and publication behavior.
No success/failure leaks owned records, iterators, callbacks or temporary output.
Imported multi-module execution agrees across claimed backends and is not replaced by host-side business logic.
A hidden input variant rejects hard-coded sample-output implementations.

Test discovery and integration

Use rg --files src tests scripts and the owning harness command cargo test --locked -p semaprax --test <existing-harness> -- --list or its existing library selector. Replace <existing-harness> with the owner established from the references. Register new focused tests in that harness, prove they execute, then run the documented full profile. Do not copy a guessed selector into CI and accept a zero-test success.

Definition of done

A useful multi-module Semaprax application passes its independent oracle.
Its requirements were not weakened to hide profile gaps.
The demo is reproducible with explicit capabilities and becomes a reusable agent acceptance task.

Guardrails for the implementing agent

Read the current AGENTS.md, affected RFC/specification, completion rows and architecture owners before implementation. This pack is pinned planning input, not permission to ignore newer code or decisions.
Use a private worktree and a private CARGO_TARGET_DIR inside it. Do not use git stash, overwrite sibling worktrees, or share compiler/test scratch output. Add tests to the existing owning harness, not a new compiler-linking top-level test binary unless repository rules require it.
Preserve source/graph/schema compatibility where contracted, left-to-right evaluation, lazy operands, ownership transfer, canonical cleanup order and sticky failure. Do not weaken tests, diagnostics, range checks or authority checks to get green.
A proposed new shape/API below is a design target, not a claim that syntax or a command already exists. Run --help and discover exact test selectors. Zero selected tests, unavailable tools and skipped required rows are not execution evidence.
Keep model output as untrusted data; compiler facts and evidence never mint capabilities. Human support/publication decisions, paid model calls and real registry/Git writes require their own explicit authority.
Reuse implemented Agent v2, migrations, iterators, closures, semantic candidates, persistent HIR cache and hosted PG grammar/metadata work. If the requested slice already landed, verify its gate and close only the residual gap rather than building a competitor implementation inside this repo.

Evidence to attach before closing

For the implementation handoff, include: exact starting and finishing commit; changed files and their owning invariants; the pre-fix failure or negative-control result; exact commands, exit status and nonzero selected test counts; target/toolchain/provider identity; artifact/log locations; and remaining limitations. Run the repository's complete required quality profile (scripts/quality.sh full on supported Unix, or the documented platform equivalent) before merge. A local focused pass is not hosted evidence and a green framework is not a completed model experiment. For architecture/ABI/ownership changes, obtain independent semantic review before promotion. Do not mark an execution issue complete using authored tests or documentation alone.

feat(app): freeze catalog-normalizer requirements and independent oracle

test(std): compose bounded Reader/Writer and typed provider outcomes

test(std): prove Reader/Writer + JSON-dec chunking does not corrupt v…

feat(catalog-normalizer): implement CNORM-010/011/012 against the fro…

docs(catalog-normalizer): record a deeper SPX-G171/SPX-H006 gap inves…

Blocked by two compiler capacity ceilings, now characterized — filed as #241

The full CNORM-001..043 pipeline does not compile, and the cause is toolchain
capacity rather than any missing language feature or dependency problem. Details
in examples/catalog-normalizer-project/README.md (e513a504).

The previous diagnosis was wrong

This was believed to be package composition — the SPX-G171 workspace-graph
pre-bound hit when combining std.data.json.dec with a non-trivial application
(#232). The pipeline was therefore rebuilt as a fully self-contained project
with an empty [dependencies], hand-porting the JSON decode and scanner
algorithms into its own namespace specifically to test that theory.

SPX-G171 fires anyway, with zero dependencies, once the combined modules
reach roughly 35–40 KB. apex-supply-chain, a real shipped multi-module
example, is about 10.5 KB. Removing the dependency did not raise the ceiling;
neither did shorter identifiers or restructured contracts.

A second, independent ceiling was also found: SPX-H006 on plain scalar code
with no owned resources — once from a recursive mutual-call cycle (fixed by
porting an iterative bit-packed state machine), and again from ordinary
classification functions with about ten sequential branches, which resisted
function splitting, boolean flattening and match-based dispatch once recombined.
A ten-branch classifier is what validation logic normally looks like.

What was done instead of committing a broken tree

All .spx and manifest changes were reverted to the merged, working state
(byte-identical to origin/main), and the investigation recorded as a README
addendum so a future attempt does not rediscover it. The already-merged
CNORM-010/011/012 slice (decode validity, length bound, ASCII trim over one
isolated JSON string token) is unchanged and still passes.

The oracle is proven able to fail

That was the credibility requirement for this issue, and it is met — 4/4 in
cargo test --locked -p semaprax --test useful_data catalog_normalizer_oracle:::

test	what it proves
`oracle_self_test_passes_the_frozen_corpus_and_every_negative_control`	oracle agrees with its frozen corpus across all six negative controls
`oracle_cli_reproduces_the_frozen_published_example`	CLI path reproduces the published output
`a_deliberately_wrong_candidate_is_rejected_by_byte_exact_comparison`	an unchecked-total candidate and an accept-duplicate-keys candidate are each run and asserted to differ from correct oracle output
`hardcoding_the_published_example_fails_a_hidden_case`	a hard-coded candidate is rejected on a held-back hidden-corpus input

The last one matters most: it closes the obvious cheat of tailoring the program
to the published example.

Status

No full-scope acceptance row could be closed. This is a genuine blocker, not a
shortcut — completion needs either a compiler-side change to the SPX-G171 /
SPX-H006 budgets for legitimate branch-heavy single-project logic, or an
implementation strategy not found in the time available. Tracked in #241.

Evidence class: local, re-runnable. Not hosted.

test(catalog-normalizer): cross-check the Semaprax candidate against …

Audited; one real gap closed (merge b9f1895c, commit fd2b1a96). Not closing — the full CNORM scope remains blocked on compiler capacity.

What existed

A CNORM-010/011/012 slice in examples/catalog-normalizer-project/ (decode validity, the 256-byte label bound, ASCII boundary trim on one isolated JSON string token), from commits 6ca9820b, 647545ae, e513a504 — all ancestors of origin/main. The full CNORM-001..043 scope (batch parsing, schema, duplicates, checked totals, canonical envelope) is documented as blocked by two independently measured ceilings: SPX-G171 (~35–40 KB of project source) and SPX-H006 (refusing roughly ten sequential branches, or a small recursive scanner). Two prior sessions hit that wall after every mitigation they tried; this pass did not re-derive it a third time — same wall, no new information, real cost. Both ceilings are tracked in #241.

The gap that mattered, and it was about the oracle

Every existing .spx test compared against bytes copied from frozen oracle fixture JSON, and the oracle-proof harness explicitly disclaimed owning any candidate cross-check. So nothing ran the built Semaprax candidate and the independent oracle against the same input at test time — the two agreed only in the sense that someone had once transcribed one into the other.

That matters because independence is this issue's entire point: an oracle that shares a code path with the implementation proves the implementation agrees with itself. Here the oracle is tests/oracle/catalog_normalizer/oracle.py — Python, independently authored, sharing no code path with the Semaprax candidate. The independence is real; it just was not being exercised.

Added (tests/useful_data/catalog_normalizer_oracle.rs, new semaprax_candidate submodule, 6 tests, no .spx source touched):

Five tests wrap each of tests.spx's exact label tokens — four valid, one deliberately unterminated — in a minimal schema-valid record, invoke oracle.py live (not --self-test, not a static fixture), and assert the oracle's own computed label and malformed_json category agree with what the candidate's token_valid/trimmed_matches claim.
One test runs the actual compiled candidate through semaprax test semaprax.toml, so the slice is proven against real interpreter execution rather than source verification alone.

Evidence class: local re-runnable. cargo test --locked -p semaprax --test useful_data catalog_normalizer_oracle::semaprax_candidate:: → 6 passed, 0 failed.

Remaining for closure: CNORM-001..043, behind either a compiler-capacity change (#241 — a maintainer decision) or an implementation strategy neither this nor two prior sessions found.

docs(catalog-normalizer): reconfirm the SPX-G171 margin with a sharpe…

Add private durable source CLI and catalog batch gates (Refs #113, #116…

Pushed to main in ea330594: the catalog Project now implements input byte/record/line bounds and terminal-LF record counting in a dedicated batch module. The application gate runs all seven named source cases through interpreter, C11 O0/O2, and Core Wasm twice; canonical round-trip preserves comments, and an intentionally wrong lone-LF implementation is rejected. The frozen independent oracle/cases are unchanged.

The workspace graph also gains a final conservative construction retry that retains all AST/HIR costs and charges the largest sequential temporary imported-function clone. Review caught retained Vec backing storage; stub contract vectors now actually release it. Six unit and nine builder-boundary tests pass, including prior receipt/retry behavior, exact bounds, and nested refusal. The owning pure-projection source audit passes.

The full catalog parser still exceeds the cumulative graph construction limit in the larger application probe; this change does not claim to unlock it. JSON schema/records, error positions, duplicate IDs, checked totals, canonical all-or-nothing output, and enrichment remain unfinished. #124 stays open. No full suite or hosted run was repeated; focused local application gate passed.

Add durable priced source accounting and bound uncached graph peaks

Compiler prerequisite pushed in fbef4a8cab7bdc9ef79fb744a5f6e697b7011910 (from f2070b15); #124 remains OPEN.

Ordinary imported function stubs now clone only the retained signature instead of cloning and discarding provider bodies/contracts. Existing fitting graph receipts remain unchanged. After older estimates refuse, an uncached-only fallback charges all retained HIR plus the peak single synthetic AST (including generated defaults); cached frontends keep summed retention. The 18 MiB limit is unchanged. Independent review checked the allocation lifetimes and phase boundary; seven prebound tests, ten builder-limit tests, source audit, module-size and formatting checks passed.

The current expanded catalog application still fails SPX-G171 at graph admission; only its independent oracle id-boundary test passes. Those application/source-test changes were deliberately excluded from the commit. This compiler improvement does not prove full catalog normalization or application completion. Next diagnosis will distinguish initial prebound refusal from actual core overflow using the exact catalog source set before attempting further changes. No full/hosted gate or performance claim.

Accepted implementation commit 1e9e1d1d (main), following b8e7ccca.

Added a final uncached retained-vector fallback that preserves earlier successful budget receipts. The focused retained-output/loan-plan test passes. Actual expanded catalog-normalizer check still refuses SPX-G171 builder_bytes; this is NOT application admission or issue completion. Remaining structural peak: all synthetic resolved modules stay live before cross-module validation. Next step is compact signature/declaration facts and earlier HIR release, preserving validation.

Validation: private macOS Rust build, one job, focused gates only. Generic priced3 PASS; source live1 PASS; source policy denial/nonrefund1 PASS; retained-vector1 PASS. No full or hosted CI gates; issue remains open for stated residual scope. Logs: .agent-logs/0913-next-*-final.log and .agent-logs/0913-next-graph.log.

Pushed compact cross-module validation and sequential full-HIR release in e56a492b. The final uncached path extracts exact declaration/signature/wrapper evidence and selected output carriers before dropping each full resolver result. All carriers remain charged; original fitting modes/receipts and 18MiB cap are preserved. The initial prebound can now reach the existing compact path after earlier estimates refuse.

Compact proof regression passed. Actual catalog check STILL FAILS SPX-G171; no application completion or memory-success claim. A temporary exact-source diagnostic including bundled dec/io dependencies measured initial admission Ok((17801413,17809002)) and four core construction attempts before refusal. This proves the remaining failure is in core construction, not initial admission. Probe log .agent-logs/0913-catalog-stage-probe.log; probe was removed before commit, existing tests preserved. Existing four-file catalog WIP remains excluded. Worker continues concrete core-phase/debit diagnosis; no cap widening or full/hosted gates.

Adversarial follow-up pushed in baad894 (Refs #124). The first low-memory patch serialized backend lanes inside one test but left a second oracle-module test free to launch the same full project concurrently under libtest. That redundant interpreter invocation is removed; the canonical application gate remains the single owner of project execution, while the independent oracle module retains its byte/category checks. Static formatting and diff checks passed; backend execution remains unrun.

feat(example): prove maximal catalog output capacity

Maximal output-capacity and malformed-input tranche pushed in 0e6d428 (Refs #124). A valid 65,536-byte, 256-record construction produces exactly 65,614 plain bytes and 69,710 fixture-enriched bytes, deriving the source writer arena from request/envelope/per-record maxima rather than sampling. Catalog Wasm alone opts into 69,710; the shared provider keeps 65,536 by default and explicitly refuses +1 and unsafe integers before allocation. All 17 published malformed cases are byte-compared with the frozen oracle, and the candidate source suite now covers the remaining structural/schema categories and exact offsets without increasing its named-test count. Oracle self-test, Rust/SPX formatting, Node syntax and diff checks passed; Cargo/backends were not run.

test(example): execute maximal catalog capacity

Follow-up implementation pushed in 68c941c after the single adversarial review. The maximal 65,536-byte/256-record body is now executed through the actual retained Semaprax normalize and normalize-enriched functions and byte-compared with the frozen oracle. The Wasm host validates i64 allocation length before constructing Uint8Array; actual imported allocation rejects 69,711 and i64::MAX without allocation, while 69,710 allocates and settles. Lightweight checks: Rust formatting, Node syntax, direct allocation-bound probe, and diff check. Full Cargo/backends were intentionally not run in this implementation-focused wave.

test(catalog): bind published corpus to source execution

Pushed 77cca37. The existing serialized catalog-normalizer application gate now loads exactly the four public manifests (36 cases), rechecks frozen outputs against the live independent oracle, and invokes the real normalize/normalize-enriched source entrypoint for every public case without traversing hidden data or constructing a second graph. The focused 36-case oracle inventory test passed. The capacity-heavy multi-backend application gate is authored but was intentionally not run in this storage/RAM-constrained implementation tranche.

feat(bytes): admit expanded owned result capacity

Implementation progress pushed in 3d976386.

This fixes the catalog project’s previously invalid 69,710-byte allocation by separating the bounds coherently across claimed local backends: external roots remain 65,536 bytes; internal owned Bytes/derived views are bounded at 131,072 bytes; aggregate source-derived owned-byte payload is 2,097,152 bytes across at most 32 static owned-byte sites. Source/HIR admission, interpreter settlement, native C runtime, Core-Wasm runtime, mock hosts, owning specs, and boundary regressions were updated together. The catalog source now uses the required literal bytes_zeroed(69710usize), and its oversized record-parser cleanup expression was split into bounded helper calls without dropping cases.

Focused local evidence:

byte-capacity exact/+1 unit selector: 1/1 passed
source/HIR 69,710 admission selector: 1/1 passed
Core-Wasm 131,072/+1 hostile boundary selector: 1/1 passed
formatting/static checks passed

The owning end-to-end catalog selector now clears the former SPX-T271, cleanup replay, aggregate-payload, and static-site failures. It proceeds into execution but still reaches the existing 100,000,000-step interpreter ceiling on published case label-length-256-bytes-boundary-success after ~346s. I did not inflate fuel or weaken the frozen oracle, so #124 remains open for that execution-efficiency residual and the later full evidence gates.

perf(catalog): eliminate normalization prefix rescans

Pushed 9ae0837 as measured partial progress. Catalog normalization now carries record ranges forward through validation and writers, removes nested line-start prefix rescans, scans known UTF-8 ranges once on the hot path, reuses computed count/total values, and emits escaped strings through packed forward cursors instead of decoding each prefix again. Static project verification passed (hash 4fe0c49fafccafc6fc08c28febe0a0caaa8af4d4b5025983146623a40520ab53), along with format/diff checks. I am not claiming the boundary is fixed: the exact owning selector still FuelExhausted on records-count-256-boundary-success after 458.12s. The remaining dominant cost is the required O(n^2) exact decoded-ID duplicate comparison; the oracle, accepted bounds, and fuel limit were not weakened or raised.

perf(catalog): stream duplicate id comparison

Pushed aaacc2e. Replaced duplicate-ID comparison’s decoded-length plus restarted equality traversals with a paired forward decoder cursor that checks exact decoded equality and the 1..64-byte bound in one pass without allocation or changing the stable same-id identity. Static canonical formatting passed. The project check emitted no diagnostic but did not finish within two 30-second observation windows and was stopped to respect the constrained machine; I am not claiming the expensive catalog gate passed. This is implementation/performance progress, not closure of the remaining boundary fuel regression.

Implementation update: I exercised a larger execution-efficiency tranche against the single owning application gate, but did not push it because it did not clear the existing admission budgets.

What was tried without weakening limits:

a fixed source-offset cache for prior IDs plus an ASCII decoded-ID fast path;
a smaller variant that removed the owned cache and avoided duplicate decoder calls.

What the focused gate proved:

the owned cache initially violated the loop admission rule and, after restructuring, pushed live semantic-graph construction past the existing 64 MiB ceiling;
the lean variants fit source parsing but crossed the existing program-wide cleanup-replay work ceiling or live graph construction ceiling before backend parity could run;
no graph, cleanup, input, or interpreter-fuel ceiling was raised, and no test/oracle was weakened.

The unsuccessful tranche has been removed from main. #124 remains open with the original 256-record execution-fuel residual. The next viable implementation needs to reduce work inside the existing graph shape rather than add another cache/branching surface.

Navigation Menu

Repository navigation

[P2] [SPX-AI-025] Implement catalog-normalizer in Semaprax and pass the independent application oracle #124

SPX-AI-025 — Implement catalog-normalizer in Semaprax and pass the independent application oracle

Outcome

Current evidence and distinction from existing work

Read these owners first

Bounded scope and non-goals

Implementation steps

Required tests and failure cases

Test discovery and integration

Definition of done

Guardrails for the implementing agent

Evidence to attach before closing

Blocked by two compiler capacity ceilings, now characterized — filed as #241

The previous diagnosis was wrong

What was done instead of committing a broken tree

The oracle is proven able to fail

Status

What existed

The gap that mattered, and it was about the oracle

24 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[P2] [SPX-AI-025] Implement catalog-normalizer in Semaprax and pass the independent application oracle #124

Description

SPX-AI-025 — Implement catalog-normalizer in Semaprax and pass the independent application oracle

Outcome

Current evidence and distinction from existing work

Read these owners first

Bounded scope and non-goals

Implementation steps

Required tests and failure cases

Test discovery and integration

Definition of done

Guardrails for the implementing agent

Evidence to attach before closing

Activity

wsdt commented on Sep 12, 2026

Blocked by two compiler capacity ceilings, now characterized — filed as #241

The previous diagnosis was wrong

What was done instead of committing a broken tree

The oracle is proven able to fail

Status

wsdt commented on Sep 12, 2026

What existed

The gap that mattered, and it was about the oracle

wsdt commented on Sep 13, 2026

wsdt commented on Sep 13, 2026

wsdt commented on Sep 13, 2026

wsdt commented on Sep 13, 2026

24 remaining items

wsdt commented on Sep 21, 2026

wsdt commented on Sep 21, 2026

wsdt commented on Sep 21, 2026

wsdt commented on Sep 21, 2026

wsdt commented on Sep 21, 2026

wsdt commented on Sep 21, 2026

wsdt commented on Sep 21, 2026

wsdt commented on Sep 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions