Project ANVIL

ANVIL is an experimental general-purpose lossless-compression research project.

Its objective is not to produce another wrapper around Brotli, Zstd, or LZMA. The research target is a materially better compression / encode / decode / memory Pareto frontier, with mechanism-level advances that remain useful on real corpora.

The central research thesis is:

The cheapest byte to encode is a byte the decoder can reconstruct from information it already has.

That means ANVIL studies not only entropy coding, but the representation that exists before entropy coding: exact references, transformed references, reversible reconstruction, bounded generation rules, structure exposure, backend routing, and hardware-conscious decoder execution.

Current status

ANVIL is a research prototype. The bitstream is unstable and the project is not a production replacement for established codecs.

October 7, 2026 — evidence-graded state

Research verdict: 0 verified complete general-purpose Pareto crossings. The frozen Class-A accounting is 435 dominated, 28 degenerate, 5 unresolved front gaps (468 total). There have been real specialized advances, but none changes that classification. Read the research synthesis and open questions for the experimental reasoning, exact claim limits, reference-class distinctions, limitations, and research agenda. The research evidence index maps every important claim to its measured CI source and evidentiary scope.

Lane	Supported finding	Scope / caveat
I9/I10 BWT	Better compressed-byte totals on selected canonical corpora; auxiliary inverse BWT speeds ANVIL decoding	External decode and memory costs prevent complete front crossing
I10 structure	Exact lexical reordering improves a selected receipt dataset under the same Brotli backend	Broad/held-out gates did not generalize; G4 and G5B were adverse
I12–I16 numerical	Local modular innovations and tiled/field-local predictors substantially compress certain synthetic numerical fixtures	Known structural families, fixture reuse, strong q5 and typed-codec competition
I17 work portfolio	Fast selector reduces costly q11 search	Significant byte regret on some nonnumerical inputs
I18 hybrid	Complete synthetic arithmetic archive 19,445 B vs I17 fast 33,204 B; paired decode 381.422 MB/s vs Brotli q5 161.660 MB/s	I18 encode 4.089 MB/s vs I17 fast 11.473 MB/s; no independent real validation
I19 Phase 0/0b	Four exact original response-body snapshots and four derived non-held-out typed streams, 112,620 B verified	No I19 codec efficacy run; corpus admission blocked, UCI reserve unsealed

For I18, corrected Actions 37703141058 is the source-attested synthetic discovery run. For I19, acquisition 37704571734 and semantics 37704897319 verified input identities and projections, not compression performance. I19 source artifacts have finite GitHub retention; an exposed UCI ZIP is not a blinded heldout; numeric vectors do not losslessly reconstruct parent DLY/JSON/ZIP without additional sideband. Workflow success is not scientific frontier certification.

Next gate: source-rights and durable-original custody, another independent discovery lineage, genuinely sealed untouched holdout, frozen typed-codec competitors (Sprintz, FastLanes/FOR/PFOR, OpenZL and matched general-purpose codecs), and preregistered decoder-work / encoder-regret testing. All CPU-intensive processing and tests run through GitHub Actions; local work is limited to lightweight inspection and edits. Current numerical formats (AVI6, AVH1, AVH2) are separate research prototypes, not necessarily implemented by the primary src/anvil.cpp CLI. Do not infer a single released interoperable encoder from their presence in experiment folders.

Historical I9–I10 evidence (retained, not the current frontier summary)

The frozen Iteration-9 checkpoint demonstrated that ANVIL's ratio portfolio can beat the measured Brotli q11 and xz -9e byte totals on the project's canonical Silesia/enwik8 runs, but it produced zero complete Pareto-front crossings because the BWT-heavy path remained far too slow to decode.

Iteration 10 has now closed its first adopt-class improvement: auxiliary-index inverse BWT. On three paired same-runner targets it improved whole-codec decode by 1.364× to 2.339× for a fully charged 2–4 KiB-per-file wire increase. Across canonical Silesia + enwik8, the size-first portfolio pays 22,398 B (+0.00718% of source) with no routing changes. The legacy smaller representation remains the default for size-first mode; auxiliary BWT is retained as a distinct faster-decode Pareto option.

The same-runner external reference-cost gate has now closed that question: I10-1A still wins bytes, but is FRONT-GAP_COST, not a complete reference-front crossing. On Silesia it is 1.72x xz decode time but 4.57x xz peak RSS; on enwik8 it is 3.91x xz decode time with ~599 MiB peak RSS. Decode dependency depth and BWT working-set size therefore remain binding system axes.

A separate Grotli-derived representation lane has now produced ANVIL's first clean same-backend structured representation crossing: on the frozen CROVIA receipt corpus, byte-exact lexical shape/column reordering followed by the same Brotli q11/lgwin30 backend is 21.77% smaller than raw Brotli. The broader untyped G1 gate is still a NO-GO—only one discovery family crossed 5%, and the routed four-family portfolio improved by 1.656%, below the preregistered 3% threshold. G2's typed basis also missed its broad gate, while G3 regionization passed discovery at -5.5984% aggregate but failed its independent DrugQA held-out test at +3.1900% worse than raw Brotli, closing as PASS-G3-NARROW.

The current Grotli anatomy program has narrowed the mechanism further. G4 found NO-GO-G4 for cheap planner substitution. G5A then proved ordering/locality itself is materially causal: with the same scalar-chunk multiset, envelope, and Brotli backend, exact-shape column order was 6.1189% smaller than source order, with 80.19% of that measured ordering effect attributable to the column increment. G5B tested the coarse cross-shape extension and closed ORDINAL-ADVERSE: global same-ordinal blocking was 1.6507% larger than the exact-shape column floor on D1-D4. The next open anatomy question is therefore semantic/path-aware cross-shape compatibility—not ordinal coarsening. None of G4/G5A/G5B authorizes a production transform or a new generalization claim.

Research lineage and execution documents (including historical, deliberately frozen hypotheses) are documented in:

• Scientific synthesis and open questions — October 7, 2026 — current paper-style report, quantitative claim matrix, threats to validity, and proposed experiments

• I17–I18 source-attested adjudication — exact paired evidence and failure history

• I18 corrected quantitative closeout — complete-wire seven-file matrix, timing and RSS limitations

• I19 acquisition/typed semantics closeout — original and derived SHA-256 identities, rights and custody gaps

• I19 real-source experimental design — unexecuted hypotheses and tentative criteria

• I19 bounded oracle mathematical design — not a frozen preregistration

• docs/FRONTIER-RESET-2026-09-23.md — post-I9 synthesis and mechanism frontier

• docs/FRONTIER-RESEARCH-ADDENDUM-2026-09-23.md — post-I10-1A program-synthesis, information-theory, and CPU-native decode research addendum

• docs/I10-EXPLANATION-GAP-ORACLE-DESIGN.md — concrete oracle architecture for identifying which explanatory family ANVIL is actually missing

• docs/I10-GROTLI-REPRESENTATION-SYNTHESIS.md — Grotli-derived adaptive representation compiler architecture: structure discovery, predictor/residual synthesis, exact downstream arbitration, and raw-Brotli fallback

• docs/I10-GROTLI-G0-PREREG.md — frozen first causal experiment: exact record-aligned vXOR+Brotli versus raw Brotli

• docs/I10-GROTLI-G0-CORPUS-FREEZE.md — immutable discovery/validation split and upstream Git-object identities frozen before G0 implementation

• docs/I10-GROTLI-G0-RESULTS.md — G0 closeout: padded row-major vXOR is a decisive no-go; broader typed/shape-aware SRS remains a separate hypothesis

• docs/I10-GROTLI-G1-CEILING-PREREG.md — frozen untyped exact-shape/column causal gate

• docs/I10-GROTLI-G1-CEILING-CORPUS-FREEZE.md — immutable G1 discovery split plus still-unspent held-out DrugQA validation corpus

• docs/I10-GROTLI-G1-RESULTS.md — G1 closeout: broad base NO-GO, but a 21.77% same-Brotli class-specific column-locality win

• docs/I10-GROTLI-G2-RESULTS.md — G2 typed-column closeout: broad discovery NO-GO despite strong D2/D4 specialist wins

• docs/I10-GROTLI-G3-REGION-PREREG.md and docs/I10-GROTLI-G3-RESULTS.md — regionized representation: discovery pass, held-out PASS-G3-NARROW

• docs/I10-GROTLI-G4-PLANNER-FIDELITY-PREREG.md and docs/I10-GROTLI-G4-RESULTS.md — cheap planner-proxy hypothesis closed NO-GO-G4

• docs/I10-GROTLI-G5A-ORDERING-ATTRIBUTION-PREREG.md and docs/I10-GROTLI-G5A-RESULTS.md — same-multiset ordering attribution: ORDER-MATERIAL / COLUMN-DOMINANT

• docs/I10-GROTLI-G5B-ORDINAL-PREREG.md and docs/I10-GROTLI-G5B-ORDINAL-RESULTS.md — global same-ordinal cross-shape proxy closed ORDINAL-ADVERSE

• docs/I10-BREAKTHROUGH-PROGRAM.md — falsifiable Iteration-10 execution program

• docs/I10-AUX-UNBWT-RESULTS.md — closed I10-1A paired timing, byte economics, hardening and ruling

• docs/I10-REMOTE-BASELINE-CLOSURE.md — closed Linux/GitHub baseline and deterministic I9 identity ruling

• docs/anvil-i9-findings.md

• RESEARCH_LEDGER.md

• docs/audit-2026-09-07/06-do-not-reburn.md

What exists today

The codebase contains several research generations under one decoder/format framework:

• block framing with raw fallback and CRC-32;
• exact LZ parsing and multiple match-finder/parser experiments;
• sparse-corrected self-reference (COPY_PATCH);
• implicit transformed copy (TCOPY);
• transformation-invariant candidate experiments (PNRA);
• semantic shape and hot-op representations;
• adaptive arithmetic and static rANS backends;
• multiple rANS precision modes;
• context-switched rANS experiments;
• Huffman / default-with-exceptions / stream-selection infrastructure;
• BWT backend built on pinned libsais;
• Brotli/BWT whole-file backend routing;
• reversible transform experiments;
• strict malformed/truncation checks;
• adversarial/golden/fuzz testing;
• native Brotli/Zstd/xz comparison tooling;
• Pareto/reference-class analysis tooling.

Not every implemented mode is considered successful. Failed mechanisms are deliberately kept in the research record so they are not rediscovered and re-burned.

Iteration-9 checkpoint

The canonical I9 report records the following deterministic byte totals:

Corpus	Input	ANVIL auto	Brotli q11/lw30	xz -9e
Silesia	211,938,580 B	46,446,995 B	49,383,136 B	48,456,100 B
enwik8	100,000,000 B	23,534,368 B	24,810,180 B	24,831,656 B

Those byte results coexist with a major decode deficit. See docs/anvil-i9-findings.md for the exact reference class, provenance, timing grades, and the FRONT-GAP / FRONT-CROSSING rules.

The project does not describe the above as a general Pareto breakthrough.

Research direction — conceptual foundation

ANVIL is moving away from the question:

How do we entropy-code these bytes a little better?

and toward:

How can the decoder explain more output bytes from compact, cheap, deterministic state?

The working research abstraction is a bounded set of decoder-visible explanations:

• LITERAL — explicit irreducible residual;
• COPY — exact temporal reference;
• COPY_PATCH — reference plus sparse correction;
• TRANSFORM_COPY — reference under a compact deterministic transform;
• GENERATE — bounded rule that produces many output bytes;
• REPLAY — reconstruct an encoded/container representation from normalized data plus correction metadata.

This is a research model, not a novelty claim. Each concrete mechanism still has to pass ANVIL's prior-art and quantitative gates.

The first high-EV I9 carry-forward is now closed:

1. auxiliary-index inverse BWT — ADOPTED as an explicit faster-decode Pareto option, while the byte-smaller legacy representation remains the size-first default.

At the time of the earlier I10 research plan, the next isolated engineering experiment was:

2. native bit-exact DEFLATE reconstruction (P4.1), followed by direct roundtrip/fuzz, complete wire/code-size accounting, and same-backend causal measurement.

New mechanism work is expected to begin with anatomy/oracle probes, not full codec modes.

The live successor lane is I19 source-admission and work-constrained numerical reconstruction research; do not mistake the historical I10 next-step statement for an active experiment authorization. See the current synthesis and I19 design links above.

Research discipline

Every mechanism is expected to answer four questions before it becomes a claim:

1. What is the prior-art lineage?
2. What, precisely, is new?
3. Why should it move a real Pareto frontier?
4. Which controlled ablation proves that the mechanism caused the gain?

ANVIL also maintains a strict do-not-reburn register. A correlation, detector hit rate, lower entropy proxy, or synthetic win is not enough. Final value is measured in complete payload bytes plus runtime, memory, decoder cost, and correctness.

Hardware-conscious architecture

The decoder is treated as a dataflow engine, not just a mathematical inverse.

The project actively investigates:

• PCLMUL-backed CRC;
• SIMD-friendly mismatch verification;
• multi-state entropy decoding;
• control/data separation;
• cache-resident decode tables;
• dense integer metadata coding;
• branch reduction and fused semantic operations;
• auxiliary-index BWT inversion;
• runtime ISA dispatch with a scalar fallback.

A wire-format idea that fundamentally requires expensive pointer chasing or branch-heavy interpretation has to justify that cost against the bytes it removes.

Building

Linux

Requirements:

• CMake
• Ninja
• C++20 compiler
• Brotli development libraries for the ratio backend/helper
• Zstd development libraries for the native comparison harness

Example:

cmake -S . -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++
cmake --build build --parallel

The vendored libsais primitive is pinned under third_party/libsais.

Windows

The historical Windows research environment uses clang-cl and Ninja. See docs/CONTEXT.md and tests/host-spec.md for the exact frozen environment associated with old measurements.

CLI

The research CLI evolves with the format. Run:

anvil

for the current option set.

Typical examples:

# research-oriented ratio path
./build/anvil c input.bin output.anv --parse=ratio --ratio-backend=auto

# decode
./build/anvil d output.anv restored.bin

# verify a parser/backend path
./build/anvil verify input.bin --parse=hotop --entropy=rans

Do not assume an experimental mode is a current recommendation merely because it remains available from the CLI.

Correctness

The fuzz harness exercises roundtrips, malformed inputs, decoder-visible registry IDs, BWT goldens, and deterministic format cases.

python3 tests/fuzz.py --exe build/anvil --cases 50

Format changes must add decoder bounds and direct forced coverage. Selection-based coverage is not sufficient.

Benchmarking

CPU-heavy ANVIL research is designed to run remotely through GitHub Actions rather than on the developer workstation.

See:

• docs/github-actions-benchmark-protocol.md — normative statistical/promotion protocol
• docs/GITHUB-ACTIONS-BENCHMARKING.md — operational CI/corpus/publication companion
• .github/workflows/anvil-research-bench.yml

The workflow is manual-only and uploads raw artifacts. Shared GitHub runners are treated as a scout/ranking environment for timing, while deterministic bytes, hashes, and roundtrip results are independently reproducible quantities.

Canonical external corpora are downloaded at CI runtime and verified against pinned manifests; they are not committed to the repository.

Repository map

Path	Purpose
src/anvil.cpp	current encoder/decoder research implementation
FORMAT.md	current experimental wire-format contract
RESEARCH_LEDGER.md	experiment history, numbers, verdicts, corrections
docs/	research agenda, audits, prior-art notes, frozen iteration reports
tests/	fuzzing, corpus manifests, benchmark/verification artifacts
tools/	benchmark, Pareto, corpus, build and analysis tooling
prototypes/	isolated mechanism/oracle experiments
third_party/libsais/	pinned BWT/suffix-array primitive and upstream license

Reproducibility and claims

When documents disagree, prefer the most direct frozen artifact:

1. exact executable/CSV output and hashes;
2. current source + FORMAT.md;
3. explicit gate/ruling documents;
4. RESEARCH_LEDGER.md;
5. old context/brief documents.

The October 2026 I12–I19 lanes are primarily isolated research worktrees and CI workflows, not features necessarily present in the main anvil executable. Every published number requires a specific commit, complete frame geometry, source identity and Actions evidence. Raw-source compression and derived typed-projection compression are distinct questions. An independently reproducible compressed-byte result can coexist with noisy shared-runner performance estimates; an unmeasured memory/code-size dimension remains unknown.

Historical throughput numbers are host- and protocol-specific. Never transplant them to a different compiler/CPU and present them as current performance.

Public research snapshot

The public repository is a sanitized, fresh-history snapshot of the research tree. Local OpenCode state, build products, workstation-specific caches, and private working metadata are intentionally excluded.

The absence of a file from the public snapshot therefore does not imply it never existed in the internal research process.