Skip to content

hftbacktest 2.4.4 as the independent sim-lane cross-check engine: pinned install, upstream tests, L1 feasibility - #232

Merged
seathatflowsinourveins merged 8 commits into
mainfrom
claude/hftbacktest-crosscheck-20260925
Sep 25, 2026
Merged

seathatflowsinourveins merged 8 commits into
mainfrom
claude/hftbacktest-crosscheck-20260925

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

What

This installs hftbacktest 2.4.4 (Rust 0.9.4, MIT) as the independent cross-check engine for the simulation lane. The choice comes from the 2026-09-25 research convergence. The repo's own record (manifest-20260922.json) had deferred hftbacktest until intraday execution realism mattered. The 180–900 fills/min sim-capacity lane (blueprints/us-equities/sim-capacity/, in a separate PR) meets that condition. It sits alongside the selected NautilusTrader 2.0.0rc5 engine and does not replace it. Everything is in blueprints/us-equities/sim-crosscheck-hftbacktest/.

  • Supported install.
    • A requirements.lock hash-pinned by uv pip compile --generate-hashes: 43 packages, including hftbacktest 2.4.4 with wheel sha256 7df29f3f…, which matches PyPI.
    • Installed with uv pip sync --require-hashes.
    • pip-audit --disable-pip --no-deps found 0 findings on both the PyPI and OSV databases.
    • The lock is registered in .github/osv-scanner-lockfiles.json.
  • Upstream's own tests at tag py-v2.4.4 / rust-v0.9.4 (a244a142), unchanged.
    • The only Python test needs a data file upstream does not publish. It fails, and the failure is recorded as-is.
    • cargo test -p hftbacktest --no-default-features --features backtest --lib passes 22/22. These tests cover the L3 FIFO queue and the depth implementations, not the exchange fill logic.
    • The default-feature build fails for lack of libclang. Six examples are broken upstream, two of them unrelated to the live feature.
    • The Rust toolchain was installed with rustup and uninstalled again afterwards.
  • L1 feasibility, our own integration check on a synthetic fixture: feasible with limits.
    • Top-of-book data decides fills and expiries for marketable IOC, FOK and GTX orders.
    • Trade prints with no side are dropped, so the queue model never affects fills on raw L1 data. Passive orders fill only when the opposite quote reaches them.
    • The Ellis-Michaely-O'Hara (at-quote) and Lee-Ready (midpoint) side-inference rules are implemented, with a tape where they disagree. Inferred sides restore the trade-through fill.
    • Two biases are measured and documented: zero queue ahead on L1, and taker fills in the partial-fill exchange model never depleting the touch.
    • NoPartialFill and PartialFill are shown to diverge on oversized orders.
  • Plan for the cross-check.
    • Replay a bounded sample of the sim-capacity lane's private SIP data through both engines.
    • Before comparing, pin the settings to match, or tabulate how they differ per engine.
    • Compare fill rate, median time to fill, share of fills at the touch, and cost.
    • The comparison runs against the sim-capacity commit that merges; that lane now feeds its engine quotes only.
  • Upstream dormancy is recorded as a risk. The last push was 2025-12-23 and there are 17 open issues. The overturn checkpoint is "no upstream commits by 2026-12-23".

Review

Independent Claude review, three rounds. Round 1 found a CI blocker, where a lockfile was not registered with osv-scanner, plus overclaims. Round 2 found wording issues. The final round confirmed:

  • every scenario can be derived by hand from the Rust source at the tag;
  • the venv rebuilds exactly from the lock;
  • all hashes reproduce.

The Codex CLI is at its usage limit until 2026-09-30, so this PR has no cross-family review.

Checks

  • Tests: tests/test_sim_crosscheck_hftbacktest.py runs on system python (6 pure-Python tests pass, 17 skip cleanly) and in the venv (23/23 pass).
  • Coverage: test_osv_lockfile_coverage passes.
  • Validators and scans: the six validators and gitleaks (history and dir) pass.
  • Host cleanup: the venv and the Rust toolchain were removed after verification. The README gives the rebuild commands.

🤖 Generated with Claude Code

Scout and others added 6 commits September 25, 2026 00:12
…im lane

Pins hftbacktest==2.4.4 (py-v2.4.4/rust-v0.9.4, MIT) into an isolated venv
with a hashed uv lockfile, records upstream's own Python test result at the
pinned tag (fails on a missing private fixture; no CI runs it either), and
records that no Rust toolchain was available to run cargo test (unresolved,
not silently skipped). Builds and hand-verifies a synthetic L1 (Alpaca-SIP-
shaped, single-level depth + side-less trades) feasibility fixture against
hftbacktest's actual exchange-model source: marketable IOC fill/expiry is
decided purely from top-of-book depth (feasible), while trade-driven queue
depletion requires an aggressor side that SIP consolidated trades do not
carry (feasible-with-limits, demonstrated empirically, not just asserted).
Does not touch blueprints/us-equities/sim-capacity/ (parallel builder's work);
records the planned cross-check against it once that lane lands on main.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Installs a user-level Rust toolchain (rustup, rustc/cargo 1.98.1) and runs
hftbacktest's own #[test] suite at the pinned py-v2.4.4/rust-v0.9.4 tag:
`cargo test -p hftbacktest --no-default-features --features backtest --lib`
passes 22/22 (0 failed), covering the queue/depth-model unit tests this
cross-check's L1 conclusions rely on. The bare default-feature `cargo test`
still fails to build (the `live` feature needs libclang + C headers this
host lacks and has no sudo to install); recorded as a distinct, narrower,
unresolved gap rather than papered over.

Re-runs the L1 feasibility fixture (Scenarios A/B/C) under both
no_partial_fill_exchange and partial_fill_exchange: results are identical,
so the feasible-with-limits verdict is unchanged on this fixture (order
sizes never exceeded the touch's quoted quantity, so the two exchange
models could not diverge here -- recorded as an open gap, not claimed
covered). Adds a Lee-Ready aggressor-side inference step, presented as
proposed data preparation (not an hftbacktest feature), and shows it
restores trade-driven queue depletion on the same synthetic fixture.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…cker

HIGH: .github/osv-scanner-lockfiles.json was missing a covered_by_lockfile
entry for the new requirements.in (mover-v3 pattern), which failed
test_osv_lockfile_coverage.py. Added and re-verified.

MEDIUM, all grounded in a fresh read of the pinned Rust source:
- Corrected "8 files, 22 tests cover..." to the actual 7 files run under
  `-p hftbacktest` (connector/src/utils.rs is a different crate, excluded);
  explicitly states none of the 22 tests exercise exchange fill logic,
  IOC/FOK/GTX, ProbQueueModel, or the L2 depth-update path this fixture
  depends on, and that --lib skips 2 doctests (types.rs, latency.rs).
- Moved unrun claims (FOK, GTX, interpolated/per-symbol latency,
  FlatPerTradeFeeModel) out of "feasible": FOK and GTX are now actually run
  (Scenario G) and nonzero fees are now actually verified (Scenario H);
  interpolated/per-symbol latency and FlatPerTradeFeeModel are moved to
  "not exercised".
- Fixed the side-less-fill mechanism: depth alone (on_bid_qty_chg) never
  invokes is_filled() and never fills a resting order (Scenario D2, new,
  corrects an earlier wrong claim); passive fills need the OPPOSITE quote
  to cross (Scenario D3, new); trade-through fills are side-gated too, not
  just queue depletion (Scenario D1, new). Added the reviewer's three probes
  as tests.
- Renamed the mislabeled "Lee-Ready" rule to the correct EMO at-quote rule
  (Ellis, Michaely & O'Hara 2000; comparison at NBER w14158) and implemented
  genuine Lee-Ready (midpoint-based) separately; fixed the flat-tick
  fallback to reuse the last known direction; added a tape that actually
  exercises the tick-test branch and shows EMO/Lee-Ready can disagree.
  Documented the optimistic zero-queue-ahead bias for orders placed away
  from the touch (Scenario E, measured) and corrected the "exact
  single-level replacement" wording to price/size only, not queue position.
- Moved tests/ to the repository's top-level tests/ directory (the only
  blueprint that had its own tests/ subdirectory, which collided with the
  top-level tests package during unittest's bare discovery and so never ran
  in CI); made numpy a lazy import so the pure-Python EMO/tick-test
  classification tests run unconditionally instead of only "skipping
  cleanly".

LOW:
- A backtest-only, all-targets Rust build fails on six examples, not four;
  two (algo, custom_evhandling) are broken independent of the `live`
  feature or libclang, so installing libclang alone would not get a full
  `cargo test -p hftbacktest` to pass. Recorded the absence of
  rust-toolchain.toml/Cargo.lock upstream and this run's resolved versions
  (cargo tree, generated Cargo.lock hash).
- Added Scenario F: NoPartialFillExchange and PartialFillExchange genuinely
  diverge once order size exceeds the touch's quoted size; the cross-check
  plan now picks PartialFillExchange deliberately instead of calling the
  two equivalent.
- Relabeled the 69.2ms latency figure as the sensitivity point it is, not a
  calibration, and planned a sweep instead.
- Re-checked the maintenance precondition via the GitHub API: upstream's
  default branch is unchanged since the already-reviewed 2025-12-23 commit;
  py-v2.4.4 remains the latest tag.
- Reworded wheel provenance (PyPI-served digest match, not independent
  attestation); fixed the package count (43, not 42) and other nits.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…ions

Adds Scenario I: PartialFillExchange never depletes self.depth from a fill
(three back-to-back 100-share IOCs against an always-100 touch all fill in
full, position 300), matching upstream's own doc comment and the reviewer's
probe exactly. Corrects the cross-check plan to require matching (or an
explicit per-engine difference table for) liquidity/queue settings: reads
the actual sim-capacity branch (claude/sim-capacity-20260925, via git show,
not touched) confirming liquidity_consumption=True/queue_position=True and
the cited 180/900 fills-per-minute budgets, versus sim-paper-compare's
liquidity_consumption=False/queue_position=False; picks PartialFillExchange
deliberately as the closest-available (not equivalent) configuration.

Corrects the NautilusTrader taker-queue claim against the actual rc5
matching_engine/mod.rs source (snapshot_queue_position is only called on
the passive/maker branch, never for a taker fill) instead of an unverified
paraphrase. Fixes "coincide on A-D and G" to "A-D and G1/G3/G4" (G2, the
oversized FOK, genuinely diverges). Corrects the NBER 14158 citation
(Asquith, Oman & Safaya 2008, a short-sale misclassification study, not
"Diether, Lee & Werner" nor a literature review). Restricts the EMO rule to
exact-quote equality (was >=/<=), with new boundary tests confirming a
print beyond either quote now falls to the tick test. Actually runs EMO
inference on the D1 trade-through tape (with a seed trade so the tick test
has history) instead of asserting the result unrun.

Nits: FOK/IOC now imported from hftbacktest.order instead of hardcoded;
upstream publishes via maturin-action, not twine; the old test directory's
CI-invisibility is attributed to missing __init__.py under blueprints/,
verified with a minimal reproduction, not a name collision; algo.rs's build
failure is a Cargo examples/ packaging issue (a shared helper with no main,
also auto-discovered as its own example), not API drift like
custom_evhandling's; the "status understates the fill" caveat is scoped to
IOC only (FOK expires with exec_qty exactly 0); Scenario E is relabeled
hand-verified with its full trace, since ProbQueueModel's early-return path
never touches the probability formula here. Re-runs pip-audit with
--disable-pip --no-deps, now covering all 43 packages (was 42). Corrects
the stale "removal_commands_not_yet_run" cleanup record to reflect the
cleanup already executed, and records this round's own rebuild-then-delete
of the venv and gives rebuild commands in place of the deleted activate path.
Reframes the maintenance check as a dormancy risk (~9 months, 17 open
issues) with a concrete 2026-12-23 no-further-commits overturn checkpoint,
rather than reading as a pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
The cross-check plan and receipt had described the sim-capacity engine
configuration as of 0e0740e, which fed trade ticks into the engine. From
f47160b onward that runner feeds quotes only: in rc5, L1 trade ticks overwrite
both book sides, so 41% of IOC fills beat the NBBO when trades were fed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Scout and others added 2 commits September 25, 2026 02:03
… docs

- NautilusTrader's liquidity_consumption keeps a per-level consumed-size tally
  that resets when displayed size changes; it does not decrement the book.
- 180-900 is the sim-capacity submission budget per minute, not a fill rate.
- The cleanup log records commands and dates only; no exit codes were recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…t; note the quotes-only change

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins
seathatflowsinourveins merged commit 6568f47 into main Sep 25, 2026
23 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/hftbacktest-crosscheck-20260925 branch September 25, 2026 06:15
seathatflowsinourveins pushed a commit that referenced this pull request Sep 25, 2026
…y branch

evidence.json was rebuilt from main's copy. This branch's convergence-record
entry and its ten sim-capacity file registrations were re-applied. The CI lock
path this branch still carried was not re-registered, because #233 renamed it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Sep 25, 2026
… SIP quotes, native rate limits and full US-equity fees (#235)

A sim-lane capacity run, labelled infrastructure evidence and not strategy evidence. It replays real Alpaca SIP data for 8 symbols, 2026-09-24 14:00-14:30Z, through NautilusTrader 2.0.0rc5's simulated exchange.
- Setup: a quotes-only engine feed (rc5 L1 trade ticks overwrite both book sides); liquidity consumption; 70 ms latency plus a sweep; SEC, TAF, CAT and Elite commission fees; the native rate limiter at 180/min (paper parity) and 900/min (Elite).
- Elite results: 859/873/882 fills/min, with all 30 of 30 minutes at or above 180; 0 fills beating the touch; 0 rejects; the NautilusTrader recount agrees; about $243/min simulated cost.
- Paper parity: 171/175/180 fills/min.
- Convergence record: NautilusTrader is the selected engine, hftbacktest is the cross-check (#232), and the alternatives are rejected with overturn conditions.
Four rounds of Claude review (Codex at its usage limit until 2026-09-30); all 31 mutation probes are caught.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant