Paper engine: #215 part 2, run-loop stall, F1, a recovery exit refused before the wire, and unmodeled FEE activities

Tracks what #215 still leaves open after #486 (part 1) closed it through its closing link, plus two defects measured on 2026-09-29/30 on the frozen engine b528bb55 (which contains #486). Paper only; no live endpoint.

Open items

Sparse mover quotes freeze the transport (exits and recovery stop); attempted submit under a frozen transport has no operator resolution #215 part 2: the ambiguous GRML intent with no broker record. Recovery exits each held symbol on its own fresh quote (#215 part 1) #486's description keeps it open: an attempted submit under a frozen transport has no operator resolution when the broker never recorded the order.
Trial C run-loop stall (Sparse mover quotes freeze the transport (exits and recovery stop); attempted submit under a frozen transport has no operator resolution #215). Also kept open by Recovery exits each held symbol on its own fresh quote (#215 part 1) #486.
F1: adaptive-lane recovery port. runner.py's recovery port still gates start() on every held symbol (Recovery exits each held symbol on its own fresh quote (#215 part 1) #486's description, "until the F1 follow-up"; README-recovery.md states it).
ext-20260929-4: a recovery exit refused before the wire after a transport gap. In the after-hours trial on account 2 (2026-09-29), the exits triggered with reason transport_gap at 17:08:39 ET. BKYI's exit filled; BEX's exit was reserved and never attempted (recovery_proven_never_attempted); the engine handed off with no_exit_progress. The recovery then sold BEX 24 (rec-ext-20260929-4-0000001, filled 40.44). Its IOVA 68 exit (rec-ext-20260929-4-0000002, limit 14.38) was reserved 70 s into the 120 s recovery and recorded intent_not_sent with reason transport_proven_not_sent (receipt error SubmissionNotSent). The ledger's request table shows only 4 reads and 1 submit in that recovery, so the 150/130 per-minute caps did not defer it. The transport sanitizes the refusal to "submission prevented before HTTP request", so its cause is not recorded. The candidates are ledger.validate_pending in Controller.before_request (quote freshness or price bounds at submit time) and the transport's wire guard, while the paper log shows data-stream gaps. IOVA 68 stayed open overnight.
Unmodeled broker FEE activities block reconciliation (measured 2026-09-30). Alpaca paper posted FEE activities for 2026-09-29 sells on account 2: REG -0.19, TAF -0.27, CAT -0.01 USD. runner.reconcile fails closed on the -0.47 cash gap (cash_mismatch_or_unmodeled_fees), so every recovery of IOVA 68 (17 attempts, 06:00-06:51 ET, 0 orders) ended needs_attention. The engine's README-safety.md still says paper does not simulate regulatory fees. A fix that records FEE activities durably in the ledger's cash accounting is being built on branch trading/engine-fee-activities-20260930; it will link here.

Acceptance for closing

Each item is closed by a merged change with a failing-first test, or by a dated record that it no longer applies, cited here. Item 5 also needs account 2 recovered flat on the fixed engine, with a receipt.

Rule-text record: rebuilt base, Sol-primary clause (a), folded items,…

Trading lane update, 2026-09-30 ~23:05Z (native-agent-stack-ae, formerly 24).

Item 5, unmodeled FEE activities: fixed in #559, which is open.

The PR has 8/8 required checks green at 6db276c2. It is rebased through the hot-file protocol and waits for the OpenHands relock that turns osv-scanner green on main.
The change:
- The engine books Alpaca FEE activities (REG/TAF/CAT and the other documented sub-types) into the ledger's cash, realized P&L and realized loss. Ledger schema 3 migrates one way.
- Each trial opens at a consistent fee checkpoint (FEE list, account, FEE list). Every snapshot, recover and the adaptive next-trial read use that checkpoint's window.
- Legacy trials keep their window, so account 2's IOVA 68 residual recovers after 2026-09-29T20:25:19Z.
Measured. Both paper accounts' fees for 2026-09-29 were read through the change's own allow-listed read (GET only; ids and descriptions not recorded):
- account 1: REG -0.34, TAF -0.30, CAT -0.01 (total -0.65);
- account 2: REG -0.19, TAF -0.27, CAT -0.01 (total -0.47, exactly the gap that blocked 17 of 17 recovery attempts).
Reviews.
- Round 1: Claude Opus and gpt-6.1-sol (ultra).
- Repair: gpt-6.1-sol ultra, test-first.
- Re-checks by both families: pass.
- Both families independently found the same defect in round 1, a fee window and cash baseline taken at different moments. The repair's checkpoint fixes it.

New residuals, recorded here:

A. Fees missing from a refused-start receipt. A fee the start checkpoint books, at a start that begin_next_trial then refuses (for example, a fee that trips a cap), is in the ledger and its fee_recorded event, but in no receipt's fees_recorded. Documented in README-safety.md, Limits.
B. A timing-flaky runner test. tests.test_adaptive_paper_runner test_native_adaptive_policy_reaches_ledger_and_flat_reconciliation fails about 1 run in 24, at the base too (unknown_submit:...:SafetyError near the end of its 2 s trial window).
C. Recovery halts are now permanent. A risk cap that trips during a recovery now replaces recovery_only and is permanent for the ledger, as in a trial. The sell-only exit is still allowed. Tested; documented in README-safety.md.

Item 4, the IOVA exit refused before the wire: the candidate list is refined. Source: the 09-29 after-hours receipt, independently verified.

The intent has submit_attempted 0, no request row, and no request in the previous 60 s. The durable budget writes the row and the flag in one transaction (safety.py:1299-1311). That excludes the 150/130 caps and the wire guard.
validate_pending checks no price bound for a sell.
Not excluded by the files:
- validate_pending's quote_not_fresh (the quote was 0.021 s old at reservation; the refusal time is not recorded);
- the transport's 0.25 s owner-callback deadline on the submit hook (transport.py:1148-1167).
Also found: the paper stage's own in-process forced recovery (mover_runner.py:783-787) marked the BEX exit recovery_proven_never_attempted and ended with a TransportError before any snapshot. Its cause is not recorded.

Next on this issue. After #559 merges, the residual is recovered on the fee-aware engine at the 2026-10-01 open (pre-market attempts from 06:00 ET, then 09:45-10:30 ET). The outcome will be posted here.

Rule-text record: rebuilt base, Sol-primary clause (a), folded items,…

Rule text in every layer: the six standing clauses on AGENTS.md, the …

Token-efficiency profile accepted with the three code-navigation tool…

2026-10-01 findings from the rth-20261001 receipt build (trading lane, native-agent-stack-ae; engine dca821cc, account 1). They are recorded in the receipt on branch trading/receipts-20261001, which is not yet a PR. None changes a trial verdict.

The cause of a forced exit is not saved.
- Trial 1's four legs exited on transport_gap, about 7 minutes before their 1-hour hold ended.
- Trial 2's five legs exited on controller_stop, 0.143 s after the trial's first 30-second broker check and 35 s into the trial.
- No file records the health reason behind either exit. The files rule out a rate limit and an incomplete snapshot. They cannot rule out a stream drop or, for trial 2, a signal to the process.
- Candidate fix: persist the triggering health check, meaning its reason, the last stream timestamps and the snapshot result, in mover-paper.json and the ledger events.
The rung schedule is checked only after broker reads.
- Session 21 was refused with mover_session_beyond_rung_schedule, with no submit or cancel.
- Before the refusal, the paper stage made 13 trading-API reads and the scan read /v2/assets and /v2/calendar.
- Candidate fix: run plan_session's schedule check before any broker request.
A drawdown-ladder lockout spends sessions (protocol, not engine).
- After trial 3 the lineage drawdown was 206.47 of 250 USD (0.826). The frozen ladder gives max_leverage 0 from 0.75, so trials 4-20 each had gross_budget_usd 0.00 and ended completed_no_signals.
- They still counted against the 20-session rung_schedule, so the series stopped at 11:58:50 ET, 3h40m before its last start.
- This goes to the next separately reviewed freeze set: stop the series, or skip without counting, while the ladder gives zero leverage. The 2026-10-02 freezes record it as a known effect only.

Navigation Menu

Repository navigation

Paper engine: #215 part 2, run-loop stall, F1, a recovery exit refused before the wire, and unmodeled FEE activities #543

Open items

Acceptance for closing

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Paper engine: #215 part 2, run-loop stall, F1, a recovery exit refused before the wire, and unmodeled FEE activities #543

Description

Open items

Acceptance for closing

Activity

seathatflowsinourveins commented on Sep 30, 2026

seathatflowsinourveins commented on Oct 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions