Repository navigation
Description
Sparse mover quotes freeze the transport, so exits and recovery stop; an attempted submit under a frozen transport has no operator resolution
Evidence: blueprints/us-equities/adaptive-paper/trials/mac-2026-09-24-mover-c/ (in #205). The trial ran on Alpaca paper on 2026-09-24, 14:52–16:01 ET, and is needs_attention and not flat. Lane: trading.
1. One sparse required symbol freezes the whole transport
AlpacaPaperTransport counts as ready and healthy only while every required quote symbol has a quote younger than quote_timeout (3 s in the mover config, at most 30). For the mover trial:
- 15:49 ET. The data websocket reconnected. PFSA and SRZN marks then stayed stale, and the transport froze (
adapter_frozen). - 15:58. X1 never sent, because PFSA and SRZN were waiting on
no_fresh_quote. - 16:01. At the hard flatten, APUS's sell was refused before the wire, even though APUS was quoting continuously.
- After the close.
recovercannot start. It requires every held symbol to be fresh at once, and in a 30 s sample SRZN had 0 quotes while PFSA and GRML had 15–16 s gaps, which fails even at the 30 s maximum.
The 2026-09-23 after-hours trial stopped on the same rule (trials/20260923-post-extended-hours).
Possible directions:
- Judge freshness per symbol for exits: sell any leg with a fresh quote and hold only the stale ones.
- Let recovery price from a REST latest-quote read within the ledger's 3 s gate instead of requiring every stream to be warm.
- Keep the global freeze only for integrity failures (event loss, identity conflicts).
2. An attempted submit during the freeze needs an operator resolution the engine does not have
- 16:01:26.787. GRML's hard-flatten sell was reserved, and its HTTP attempt was recorded 2 ms later.
- What followed. No broker observation; the node stopped.
- Broker state. Alpaca's client-ID lookup returns 404 at 16:24 and 16:31. There are no open orders, and the positions match the ledger.
- Recovery. It fails closed every time, with
owned intent absent from complete snapshot(transport.py:949), thensnapshot incomplete; admissions remain frozen(:956). The message came fromrecover_diag.pyin the record.
This is correct by the safety rules (absence alone never retires an attempt), but it leaves four paper positions with no supported path to flat. Options:
- a reviewed operator command that retires an attempted intent as
broker_absentwith recorded evidence (client-ID 404 plus a complete order-history snapshot since trial start, at least N minutes after the attempt), followed byrecover; or - a documented broker-side close plus a ledger re-baseline.
Separately, a submission attempted while the transport is frozen should be refused before the wire with a not_sent guarantee, as APUS's was. GRML's attempt came about 150 ms before APUS's pre-wire refusal (16:01:26.789 versus .937).
Current state (this Mac's paper account)
APUS 37, SRZN 6, GRML 13 and PFSA 67 are held; the mover ledger is recovery_only. Nothing was overridden. The next mover trial on this account is blocked until this is resolved.
Activity
Recovery exits each held symbol on its own fresh quote (#215 part 1)
Recovery exits each held symbol on its own fresh quote (#215 part 1) (#…
seathatflowsinourveins commentedon Sep 30, 2026
Tracked from here on in #543 (part 2: the ambiguous GRML intent with no broker record; the trial C run-loop stall; F1; plus a pre-wire recovery-exit refusal and unmodeled broker FEE activities, both measured on 2026-09-29/30). #486 closed this issue through its closing link but fixed only part 1.