Skip to content

Sparse mover quotes freeze the transport (exits and recovery stop); attempted submit under a frozen transport has no operator resolution #215

Closed

Description

@seathatflowsinourveins

Sparse mover quotes freeze the transport, so exits and recovery stop; an attempted submit under a frozen transport has no operator resolution

Evidence: blueprints/us-equities/adaptive-paper/trials/mac-2026-09-24-mover-c/ (in #205). The trial ran on Alpaca paper on 2026-09-24, 14:52–16:01 ET, and is needs_attention and not flat. Lane: trading.

1. One sparse required symbol freezes the whole transport

AlpacaPaperTransport counts as ready and healthy only while every required quote symbol has a quote younger than quote_timeout (3 s in the mover config, at most 30). For the mover trial:

  • 15:49 ET. The data websocket reconnected. PFSA and SRZN marks then stayed stale, and the transport froze (adapter_frozen).
  • 15:58. X1 never sent, because PFSA and SRZN were waiting on no_fresh_quote.
  • 16:01. At the hard flatten, APUS's sell was refused before the wire, even though APUS was quoting continuously.
  • After the close. recover cannot start. It requires every held symbol to be fresh at once, and in a 30 s sample SRZN had 0 quotes while PFSA and GRML had 15–16 s gaps, which fails even at the 30 s maximum.

The 2026-09-23 after-hours trial stopped on the same rule (trials/20260923-post-extended-hours).

Possible directions:

  • Judge freshness per symbol for exits: sell any leg with a fresh quote and hold only the stale ones.
  • Let recovery price from a REST latest-quote read within the ledger's 3 s gate instead of requiring every stream to be warm.
  • Keep the global freeze only for integrity failures (event loss, identity conflicts).

2. An attempted submit during the freeze needs an operator resolution the engine does not have

  • 16:01:26.787. GRML's hard-flatten sell was reserved, and its HTTP attempt was recorded 2 ms later.
  • What followed. No broker observation; the node stopped.
  • Broker state. Alpaca's client-ID lookup returns 404 at 16:24 and 16:31. There are no open orders, and the positions match the ledger.
  • Recovery. It fails closed every time, with owned intent absent from complete snapshot (transport.py:949), then snapshot incomplete; admissions remain frozen (:956). The message came from recover_diag.py in the record.

This is correct by the safety rules (absence alone never retires an attempt), but it leaves four paper positions with no supported path to flat. Options:

  • a reviewed operator command that retires an attempted intent as broker_absent with recorded evidence (client-ID 404 plus a complete order-history snapshot since trial start, at least N minutes after the attempt), followed by recover; or
  • a documented broker-side close plus a ledger re-baseline.

Separately, a submission attempted while the transport is frozen should be refused before the wire with a not_sent guarantee, as APUS's was. GRML's attempt came about 150 ms before APUS's pre-wire refusal (16:01:26.789 versus .937).

Current state (this Mac's paper account)

APUS 37, SRZN 6, GRML 13 and PFSA 67 are held; the mover ledger is recovery_only. Nothing was overridden. The next mover trial on this account is blocked until this is resolved.

Activity

  1. seathatflowsinourveins commented on Sep 30, 2026

    @seathatflowsinourveins
    CollaboratorAuthor

    Tracked from here on in #543 (part 2: the ambiguous GRML intent with no broker record; the trial C run-loop stall; F1; plus a pre-wire recovery-exit refusal and unmodeled broker FEE activities, both measured on 2026-09-29/30). #486 closed this issue through its closing link but fixed only part 1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    lane:tradingTrading lane: sim, paper and live north star

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions