Repository navigation
trading: keep the SDK quote-stream mute watchdog off for extended-hours runs - #812
Conversation
…rs runs The transport passed data_timeout=quote_timeout to alpaca-py's StockDataStream for every run. alpaca-py 0.44.0 (DataStream.__init__ and _consume) then closes the socket itself and reconnects after that many seconds without market data, and documents the option as off by default because a legitimately quiet subscription would otherwise reconnect periodically. Each such close reaches _connection as quotes_disconnected, and both runners stop on any freeze reason other than stale quotes (mover_runner.py, runner.py). In thin extended-hours sessions the whole subscription can stay quiet longer than the quote timeout: trial 20260923-post-extended-hours logged the 3 s reconnect and stop, and Account 2's 2026-10-06 recovery logged "no data received on data stream for 30.0s, reconnecting" followed by quotes_disconnected. quote_stream_data_timeout() keeps the watchdog at quote_timeout for regular-session runs and the SDK default (None) for runs that allow extended hours. No limit changes: quote_max_age_seconds, the quote_stale watchdog and the order-time quote age still gate every decision on freshness. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…tor test; narrower fallback wording The command center's #812 review (GPT read at 96e0ccc: two P2, one P3): - P2: CI could miss a regression in the real constructor wiring, because both constructor assertions were SDK-gated. A new ungated test runs the real constructor with its SDK seams stubbed (client, protocol guard, stream classes, the scoped DataFeed import) for (False, 30) and (True, None); reverting only the wiring now fails it in SDK-free CI. - P2: saved receipts omitted the tolerated transport freeze. The transport records its selected data_timeout (quote_data_timeout) and exposes stream_health(): frozen, reasons, fresh_quotes and data_timeout_seconds, snapshotted as stop() begins, before teardown. Both runners keep it in their outcomes (runner.py's failure path too, before recovery replaces the port), and the mover receipt's native allowlist forwards it. - P3: the README and the helper docstring say a persistently mute transport stays frozen until a new transport is built, and that recovery builds one only for residuals, not after an accepted overnight hold. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
…fore the final hot-file commit Co-Authored-By: Claude Opus 5.5 <[email protected]>
… (hot-file protocol: every hot-file edit in the last commit) Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
Claude session native-agent-stack-5f: landing at head Observed main Required checks at this head: 7 pass . Unresolved review threads: 0. |
|
Claude session native-agent-stack-5f: post-merge observation. Landed as |
#812 and #809 both change transport.py, tests/test_adaptive_paper_transport.py and source-hashes.json. transport.py and the test file merged without conflict (#812's quote_stream_data_timeout, stream_health and stop() snapshot; #809's readiness reasons and the snapshot's from None). source-hashes.json takes main's copy with the four changed entries recomputed from the merged bytes (recovery.py, transport.py and both test files), and manifests/evidence.json takes main's registry with this branch's five owned rows re-registered (hot-file protocol). Co-Authored-By: Claude Opus 5.5 <[email protected]>
Content resolutions (W1 and W1b both rewrote Grafana's status text): - manifests/stack.json grafana.freshness: W1b's text (13.2.3 qualified in an isolated prefix; host activation separate) plus main's NativeStack2604 release hold (open regression reports grafana/grafana#133835 and #133856, docs/decisions/2026-10-04-2604-e2e-fix-wave.md:117). - observability/README.md: main's two OpenTelemetry Collector rows (deployed 0.161.0, selected 0.162.0) and W1b's Storage-and-viewing row, with the same release hold added. - catalogs/landscape/upstream-snapshot.json: main's two added limits, then W1b's scoped_refreshes block. Generated and derived outputs, through their unchanged generators: docs/new-host-grand-list.md, the component evidence matrix, and the new-WSL handbook outputs with one receipt regeneration entry (handbook_regen.py; main's 25 entries and the branch's kept). The land-slot fix: the architecture catalog's 134 pin_source line citations re-pointed from tests/test_architecture_pin_source.py's own expected_ranges() (51 sources changed), which CI had found stale. Registry: main's copy, the branch's 37 owned paths registered and its two receipts carried (hot-file protocol). Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ontext (#809) ### Scope - **What this PR changes:** a recovery receipt now records why recovery failed, not only the exception type. - **Why:** account 2's 2026-10-06 recovery receipts said only `TransportError`, which hid a stream-readiness and quote-freshness failure. - **Base commit:** origin/main at branch time. - **Lane:** `lane:trading`. - **Owned paths touched:** - `blueprints/us-equities/adaptive-paper/recovery.py`; - `blueprints/us-equities/adaptive-paper/transport.py`; - `blueprints/us-equities/adaptive-paper/source-hashes.json`; - `tests/test_adaptive_paper_recovery.py`; - `manifests/evidence.json` (the last commit). **What changes:** - **`recovery.py`.** `errors` keeps its stable codes, so every consumer and test is unchanged. A new `error_details` list gives each failure's: - `code` (the same as the `errors` entry) and `type`; - `message`, only when the engine authored it (a `transport` or `safety` exception); - `context_type` of the chained or suppressed context. Raising `from None` still keeps `__context__`. The context's message follows the same engine-only rule. - **What never enters a receipt:** foreign exception text (SDK, network, OS). It can quote a request or a credential; the existing no-secret test (`test_ambiguous_exit_is_retained_never_retried_or_rejected`) enforces this. - **Redaction (`_text`):** home paths and long tokens are redacted over the whole text, and only then is the text stopped at 300 characters, so a token that crosses the cut never survives as a fragment. The engine's own words stay readable: lowercase words joined by `_` or `/`, such as `quotes_subscription_rejected` or `authentication/subscription/quote`. Any other long token, with digits, capitals, `+`, `=` or `-`, is redacted. - **Cancellation** records `{"code": "recovery_cancelled", "type": "CancelledError"}`, so every `errors` entry has a detail. - **`transport.py`:** - the start-readiness failure names its health freeze reasons, for example `quotes_disconnected,quote_stale`; - `snapshot()` still raises `from None`, now with a comment saying why. No caller's traceback or log prints the failing read's foreign text, and `__context__` still gives the receipt that read's type. - **`source-hashes.json`:** pins the changed engine files and tests. The frozen trial engine (`dca821cc`) is unchanged. This takes effect only for a later engine pin. ### SOTA sources - The engine's own recovery and transport contracts: - `blueprints/us-equities/adaptive-paper/README-recovery.md` (receipts, `needs_attention`); - `README-transport.md` (health freeze reasons); - the existing receipt tests in `tests/test_adaptive_paper_recovery.py`. - Python's exception-chaining semantics (`raise ... from`, `__cause__`, `__context__`, `__suppress_context__`): https://docs.python.org/3.13/library/exceptions.html#exception-context - The incident: the command center's items `task-native-agent-stack-5f-20261006T212903Z` (item 4) and `task-native-agent-stack-5f-20261006T213803Z` (item 4). A diagnostic run of the frozen engine with only the error label formatted showed the hidden reasons: `stream authentication/subscription/quote readiness failed`, `quotes_disconnected`, `quote_stale`, and `SafetyError: recovery_quote_not_fresh`. ### Evidence-class table | Claim | Evidence class | Command / receipt | | --- | --- | --- | | `error_details` carries engine messages and contexts, and redacts foreign text | local_integration | `tests/test_adaptive_paper_recovery.py`: `test_receipt_carries_the_message_and_its_context` (chained and suppressed), `test_engine_authored_context_is_kept_and_redacted`, `test_a_clean_recovery_has_no_error_details` | | The review fixes: each subscription reason survives in the receipt; unknown long tokens are still redacted; redaction runs before the cap; cancellation has a detail, also with a teardown failure | local_integration | `test_readiness_failure_keeps_each_subscription_reason` (all three reasons), `test_engine_codes_stay_and_other_long_tokens_are_redacted`, `test_redaction_runs_before_the_length_cap` (`message` and `context_message`), `test_task_cancellation_stops_transport_and_retains_ambiguity`, `test_cancellation_then_teardown_failure_keeps_both_details` | | A snapshot failure keeps the foreign read out of tracebacks | local_integration (engine venv with alpaca-py; CI skips it) | `tests/test_adaptive_paper_transport.py`: `test_snapshot_failure_keeps_the_foreign_read_out_of_tracebacks` (`__cause__` None, context suppressed, `traceback.format_exception` without the foreign text) | | The new tests detect the reviewed defects | negative control | the new test files against the previous head `2cdd1faaf`: 8 recovery assertions fail (the three reasons, the token rule, both boundary fields, both cancellation details), and the transport test fails on the chained `ConnectionError` | | No existing behavior changes | local_integration | CI-equivalent venv: all adaptive-paper modules, 1248 tests, with the 5 `credential_race` failures below; engine venv: the transport module, 117 tests, with 3 errors that are identical on main (no `exchange_calendars` in that venv); trading gates and source hashes: 63 OK | | 5 `test_adaptive_paper_credential_race` mutation-driver failures are environmental | local_integration | the same 5 fail on a fresh `origin/main` worktree in this host's venv; CI is the authority | ### Local commands run ``` $ <CI-equivalent venv>/bin/python -m unittest discover -s tests -t . -p "test_adaptive_paper_*.py" Ran 1248 tests ... FAILED (failures=5, skipped=358) # the 5 credential_race mutation-driver failures, also on main $ python3 scripts/validate.py {"components": 69, "hashed_files": 10355, "profiles": 4, "receipts": 214, "status": "passed"} $ python3 scripts/evidence_manifest.py --check {"files": 10355, "status": "passed"} ``` ### Decision record The command center's items cited above, and its review of this PR at `2cdd1faaf` (ACK after one P2 and two P3 fixes, and a check that foreign causes stay out of logs). The stream self-close fix is separate: #812. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
### Scope Complete the reopened C5 source audit: review six names and three instruction documents, retain native defaults for the names, and apply their nine current disposition rows. The unchanged main-catalog watch returned nine unreviewed items; the candidate now returns zero with all document hashes matching. - Base commit: b4e1fa6. Rebase-only after #812 landed; all four C5 content files are byte-identical to the initially published345b8d00. - Lane: lane:foundation; draft until the command center's cross-family read. - Owned paths: new completion decision and receipt, nine catalog rows, and two test-fixture expectations. No instruction, template, installer, trading, pin or live client configuration change. - A21 places completion in this separate PR. #805 stays exactly at23d4366e; its partial decision and recorded observations remain unchanged and are linked by immutable path/commit. - Shared-file neighbours: #802, #813, #770, #769 and #764. Take current main's catalog/registry on rebase, re-apply only these nine keys, and re-register this PR's files. Rebase after #805 lands; no peer branch edits. ## SOTA sources - [openai/[email protected]/d27764b8 — guardian feature defaults](https://github.com/openai/codex/blob/d27764b82f7118f674371e6d6e76271d9d606edb/codex-rs/features/src/lib.rs#L1666-L1688): both UnderDevelopment/default false. Installed0.160.1 features list agrees. - [Tagged Guardian history guard](https://github.com/openai/codex/blob/d27764b82f7118f674371e6d6e76271d9d606edb/codex-rs/core/src/guardian/reviewer_config.rs#L48-L62) and [root-handoff declaration](https://github.com/openai/codex/blob/d27764b82f7118f674371e6d6e76271d9d606edb/codex-rs/features/src/lib.rs#L333-L334). Unchanged scenario files are source-reviewed, not executed. - [Codex0.160.1 release note](https://github.com/openai/codex/releases/tag/rust-v0.160.1), published2026-10-05, and [anthropics/claude-code@fbe20e00 —2.1.292 changelog](https://github.com/anthropics/claude-code/blob/fbe20e00e2851fc01506f54f98a8f0b875af3847/CHANGELOG.md#L6), alongside the installed-client source proof retained by [#80523d4366](https://github.com/seathatflowsinourveins/native-agent-stack/blob/23d4366e26910ae43783518bd6807625a9e5e409/evidence/artifacts/ns2604-context-audit-20261006/review.json). - Current vendor bodies, reread2026-10-06: [Claude memory](https://code.claude.com/docs/en/memory), [skills](https://code.claude.com/docs/en/skills#skill-content-lifecycle), [environment variables](https://code.claude.com/docs/en/env-vars), [CA trust](https://code.claude.com/docs/en/network-config#ca-certificate-store), and [official Codex AGENTS Markdown](https://learn.chatgpt.com/docs/agent-configuration/agents-md.md). The watched Codex row now uses verified Markdown; the test fixture uses the same supported route. - [Repository hot-file protocol@e28d0eec](https://github.com/seathatflowsinourveins/native-agent-stack/blob/e28d0eec112527ac02659ac23753533b8ed39a73/docs/lanes.md#L94-L105) and [disposition schema](https://github.com/seathatflowsinourveins/native-agent-stack/blob/e28d0eec112527ac02659ac23753533b8ed39a73/docs/upstream-surface-watch.md#L174-L193). ### Evidence-class table | Claim | Evidence class | Command / receipt | | --- | --- | --- | | Nine source decisions complete; native overrides declined | source_review | New completion review.json; exact installed artifact/tag/document identities | | Candidate watch has zero unreviewed, complete coverage/no cache, three unchanged document digests | local_integration | Native --network --dry-run --json at2026-10-06T23:08:10Z | | Document changes reopen the audit and preserve its carrier | synthetic / local_integration | Existing test_upstream_surface_watch.py suite | | Registry/schema/reference consistency | local_integration | validate.py; no live provider/GPU execution | ### Local commands run All substantive commands ran at nice19/ionice3. - python3 scripts/upstream_surface_watch.py --check-dispositions: exit0,311 rows. - python3 scripts/upstream_surface_watch.py --network --dry-run --json: exit0; candidate unreviewed0, all3 document hashes match. Before application: exit0/unreviewed9, retained separately. - python3 -m unittest discover -s tests -p test_upstream_surface_watch.py: exit0,114 tests,6 skips for optional cached upstream-input checks. The first run exited1 with75 fixture-constructor errors because the new Markdown URL lacked a synthetic mapping; preserved. Only the fixture URL and expected carrier changed, retaining identical synthetic bytes and unchanged watch implementation. - Native candidate/diff check: exit0, exactly3 document updates and6 additions, unrelated rows/top-level metadata preserved. - python3 scripts/validate.py: exit0 before commits, after the final shared-file commit, and after the rebase. Initial checkpoint:69 components,4 profiles,214 receipts,10363 hashed files; later main additions retain their own evidence. Two earlier watch usage failures exit2 before fetch because the long worktree could not fit the160-character details summary. Both are retained; a supported per-invocation Worktrunk short owned path/no-hooks resolves the command without saved settings changes. ### Decision record docs/decisions/2026-10-06-harness-context-budget-completion.md and evidence/artifacts/ns2604-context-audit-completion-20261006/review.json. Keep native defaults; revisit on a supported release and qualified host need, or any changed instruction-body digest. Previous reviewed body bytes were unavailable, so no historical semantic delta is invented. Source review does not establish runtime acceptance or token savings. ### Host evidence No evidence/hosts file, host installation or platform-status change. Local catalog-candidate zero is separate from landed-main or deployed-client state. ### Checklist - [x] No workflow or action changes. - [x] No credential values/raw conversations/private active configurations committed. - [x] No new paid hosting or billing surface. - [x] Peer-owned files/worktrees and immutable observations preserved. - [x] Preserve unrelated rows, update the three existing document rows by key, and append only newly reviewed name rows. - [x] Shared catalog and evidence registry are committed last.
Scope
data_timeout) only for regular-session runs. A run that allows extended hours keeps the SDK default, which is off.quotes_disconnectedfreeze, and the runners stop on that freeze.67c6b6f94.lane:trading.blueprints/us-equities/adaptive-paper/transport.py,runner.pyandmover_runner.py;blueprints/us-equities/adaptive-paper/README-transport.md;blueprints/us-equities/adaptive-paper/source-hashes.json;tests/test_adaptive_paper_transport.pyandtests/test_adaptive_paper_mover.py;manifests/evidence.json(the last commit).The mechanism, from source:
DataStream._consumecloses the socket itself and reconnects with backoff afterdata_timeoutseconds without market data. Its constructor docstring leaves the option off by default, "since a legitimately quiet subscription (for example news or infrequent bars) would otherwise trigger periodic reconnects".transport.pypasseddata_timeout=quote_timeoutfor every run: 30 s for the mover, and the 3 s quote age forrunner.py._connection, which freezes health asquotes_disconnectedonce the stream had been acknowledged.mover_runner.pyandrunner.py, the"stale" not incheck).Observed:
20260923-post-extended-hours. Its README records the 3 s reconnect followed by a stop, and left the remedy open.no data received on data stream for 30.0s, reconnecting, thenquotes_disconnected. In that run only SPY ever quoted.What changes:
quote_stream_data_timeout(quote_timeout, extended_hours_allowed)returnsquote_timeoutfor a regular-session run (unchanged behavior) andNone, the SDK default, when the run allows extended hours. The constructor passes it to the quote stream only. The orders stream has no such option.README-transport.mdstates the rule, the cost and the evidence.source-hashes.jsonpins the changed engine file and test.Nothing else changes:
quote_stalewatchdog and the order-time quote age (3 s) still gate every entry and exit on freshness.dca821cc) is unchanged. This takes effect only for a later engine pin.Review fixes (the command center's GPT read at
96e0ccc95: two P2, one P3):QuoteStreamDataTimeout.test_the_constructor_passes_the_policy_to_the_quote_streamruns the real constructor with its SDK seams stubbed (client, protocol guard, stream classes and the scopedDataFeedimport), for(False, 30)and(True, None). CI runs it, so reverting only the constructor's wiring fails it.quote_data_timeout) and exposesstream_health():frozen,reasons,fresh_quotesanddata_timeout_seconds(null is the SDK default).stop()begins, before teardown, so every outcome path reads the terminal state.stream_healthin their outcomes;runner.py's failure path reads it before recovery replaces the port. The mover receipt'snativeallowlist forwards it.The cost, for review.
quote_stale, with no entry and no quote-priced exit, until a new transport is built. Recovery builds one only for residuals.quotes_disconnected, then a stop. The stop came sooner.Not claimed:
recovery_quote_not_fresh, because OPCH never printed a quote no older than 3 s. This PR changes neither.SOTA sources
v0.44.0(commitcc4cb3b7ba50ae250e621983c2779047fb16bb28), the installed pin:alpaca/data/live/websocket.py:DataStream.__init__docstring, lines 93-99;_consume, lines 231-283;_run_forever's stale-reconnect backoff;alpaca/data/live/stock.py:StockDataStreampassesdata_timeoutthrough, defaultNone.websocket.pyis byte-identical to the tag's: sha256dcd0afd9c80fafb18095f02c3d929b98952fa86ae0ac24f4a42be8f0c5977c3efor both.README-transport.md(stream health);mover_runner.pyandrunner.py(the reconciliation stop rule);blueprints/us-equities/adaptive-paper/trials/20260923-post-extended-hours/README.md(the earlier finding).task-native-agent-stack-5f-20261006T213803Z(the stream self-close fix, with a test, through normal review and landing). The 09-23 trial README asks for "a preregistered limits decision, tests and independent review" for an after-hours remedy. This PR changes no limit: it returns an optional SDK watchdog to its upstream default for thin-session runs. The tests are below, and independent review follows the normal path.Evidence-class table
quote_timeout, and extended-hours runs passNoneQuoteStreamDataTimeout.test_regular_session_opts_into_the_watchdog_at_the_quote_timeout,test_extended_hours_keep_the_sdk_default(pure; no SDK needed)ConfiguredQuoteFeed.test_regular_session_quote_stream_keeps_the_data_timeout_watchdog,test_extended_hours_quote_stream_keeps_the_sdk_default_data_timeoutQuoteStreamDataTimeout.test_the_pinned_sdk_default_is_offQuoteStreamDataTimeout.test_the_constructor_passes_the_policy_to_the_quote_streamstream_health()keeps the view taken asstop()beganQuoteStreamDataTimeout.test_stream_health_keeps_the_view_taken_as_stop_began(a freeze added after teardown starts is not in the view)stream_healthtests/test_adaptive_paper_mover.py:ReceiptAndCommands.test_receipt_carries_orders_pnl_and_reconciliation_without_account_datatransport.py(engine venv):AssertionError: 30 is not None. The review-fix tests against96e0ccc95: 4 errors (nostream_health, noquote_data_timeout, no allowlist entry). The currenttransport.pywith only the constructor's wiring reverted, SDK-free:AssertionError: 30 != Noneexchange_calendarsin that venv); CI-equivalent venv with an owned TMPDIR: all adaptive-paper modules, 1247 tests, with the 5test_adaptive_paper_credential_racemutation-driver failures, which also fail on main here (the driver's subprocess interpreter cannot importexchange_calendars, so its pristine baseline fails; CI is the authority); trading gates and source-hash tests: 63 OKno data received on data stream for 30.0s, reconnecting, thenFREEZE_HEALTH reason=quotes_disconnected; the trial config hasextended_hours: true,stream_quote_timeout_seconds: 30, feedsipLocal commands run
Decision record
The command center's item cited above. Whether a new engine pin carries this to Account 2's 2026-10-07 recovery attempts is the command center's decision. Those attempts run the frozen
dca821cc.🤖 Generated with Claude Code