Skip to content

Live runner: service operation, cumulative report export and live redelivery - #331

Merged
luisleo526 merged 13 commits into
mainfrom
lv/runner-service
Oct 5, 2026
Merged

luisleo526 merged 13 commits into
mainfrom
lv/runner-service

Conversation

@luisleo526

Copy link
Copy Markdown
Collaborator

Summary

Makes the native live runner pineforge-live operable as a long-running service and proves that its cumulative results equal the batch. Runner only: the engine library, its headers and the corpus are unchanged.

  • Cumulative report, persisted and exportable.
    • A versioned cumulative report (equity, open profit, closed trades and the engine's report fields) is committed in the same transaction as each input, as append-only deltas.
    • pineforge-live report --ledger L [--deployment D] [--at-input N] rebuilds it exactly, at any committed cursor and after a restart. An optional --report-jsonl mirror is also available.
    • Over identical confirmed bars, warmup, strategy and settings, the runner's actions, equity, open profit and trades equal run_backtest_full. The one documented difference is a cursor in the middle of a script-timeframe bucket.
  • Health and readiness.
    • --status-file PATH is an atomically replaced JSON holding liveness, readiness (validated strategy and warmup, recovered ledger, verified input prefix, no unhealed input gap, storage below budget) and metrics (committed input, last sequence, source lag, queue and ledger bytes, report cursor, per-target pending deliveries).
    • Writes are throttled to --status-interval.
    • pineforge-live probe --status-file PATH --max-age S [--ready] returns 0 or 1 for container exec probes.
  • SIGTERM/SIGINT drain. The runner finishes or rolls back the current atomic message, persists its cursor, outbox and report, stops intake, drains deliveries for at most one total timeout, and exits with a documented code.
  • Configurable feed deadlines: --feed-idle-timeout and --feed-message-timeout. The defaults are unchanged.
  • Live redelivery. An atomically created request in a control directory asks a running runner to resend failed or selected deliveries. There is no new network listener. The same delivery_id and the append-only audit apply, and computation is never blocked. Malformed or unremovable control entries are skipped and reported, never fatal. Offline redeliver still works and refuses while a runner holds the lock.
  • Storage budget. --max-ledger-bytes: when it is exceeded, readiness goes false and the runner stops cleanly after the current message, dropping nothing.

Measurements

Workload Ledger before → after Throughput before → after
3,000 one-minute bars 1,051 MB → 11 MB 68 → 502 msg/s
30,000 one-minute bars 9.7 GB → 110 MB 22 → 481 msg/s
30,000 ticks 1.85 GB → 51 MB 172 → 314 msg/s

Recovery at 3,000 bars falls from 30 s to 1.3 s. The status file costs under 1% of throughput.

Verification

  • Report equals batch for 5 strategies at every aligned cursor and after a restart, with tamper detection on replay.
  • Runner ASan+UBSan and ThreadSanitizer: 16/16 each.
  • Every SIGTERM phase is tested, along with probes, deadlines, live redelivery including malicious control entries, and the storage stop.
  • Engine bundles pass; corpus parity is 312/312 with 0 moved.
  • Two independent reviews: MERGE-WITH-FIXES, then fixes, then MERGE.

🤖 Generated with Claude Code

self.send_response(503 if mode == "fail" else 204)
self.send_header("Content-Length", "0")
self.end_headers()
except OSError:
self.send_response(200)
self.send_header("Content-Length", "0")
self.end_headers()
except OSError:
self.wfile.flush()
else:
time.sleep(3)
except OSError:
result = predicate()
if result:
return result
except (FileNotFoundError, json.JSONDecodeError, sqlite3.OperationalError):
@luisleo526
luisleo526 force-pushed the lv/runner-service branch 2 times, most recently from 889f4c5 to 069ca7f Compare October 4, 2026 23:30
@luisleo526
luisleo526 merged commit bfcf50a into main Oct 5, 2026
17 checks passed
@luisleo526
luisleo526 deleted the lv/runner-service branch October 5, 2026 01:11
luisleo526 added a commit that referenced this pull request Oct 5, 2026
* docs: release notes and version scoping for v1.2.0

CHANGELOG: the Unreleased section becomes 1.2.0 (2026-10-05), a minor
release that adds the compiled execution-capabilities extension to the C
ABI (PF_CAPABILITIES_API_VERSION, strategy_capabilities_api_version,
strategy_capabilities_receipt). One bullet per change since v1.1.0 with
its PR (#328, #330, #331, #332, #333, #334, #335, #336, #337, #338 and the
session-clock change, written #VCP until its PR number exists), the
recipe additions under State hashes (#328, #330, #335; domain tags and
epoch unchanged), Report keys (docker/run_json.py is byte-identical to
v1.1.0's, so no key changes) and Migration.

Version scoping: README, install and public-contract pages name v1.2.0
and codegen 1.2.0; "since the next release" and "fixed on main" in the
capabilities guide and the runner guide become 1.2.0 and 1.1.0 (#325).
Lines that state what an earlier release shipped stay.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* docs: 1.2.0's grades from the facts tokens, #339, and codegen's license

- CHANGELOG: the session-clock change is #339 (merged as 52292db), in place
  of the #VCP token of 3082fbd; the "What a script author sees" bullet of
  3082fbd now states release 1.2.0's grades (registry baseline
  pineforge-parity-baseline-20261005-engine-52292db9: 7,970 excellent / 19
  strong of 7,989, none below strong) instead of the main scoreboard at
  6b77f06.
- README: the release sentence names 1.2.0 and renders releases[1.2.0], and
  the main scoreboard renders the same baseline, from pineforge-release
  facts/facts.json at 0c2f6838 (the facts export for the 1.2.0 release).
- README, LEGAL.md, docs/pages/index.md: pineforge-codegen is under the
  PineForge Source License 1.1 from its release 1.2.0; releases up to 1.1.0
  keep the PolyForm Noncommercial terms they shipped with.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant