Skip to content

feat: add causal runtime services and Zipline migration workflows - #164

Merged
stefan-jansen merged 116 commits into
mainfrom
codex/runtime-zipline-migration
Oct 21, 2026
Merged

stefan-jansen merged 116 commits into
mainfrom
codex/runtime-zipline-migration

Conversation

@stefan-jansen

Copy link
Copy Markdown
Contributor

Summary

Add opt-in causal market/history, prepared inputs and calendar callbacks to the existing Engine and Strategy, with a Zipline callback and BarData adapter. Native orders, execution limits, account state and results remain authoritative.

  • Read-only native current/history/session/staleness and instrument views preserve physical clocks, warmup, publication cutoffs and prepared adjustment perspective. Native history remains Polars; legacy Pandas shapes are projected at the adapter boundary.
  • Original callback bodies and idiomatic native ports cover moving-average, OLMAR, minute scheduling, prepared Pipeline panels, registered CSV sources and declared benchmark returns.
  • Native quantity/value/percent orders, conditional orders, partial fills, incremental fees, per-asset costs, volume budgets, trading controls, account controls and session cancellation have explicit methods and full order/account traces.
  • Dated futures metadata, costs, causal volatility-volume inputs and the explicitly selected legacy execution window use native contracts. The existing signed-notional account default is preserved.
  • Documentation examples execute against the built package. Zipline is a reference/preparation dependency, not a production dependency of the migrated run.

Advances #156. Raw corporate-action accounting (#157), variation-cash settlement (#158), dated roll execution (#159) and the final installed migration/Diagnostic closure (#160) remain separate accounting and integration work. Split/roll interaction acceptance is retained until those owning changes are qualified. This PR does not establish full Zipline replacement or change the preserved 0.1.13 release candidate.

Financial and compatibility review

Reference fixtures pin published Zipline Reloaded 3.2.0. The pre-existing performance comparisons retain their own immutable framework versions, including Zipline 3.1.1. These identities are intentionally distinct.

Material differences are documented and tested: pre-session reads do not reproduce demonstrated reference lookahead; total consumed volume cannot exceed actual bar volume; all orders eligible for session-end cancellation expire together; native signed-notional futures cash is distinct from reference variation cash. Original Pipeline expressions and chunked factor computation move to the prepared Engineer route rather than being accepted and ignored.

Verification

Reviewed head: 8f3e5466153f516c37dfa70e56920a8baa8938b5.

  • Complete source suite: 2,947 passed, 55 optional dependency/in-process comparison skips, five benchmark tests deselected. Global and every critical module coverage gate pass, including fill execution at 98.13% statements and result reading at 88.08% branches.
  • Fresh correctness: all 66 synthetic and 17 production comparisons pass with published Specs 0.2.0 against hash-verified immutable reference outputs. All 17 current production output identity sets match the newly measured Backtest outputs.
  • Material runtime changes were measured at b56e5cf with one warmup and ten fresh measured processes for each of the 17 Backtest configurations. Original reference samples are unchanged. Subsequent convenience-constructor, capture-only metadata, validator/docs and packaging repairs have a separate impact review supported by unchanged timed-method ASTs and fresh correctness. No licensed authentication or reference timing rerun occurred.
  • Ruff across source/tests/release scripts, type checks, compatibility snapshot, generated claims/changelog checks and all hooks pass.
  • Pinned dependency and source security scans pass without findings or exceptions.
  • Both distribution formats pass exact contents, reproducible rebuilds and metadata checks. The sdist retains its required fixtures, native examples and original-source attribution.
  • Source and installed-wheel guide examples execute. Strict documentation build, version/source identity and 11,098 rendered content links plus 44 external destinations pass.
  • Final installed wheel: 405 runtime, migration and execution-contract tests pass. Producer workflows use the actual Data c3b9db80 and Engineer 093b72d6 artifacts with published Specs 0.2.0. The isolated production environment has neither Zipline nor PyArrow.
  • Real failing-base/passing-fix evidence covers nanosecond clocks, fee quote state, affordability, history/publication boundaries, contract units and the final three runtime defects. Seven producing-code fault controls also fail the execution/account-export safeguards while the correct version passes.

Hosted supported-platform and package qualification runs on this PR head before ordinary merge. Local passing checks are not a substitute for those required checks.

AI assistance

OpenAI Codex assisted with implementation, tests, documentation and qualification orchestration. The maintainer remains responsible for reviewing the source, independent accounting/timing oracles, real reference workflows, regression evidence and final checks. Historical tests, current tests, retained references and new measurements are recorded separately.

Adopt the immutable Specs temporal contract, keep legacy config payloads readable, and make explicit None select observation semantics. Qualify the next API boundary separately from the retained 0.1.13 candidate.
Regenerate lock metadata offline after the direct immutable source dependency. Preserve all other package versions and source identities; the public locked Zipline builder now succeeds.
Include first-period execution costs in returns and drawdown without adding a synthetic bar. Store the baseline in schema 3 and preserve schema 2 and beta readers. Adds real Engine failing-base regression coverage for issue #155. Also includes the requested maintainer batching guidance.
…xports

Enable explicit bar capture while retaining complete account observation keys, including all-cash dates. Extend the existing schema 3 artifact with optional holdings and strict keys, retain unavailable legacy capture, and preserve equal-time execution order. Advances #155; capital flows and installed consumer qualification remain open.
…ication

Reproduce the shared Python 3.15 failure through uv resolution. Bind selected packages, artifacts, markers and lock declarations while excluding the resolver prerelease permission alone. Preserve raw identities and all 17 accepted measurement records, and keep actual dependency drift rejection. Full suite: 2320 passed, 14 skipped. Fixes qualification of #151.
Apply validated delta/target schedules before decisions using actual marked NAV and preserve pre/post-flow valuations. Chain subperiod returns, retain reported NAV separately from investment wealth, and persist ordered flow applications. Extend accounting invariants to include independently recorded external capital without changing their tolerance. Adds public Engine and artifact regressions and a runnable guide. Advances #155; full qualification and migration integration remain open.
Use a real uv-generated self-contained lock instead of requiring every platform package and build backend in the runner cache. Preserve the failing-base producer operation and the actual dependency-drift rejection control. Addresses the clean-runner failures at 7233675 without changing validator or runtime behavior.
Capture the multiplier and declared currency used for each fill and export signed quantity. Preserve unknown legacy units and cost inclusion rather than reconstructing them from later configuration. Adds actual Engine/Parquet unit checks. Advances #155.
Capture post-fill and last-observed-session holdings through the existing account and fill journal, with explicit completeness and sequence keys. Keep investment returns on the original bar timeline and roll back observations with failed immediate callbacks. Reuse one marked snapshot implementation; preserve a valid zero mark in both carried valuations and open-trade PnL. The zero-mark regression fails twice on b49307a; focused account/lifecycle checks pass. Advances #155.
Keep detailed mark provenance out of ordinary valuation loops and enable timestamp history only when requested. Custom/legacy feeds may omit the new optional capital-events attribute. The real legacy feed run fails before the fix and passes afterward; 39 account/lifecycle/runtime checks pass, including the existing runtime regression gate. Advances #155.
…sions

Add Broker.record using the existing journal, callback transaction and portable artifact encoding. Retain carried values on actual bar keys, preserve nonfinite values, expose decoded records, and reject unsupported values before mutation. Persist the optional versioned component with strict observation checks and a runnable guide. Advances #155 and the record adapter foundation of #156; 131 focused checks pass.
Copilot AI balanced review requested due to automatic review settings October 21, 2026 21:54

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@stefan-jansen
stefan-jansen merged commit c5d4309 into main Oct 21, 2026
43 checks passed
@stefan-jansen
stefan-jansen deleted the codex/runtime-zipline-migration branch October 21, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants