Skip to content

feat: complete reproducible Zipline migration and analytical handoffs - #169

Merged
stefan-jansen merged 159 commits into
mainfrom
codex/reproducible-migration-workflows
Oct 22, 2026
Merged

stefan-jansen merged 159 commits into
mainfrom
codex/reproducible-migration-workflows

Conversation

@stefan-jansen

@stefan-jansen stefan-jansen commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Migration workflows

Closes #160. Closes #156, completing the integration acceptance retained after #164 and the accounting/lifecycle batches #166, #167 and #168.

  • Accept Pandas and eager/lazy Polars prices/context at one boundary. Reject ambiguous/null/duplicate event keys and preserve equal-time source ordering.
  • Consume verified immutable Data snapshots, including captured Engineer feature declarations. Retain the exact source manifest, consumed table map and normalized identities through native result artifacts; reject changed verified frames before callbacks.
  • Add ContinuousFuture/PreparedFutures, continuous_future, current_chain and selected dated-contract reads over Data's causal inputs. The existing native engine still owns orders, fills and accounts. Research history is explicitly registered and cannot become an execution feed.
  • Run offline Data preparation -> Engineer features -> Engine -> saved result -> Diagnostic holdings/activity/returns, plus continuous-futures migration and original Zipline 3.2 result conversion. Document immutable source versions, input requirements and actual method differences. Production examples do not require Reloaded packages.
  • Exercise the same pinned optional producer/consumer cohort in coverage and clean installed Python 3.12/3.13/3.14 CI on Linux, macOS and Windows. Existing locked/platform/package/security gates remain required. The installed acceptance suite rejects every skipped workflow, and coverage preserves and checks its intentionally installed optional cohort.

Executed evidence

Runtime source: c8cf7b1be2c3d3eb8f532f11a4f2ad58a3b19051. Subsequent commits contain tests, CI, documentation and generated qualification evidence; all 112 runtime modules remain byte-identical.

  • Full source suite: 3,270 passed, 9 optional cross-engine skips, 5 benchmark deselections. Global statement/branch coverage 90.67%/81.12%; every critical-module threshold passes.
  • Built wheel on Python 3.14 and independently built/installed sdist on Python 3.12: the final selected suite passes 44 installed migration tests per supported environment, no skips, warnings as errors. Dependency consistency checks pass. The five-table snapshot case verifies split quantity, contribution cash, NAV, signals/context and result round-trip.
  • Five installed Engine -> strict artifact -> Diagnostic account journeys pass, including first-period fees, all cash, long/short, external contribution and minute round-trip. Both documented offline CLIs execute against the installed package.
  • Actual original published Zipline 3.2 captures validate volume and declared first-notice selection, raw fills and additive/multiplicative/no-adjustment histories. Same-session versus preceding-session paired roll marks are explicit choices. No universal arbitrary-chain parity claim is added.
  • Failing-base evidence includes six real panel/CSV key workflows before input repair, ignored captured feature metadata before its fix, and three aliased continuous-query failures on the actual pre-fix wheel. Corrected reads reject the wrong query before orders; the tests pass on the fix.
  • Fresh current Backtest correctness: 66 synthetic and 17 production pairs pass against hash-verified immutable original outputs. Every production output identity equals the accepted measured output. Performance samples and measurement digest remain byte-for-byte unchanged after an explicit nonmaterial-impact source review; no reference performance engine or licensed authentication was rerun.
  • Strict MkDocs, 11,890 rendered links/52 external destinations, installed documentation examples, Ruff, ty, all hooks, reproducible distributions, Twine, locked dependency audit and source security checks pass locally.

The pinned Data and Diagnostic builds declare Python below 3.15; this complete producer/consumer cohort uses Python 3.12 through 3.14. Backtest core Python 3.15 qualification runs separately. Published optional Diagnostic 0.1.4 lacks the new holdings API, so only that newer consumer case requires the holdings capability; the mandatory pinned suite executes it.

The integration sources are recorded in validation/successor-integration-requirements.txt. Diagnostic's source distribution label is 0.1.9; the guide explicitly distinguishes the pinned Git source from the previously published wheel. This PR does not publish or deploy a release and preserves the separately pending 0.1.13 candidate.

Hosted qualification must complete on the final head before ordinary guarded merge.

AI assistance

OpenAI Codex assisted with implementation, documentation, source review, regression reproduction and qualification. The maintainer remains responsible for the changes and reported evidence.

Adopt the immutable Specs temporal contract, keep legacy config payloads readable, and make explicit None select observation semantics. Qualify the next API boundary separately from the retained 0.1.13 candidate.
Regenerate lock metadata offline after the direct immutable source dependency. Preserve all other package versions and source identities; the public locked Zipline builder now succeeds.
Include first-period execution costs in returns and drawdown without adding a synthetic bar. Store the baseline in schema 3 and preserve schema 2 and beta readers. Adds real Engine failing-base regression coverage for issue #155. Also includes the requested maintainer batching guidance.
…xports

Enable explicit bar capture while retaining complete account observation keys, including all-cash dates. Extend the existing schema 3 artifact with optional holdings and strict keys, retain unavailable legacy capture, and preserve equal-time execution order. Advances #155; capital flows and installed consumer qualification remain open.
…ication

Reproduce the shared Python 3.15 failure through uv resolution. Bind selected packages, artifacts, markers and lock declarations while excluding the resolver prerelease permission alone. Preserve raw identities and all 17 accepted measurement records, and keep actual dependency drift rejection. Full suite: 2320 passed, 14 skipped. Fixes qualification of #151.
Apply validated delta/target schedules before decisions using actual marked NAV and preserve pre/post-flow valuations. Chain subperiod returns, retain reported NAV separately from investment wealth, and persist ordered flow applications. Extend accounting invariants to include independently recorded external capital without changing their tolerance. Adds public Engine and artifact regressions and a runnable guide. Advances #155; full qualification and migration integration remain open.
Use a real uv-generated self-contained lock instead of requiring every platform package and build backend in the runner cache. Preserve the failing-base producer operation and the actual dependency-drift rejection control. Addresses the clean-runner failures at 7233675 without changing validator or runtime behavior.
Capture the multiplier and declared currency used for each fill and export signed quantity. Preserve unknown legacy units and cost inclusion rather than reconstructing them from later configuration. Adds actual Engine/Parquet unit checks. Advances #155.
Capture post-fill and last-observed-session holdings through the existing account and fill journal, with explicit completeness and sequence keys. Keep investment returns on the original bar timeline and roll back observations with failed immediate callbacks. Reuse one marked snapshot implementation; preserve a valid zero mark in both carried valuations and open-trade PnL. The zero-mark regression fails twice on b49307a; focused account/lifecycle checks pass. Advances #155.
Keep detailed mark provenance out of ordinary valuation loops and enable timestamp history only when requested. Custom/legacy feeds may omit the new optional capital-events attribute. The real legacy feed run fails before the fix and passes afterward; 39 account/lifecycle/runtime checks pass, including the existing runtime regression gate. Advances #155.
…sions

Add Broker.record using the existing journal, callback transaction and portable artifact encoding. Retain carried values on actual bar keys, preserve nonfinite values, expose decoded records, and reject unsupported values before mutation. Persist the optional versioned component with strict observation checks and a runnable guide. Advances #155 and the record adapter foundation of #156; 131 focused checks pass.
Copilot AI balanced review requested due to automatic review settings October 22, 2026 04:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@stefan-jansen
stefan-jansen merged commit 3141fbc into main Oct 22, 2026
52 checks passed
@stefan-jansen
stefan-jansen deleted the codex/reproducible-migration-workflows branch October 22, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Installed bundle-free migration guide and workflow closure Port and compare original Zipline user journeys

2 participants