Authoritative per-asset histories, initial-capital returns and versioned persistence #155

User workflow and ownership

Actual Engine events -> observed holdings/account/fill/flow tables + manifest -> same BacktestResult reload -> Diagnostic public holdings/activity/returns consumer. Existing fills retain sequence by list/row order; add explicit persisted sequence before relying on cross-format ordering.

Backtest owns runtime simulation and actual account exports. Data owns prepared inputs/adjustments, Engineer owns features, Diagnostic owns analytics. Existing Strategy, Engine, LifecycleDispatcher and AccountState remain authoritative. No mandatory new strategy base or Reloaded production dependency.

Current behavior and implementation

missing capability. Current interfaces: Engine._record_portfolio_state, BacktestResult, BacktestResult.to_portfolio_state_dataframe, BacktestResult.to_fills_dataframe, BacktestResult.to_equity_dataframe, BacktestResult.to_parquet/from_parquet, BacktestExporter.

Source locations: engine.py, result.py, export.py, analytics/equity.py, analytics/bridge.py.

• Fix the reproduced first-period fee/initial-capital omission across first return, cumulative
  return, total return, initial-capital metrics and initial drawdown. Retain actual observation
  timestamps; do not append a fabricated market bar to make the arithmetic work.
• Add the proposed holdings and recorded-variable views. Extend existing portfolio/fill views
  with stable IDs, observation/event sequence, sampling, availability, units and account method.
• Record initial capital, external contributions/withdrawals, internal income/funding/fees and
  marks separately. For flows, specify event ordering and use independently valued subperiods for
  exact time-weighted returns. Missing required valuations mean unavailable exact returns, not
  a silently assumed denominator.
• Add native capital-flow events, not just an export column. Support declared delta and target
  capital changes, computing a target's applied delta from the marked pre-flow NAV at its declared
  event phase. Apply cash changes before the corresponding decisions when the reference requires
  it. The proposed DataFeed(capital_flows=...) input joins the same ordered account event path;
  income/funding is not an external contribution.
• Extend the existing Parquet/artifact schema and backward reader, including old beta artifacts.
  Old files without holdings remain readable with an explicit unavailable component.
• Keep detailed history opt-in through declared sampling. Migration examples enable the capture
  they promise. Use complete snapshot/account observation keys to retain genuinely all-cash dates.

Acceptance: the retained $10 first-bar fee on $10,000 fails on the relevant supported base and
passes after correction with return -0.1%. No-trade, first-bar-only, intraday open-flat,
partial/open lots, equal-time fills and external-flow cases reconcile independently. Real Engine
output survives strict save/load and feeds the actual public analytical consumer. No fabricated
Backtest-like object or reconstruction from closed trades counts as final producer acceptance.
For example, a portfolio grows from $10,000 to $11,000, receives a $1,000 contribution and then
falls from $12,000 to $10,800. Its compounded investment return is -1%. The executed fixture must
generate those valuations through actual positions/flows, not a fabricated result.

Acceptance and independent oracles

• 10000 initial capital, 10 shares at100 and10 fee -> cash8990, holdings1000, NAV9990, first return-0.001 and total return-0.001; unchanged prices must not hide the first loss.
• No trades -> every observed date retained, cash/NAV10000, zero return, complete zero holdings, no fabricated execution.
• Same-time sell/buy order and partial/open lots retain unique event sequence, exact quantities, multipliers, fees and status through export/reload.
• Intraday buy then sell with flat EOD holdings still has nonzero turnover/costs and observed exposure at declared intraday snapshots.
• Two complete snapshots with signed weights and external flow produce independently computed start-period exposures and flow-adjusted returns; missing snapshots must reject completeness claims.
• Failing-base/passing-fix public Engine regression for initial capital; source tests plus isolated built wheel/sdist install, real Parquet round trips including legacy artifacts, and actual Diagnostic invocation. Do not construct Backtest-like objects to claim final integration.
• Invalid or ambiguous inputs fail before account/output mutation; unknown information is not fabricated.
• Existing default behavior, actual legacy artifact readers, keys/units, and captured accepted performance samples retain their qualified scope.

Dependencies

• Agreement on producer-account-history contract with Diagnostic; captured fixtures permit consumer development independently.
• Calendar branch for full v2 integration; generic holdings capture and first-return reproduction can proceed now.
• Instrument metadata from caller/Data; absent stable identity/FX must be explicit, not invented.

Source requirements

• Coverage row 4: Empyrical / aggregate_returns (primary owner Diagnostic).
• Coverage row 77: Empyrical / perf_attrib (primary owner Diagnostic).
• Coverage row 78: Empyrical / compute_exposures (primary owner Diagnostic).
• Coverage row 168: Pyfolio / Analysis-ready holdings and account-state exports (primary owner Backtest).
• Coverage row 169: Pyfolio / Intraday state and execution provenance (primary owner Backtest).
• Coverage row 170: Pyfolio / Accounting capability and migration boundaries (primary owner Backtest).
• Coverage row 175: Pyfolio / Calendar and annualization propagation (primary owner Diagnostic).
• Coverage row 185: Zipline / Zipline result conversion and analytical conventions (primary owner Diagnostic).

Evidence and completion

Retain failing-base/passing-fix public Engine evidence for defects; independent arithmetic and matched native reference configurations for new behavior. Test actual installed producer-to-consumer workflows, not reconstructed account objects. All applicable final-head source, type, coverage, package, security, platform and strict documentation checks must pass before ordinary guarded merge. Missing essential workflows remain unfinished. No release or deployment is included.

Premises: none - This projection preserves accepted implementation requirements and explicit independent acceptance oracles; it introduces no untested factual assumption as a design prerequisite.

Closes: none
Advances: none

AI assistance: This implementation specification was prepared with OpenAI Codex using source inspection, retained Engine reproductions and the accepted maintainer-reviewed cross-library scope. The implementing maintainer remains responsible for review and executed evidence.

Plan reference: successor implementation projection, Issue 2; batch analysis-ready-results.

Additional reproduced account defect

Observed zero marks must stay zero in account valuations and open-trade P&L. A real Engine with10000 starting cash buys10TEST at100, observes TEST at0 on the next bar, and then either observes0 again or receives only another asset. Expected NAV is10000,9000,9000 and open P&L is-1000. On supported base b49307a, open P&L incorrectly becomes0; when TEST is absent on the last bar, NAV incorrectly returns to10000. Both public Engine regressions fail on that base and pass with the unified marked snapshot correction in7d1823e. No zero-price execution or invented liquidation is required. Preserve the observation time/source of carried marks.