Skip to content

Commit 9493bcb

Browse files
feat: add causal raw corporate-action accounting (#166)
Closes #157
1 parent c5d4309 commit 9493bcb

297 files changed

Lines changed: 9999 additions & 274 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -360,7 +360,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
360360

361361
Backtest measured 2026-10-21; reference measurements dated 2026-10-13 were retained unchanged. Host: `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
362362

363-
A performance-impact review retains these measurements for engine source `d1aff90984b7`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
363+
A performance-impact review retains these measurements for engine source `d6b013536ecd`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
364364

365365
### Synthetic diagnostic scenarios
366366

‎docs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -179,7 +179,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
179179

180180
Backtest measured 2026-10-21; reference measurements dated 2026-10-13 were retained unchanged. Host: `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
181181

182-
A performance-impact review retains these measurements for engine source `d1aff90984b7`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
182+
A performance-impact review retains these measurements for engine source `d6b013536ecd`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
183183

184184
### Synthetic diagnostic scenarios
185185

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
# Raw corporate actions
2+
3+
Use `DataFeed(corporate_actions=..., input_manifest=...)` or the same keywords on
4+
`run_backtest` to apply declared events to raw prices. Backtest changes the existing
5+
account, positions and pending orders. Data preparation belongs in ML4T Data;
6+
adjusted historical series alone do not create dividend cash or payment entitlements.
7+
8+
This interface is part of the next-version source boundary. It is not present in
9+
the published 0.1.12 package or the preserved 0.1.13 release candidate. Use a wheel
10+
built from the source revision containing this page. The example needs only Backtest
11+
and its normal dependencies; Zipline and ML4T Data are not required at runtime.
12+
13+
## Run a split followed by a dividend
14+
15+
All timestamps below are explicit UTC instants, so this example disables calendars.
16+
The account buys ten shares for $1,000, receives a 2:1 split, earns a $20 dividend
17+
on twenty shares, then receives the payment. Receivables contribute to NAV but
18+
cannot be spent before payment.
19+
20+
<!-- ml4t-doc-test: raw-corporate-actions -->
21+
```python
22+
from datetime import UTC, datetime, timedelta
23+
import polars as pl
24+
from ml4t.backtest import BacktestConfig, ContractSpec, Strategy, run_backtest
25+
26+
times = [datetime(2025, 3, 3, 21, tzinfo=UTC) + timedelta(days=i) for i in range(5)]
27+
prices = pl.DataFrame({
28+
"timestamp": times, "asset": ["A"] * 5,
29+
"close": [100.0, 100.0, 50.0, 49.0, 49.0],
30+
})
31+
common = {
32+
"source": "declared-example", "revision_id": "r1", "asset": "A",
33+
"available_at": times[0], "currency": "USD", "share_basis": "pre_event",
34+
}
35+
events = pl.DataFrame([
36+
{**common, "event_id": "split", "action_kind": "split",
37+
"effective_at": times[2], "split_ratio": 2.0,
38+
"split_ratio_convention": "new_per_old"},
39+
{**common, "event_id": "dividend", "action_kind": "cash_dividend",
40+
"effective_at": times[3], "payment_at": times[4], "cash_per_share": 1.0},
41+
])
42+
43+
class Buy(Strategy):
44+
def on_data(self, timestamp, data, context, broker):
45+
if timestamp == times[0]:
46+
broker.submit_order("A", 10)
47+
48+
result = run_backtest(
49+
prices, Buy(),
50+
config=BacktestConfig(calendar=None, initial_cash=10000,
51+
commission_rate=0, slippage_rate=0),
52+
contract_specs={"A": ContractSpec("A", currency="USD")},
53+
corporate_actions=events, input_manifest={"price_basis": "raw"},
54+
holdings_sampling="bar",
55+
)
56+
account = result.to_portfolio_state_dataframe()
57+
assert account["cash"].to_list() == [10000, 9000, 9000, 9000, 9020]
58+
assert account["receivables_value"].to_list() == [0, 0, 0, 20, 0]
59+
assert account["equity"].to_list() == [10000] * 5
60+
assert result.to_actions_dataframe().filter(pl.col("operation") == "split")[
61+
"quantity_after"
62+
].to_list() == [20]
63+
assert result.to_entitlements_dataframe()["paid"].to_list() == [True]
64+
print("20 shares, cash 9020, NAV 10000; one dividend paid")
65+
```
66+
67+
<!-- ml4t-doc-output: raw-corporate-actions -->
68+
```text
69+
20 shares, cash 9020, NAV 10000; one dividend paid
70+
```
71+
72+
## Event fields
73+
74+
Pass a Polars frame. Every event requires nonempty `source`, `event_id`,
75+
`revision_id` and stable `asset`, plus `action_kind`, `effective_at`, a physical
76+
`available_at`, and `currency` matching the instrument's explicit `ContractSpec`.
77+
There is no implicit FX conversion. Source and destination instruments must use
78+
the same currency for security delivery.
79+
80+
| Kind | Economic terms | Account effect |
81+
|---|---|---|
82+
| `split` | Positive `split_ratio`, `split_ratio_convention="new_per_old"` | Rebase quantities, basis and pending order prices. Original fills remain unchanged. |
83+
| `cash_dividend`, `cash_distribution` | Nonnegative `cash_per_share`, `share_basis`, payment clock | Capture eligible shares once, then credit cash or debit a short liability at payment. |
84+
| `stock_dividend`, `stock_distribution` | Positive `quantity_ratio`, stable `destination_asset`, `share_basis`, payment clock | Capture the security entitlement and deliver to the existing destination position. |
85+
| `merger` | Nonnegative `quantity_ratio` and `cash_component`, destination when ratio is nonzero, `share_basis` | Remove the source position, cancel source orders and deliver the declared cash/security consideration. Further source orders are rejected. |
86+
87+
`quantity_ratio` is destination shares per source share. `cash_component` is cash
88+
per source share. Native conversion carries residual source book basis into the
89+
security leg and recognizes cash exceeding that basis as realized gain. A cash-only
90+
conversion realizes the difference from source basis. These are simulation book
91+
rules, not a tax-lot or statutory basis allocation service.
92+
93+
For a native split, fractional shares are retained. To receive cash instead, declare
94+
`fractional_policy="cash_in_lieu"` and a positive `cash_in_lieu_price` in the new
95+
share units. Partial orders adjust total and already-filled quantities independently;
96+
`requested_quantity` and original fills retain their original units.
97+
98+
## Clocks and simultaneous events
99+
100+
The native `declared_terms` policy uses explicit `entitlement_at`, then `ex_date`,
101+
then `entitlement_date`, then `effective_at` for distribution eligibility.
102+
`zipline_3_2` uses `ex_at` in place of `entitlement_at`. A record date does not
103+
replace an explicitly declared ex date. Payment uses `payment_at`, then
104+
`payment_date`; a merger without a payment clock pays at its effective instant.
105+
106+
Civil `Date` labels require a declared calendar and a valid session. They apply
107+
from UTC session midnight, before premarket callbacks. Physical instants retain
108+
their declared instant and must match the feed's timezone awareness. With
109+
`calendar=None`, use physical clocks for every clock actually used by the event.
110+
No event can be published after its effective/eligibility instant.
111+
112+
At one clock, Backtest captures `pre_event` entitlements, applies splits and
113+
conversions, captures `post_event` entitlements, then pays due obligations. Event
114+
identities provide a deterministic order within each phase. The group is atomic:
115+
an invalid resulting position restores cash, holdings, pending orders and journals.
116+
Actions run before strategy callbacks and subsequent order execution at that clock.
117+
118+
An exact repeated revision applies once. Conflicting terms under the same revision
119+
fail. A later publication may replace earlier terms only before the earliest
120+
effective instant. Different revisions competing for the latest publication clock
121+
are ambiguous.
122+
Retroactive corrections require a separately declared forward event.
123+
124+
## Input basis and audit outputs
125+
126+
Declare `input_manifest={"price_basis": "raw"}` or pass an actual ML4T Data input
127+
snapshot manifest with raw bar-table metadata. Adjusted or unknown bar basis is
128+
rejected before the strategy runs. ML4T Data's
129+
`ml4t.data.adjustments.prepare_action_inputs` validates and normalizes event facts;
130+
the caller must supply genuinely known publication, currency and missing economic
131+
terms. A bundle export's null publication field is not permission to invent one.
132+
133+
`to_actions_dataframe()` retains application times, revision identity, economic
134+
terms, cash changes and order adjustments. `to_entitlements_dataframe()` retains
135+
eligibility and payment facts even after the source position is sold. Account
136+
observations include actual `receivables_value`. Persist these outputs with
137+
`result.to_parquet(...)`; action-enabled results use artifact schema 5 with
138+
required action and entitlement tables and content verification. Older artifacts
139+
remain readable, with missing action history reported as unavailable.
140+
141+
Adjusted account marks preserve the original quote observation timestamp.
142+
`broker.get_last_price()` and runtime history keep the raw observations. An
143+
account basis adjustment is not an executable new quote. Security delivery requires
144+
an observed destination mark; it does not create a market observation.
145+
146+
## Zipline migration differences and tested scope
147+
148+
Set `BacktestConfig(corporate_action_policy="zipline_3_2")` for the tested legacy
149+
rounding and recognition method. The callback `run_algorithm` adapter selects this
150+
method. It truncates fractional shares toward zero, uses rounded reported basis
151+
for split cash-in-lieu, and recognizes distributions at payment. Native
152+
`declared_terms` recognizes unpaid cash/security receivables when earned.
153+
154+
The retained original is published `zipline-reloaded==3.2.0`, source
155+
`cf649ac06fddce76f6caa2403cd8494ac84f72ad`, wheel SHA-256
156+
`1d7a3c4f547c02e9a3c3cc1f8ea2394641124c1fdb35ef27cdf30fa488d3899b`.
157+
Actual snapshots were exported and native action facts prepared by ML4T Data
158+
source `c3b9db80a66397420fb27adeccc762559dd41e07`
159+
(`0.2.2.dev21+gc3b9db80a`) with published Specs 0.2.0. These development Data
160+
artifacts are qualification identities, not a claim that an older published Data
161+
release contains these interfaces.
162+
163+
`tests/test_zipline_corporate_actions.py` consumes those exported inputs and compares
164+
closing cash, NAV and positions for long/short splits, reverse splits, partial orders,
165+
cash payments after sale, own/cross-stock payments and destination position changes.
166+
The original reports transaction commission as `None` under `NoCommission`; native
167+
fills report explicit zero, while original and native order commission are both zero.
168+
After a security payment covers a destination position, the callback facade can
169+
retain the original's zero-quantity position entry and reported basis. Native
170+
holdings remain flat; the facade projects that presentation from the payment journal.
171+
172+
Original daily premarket valuation can already read that day's close. Changing only
173+
the ex-day close by $7 changes its premarket mark by $7. Backtest retains the causal
174+
previous adjusted mark until the quote arrives. Those premarket values deliberately
175+
differ; closing ledger agreement does not establish premarket information parity.
176+
Native mixed cash/security mergers use independent accounting oracles. The retained
177+
original comparisons do not establish general merger, tax-lot or multi-currency parity.
178+
179+
Continue with [Zipline migration](migrate-from-zipline.md) for callbacks and prepared
180+
inputs, or [Results and Analysis](results.md) for account exports.

‎docs/user-guide/migrate-from-zipline.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -207,12 +207,12 @@ native key; use `int(asset)` only when an actual legacy integer SID was supplied
207207

208208
## Match ledger and corporate-action assumptions
209209

210-
`Engine` does not automatically turn arbitrary `split_ratio` or `cash_dividend`
211-
columns into share adjustments or cash payments. Prepare the input price and
212-
share-unit basis explicitly upstream. Adjusted bars alone do not reproduce an
213-
unadjusted ledger's dividend entitlement, payment date or split-adjusted share
214-
count. Keep an action-sensitive migration on the reference engine until its
215-
required event workflow is supported and compared.
210+
Pass explicit raw event facts through `DataFeed(corporate_actions=...,
211+
input_manifest=...)`. The [corporate-action guide](corporate-actions.md) runs a
212+
split and delayed dividend payment, specifies event clocks and input basis, and
213+
documents the tested `zipline_3_2` method. Arbitrary event columns in a price frame
214+
do not activate accounting. Adjusted bars alone do not reproduce raw share counts
215+
or payment entitlements. Match your event terms and method to the tested scope.
216216

217217
Contract multipliers and fixed margin schedules support contract P&L and
218218
buying-power checks. Backtest's current leveraged ledger still moves full trade
@@ -358,8 +358,8 @@ all Zipline workflows are supported.
358358
This initial adapter does not yet implement
359359
the original Pipeline construction API. Use the [prepared panel route](#deliver-prepared-pipeline-results)
360360
for factor delivery and the [frozen CSV route](#register-an-offline-csv-source)
361-
for auxiliary inputs. Raw corporate actions and futures variation-cash settlement
362-
remain separate unfinished accounting work.
361+
for auxiliary inputs. [Raw corporate actions](corporate-actions.md) use explicit
362+
event inputs; futures variation-cash settlement remains separate unfinished work.
363363
Native scheduler and prepared feature/history routes remain available as
364364
shown above. `set_benchmark(None)` explicitly disables a benchmark; omission
365365
records it as not supplied unless prepared benchmark data is passed. An asset

‎docs/user-guide/profiles.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,7 @@ The table reports engine-call wall time for all 17 correctness-passing pairs. Th
218218

219219
Backtest measured 2026-10-21; reference measurements dated 2026-10-13 were retained unchanged. Host: `Linux-7.0.0-38-generic-x86_64-with-glibc2.43` with 24 logical CPUs. Each side used one isolated warm-up process and ten isolated measured processes. The timer includes only the engine call; it excludes input loading, model inference, target construction, adapter preparation, result extraction, serialization, reporting. These measurements apply only to the named strategy, framework version, frozen input bundle, and machine. Raw samples and bootstrap intervals are retained in [real-strategy performance evidence](https://github.com/ml4t/backtest/blob/main/validation/REAL_STRATEGY_PERFORMANCE.json).
220220

221-
A performance-impact review retains these measurements for engine source `d1aff90984b7`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
221+
A performance-impact review retains these measurements for engine source `d6b013536ecd`. The measured source, locked dependencies, review rationale and current correctness evidence are recorded with the raw samples. Measurements are repeated when changes materially affect timed execution; a new source revision alone does not require a benchmark rerun. Dependency validation checks selected packages and artifacts; allowing prereleases without changing those selections does not invalidate the measurements.
222222

223223
### Synthetic diagnostic scenarios
224224

‎mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,7 @@ nav:
127127
- Migrate from Zipline: user-guide/migrate-from-zipline.md
128128
- Risk Management: user-guide/risk-management.md
129129
- Account Policies: user-guide/accounts.md
130+
- Corporate Actions: user-guide/corporate-actions.md
130131
- Rebalancing: user-guide/rebalancing.md
131132
- Data Feed: user-guide/data-feed.md
132133
- Results & Analysis: user-guide/results.md

0 commit comments

Comments
 (0)