Skip to content

Commit c3b9db8

Browse files
feat(storage): reconstruct portable inputs and exchange legacy bundles (#130)
Capture verified portable native input sets and qualify optional standard Zipline conversion, installed consumers and calendar/precision preservation. Final f14de81 passed all applicable gates. Closes #122.
1 parent b6b0720 commit c3b9db8

13 files changed

Lines changed: 2483 additions & 0 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,39 @@ jobs:
9898
uv export --locked --all-extras --all-groups --no-emit-project --no-hashes --output-file requirements-audit.txt
9999
uv run pip-audit --requirement requirements-audit.txt
100100
101+
bundle-migration:
102+
name: Installed bundle migration
103+
runs-on: ubuntu-latest
104+
steps:
105+
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
106+
with:
107+
persist-credentials: false
108+
fetch-depth: 0
109+
110+
- name: Install Python
111+
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
112+
with:
113+
python-version: "3.12"
114+
115+
- name: Install uv
116+
uses: astral-sh/setup-uv@1c37ad07a6a961277cf70c0d37d6f313000f5884 # v10.3.0
117+
with:
118+
version: ${{ env.UV_VERSION }}
119+
enable-cache: true
120+
121+
- name: Build immutable Data wheel
122+
run: uv build --wheel
123+
124+
- name: Qualify optional standard bundle conversion
125+
run: |
126+
wheels=(dist/*.whl)
127+
test "${#wheels[@]}" -eq 1
128+
uv run --isolated --no-project --python 3.12 \
129+
--with "${wheels[0]}" \
130+
--with 'zipline-reloaded @ git+https://github.com/stefan-jansen/zipline-reloaded@cf649ac06fddce76f6caa2403cd8494ac84f72ad' \
131+
--with pandas==2.3.3 --with pytest --with pytest-timeout \
132+
python scripts/verify_bundle_migration.py
133+
101134
compatibility:
102135
name: Supported Platforms
103136
uses: ./.github/workflows/compatibility.yml

‎docs/api/index.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -567,3 +567,40 @@ shows symbol normalization,pre-entry observations and actual source revisions.
567567
options:
568568
show_source: false
569569
heading_level: 3
570+
571+
## Portable input capture
572+
573+
Capture selected tables and caller-declared effective configuration, then verify all members before
574+
returning reconstructed native frames. See the [storage contract](../user-guide/storage.md#portable-input-snapshots)
575+
and [migration recipe](../getting-started/migration.md#reconstruct-inputs-after-source-pruning).
576+
Optional bundle exchange uses standard Zipline readers/writers only in the conversion environment.
577+
578+
::: ml4t.data.storage.SnapshotTable
579+
options:
580+
show_source: false
581+
heading_level: 3
582+
583+
::: ml4t.data.storage.InputSnapshot
584+
options:
585+
show_source: false
586+
heading_level: 3
587+
588+
::: ml4t.data.storage.export_snapshot
589+
options:
590+
show_source: false
591+
heading_level: 3
592+
593+
::: ml4t.data.storage.read_snapshot
594+
options:
595+
show_source: false
596+
heading_level: 3
597+
598+
::: ml4t.data.compat.zipline.export_bundle
599+
options:
600+
show_source: false
601+
heading_level: 3
602+
603+
::: ml4t.data.compat.zipline.make_bundle_ingest
604+
options:
605+
show_source: false
606+
heading_level: 3

‎docs/getting-started/migration.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -472,3 +472,95 @@ Data-produced inputs. They used Python `3.12`, pandas `2.3.3`, Polars `2.0.0` an
472472
pandas-market-calendars `5.4.0` and yfinance `1.5.2`. Older yfinance versions admitted by the
473473
former `>=0.2.0` declaration lack the exception API used by Data's empty-download recovery;
474474
the Yahoo extra now requires the qualified `>=1.5.2` range.
475+
476+
## Reconstruct inputs after source pruning
477+
478+
Retain the input tables and declarations instead of referring to mutable source paths. Run the
479+
offline [portable snapshot example](https://github.com/ml4t/data/blob/a98cede0a36ba6856a010456f302e3e987d255de/examples/portable_snapshot_migration.py)
480+
from a source checkout:
481+
482+
```bash
483+
uv run python examples/portable_snapshot_migration.py
484+
```
485+
486+
The recipe reuses the dated-history and individual-futures examples. It captures six tables,
487+
their separate source cutoffs and revisions, stable identity namespaces, units, raw price/share
488+
basis, feed declarations and effective selection configuration. It then prunes source generations,
489+
changes the original futures file, relocates the archive, and reconstructs the same prepared inputs.
490+
The checks compare equity keys, ticker changes, action facts, adjusted prices/volume, selected
491+
contracts and paired roll marks. Missing quotes remain missing. No Reloaded package, credentials
492+
or network is needed. Read the [snapshot storage contract](../user-guide/storage.md#portable-input-snapshots)
493+
for required metadata and integrity rules.
494+
495+
Effective configuration must be supplied by the caller. A Data snapshot does not establish a
496+
common historical publication time for independently sourced tables, discover omitted execution
497+
policies, or certify an account result. Backtest owns runtime history, execution and account
498+
reconstruction; Engineer owns feature computation and Diagnostic owns analytical fill views.
499+
500+
### Convert an existing Zipline bundle once
501+
502+
In a separate conversion environment with Zipline Reloaded installed, load the original bundle
503+
through its standard API and export selected stable source IDs and inclusive reader bounds:
504+
505+
```python
506+
from zipline.data import bundles
507+
from ml4t.data.compat.zipline import export_bundle
508+
509+
with bundles.load("my-existing-bundle") as bundle:
510+
snapshot = export_bundle(
511+
bundle,
512+
"captured-bundle",
513+
namespace="original-bundle-permanent-ids",
514+
source_revision="original-ingestion-id",
515+
sids=[1, 2],
516+
start="2024-01-02",
517+
end="2024-01-04",
518+
frequencies=("daily",),
519+
)
520+
```
521+
522+
Keep the namespace stable across captures; exported identities combine it with the original sid.
523+
The bridge reads standard asset, bar and adjustment readers without modifying the source bundle.
524+
Native tables include raw observations, source history-adjustment factors, dated asset definitions,
525+
symbol lookup intervals, and corporate-action facts. A bounded stock-dividend selection also retains
526+
the payment asset's identity metadata, without creating price observations for it.
527+
528+
Daily bounds are civil session labels. Minute bounds are UTC instants; use `frequency_bounds`
529+
when selecting daily and minute data together. Minute timestamps retain the source's right-labeled
530+
bar-close convention. Source reader zeros in volume remain zeros without an inferred no-trade
531+
status. Source schedule/provider declarations remain separate from Data's selected calendar;
532+
`calendar=None` disables the native schedule without changing the captured source grid.
533+
534+
Zipline split price factors are converted to native new-shares-per-old-share terms. Source
535+
history factors remain a declared, supplied adjustment method; they are not recomputed as a
536+
different vendor's total-return definition. Publication times, quote currency, merger consideration
537+
and original listing endpoints that public readers cannot establish remain explicitly unavailable.
538+
Symbol ownership lookup intervals must not be interpreted as original listing intervals.
539+
Capturing unknown publication times does not make a bundle suitable for causal known-at claims.
540+
541+
For staged migration, `make_bundle_ingest("captured-bundle")` returns a standard Zipline bundle
542+
ingestion callback. Register it with the same source calendar and a range covering the captured
543+
bars, then use normal `bundles.ingest` and `bundles.load`. The callback verifies all files and
544+
prepares all tables before invoking writers. Asset metadata, source action-table indexes and raw
545+
daily/minute values round-trip through standard writers. A target minute writer must have enough
546+
price precision for the source; incompatible scales fail before writes instead of rounding prices.
547+
Configure `BcolzMinuteBarWriter`'s per-sid scale when needed. Native snapshots impose no legacy
548+
integer price scale. Reverse ingestion is optional and is not required for native reconstruction.
549+
550+
The actual optional conversion checks use immutable Zipline Reloaded `3.2.0` source
551+
[`cf649ac06`](https://github.com/stefan-jansen/zipline-reloaded/commit/cf649ac06fddce76f6caa2403cd8494ac84f72ad),
552+
Python `3.12`, pandas `2.3.3`, Specs `0.2.0` and Polars `2.0.0`. They construct a real bundle
553+
with renamed equities, two venues, two futures, missing quotes, split/cash/stock actions and daily
554+
and minute observations, then compare native preparation and standard re-ingestion. The native
555+
recipe requires no legacy runtime. This is unreleased source functionality; it does not claim that
556+
published Data `0.2.1` provides these APIs or that Data replaces Zipline's runtime or broker.
557+
558+
Clean wheel and source-distribution installs qualified Data `0.2.2.dev27+g90c06ba9c` for native
559+
reconstruction, the optional conversion and the selected dependency floors. Reconstructed tables
560+
also passed actual Engineer labeling, Diagnostic forward returns and Backtest contract/position
561+
checks using Engineer `0.1.7.dev29+gc69e7bada`, Diagnostic `0.1.9` and Backtest
562+
`0.1.13.dev22+g12a2c483f`. That Backtest check verifies a USD500 marked position result, not
563+
cash settlement or corporate-action account execution. The source recipes and input contracts
564+
remain usable without any of those consumers. CI's separate `Installed bundle migration` job
565+
keeps the optional conversion qualified against the pinned Zipline source without adding it to
566+
Data's production dependencies.

‎docs/user-guide/storage.md‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,3 +268,77 @@ Both backends use two mechanisms for safe concurrent access:
268268
when multiple processes write simultaneously. Lock timeout is 10 seconds.
269269

270270
These are enabled by default and can be toggled via `StorageConfig`.
271+
272+
## Portable input snapshots
273+
274+
Use `export_snapshot` to retain explicitly selected inputs independently of mutable storage
275+
generations. Each `SnapshotTable` accepts a Polars frame, a lazy frame, or a storage backend plus
276+
`key`. Backend capture holds the existing key lock and copies the current committed generation,
277+
including its custom metadata. Different keys retain their own revision declarations; export does
278+
not establish a common historical vintage.
279+
280+
```python
281+
from ml4t.data.storage import SnapshotTable, export_snapshot, read_snapshot
282+
283+
snapshot = export_snapshot(
284+
"retained-inputs",
285+
{
286+
"prices": SnapshotTable(
287+
storage,
288+
role="prices",
289+
key="equities/daily/AAPL",
290+
metadata={
291+
"source_revision": "captured-provider-revision",
292+
"identity_namespace": "my-permanent-asset-ids",
293+
"units": {"close": "USD/share", "volume": "shares"},
294+
"as_of": None,
295+
"price_basis": "raw",
296+
"share_basis": "source",
297+
"revisions": {},
298+
},
299+
)
300+
},
301+
effective_config={"price_field": "close"},
302+
artifacts={},
303+
)
304+
restored = read_snapshot("retained-inputs")
305+
assert restored.snapshot_id == snapshot.snapshot_id
306+
prices = restored.tables["prices"]
307+
```
308+
309+
The seven metadata fields shown above are required. Use `None` for unavailable facts, rather than
310+
inferring publication times, currencies or adjustment rules. `source_revision` must be a nonempty
311+
string. A known `identity_namespace` is also a nonempty string; explicit `None` records that the
312+
namespace is unknown and does not establish stable cross-source identity. `units` and `revisions`
313+
are mappings. All declarations,
314+
effective configuration and artifact descriptions must contain finite JSON values. Machine paths
315+
and credential-bearing keys or URLs are rejected. This check cannot identify arbitrary secrets
316+
hidden inside ordinary strings; supply public provenance only.
317+
318+
Pass existing `FeedSpec` column and clock declarations through `feed_spec`. A function's explicit
319+
`calendar` wins over the feed declaration; omission uses the feed calendar, then NYSE.
320+
`calendar=None` disables schedules; `"24/7"` selects a real schedule. The manifest keeps the
321+
original feed and resolved selection separately, including the schedule-provider version.
322+
`ArtifactSpec` declarations can accompany tables; artifact storage locations become portable.
323+
Tables with the default `timestamp` column also capture their default or metadata-only clock
324+
without a separate feed object. Mappings without an explicit time-contract version use version 2,
325+
including the distinction between omitted and null calendars. Explicitly versioned feed contracts
326+
retain their declared compatibility semantics. Reference tables without a declared or default
327+
clock column remain unclocked.
328+
329+
The destination must be new, or already contain the same verified snapshot ID. A different
330+
snapshot never overwrites an existing destination. Export flushes staged files, verifies the
331+
complete archive, and then publishes the directory. It does not update a source `CURRENT` pointer.
332+
The archive uses relative member paths and survives relocation and source-generation pruning.
333+
Reading verifies the manifest and every member hash before parsing any table; missing, changed,
334+
escaping or symlinked members fail without returning partial inputs.
335+
336+
The snapshot ID identifies physical input bytes, schemas, roles, clock declarations, source and
337+
unit metadata, effective configuration, artifacts and software versions. Capture time and random
338+
storage-generation identifiers do not change that ID. Equivalent financial values stored with
339+
different bytes can have different IDs. Hashes detect accidental modification; they do not
340+
authenticate an archive supplied by an untrusted party. These APIs prepare inputs and do not
341+
serialize an executable strategy, credentials, broker state or a complete account ledger.
342+
343+
See the [reconstruction recipe](../getting-started/migration.md#reconstruct-inputs-after-source-pruning)
344+
for an offline equity-and-futures workflow and the optional legacy bundle transition.

0 commit comments

Comments
 (0)