Skip to content

Commit 2ea336f

Browse files
feat(migration): prepare optional inputs and qualify installed workflows (#116)
Prepare precise optional market/loading/classification inputs and complete the Engineer-owned migration workflows, including actual source-archive recipe execution and declared producer/consumer boundaries. Correct reproduced conversion, currency, identity and packaging defects with regression evidence; preserve native defaults and shared temporal semantics. Closes #111 Closes #112 AI assistance: Codex assisted with implementation, tests, documentation and qualification. Exact source/artifact identities, real workflow results and final-revision gates were reviewed in the owning maintainer session.
1 parent dd26bca commit 2ea336f

13 files changed

Lines changed: 2475 additions & 22 deletions

‎docs/api/pipeline-ports.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ Use the [migration how-to](../user-guide/pipeline-migration.md) to prepare input
7373
| `ArrayPredicate`, `NumExprFilter` | Native elementwise Boolean expressions | Preserve declared null behavior; do not execute arbitrary legacy expression strings. |
7474
| `Filter.if_else` | `pl.when(predicate).then(left).otherwise(right)` | Declare unknown-predicate handling explicitly. |
7575
| `CustomFilter`, base `Filter` | Boolean expressions or reusable registered features | No graph interpreter; supply required history and knowledge clocks. |
76-
| Classifier `eq`, `ne`, `element_of` | Equality, inequality, `.is_in(...)` | Map missing sentinel to null; fill null Boolean outputs as intended. |
76+
| Classifier `eq`, `!=` (`__ne__`), `element_of` | Equality, inequality, `.is_in(...)` | Map missing sentinel to null; fill null Boolean outputs as intended. |
7777
| `startswith`, `endswith`, `has_substring` | `.str.starts_with`, `.str.ends_with`, `.str.contains(..., literal=True)` | Literal substring, not accidental regex interpretation. |
7878
| `matches` | Explicit `re.compile(pattern).match(value)` mapping | Legacy Python re.match starts at the beginning and supports Python regex features. Polars unanchored Rust regex is not a general equivalent. |
7979
| `relabel` | `.replace_strict(mapping, default=None)` or explicit mapping function | Preserve nulls and declare behavior for unknown labels. |
Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Prepare optional market, sector and loading inputs
2+
3+
Use these helpers when an existing research workflow supplies Pyfolio-style
4+
market/loadings containers or already keyed Polars tables. Engineer prepares the
5+
inputs. Diagnostic owns capacity, sector exposures and supplied-loading analysis;
6+
Data owns revision selection, corporate adjustments and source identity history.
7+
No Reloaded package is required by these helpers.
8+
9+
## Convert the container before preparing derived values
10+
11+
`market_to_long` maps a pandas frame indexed by `(date, field)`, with assets in
12+
columns, to the timestamp/entity/price/volume columns declared by `feed_spec`.
13+
The default field selectors are `price` and `volume`. `loadings_to_long` maps
14+
`(date, asset)` rows and factor columns to keyed `factor`/`loading` rows. Both
15+
accept native Polars tables, validate keys and preserve their eager/lazy form.
16+
17+
Legacy containers do not declare publication time, monetary units, corporate
18+
adjustment basis or completed bars. Supply those facts from the source contract;
19+
the conversion does not invent them. Opaque legacy Asset objects require an
20+
explicit `entity_map` to stable integer/string IDs. Compound entity mappings use
21+
tuples matching `FeedSpec.entity_col`. Numeric and timestamp dtypes are retained;
22+
incompatible asset/factor dtypes must be normalized explicitly, rather than
23+
silently casting precise integers to floats. These narrow container conversions
24+
preserve numeric nullable arrays and physical datetime keys without requiring
25+
PyArrow. They do not provide a generic arbitrary-object serialization service.
26+
27+
Run `uv run python examples/optional_input_migration.py` for a complete container
28+
conversion, liquidity preparation, supplied-loading selection and static-sector
29+
recipe. The example declares a synthetic source's publication/completion facts.
30+
It does not claim that real daily prices are known before their close.
31+
32+
## Prepare completed, known market inputs
33+
34+
This executable native example uses observation rows deliberately. With a named
35+
calendar, declare the actual time-bar grid; omitted `calendar` resolves the
36+
explicit feed metadata, then NYSE. Explicit `calendar=None` disables schedules
37+
and counts rows, including gaps. A named continuous schedule still requires its
38+
scheduled bars and is not row mode.
39+
40+
<!-- ml4t-exec -->
41+
```python
42+
from datetime import UTC, date, datetime
43+
import polars as pl
44+
from ml4t.specs import FeedSpec
45+
from ml4t.engineer.features.inputs import prepare_market_inputs
46+
47+
days = [date(2025, 7, d) for d in (7, 8, 9)]
48+
raw = pl.DataFrame({
49+
"timestamp": days, "asset": [17] * 3,
50+
"close": [10.0, 20.0, 30.0], "volume": [100.0, 0.0, 200.0],
51+
"currency": ["USD"] * 3, "multiplier": [1.0] * 3,
52+
"volume_unit": ["shares"] * 3,
53+
"available_at": [datetime(2025, 7, d.day, 20, tzinfo=UTC) for d in days],
54+
"completed_at": [datetime(2025, 7, d.day, 20, tzinfo=UTC) for d in days],
55+
"decision_time": [datetime(2025, 7, d.day, 13, tzinfo=UTC) for d in days],
56+
})
57+
result = prepare_market_inputs(
58+
raw.lazy(),
59+
feed_spec=FeedSpec(time_contract_version=2, calendar=None, entity_col="asset"),
60+
price_basis="raw", volume_basis="raw", purpose="causal",
61+
liquidity_window=2, completed_at_col="completed_at",
62+
).collect()
63+
assert result["lagged_mean_notional_volume"].to_list() == [None, None, 500.0]
64+
assert result["notional_volume"].to_list() == [None, None, None]
65+
assert result["liquidity_covered"].to_list() == [False, False, True]
66+
assert result.select(raw.columns).equals(raw)
67+
```
68+
69+
The third decision can use the two completed prior products, 1,000 and zero. It
70+
cannot use that day's unpublished price/volume product. Missing volume remains
71+
missing; a genuine zero is retained. Unknown currency, volume unit or monetary
72+
multiplier leaves dependent liquidity uncovered. A contract multiplier is
73+
explicit, not an assumed equity unit. No FX conversion occurs.
74+
Each stable entity's liquidity history must use one quote currency. Mixed
75+
currencies fail atomically and require explicit normalization upstream; units
76+
are not made comparable by averaging. Different entities may use different
77+
currencies. With no requested liquidity feature, their raw tables remain unchanged.
78+
79+
## Method and output reference
80+
81+
| Input or output | Contract |
82+
|---|---|
83+
| `purpose` | Required `causal` or `retrospective`; retrospective values are not signals. |
84+
| `price_basis`, `volume_basis` | Required compatible declarations for derived liquidity. Data/callers prepare adjustments upstream. |
85+
| `currency`, `multiplier`, `volume_unit` | Explicit per-row supplied units; column roles are configurable. |
86+
| `liquidity_window=None` | Validate and return the native market table unchanged. |
87+
| `notional_volume` | Float64 `price * volume * multiplier`; original financial columns remain unchanged. Causal output requires known completed input. |
88+
| `lagged_mean_notional_volume` | Strict complete-window mean of those products, shifted once by `completed_lag` (default one). |
89+
| `liquidity_available_at`, `liquidity_covered` | Latest contributing publication/completion instant and actual usable coverage. Unknown clocks cannot qualify causal output. |
90+
| `liquidity_window`, `liquidity_lag`, `liquidity_window_unit` | Explicit observation/expected-bar window and lag, not elapsed duration. |
91+
| `liquidity_price_basis`, `liquidity_volume_basis`, `preparation_purpose` | Retained financial definition and information-set declaration. |
92+
93+
Pyfolio Reloaded's capacity denominator is `mean(price * volume).shift(1)` for
94+
unit equity multipliers. Current price times lagged average volume gives a
95+
different answer. Engineer prepares the denominator only; it does not normalize
96+
monetary holdings into weights or calculate days to liquidate. A second
97+
liquidity preparation rejects already derived output columns to prevent double
98+
lagging. A zero denominator remains covered zero here; the receiving capacity
99+
calculation decides that liquidation is undefined.
100+
101+
## Select supplied loadings and classifications
102+
103+
`prepare_loading_inputs(facts, requests=..., factors=[...], feed_spec=...,
104+
purpose=...)` retains every explicitly requested asset/date/factor, even when no
105+
loading exists. Facts have the feed's timestamp/entity keys, string `factor`,
106+
numeric `loading` and aware `available_at`. Requests have observation keys and
107+
aware `decision_time`. `effective_time_col` selects a separate request key when
108+
the applicable session differs from the observation cutoff. No earlier loading
109+
is carried to an absent date; missing loading is not zero exposure.
110+
111+
`prepare_classification_inputs` accepts supplied start-inclusive/end-exclusive
112+
`effective_from`/`effective_to` facts, or a mapping with `static=True`. Causal
113+
static maps also require a real `static_known_at` instant. Static maps describe
114+
a fixed classification, not historical sector truth. Expired or unknown sectors
115+
remain uncovered. A declared request effective clock is used exactly, without
116+
UTC date flooring or observation-spacing inference.
117+
118+
Both preparers retain the original request columns plus the selected value and
119+
its `_available_at`, `_covered`, `_known_at_decision`, `_effective_from`,
120+
`_effective_to`, `_purpose` and `_static` columns. The suffixes use the selected
121+
`output` name (`loading` or `sector` by default). Causal selection excludes
122+
future publication; retrospective selection retains the actual time and a false
123+
known-at-decision flag where appropriate. Multiple applicable known source
124+
revisions fail explicitly; Data/callers must select the intended revision.
125+
Output names cannot overwrite a factor key. Compound static-map tuple keys must
126+
match the declared entity columns exactly; extra tuple elements are not discarded.
127+
`None` optional facts produce uncovered request rows and do not remove unrelated
128+
price features. Explicit requested factors define loading coverage, not future
129+
factor names or assets discovered in the supplied history.
130+
131+
## Compatibility and qualification boundary
132+
133+
These APIs are unreleased development additions after Engineer 0.1.6. The
134+
native examples need no Pyfolio, Data or Diagnostic installation. The original
135+
capacity method reference is Pyfolio Reloaded 0.10.0; matching its denominator
136+
does not establish equivalence of a portfolio tear sheet, intraday estimate or
137+
order simulation. Do not relabel retrospective external tables as causal, or
138+
silently reinterpret total-return prices with unadjusted share volumes.

‎docs/user-guide/pipeline-migration.md‎

Lines changed: 78 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -132,27 +132,84 @@ decision key and pass `history_decision_time_col`. Do not reuse retrospectively
132132
adjusted history as a causal snapshot. Store Data provenance alongside the
133133
FeatureSpec and typed frame.
134134

135-
The installed migration checks qualified Engineer development source
136-
`cce590132846dbc7d558971b2d5a7f703d5cd9a2` against Zipline Reloaded 3.2.0
137-
(`cf649ac06fddce76f6caa2403cd8494ac84f72ad`), pandas 2.3.3 and NumPy 2.5.4.
138-
Both original momentum PipelineEngine journeys used the real shifted pricing
139-
loader and matched 135 keyed rows, RSI values and long/short selections each,
140-
with complete quotes and a known-missing close, listing/delisting and supplied
141-
lifetimes. This is development qualification, not an available release or proof
142-
of order, fill, commission, leverage or portfolio-performance equivalence.
143-
144-
Development source `3c5665cd2d08bad2dc5dc25b4a97573a84a7de88` additionally
145-
passed both missingness profiles with actual CustomFactor, CustomFilter and
146-
CustomClassifier terms, and 33 installed operation comparisons covering all 19
147-
elementary math functions, seven daily summaries, five return-definition/window
148-
cases and both weekday-event classes. `examples/pipeline_expression_ports.py`
149-
contains the executable math and daily-summary ports. Data's unpublished
150-
`0.2.2.dev41+gd48d849f7` artifact at
151-
`d48d849f77d3ff75d05383cbaaed9b42f6ca95b7` fed actual known-at histories into
152-
Engineer source `cce590132846dbc7d558971b2d5a7f703d5cd9a2`: an appended future
153-
correction left the earlier mean 199 unchanged, and the later adjusted history
154-
retained stable asset 17 across OLD/NEW symbols with a final two-price mean 99.5.
155-
These artifacts are qualification inputs, not published install recommendations.
135+
## Port the complete preparation workflow
136+
137+
Run `uv run python examples/reloaded_workflows.py` from the source checkout for
138+
precise wide-price conversion, a combined screen and a multi-output custom
139+
factor port. The example captures Data-produced split-adjusted observations
140+
across OLD/NEW symbols for stable asset 17. Its source as-of is July 7, 2025 at
141+
21:00 UTC; its signal-effective session is July 8. It does not claim the later
142+
adjusted history was tradable on earlier observation dates.
143+
144+
| Existing workflow | Native route | Required input or material difference |
145+
|---|---|---|
146+
| Wide prices to trailing returns | `precise_wide_returns` recipe using `market_to_long` and native `returns` | Preserve integer ticks before division. Native output retains the leading null row; legacy simple-return presentation drops the first row. This recipe explicitly counts rows. |
147+
| RSI top/bottom selection | `momentum_screen` in `examples/pipeline_migration.py` | Retain input warmup and decision-effective membership; a screen must not erase feature history. |
148+
| Momentum/volatility/liquidity/sector screen | `snapshot_screen` in `examples/reloaded_workflows.py` | Source-selected split-compatible prices/volume, currency, multiplier, actual publication/completion and dated universe facts. Optional volume must not invalidate independent price-only features. |
149+
| Multi-output CustomFactor | `custom_bands`, then `pl.col("bands").struct.field("mid")` | Each required-input group keeps its own eligibility. Missing band inputs do not silently become usable outputs or remove another feature's history. |
150+
| CustomFilter/CustomClassifier | Typed Boolean/category expressions and supplied dated facts | Rewrite callbacks explicitly; preserve missing categories and known-at times. |
151+
| `term.alias(name)` | `expression.alias(name)` | Rename the native expression/output; no legacy Term object is created. |
152+
| `term.downsample(frequency)` | Explicit sampling sessions and `sampled_values` | Use the full declared exchange schedule, including the preceding sample when output starts mid-period. Carry sampled feature rows, not raw quotes. |
153+
| Factor keys plus prices for analysis | [Native multi-horizon outcomes](labeling.md#prepare-keyed-multi-horizon-outcomes) | Diagnostic owns the Alphalens facade/cleaner. Select the return columns explicitly and retain actual endpoints, availability, units and basis. |
154+
| Pyfolio-style market/loading containers | [Optional input preparation](optional-input-migration.md) | Data/callers supply source revisions, compatible units and knowledge times. Diagnostic owns capacity, exposures and attribution. |
155+
156+
Numeric recipes can replace individual built-in constructor calls. An opaque
157+
Pipeline, custom callback or mutable execution environment requires a native
158+
table/expression port, not a global import replacement. Orders, fills, costs,
159+
account state and strategy delivery remain outside Engineer.
160+
161+
Empyrical Reloaded 0.7.0 with pandas 2.3.3 has container-dependent return behavior:
162+
its pandas `pct_change` path can round away large-integer ticks and implicitly
163+
fill missing prices. Native preparation preserves the integer differences and
164+
does not fill quotes. The qualified integer-ndarray path and independent integer
165+
quotients establish the precision expectation; rounded or implicitly filled
166+
pandas results are not the native correctness target.
167+
168+
The combined example deliberately uses `calendar=None` for its standalone
169+
base-dependency run. Omitting that argument preserves the captured NYSE feed
170+
and validates its declared daily grid with the real calendar extra. Neither
171+
route infers July 4 from missing observations.
172+
173+
<!-- ml4t-exec -->
174+
```python
175+
import runpy
176+
177+
recipes = runpy.run_path("examples/reloaded_workflows.py")
178+
prices, decisions, universe, feed = recipes["captured_snapshot"]()
179+
result = recipes["snapshot_screen"](
180+
prices, decisions, universe, feed_spec=feed, calendar=None,
181+
)
182+
assert result["asset"].to_list() == [17]
183+
assert result["screen"].to_list() == [True]
184+
assert result["lagged_mean_notional_volume"].to_list() == [100343.0]
185+
```
186+
187+
For sampled terms, select actual week-, month-, quarter- or year-start exchange
188+
sessions before the backward join. Do not choose the first observed quote in
189+
a partial period. Preserve a missing sample row: filling only nonnull values
190+
would retain a stale earlier sample. Keep input warmup before sampling, and
191+
retain `sample_session` and `sample_available_at` separately from the receiving
192+
decision. `sampled_values` rejects future-known values and ambiguous keys.
193+
194+
## Qualification profiles
195+
196+
The original workflow reference is Zipline Reloaded 3.2.0 at
197+
[`cf649ac`](https://github.com/stefan-jansen/zipline-reloaded/tree/cf649ac06fddce76f6caa2403cd8494ac84f72ad),
198+
with pandas 2.3.3 and NumPy 2.5.4. Original/custom PipelineEngine profiles use
199+
the real shifted pricing loader, missing quotes, listing/delisting and supplied
200+
lifetimes. Separate alias/cadence profiles cover all four sampling frequencies.
201+
The comparisons do not establish order/account/performance equivalence.
202+
203+
The Data producer reference is unpublished `0.2.2.dev41+gd48d849f7` at
204+
[`d48d849`](https://github.com/ml4t/data/tree/d48d849f77d3ff75d05383cbaaed9b42f6ca95b7).
205+
Data selects revisions/adjustments and retains stable IDs. Actual installed
206+
handoffs cover future corrections, raw/split-compatible products and rename
207+
history. The Pyfolio input-method reference is Reloaded 0.10.0, not an Engineer
208+
capacity implementation. These are qualification profiles, not dependency pins
209+
or published install recommendations. Immutable Engineer artifact identities,
210+
exact recipe revisions and final checks are recorded in the owning
211+
[factor/screen PR](https://github.com/ml4t/engineer/pull/115) and
212+
[input/migration PR](https://github.com/ml4t/engineer/pull/116).
156213

157214
See the [operation reference](../api/pipeline-ports.md) for individual methods,
158215
defaults and material differences, and [native features](features.md) for the

0 commit comments

Comments
 (0)