|
| 1 | +# Prepare optional market, sector and loading inputs |
| 2 | + |
| 3 | +Use these helpers when an existing research workflow supplies Pyfolio-style |
| 4 | +market/loadings containers or already keyed Polars tables. Engineer prepares the |
| 5 | +inputs. Diagnostic owns capacity, sector exposures and supplied-loading analysis; |
| 6 | +Data owns revision selection, corporate adjustments and source identity history. |
| 7 | +No Reloaded package is required by these helpers. |
| 8 | + |
| 9 | +## Convert the container before preparing derived values |
| 10 | + |
| 11 | +`market_to_long` maps a pandas frame indexed by `(date, field)`, with assets in |
| 12 | +columns, to the timestamp/entity/price/volume columns declared by `feed_spec`. |
| 13 | +The default field selectors are `price` and `volume`. `loadings_to_long` maps |
| 14 | +`(date, asset)` rows and factor columns to keyed `factor`/`loading` rows. Both |
| 15 | +accept native Polars tables, validate keys and preserve their eager/lazy form. |
| 16 | + |
| 17 | +Legacy containers do not declare publication time, monetary units, corporate |
| 18 | +adjustment basis or completed bars. Supply those facts from the source contract; |
| 19 | +the conversion does not invent them. Opaque legacy Asset objects require an |
| 20 | +explicit `entity_map` to stable integer/string IDs. Compound entity mappings use |
| 21 | +tuples matching `FeedSpec.entity_col`. Numeric and timestamp dtypes are retained; |
| 22 | +incompatible asset/factor dtypes must be normalized explicitly, rather than |
| 23 | +silently casting precise integers to floats. These narrow container conversions |
| 24 | +preserve numeric nullable arrays and physical datetime keys without requiring |
| 25 | +PyArrow. They do not provide a generic arbitrary-object serialization service. |
| 26 | + |
| 27 | +Run `uv run python examples/optional_input_migration.py` for a complete container |
| 28 | +conversion, liquidity preparation, supplied-loading selection and static-sector |
| 29 | +recipe. The example declares a synthetic source's publication/completion facts. |
| 30 | +It does not claim that real daily prices are known before their close. |
| 31 | + |
| 32 | +## Prepare completed, known market inputs |
| 33 | + |
| 34 | +This executable native example uses observation rows deliberately. With a named |
| 35 | +calendar, declare the actual time-bar grid; omitted `calendar` resolves the |
| 36 | +explicit feed metadata, then NYSE. Explicit `calendar=None` disables schedules |
| 37 | +and counts rows, including gaps. A named continuous schedule still requires its |
| 38 | +scheduled bars and is not row mode. |
| 39 | + |
| 40 | +<!-- ml4t-exec --> |
| 41 | +```python |
| 42 | +from datetime import UTC, date, datetime |
| 43 | +import polars as pl |
| 44 | +from ml4t.specs import FeedSpec |
| 45 | +from ml4t.engineer.features.inputs import prepare_market_inputs |
| 46 | + |
| 47 | +days = [date(2025, 7, d) for d in (7, 8, 9)] |
| 48 | +raw = pl.DataFrame({ |
| 49 | + "timestamp": days, "asset": [17] * 3, |
| 50 | + "close": [10.0, 20.0, 30.0], "volume": [100.0, 0.0, 200.0], |
| 51 | + "currency": ["USD"] * 3, "multiplier": [1.0] * 3, |
| 52 | + "volume_unit": ["shares"] * 3, |
| 53 | + "available_at": [datetime(2025, 7, d.day, 20, tzinfo=UTC) for d in days], |
| 54 | + "completed_at": [datetime(2025, 7, d.day, 20, tzinfo=UTC) for d in days], |
| 55 | + "decision_time": [datetime(2025, 7, d.day, 13, tzinfo=UTC) for d in days], |
| 56 | +}) |
| 57 | +result = prepare_market_inputs( |
| 58 | + raw.lazy(), |
| 59 | + feed_spec=FeedSpec(time_contract_version=2, calendar=None, entity_col="asset"), |
| 60 | + price_basis="raw", volume_basis="raw", purpose="causal", |
| 61 | + liquidity_window=2, completed_at_col="completed_at", |
| 62 | +).collect() |
| 63 | +assert result["lagged_mean_notional_volume"].to_list() == [None, None, 500.0] |
| 64 | +assert result["notional_volume"].to_list() == [None, None, None] |
| 65 | +assert result["liquidity_covered"].to_list() == [False, False, True] |
| 66 | +assert result.select(raw.columns).equals(raw) |
| 67 | +``` |
| 68 | + |
| 69 | +The third decision can use the two completed prior products, 1,000 and zero. It |
| 70 | +cannot use that day's unpublished price/volume product. Missing volume remains |
| 71 | +missing; a genuine zero is retained. Unknown currency, volume unit or monetary |
| 72 | +multiplier leaves dependent liquidity uncovered. A contract multiplier is |
| 73 | +explicit, not an assumed equity unit. No FX conversion occurs. |
| 74 | +Each stable entity's liquidity history must use one quote currency. Mixed |
| 75 | +currencies fail atomically and require explicit normalization upstream; units |
| 76 | +are not made comparable by averaging. Different entities may use different |
| 77 | +currencies. With no requested liquidity feature, their raw tables remain unchanged. |
| 78 | + |
| 79 | +## Method and output reference |
| 80 | + |
| 81 | +| Input or output | Contract | |
| 82 | +|---|---| |
| 83 | +| `purpose` | Required `causal` or `retrospective`; retrospective values are not signals. | |
| 84 | +| `price_basis`, `volume_basis` | Required compatible declarations for derived liquidity. Data/callers prepare adjustments upstream. | |
| 85 | +| `currency`, `multiplier`, `volume_unit` | Explicit per-row supplied units; column roles are configurable. | |
| 86 | +| `liquidity_window=None` | Validate and return the native market table unchanged. | |
| 87 | +| `notional_volume` | Float64 `price * volume * multiplier`; original financial columns remain unchanged. Causal output requires known completed input. | |
| 88 | +| `lagged_mean_notional_volume` | Strict complete-window mean of those products, shifted once by `completed_lag` (default one). | |
| 89 | +| `liquidity_available_at`, `liquidity_covered` | Latest contributing publication/completion instant and actual usable coverage. Unknown clocks cannot qualify causal output. | |
| 90 | +| `liquidity_window`, `liquidity_lag`, `liquidity_window_unit` | Explicit observation/expected-bar window and lag, not elapsed duration. | |
| 91 | +| `liquidity_price_basis`, `liquidity_volume_basis`, `preparation_purpose` | Retained financial definition and information-set declaration. | |
| 92 | + |
| 93 | +Pyfolio Reloaded's capacity denominator is `mean(price * volume).shift(1)` for |
| 94 | +unit equity multipliers. Current price times lagged average volume gives a |
| 95 | +different answer. Engineer prepares the denominator only; it does not normalize |
| 96 | +monetary holdings into weights or calculate days to liquidate. A second |
| 97 | +liquidity preparation rejects already derived output columns to prevent double |
| 98 | +lagging. A zero denominator remains covered zero here; the receiving capacity |
| 99 | +calculation decides that liquidation is undefined. |
| 100 | + |
| 101 | +## Select supplied loadings and classifications |
| 102 | + |
| 103 | +`prepare_loading_inputs(facts, requests=..., factors=[...], feed_spec=..., |
| 104 | +purpose=...)` retains every explicitly requested asset/date/factor, even when no |
| 105 | +loading exists. Facts have the feed's timestamp/entity keys, string `factor`, |
| 106 | +numeric `loading` and aware `available_at`. Requests have observation keys and |
| 107 | +aware `decision_time`. `effective_time_col` selects a separate request key when |
| 108 | +the applicable session differs from the observation cutoff. No earlier loading |
| 109 | +is carried to an absent date; missing loading is not zero exposure. |
| 110 | + |
| 111 | +`prepare_classification_inputs` accepts supplied start-inclusive/end-exclusive |
| 112 | +`effective_from`/`effective_to` facts, or a mapping with `static=True`. Causal |
| 113 | +static maps also require a real `static_known_at` instant. Static maps describe |
| 114 | +a fixed classification, not historical sector truth. Expired or unknown sectors |
| 115 | +remain uncovered. A declared request effective clock is used exactly, without |
| 116 | +UTC date flooring or observation-spacing inference. |
| 117 | + |
| 118 | +Both preparers retain the original request columns plus the selected value and |
| 119 | +its `_available_at`, `_covered`, `_known_at_decision`, `_effective_from`, |
| 120 | +`_effective_to`, `_purpose` and `_static` columns. The suffixes use the selected |
| 121 | +`output` name (`loading` or `sector` by default). Causal selection excludes |
| 122 | +future publication; retrospective selection retains the actual time and a false |
| 123 | +known-at-decision flag where appropriate. Multiple applicable known source |
| 124 | +revisions fail explicitly; Data/callers must select the intended revision. |
| 125 | +Output names cannot overwrite a factor key. Compound static-map tuple keys must |
| 126 | +match the declared entity columns exactly; extra tuple elements are not discarded. |
| 127 | +`None` optional facts produce uncovered request rows and do not remove unrelated |
| 128 | +price features. Explicit requested factors define loading coverage, not future |
| 129 | +factor names or assets discovered in the supplied history. |
| 130 | + |
| 131 | +## Compatibility and qualification boundary |
| 132 | + |
| 133 | +These APIs are unreleased development additions after Engineer 0.1.6. The |
| 134 | +native examples need no Pyfolio, Data or Diagnostic installation. The original |
| 135 | +capacity method reference is Pyfolio Reloaded 0.10.0; matching its denominator |
| 136 | +does not establish equivalence of a portfolio tear sheet, intraday estimate or |
| 137 | +order simulation. Do not relabel retrospective external tables as causal, or |
| 138 | +silently reinterpret total-return prices with unadjusted share volumes. |
0 commit comments