Repository navigation
Prepare dated feature panels, grouped transforms and selection screens #109
Description
User workflow and ownership
Implement the accepted Engineer preparation/migration unit below. Preserve existing native defaults, dataframe contracts and independent use with supplied data. Data owns source adjustments/revision selection; Diagnostic owns the only Alphalens facade/cleaner and analytical workflows; Backtest owns simulation/accounting. Engineer produces native features, eligibility and outcomes, not competing engines.
Dependency: existing compute_features plus Issue 2 for each method used in final parity; the
information-set and transform work can proceed with existing qualified kernels. Reuse column/feed
contracts and existing grouped ordering. Introduce the bounded native composition and the
preparation part of compat/zipline.py; do not implement a graph evaluator or require the store.
Consume Data-selected source revisions/adjustment perspective and determine feature eligibility from supplied availability, effective membership and classifications. Do not implement source revision/adjustment selection in Engineer.
Keep supplied raw observations and adjustment basis intact. Make factor input masks and final
screen distinct. A universe entry does not automatically remove usable pre-entry history. Preserve
session labels as identifiers and event/availability/decision instants as separate keys. Reject
ambiguous duplicates, missing declarations and unusable coverage before persistence. Reuse
OfflineFeatureStore's single-clock backward join only when that clock actually represents the
necessary knowledge ordering; it does not establish two-clock correctness by itself.
Provide grouped ranks, top/bottom, demeaning, z-scores, winsorization, quantiles, percentile bounds,
peer counts and boolean/rolling filters through native expressions. Resolve ties deterministically
with stable IDs and explicit rank/interpolation policy. Qualify all missing/constant/small-group
cases. Cross-sectional z-scores use the current eligible group, not a train-fitted scaler. A
custom feature/filter/classifier port uses existing extension mechanisms and has an executable
example, validation and clear unsupported behavior, not arbitrary object execution.
Output ordinary keyed feature tables with availability/membership and requested selections.
Persist full time/information-set/basis declarations through minimally extended owned artifact
definitions after a demonstrated round-trip failing test. Preserve Arrow/Parquet dtypes and JSON/
YAML omitted/null semantics. The caller can select pandas MultiIndex output explicitly at the
adapter boundary, while native compute_features remains directly usable.
Proposed focused files: tests/test_feature_panel_preparation.py and
tests/compat/test_zipline_selection.py. Verify with
uv run pytest tests/test_feature_panel_preparation.py tests/compat/test_zipline_selection.py -q.
Use the dated split/rename/late-publication fixture and actual RSI top/bottom-three example.
Compare exact eligible keys, warmup, group membership, masks, ranks and values. Future data,
revisions not yet known and ineligible assets cannot revise past causal features. Reordering rows
or assets preserves results after key sorting. Adding a genuinely eligible contemporary asset may
change cross-sectional ranks; never assert impossible rank invariance across different universes.
Changing only final screen must not change the surviving asset's rolling history incorrectly.
Premises
Input: dated feature panel
| Question | Answer | Kind |
|---|---|---|
| record | Market, universe and classification records include their observation/effective keys and a declared availability instant. | definitional |
| absent-record | Missing/unavailable/revoked membership or data is not an eligible default; coverage and nulls remain visible. | definitional |
| ordering-spacing | Explicit decision instants and existing feed grid determine history; late revisions are selected per decision. | definitional |
| entity-identity | Stable IDs survive ticker renames and define grouping; membership/group assignment is date-effective. | definitional |
| known-at | Every contributing revision and feature is known no later than its decision; no hindsight revision or label enters signals. | definitional |
| institution | Supplied universe/availability/basis provenance and the declared FeedSpec schedule govern; Engineer does not invent provider history. | definitional |
Claims
| Id | Claim | How it could be false |
|---|---|---|
| causal-panel | Earlier prepared features use only the information set available at their decision. | Future publication, restatement, split knowledge or classification changes earlier output. |
| selection-equivalence | Native and supported adapter screens select the same keys under matched definitions and universe. | Ties, masks, group normalization, warmup or asset order select a different asset. |
| feature-contract-roundtrip | Feature persistence and conversion retain time, identity, availability and basis declarations. | A saved artifact loses None/omission, timezone, precision, knowledge clock or basis. |
Closes: dated-feature-panel.record, dated-feature-panel.absent-record, dated-feature-panel.ordering-spacing, dated-feature-panel.entity-identity, dated-feature-panel.known-at, dated-feature-panel.institution, causal-panel, selection-equivalence, feature-contract-roundtrip
Intake and completion
- Reproduce public producer defects before repairs and retain exact failing/passing revisions.
- Complete the methods, negative/error cases, runnable tests and migration route above.
- Verify actual installed outputs and the applicable final-revision quality/platform gates.
- Record producer/consumer dependencies separately from standalone qualification.
Plan reference: accepted Engineer preparation and Reloaded migration plan, unit 3 in milestone unreleased - Engineer preparation and Reloaded migration.
AI assistance: Codex prepared this intake projection from the accepted maintainer plan; implementation and independent test evidence are still required.
Activity
feat(compat): map Zipline numeric factor names to native expressions
feat(features): port finite Pipeline factors and causal selection wor…