Skip to content

Commit 0e78652

Browse files
fix(api)!: validate declared bar grids before feature computation (#90)
* test(api): reproduce missing-bar feature failures for issue 76 * feat(core): validate declared bar grids against real schedules for issue 76 * feat(api): validate shared v2 bar grids before feature computation * fix(api): reject lazy null groups and document the grid migration * docs(api): make row-mode examples explicit and cover schema rejection * build(deps): pin merged Specs time contract for reproducible qualification * fix(core): reject sparse fine grids without allocating every expected bar * fix(build): qualify immutable dependencies but reject them for publication * fix(core): reject off-grid nanoseconds before datetime conversion
1 parent 6b6d02c commit 0e78652

21 files changed

Lines changed: 1107 additions & 102 deletions

‎README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -56,15 +56,16 @@ ohlcv = pl.DataFrame(
5656
}
5757
)
5858

59-
features = compute_features(ohlcv, ["rsi", "macd", "atr"])
59+
features = compute_features(ohlcv, ["rsi", "macd", "atr"], calendar=None)
6060

6161
assert {"rsi", "macd", "atr"} <= set(features.columns)
6262
assert features.height == ohlcv.height
6363
```
6464

6565
`compute_features()` returns the input columns with the requested feature columns
66-
appended. Use the feature registry to inspect categories and parameters before building
67-
larger pipelines.
66+
appended. This example explicitly uses observation-row mode with `calendar=None`;
67+
periods count rows, not scheduled sessions or elapsed days. Use the feature registry
68+
to inspect categories and parameters before building larger pipelines.
6869

6970
## Supported Workflows
7071

‎docs/getting-started/quickstart.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ ohlcv = pl.DataFrame(
3333
}
3434
)
3535

36-
features = compute_features(ohlcv, ["rsi", "macd", "atr"])
36+
features = compute_features(ohlcv, ["rsi", "macd", "atr"], calendar=None)
3737

3838
added = [name for name in features.columns if name not in ohlcv.columns]
3939
print(f"rows={features.height}")

‎docs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ df = pl.DataFrame({
6161
"close": close,
6262
"volume": [100_000 + i * 100 for i in range(100)],
6363
})
64-
features = compute_features(df, ["rsi", "macd", "atr"])
64+
features = compute_features(df, ["rsi", "macd", "atr"], calendar=None)
6565

6666
assert {"rsi", "macd", "atr"} <= set(features.columns)
6767
assert features.select(["rsi", "macd", "atr"]).drop_nulls().height > 0

‎docs/user-guide/dataset-builder.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -202,7 +202,7 @@ from ml4t.engineer.labeling import triple_barrier_labels
202202

203203
# 1. Compute features
204204
df = pl.read_parquet("spy_daily.parquet")
205-
features_df = compute_features(df, ["rsi", "macd", "atr", "bollinger_bands"])
205+
features_df = compute_features(df, ["rsi", "macd", "atr", "bollinger_bands"], calendar=None)
206206

207207
# 2. Create labels
208208
config = LabelingConfig.triple_barrier(

‎docs/user-guide/features.md‎

Lines changed: 93 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ df = pl.DataFrame({
4848
})
4949

5050
# 1. List of names (default parameters)
51-
result = compute_features(df, ["rsi", "macd", "atr"])
51+
result = compute_features(df, ["rsi", "macd", "atr"], calendar=None)
5252

5353
# 2. List of dicts (custom parameters)
5454
result = compute_features(df, [
@@ -58,30 +58,113 @@ result = compute_features(df, [
5858
"name": "bollinger_bands",
5959
"params": {"period": 20, "nbdevup": 2.5, "nbdevdn": 2.5},
6060
},
61-
])
61+
], calendar=None)
6262

6363
assert {"rsi", "sma", "bollinger_bands"} <= set(result.columns)
6464
```
6565

6666
Features are computed in dependency order (topological sort). Circular dependencies raise `ValueError`. The return type matches the input: `DataFrame` in, `DataFrame` out; `LazyFrame` in, `LazyFrame` out.
6767

68-
Input is sorted by asset and timestamp before computation. Common asset columns such as
69-
`asset_id`, `symbol`, and `ticker` are detected automatically, and every rolling or
70-
lagged feature is evaluated independently for each asset. If no timestamp column
71-
exists, pass `assume_sorted=True` to make the row-order assumption explicit.
68+
Common asset columns such as `asset_id`, `symbol`, and `ticker` are detected automatically,
69+
and every rolling or lagged feature is evaluated independently for each asset.
70+
The examples explicitly select `calendar=None`: their periods count observations and
71+
input order is preserved, even when timestamps are irregular. Sort prices yourself before
72+
deriving returns if chronological order is intended. Without timestamps, also pass
73+
`assume_sorted=True` to assert that the supplied row order is correct.
7274

7375
Use `output` when requesting more than one configuration of the same feature:
7476

7577
```python
7678
result = compute_features(df, [
7779
{"name": "sma", "params": {"period": 20}, "output": "sma_20"},
7880
{"name": "sma", "params": {"period": 50}, "output": "sma_50"},
79-
])
81+
], calendar=None)
8082
```
8183

8284
Unknown parameters, repeated output names, and output names that replace input columns
8385
raise `ValueError` before feature execution.
8486

87+
### Calendar and bar grid
88+
89+
The calendar-aware default targets the next minor version (0.2), separate from the 0.1.7 patch.
90+
Development and qualification currently pin the merged Specs v2 implementation at
91+
`701bde68744035050ede386efce498234c661c21`. The same immutable Git dependency is declared
92+
in built package metadata; published Specs 0.1.5 does not provide this API. Before a PyPI
93+
release, replace that source dependency with the verified published Specs version containing
94+
the v2 contract and repeat installed-package qualification.
95+
96+
Calendar selection
97+
uses the explicit `calendar` argument, then declared feed metadata, then NYSE. Explicit
98+
`calendar=None` disables schedule interpretation. A named `"24/7"` calendar instead defines
99+
expected bars and detects outages, including on weekends.
100+
101+
Calendar-aware execution requires a shared `FeedSpec(time_contract_version=2)` with an
102+
explicit fixed intraday frequency and bar-grid policy. The initial implementation supports
103+
time bars anchored at each tradable interval's open. It uses real holidays, early closes and
104+
breaks from the selected schedule. Daily/session-label grids and session or elapsed-duration
105+
windows are not supported by this validator; they fail rather than being treated as row counts.
106+
Naive timestamps need a declared timezone. Aware timestamps already declare their timezone.
107+
108+
Validate prices before deriving returns. An empty feature list performs the same grid
109+
validation and chronological sorting without adding feature columns:
110+
111+
<!-- ml4t-exec -->
112+
```python
113+
from datetime import datetime, timedelta, timezone
114+
115+
import polars as pl
116+
from ml4t.engineer import compute_features
117+
from ml4t.specs import FeedSpec
118+
119+
contract = FeedSpec(
120+
time_contract_version=2,
121+
calendar="24/7",
122+
entity_col="symbol",
123+
data_frequency="1h",
124+
timezone="UTC",
125+
bar_type="time",
126+
timestamp_semantics="bar_close",
127+
bar_grid_anchor="session_open",
128+
bar_label="right",
129+
partial_bar_policy="include",
130+
)
131+
close = [100.0 + i for i in range(8)]
132+
prices = pl.DataFrame({
133+
"timestamp": [datetime(2025, 7, 11, tzinfo=timezone.utc) + timedelta(hours=i)
134+
for i in range(8)],
135+
"symbol": "BTC",
136+
"open": close, "high": close, "low": close, "close": close, "volume": 1.0,
137+
})
138+
checked = compute_features(prices, [], feed_spec=contract)
139+
with_returns = checked.with_columns(
140+
pl.col("close").log().diff().over("symbol").alias("returns")
141+
)
142+
result = compute_features(with_returns, [{
143+
"name": "realized_volatility",
144+
"params": {"period": 3, "annualize": False},
145+
}], feed_spec=contract.to_dict(), window_unit="expected_bar")
146+
assert result.height == prices.height
147+
assert result["realized_volatility"][3] is not None
148+
```
149+
150+
Missing, duplicate or off-grid bars fail before any feature executes. The validator neither
151+
fills prices nor changes row counts. If normalization is needed, explicitly construct the
152+
intended schedule grid with null prices first, then derive returns. Reindexing an already-derived
153+
return series leaves a cross-gap return intact; finite returns adjacent to missing prices are
154+
rejected. Completeness is checked between each asset's observed endpoints, not outside unknown
155+
dataset coverage.
156+
157+
Use the canonical shared `to_dict()` methods for JSON/YAML handoffs. Omitted selectors and
158+
explicit null remain different in version 2. Legacy version-1 null metadata is unspecified,
159+
not an intentional opt-out; migrate it explicitly or pass `calendar=None` for row mode.
160+
This does not change legacy Specs bridges or the numerical feature kernels.
161+
162+
`window_unit="observation"` belongs to row mode; `"expected_bar"` belongs to a validated grid.
163+
Neither means sessions or elapsed hours. A shortened final bar still counts as one expected
164+
bar when `partial_bar_policy="include"`. Annualization remains a separate convention: for
165+
example, `realized_volatility(trading_periods=252)` retains its documented numerical default.
166+
No calendar-derived annualization factor is silently substituted.
167+
85168
> **Book**: [The ml4t Library Ecosystem](https://github.com/stefan-jansen/machine-learning-for-trading/blob/d2edec54b1c7a6a9d7a97d8129eb05db4491e1eb/07_defining_the_learning_task/10_ml4t_library_ecosystem.ipynb) calls `compute_features()` with names and parameter dictionaries, and compares the library RSI with a manual implementation.
86169
87170
## Category Reference
@@ -375,7 +458,8 @@ features:
375458
- name: yang_zhang_volatility
376459
```
377460
378-
Load with `compute_features(df, "features.yaml")`. The YAML format supports version comments and parameter documentation inline.
461+
Load row-based features with `compute_features(df, "features.yaml", calendar=None)`.
462+
The YAML format supports version comments and parameter documentation inline.
379463

380464
## Input Requirements
381465

@@ -415,7 +499,7 @@ meta = get_registry().get("rsi")
415499
print(meta.parameters) # {'period': 14}
416500
417501
# Override
418-
result = compute_features(df, [{"name": "rsi", "params": {"period": 20}}])
502+
result = compute_features(df, [{"name": "rsi", "params": {"period": 20}}], calendar=None)
419503
```
420504

421505
Invalid parameters raise `ValueError` with the valid parameter names.

‎docs/user-guide/fractional-differencing.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -142,7 +142,7 @@ print(f"p-value: {adf_result[1]:.6f}")
142142
# Use alongside other features
143143
from ml4t.engineer import compute_features
144144

145-
features = compute_features(df, ["rsi", "macd", "atr"])
145+
features = compute_features(df, ["rsi", "macd", "atr"], calendar=None)
146146
features = features.with_columns(
147147
ffdiff("close", d=optimal_d).alias("close_ffd")
148148
)
@@ -155,7 +155,7 @@ Fractional differencing is also available through the standard `compute_features
155155
```python
156156
result = compute_features(df, [
157157
{"name": "ffdiff", "params": {"d": 0.4}, "output": "close_ffd"},
158-
])
158+
], calendar=None)
159159
```
160160

161161
Optimal-d search is data-dependent, so it remains a direct two-step operation:

‎docs/user-guide/ml-readiness.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -161,7 +161,7 @@ from ml4t.engineer.core.registry import get_registry
161161
from ml4t.engineer.preprocessing import StandardScaler
162162

163163
# Compute mixed features
164-
result = compute_features(ohlcv, ["rsi", "sma", "atr", "macd"])
164+
result = compute_features(ohlcv, ["rsi", "sma", "atr", "macd"], calendar=None)
165165

166166
# Separate normalized and non-normalized
167167
registry = get_registry()

‎examples/complete_workflow_example.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ def main() -> None:
4545
"output": "bollinger_20",
4646
},
4747
],
48+
calendar=None,
4849
)
4950
labels = triple_barrier_labels(
5051
engineered,

‎examples/pipeline_example.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ def main() -> None:
3838
{"name": "rsi", "params": {"period": 14}, "output": "rsi_14"},
3939
{"name": "atr", "params": {"period": 14}, "output": "atr_14"},
4040
],
41+
calendar=None,
4142
)
4243
feature_columns = ["sma_20", "ema_12", "rsi_14", "atr_14"]
4344
usable = engineered.drop_nulls(feature_columns)

‎pyproject.toml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,10 @@ build-backend = "hatchling.build"
88
[tool.hatch.version]
99
source = "vcs"
1010

11+
[tool.hatch.metadata]
12+
# Qualification pin until the shared time-contract v2 API is published on PyPI.
13+
allow-direct-references = true
14+
1115
[tool.hatch.build.targets.wheel]
1216
packages = ["src/ml4t"]
1317
namespaces = true
@@ -72,7 +76,7 @@ requires-python = ">=3.12,<3.15"
7276

7377
# Core dependencies
7478
dependencies = [
75-
"ml4t-specs>=0.1.0,<0.2.0",
79+
"ml4t-specs @ git+https://github.com/ml4t/specs.git@701bde68744035050ede386efce498234c661c21",
7680

7781
# Data processing (Polars-first)
7882
"polars>=0.20.0",

0 commit comments

Comments
 (0)