Repository navigation
Rolling features count rows as time across missing bars #76
Description
Problem
Every registered feature with a lookback counts rows: realized_volatility(period=24) is the
standard deviation of the last 24 rows. When a group has missing bars, a window that spans the
gap covers more time than its period says, and a return computed across the gap reads a
multi-day move as one bar's move. Nothing in compute_features sees the gap: the timestamps
are sorted, never compared with the spacing the window assumes.
Shipped instance: the ML4T R2P course's crypto case computed price_vol_7d as a 168-row
rolling standard deviation of hourly log returns. Across 49 to 73 hour exchange data gaps,
152 feature rows read a multi-day move as one hour, inflating volatility by up to 23%.
Reproduction (public API, main at a441fb4)
from datetime import datetime, timedelta
import polars as pl
from ml4t.engineer import compute_features
start = datetime(2024, 3, 1)
hours = [start + timedelta(hours=h) for h in range(48)]
hours += [start + timedelta(hours=48 + 49 + h) for h in range(48)] # 49-hour outage
close = [100.0 + 0.1 * i for i in range(48)] + [130.0 + 0.1 * i for i in range(48)]
df = pl.DataFrame(
{"timestamp": hours, "symbol": "BTC", "open": close, "high": close, "low": close,
"close": close, "volume": 1.0}
).with_columns(pl.col("close").log().diff().over("symbol").alias("returns"))
out = compute_features(
df,
[{"name": "realized_volatility", "params": {"period": 24, "annualize": False},
"output": "vol_24h"}],
group_col="symbol",
timestamp_col="timestamp",
)Observed: vol_24h five hours after the outage is 0.0440; its largest value before the
outage is 6.9e-06. No error and no warning.
Expected: a 24-hour volatility either covers 24 hours, with the window empty (null) where
bars are missing, or the call refuses and says how to make it do so.
Why the library cannot decide alone
A gap is only missing data relative to a calendar. Hourly CME futures have a weekend gap and a
daily break; daily equities skip weekends and holidays; crypto trades around the clock. The
timestamps of a correct CME series and of a crypto series with an outage look alike, so any
threshold on spacing either refuses correct session data or passes the defect.
Proposed behavior (for agreement here before implementation)
compute_features(..., calendar=None): a new keyword-only argument naming the calendar the
rows follow ("crypto"or an exchange calendar, ascore.calendarsandml4t.specs
FeedSpec.calendaralready name them). When given, each group's timestamps are compared
with the calendar's steps at the data's frequency before any feature whose lookback counts
rows. A missing step raisesValueErrornaming the group, the first missing timestamp and
the fix: reindex the group to the calendar grid with null rows for missing bars, after which
a window of N rows always spans N calendar steps and windows over missing bars are null.- Without
calendar, values are unchanged. When a row-window feature runs over a group whose
largest step exceeds its typical step, a warning names the group, the gap and the window,
and says that passingcalendarmakes the check exact. - In 0.2,
calendarbecomes required for row-window features over timestamped data, so the
unit of every window is stated.
Compatibility: no computed value changes in 0.1.x; one new optional argument and one new
warning. Item 3 is a breaking change and is listed for discussion only.
Alternatives considered: computing windows by duration (rolling_std_by) changes every
registered feature's definition from rows to time; reindexing silently inside
compute_features changes row counts and output length without the caller asking.
Activity
stefan-jansen commentedon Oct 15, 2026
Triage: feature, normal priority, blocked on an explicit shared Specs/FeedSpec input contract, with cross-library compatibility impact. Affected published release: 0.1.6 (baseline a441fb4). The proposed opt-in validation extends the current row-based contract; it does not justify silently changing existing window definitions.
Reproduced through the public compute_features workflow. The example returns 96 rows, volatility 0.04399177755209451 five rows after the gap, a pre-gap maximum of 0.000006905079154816273, and no warning.
The safe workaround was also checked. Reindex the prices onto the explicitly intended hourly grid with null prices before calculating returns, then compute the features. That produces 145 rows and null volatility five hours after the outage. Reindexing an already-computed returns column does not remove the cross-gap return.
Recommendation: retain the current row-window definition and add opt-in continuity validation only after its input contract is explicit. A calendar name alone is insufficient: the validator also needs expected frequency, grid alignment, timestamp convention (bar start or end), and the relevant session/break rules. Inferring the contract from the median or largest observed timestamp difference would not distinguish a legitimate session break from missing data. I do not recommend a mandatory calendar argument or an inferred-gap warning as part of the current patch release.
This issue remains open for that contract decision. The implementation should then prove missing/duplicate-bar rejection, valid session breaks and DST, independent groups, empty/singleton input, null prices/returns, and supported eager/lazy workflows. The eventual validator should report the affected group and first missing expected timestamp without silently changing row counts or numerical definitions. Shared FeedSpec contract changes belong in ml4t/specs before Engineer adopts them.
AI assistance: Codex inspected the implementation, ran the original and normalized public workflows, and prepared this maintenance assessment.
Current-main verification at 6b6d02c: the isolated hosted wheel reproduces the same 96-row result, 0.04399177755209451 volatility and no warning. Reindexing prices before returns still produces 145 rows and null volatility five hours after the outage. The recommendation and contract blocker above are unchanged.
stefan-jansen commentedon Oct 17, 2026
Post-merge public workflow audit found one remaining row-mode contract defect in main0e78652. With valid OHLCV rows but no timestamp column, compute_features(prices,['sma'],calendar=None) still raises DataSchemaError asking for a time column or assume_sorted=True. Passing the same rows with timestamps, or adding assume_sorted=True, succeeds. The shared contract defines explicitNone as input observation order, so no temporal-order assertion or timestamp is needed merely to count rows.
Reopening existing issue76 for a focused follow-up rather than creating another owning issue for the same true-row-mode requirement. The merged missing-bar/grid repair in PR90 remains meaningful and qualified. The follow-up will test eager/lazy no-timestamp row workflows, include timestamp-spacing/order controls and preserve calendar-aware declaration failures. It will not change numerical kernels or expand supported calendar window types. Complete applicable hosted checks remain required before ordinary merge; no release/deploy or archive action is requested.
AI assistance: Codex found this failure while reproducing the independently owned Hurst core dependency bug, checked the actual public feature pipeline and prepared the focused regression/fix.
stefan-jansen commentedon Oct 17, 2026
The accepted remaining feature-time work stays in this owning issue. The original proposal's omitted-calendar and gap-warning suggestions have been superseded by the shared contract: explicit calendar argument > explicitly declared feed metadata > NYSE, with explicit None selecting input observation rows. Frequency/holidays are not inferred from observed spacing.
Implemented so far: PR90 validates a declared fixed-intraday grid before feature computation and rejects invalid supplied returns adjacent to missing prices. PR98 separately repairs timestamp-free row mode. Their retained public producer-workflow base/fix evidence and full hosted qualification do not establish the following remaining capabilities:
PR98 now references rather than automatically closes this issue. Remaining implementation must get focused failing-base/passing-fix public workflow evidence and applicable supported artifact/source/docs checks. These accepted requirements are not complete merely because unsupported modes are documented or an earlier PR is green.
AI assistance: Codex reconciled current source, shared contract and accepted requirements. Existing executed evidence retains its exact revision; the remaining cases above are requirements, not claimed new passing tests.
feat(api): distinguish causal session and elapsed feature windows (#76)
test(api): retain independent annualization consumer regressions (#76)
test(api): retain scheduled sessions with no bars under exclusion pol…
fix(temporal)!: qualify shared calendar grids, feature windows and la…