Skip to content

Commit a84453f

Browse files
author
Stefan Jansen
committed
Merge current main into intraday grid repair and verify updater integration
2 parents 59238a0 + 721adee commit a84453f

17 files changed

Lines changed: 851 additions & 101 deletions

‎docs/api/index.md‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# API Reference
22

3-
Complete API documentation for the `ml4t-data` library, auto-generated from
3+
API documentation for the `ml4t-data` library, generated from
44
source docstrings via [mkdocstrings](https://mkdocstrings.github.io/).
55

66
---
@@ -420,3 +420,42 @@ ML4TDataError
420420
options:
421421
show_source: false
422422
heading_level: 4
423+
424+
## Futures Expiry Rolls
425+
426+
`TimeBasedRoll` selects the nearest available contract until its configured pre-expiry trigger.
427+
`FirstNoticeDateRoll` delegates the same calendar selection while retaining its existing
428+
first-notice offset convention and five-session cash-settled fallback.
429+
430+
| Mode | Offset units | Calendar behavior |
431+
|------|--------------|-------------------|
432+
| `use_business_days=True` (default) | Authoritative sessions strictly before expiration | Explicit `calendar`, then `feed_spec`, then NYSE |
433+
| `use_business_days=False` | Elapsed civil days | No session counting; an explicitly disabled calendar is allowed |
434+
| Session-count mode with `calendar=None` | Unsupported observation-based expiry offset | Raises clearly; does not substitute NYSE or 24/7 |
435+
436+
The legacy `use_business_days` name now counts the selected exchange's sessions rather than
437+
weekdays. `calendar="24/7"` counts a real continuous schedule. Contract exchange identity does not
438+
select a calendar. Pass the feed declaration explicitly when constructing a roll strategy from
439+
stored metadata. Named calendars are validated before selection.
440+
441+
An offset of zero uses expiration itself. A missing scheduled trigger does not move the trigger
442+
backward to an observed date: selection switches at the first eligible later observation, subject
443+
to the existing contract availability and paired-price checks. Explicit elapsed-day mode also
444+
stops snapping its target backward. Neither mode reinterprets missing data as a closure or changes
445+
elapsed days into row counts.
446+
447+
::: ml4t.data.futures.roll.TimeBasedRoll
448+
options:
449+
show_source: false
450+
heading_level: 3
451+
members:
452+
- __init__
453+
- select_contracts
454+
455+
::: ml4t.data.futures.roll.FirstNoticeDateRoll
456+
options:
457+
show_source: false
458+
heading_level: 3
459+
members:
460+
- __init__
461+
- select_contracts

‎docs/user-guide/incremental-updates.md‎

Lines changed: 35 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -49,10 +49,14 @@ ml4t-data update --symbol AAPL --no-fill-gaps --storage-path ./data
4949
```
5050

5151
Daily detection counts missing sessions from the selected exchange schedule, not elapsed days.
52-
The updater currently selects NYSE for non-crypto data and 24/7 for crypto. Its legacy intraday
53-
path lacks the complete grid declaration required by the detector and now fails clearly; use
54-
`--no-fill-gaps` or `fill_gaps=False` for those updates. Direct detection with a complete declaration
55-
is described below. [Trading-session completion](sessions.md) is a separate one-minute operation.
52+
The Python updater resolves its explicit `calendar`, then supplied feed or stored metadata, then
53+
NYSE. Asset class and descriptive exchange metadata do not select a schedule. Continuous feeds
54+
must declare `calendar="24/7"`; `calendar=None` disables schedule-derived detection and filling.
55+
Intraday detection requires the complete version-2 grid declaration described below, supplied
56+
through `feed_spec` or retained in stored metadata. Undeclared intraday datasets must migrate their
57+
metadata or use `--no-fill-gaps` / `fill_gaps=False` for updates.
58+
[Trading-session completion](sessions.md) is a separate one-minute operation. The acquisition-range
59+
planner's intraday limitations are described separately below.
5660

5761
Detection does not itself fill or modify observations. The updater's separately enabled forward
5862
fill uses a regular elapsed-time grid, not the selected exchange schedule, and can insert closed
@@ -75,15 +79,21 @@ applies, then NYSE. `calendar="24/7"` expects a daily observation on weekends an
7579
feed declaration selects a calendar. Version-1 feed metadata with `calendar=None` still means
7680
unspecified; version-2 metadata preserves an explicit null as disabled after serialization.
7781

78-
Daily inputs are session labels (Polars Date, or the date component of Datetime values in their
79-
existing timezone). Non-midnight labels remain supported. Exchange holidays and weekends do not
82+
Undeclared daily timestamps and explicit `session_label` inputs are labels (Polars Date, or the
83+
date component of Datetime values). Aware labels restore a declared `timezone` before extracting
84+
dates, including after storage's canonical UTC read. Naive labels remain civil identifiers rather
85+
than localized event clocks. Non-midnight labels remain supported. Exchange holidays and weekends do not
8086
count as missing NYSE sessions; a shortened session still requires one daily observation. Gap bounds
8187
remain the two observed timestamps, while `missing_periods` counts only scheduled dates strictly
8288
between them. Inspect each asset separately; observed dates from other assets must not define the
8389
schedule. This operation does not validate off-session observations or discover gaps outside the
8490
observed range.
8591

86-
Explicit event-time/bar-close metadata is rejected rather than reinterpreted as session labels.
92+
Explicit `event_time`, `bar_close` or bar-side metadata instead uses native authoritative session
93+
mapping, even with daily frequency or midnight timestamps. Right-labeled continuous midnight belongs
94+
to the preceding session; event time belongs to the new session. Out-of-session instants fail clearly.
95+
Gap classification does not add a new off-session rejection policy for legacy daily labels. The
96+
requested daily frequency cannot silently override a declared intraday frequency.
8797
Unknown calendars and malformed daily timestamps fail without modifying the input. `fill_gaps()`
8898
remains a separate elapsed-grid operation, not calendar-aware imputation.
8999

@@ -148,10 +158,11 @@ or feed declaration is present. `exclude_weekends=True` now follows NYSE holiday
148158
Omission follows NYSE, so callers requiring the old elapsed-day counts should pass `calendar="24/7"`.
149159
Intraday/weekly range planning retains its legacy elapsed-time behavior; named calendar/feed
150160
declarations for those frequencies are not supported by this planner yet. `calendar=None` disables
151-
classification. Session-label dates retain their existing timezone's civil date. Explicit event-time
152-
or bar-close declarations are rejected for daily planning rather than silently relabeled. Declare
153-
the feed's `timezone` when a non-UTC label must survive storage's canonical UTC reads; the planner
154-
restores that declared timezone before interpreting dates.
161+
classification. Session-label dates use their declared timezone, or their existing timezone when
162+
undeclared. Explicit event-time/bar-close/bar-side declarations map to authoritative session labels
163+
before planning, including for provider-row selection during backfill. Out-of-session instants fail
164+
clearly; original provider timestamps are retained in storage. Declare the feed's `timezone` when a
165+
non-UTC label must survive storage's canonical UTC reads.
155166

156167
If a provider's history limit cannot reach the stored dataset, the updater records
157168
`provider_history_limited` and skips gap filling across the unavailable interval.
@@ -182,6 +193,19 @@ key = manager.update(
182193
`DataManager.update()` returns the storage key. It raises when storage is not configured, stored data
183194
is empty, metadata is invalid, or the provider or storage operation fails.
184195

196+
`load()`, `import_data()` and `update()` accept `calendar` and a shared `feed_spec`. New ingestion
197+
without a feed writes version-2 temporal metadata: an omitted calendar remains omitted, while
198+
explicit `None` remains disabled after storage and model projection. An existing version-1 feed
199+
is not automatically reinterpreted; its null calendar still means inheritance. An explicit calendar
200+
override writes version 2. Calendar names and conflicting frequency/bar-type declarations fail
201+
before a replacement generation is published.
202+
203+
Feed timezone, timestamp semantics and bar-grid declarations survive ingestion. A non-null
204+
`session_start_time` override is not supported by the native schedule adapter and is rejected.
205+
The requested frequency and bar type must agree with declared feed metadata. These declarations
206+
do not themselves validate OHLCV completeness or transform timestamps. Gap detection remains
207+
separate from the updater's optional elapsed-grid imputation described above.
208+
185209
## Inspect Stored State
186210

187211
List keys and inspect one dataset:

‎src/ml4t/data/calendar/crypto.py‎

Lines changed: 0 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -316,17 +316,6 @@ def validate_continuity(
316316
f"({duration.total_seconds() / 3600:.1f} hours)"
317317
)
318318

319-
# Check for weekend data (crypto should have it)
320-
timestamps = df["timestamp"].to_list()
321-
weekend_dates = [
322-
ts
323-
for ts in timestamps
324-
if ts.weekday() in [5, 6] # Saturday, Sunday
325-
]
326-
327-
if not weekend_dates and len(timestamps) >= 7:
328-
issues.append("Missing weekend data (crypto trades 24/7)")
329-
330319
return issues
331320

332321
def validate_volume_profile(

‎src/ml4t/data/core/models.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,11 @@ def serialize_time_declaration(self, handler: Any) -> dict[str, Any]:
149149
data.pop(name, None)
150150
return data
151151

152+
@property
153+
def data_frequency(self) -> str | None:
154+
"""Project a physical frequency without the legacy unknown or event-count label."""
155+
return self.bar_params.get("frequency") if self.bar_type == "time" else None
156+
152157
@property
153158
def frequency(self) -> str:
154159
"""Get frequency string for backward compatibility.

‎src/ml4t/data/data_manager.py‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -339,7 +339,9 @@ def load(
339339
bar_type: str = "time",
340340
bar_threshold: int | None = None,
341341
exchange: str = "UNKNOWN",
342-
calendar: str | None = None,
342+
calendar: str | None | CalendarUnset = UNSET_CALENDAR,
343+
*,
344+
feed_spec: Any | None = None,
343345
) -> str:
344346
"""Load data from provider and store it."""
345347
if not self._storage_manager:
@@ -357,6 +359,7 @@ def load(
357359
bar_threshold,
358360
exchange,
359361
calendar,
362+
feed_spec=feed_spec,
360363
)
361364

362365
def import_data(
@@ -369,7 +372,9 @@ def import_data(
369372
bar_type: str = "time",
370373
bar_threshold: int | None = None,
371374
exchange: str = "UNKNOWN",
372-
calendar: str | None = None,
375+
calendar: str | None | CalendarUnset = UNSET_CALENDAR,
376+
*,
377+
feed_spec: Any | None = None,
373378
) -> str:
374379
"""Import external data into storage with metadata."""
375380
if not self._storage_manager:
@@ -386,6 +391,7 @@ def import_data(
386391
bar_threshold,
387392
exchange,
388393
calendar,
394+
feed_spec=feed_spec,
389395
)
390396

391397
def update(
@@ -399,6 +405,9 @@ def update(
399405
initial_start: str | None = None,
400406
initial_end: str | None = None,
401407
initial_load_days: int = 365,
408+
*,
409+
calendar: str | None | CalendarUnset = UNSET_CALENDAR,
410+
feed_spec: Any | None = None,
402411
) -> str:
403412
"""Update existing data with incremental fetch."""
404413
if not self._storage_manager:
@@ -415,6 +424,8 @@ def update(
415424
initial_start,
416425
initial_end,
417426
initial_load_days,
427+
calendar=calendar,
428+
feed_spec=feed_spec,
418429
)
419430

420431
# ========================================================================

‎src/ml4t/data/futures/roll.py‎

Lines changed: 57 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,13 @@
66
from abc import ABC, abstractmethod
77
from dataclasses import dataclass
88
from datetime import date, timedelta
9+
from typing import Any
910

1011
import polars as pl
1112

1213
from ml4t.data.futures.schema import ContractSpec
14+
from ml4t.data.sessions.assigner import SessionAssigner
15+
from ml4t.specs import UNSET_CALENDAR, CalendarUnset, FeedSpec
1316

1417

1518
@dataclass(frozen=True)
@@ -99,17 +102,34 @@ def select_contracts(
99102

100103

101104
class TimeBasedRoll(RollStrategy):
102-
"""Select the nearest contract until its configured pre-expiry roll date."""
105+
"""Select the nearest contract until its configured pre-expiry roll date.
106+
107+
``use_business_days=True`` counts authoritative sessions strictly before expiration.
108+
Explicit ``calendar`` overrides ``feed_spec``, then NYSE; an exchange identity does not
109+
select a calendar. A missing target session never moves the trigger to an earlier observation.
110+
``use_business_days=False`` selects elapsed civil days, not an observation count.
111+
Calendar-free row-based expiry offsets are unsupported in session-count mode.
112+
"""
103113

104114
def __init__(
105115
self,
106116
days_before_expiration: int = 5,
107117
use_business_days: bool = True,
108-
):
118+
*,
119+
calendar: str | None | CalendarUnset = UNSET_CALENDAR,
120+
feed_spec: Any | None = None,
121+
) -> None:
109122
if days_before_expiration < 0:
110123
raise ValueError("days_before_expiration cannot be negative")
111124
self.days_before_expiration = days_before_expiration
112125
self.use_business_days = use_business_days
126+
self.feed_spec = FeedSpec.from_any(feed_spec)
127+
self.calendar = self.feed_spec.resolve_calendar(calendar=calendar)
128+
if self.calendar is None and use_business_days:
129+
raise ValueError("calendar-free row-based expiry rolls are not supported")
130+
self._schedule_calendar = (
131+
SessionAssigner(self.calendar).calendar if self.calendar is not None else None
132+
)
113133

114134
def select_contracts(
115135
self,
@@ -122,7 +142,7 @@ def select_contracts(
122142
return _empty_selections()
123143
contracts = dict(_contract_expirations(data))
124144
roll_dates = {
125-
symbol: self._calculate_roll_date(expiration, dates)
145+
symbol: self._calculate_roll_date(expiration)
126146
for symbol, expiration in contracts.items()
127147
}
128148
available_by_date = _available_symbols_by_date(data)
@@ -139,38 +159,54 @@ def select_contracts(
139159
)
140160
return _selection_frame(records)
141161

142-
def _calculate_roll_date(self, expiration: date, trading_dates: list[date]) -> date:
143-
"""Calculate the last configured roll date on the observed calendar."""
144-
target = expiration
145-
if self.use_business_days:
146-
days_counted = 0
147-
while days_counted < self.days_before_expiration:
148-
target -= timedelta(days=1)
149-
if target.weekday() < 5:
150-
days_counted += 1
151-
else:
152-
target -= timedelta(days=self.days_before_expiration)
153-
if target >= trading_dates[-1] or target < trading_dates[0]:
154-
return target
155-
observed = [trading_date for trading_date in trading_dates if trading_date <= target]
156-
return max(observed) if observed else target
162+
def _calculate_roll_date(self, expiration: date) -> date:
163+
"""Calculate a schedule or explicit elapsed-day trigger independently of observations."""
164+
if not self.use_business_days:
165+
return expiration - timedelta(days=self.days_before_expiration)
166+
if self.days_before_expiration == 0:
167+
return expiration
168+
span = max(14, self.days_before_expiration * 2)
169+
while True:
170+
try:
171+
start = expiration - timedelta(days=span)
172+
except OverflowError as error:
173+
raise ValueError(
174+
"not enough calendar history for the requested expiry offset"
175+
) from error
176+
schedule = self._schedule_calendar.schedule(
177+
start_date=start, end_date=expiration - timedelta(days=1)
178+
)
179+
if len(schedule) >= self.days_before_expiration:
180+
return schedule.index[-self.days_before_expiration].date()
181+
span *= 2
157182

158183

159184
class FirstNoticeDateRoll(RollStrategy):
160185
"""Select physical contracts before the configured first-notice interval."""
161186

162-
def __init__(self, days_before_first_notice: int = 1):
187+
def __init__(
188+
self,
189+
days_before_first_notice: int = 1,
190+
*,
191+
calendar: str | None | CalendarUnset = UNSET_CALENDAR,
192+
feed_spec: Any | None = None,
193+
) -> None:
163194
if days_before_first_notice < 0:
164195
raise ValueError("days_before_first_notice cannot be negative")
165196
self.days_before_first_notice = days_before_first_notice
197+
self._time_roll = TimeBasedRoll(calendar=calendar, feed_spec=feed_spec)
166198

167199
def select_contracts(
168200
self, data: pl.DataFrame, contract_spec: ContractSpec | None = None
169201
) -> pl.DataFrame:
170202
if contract_spec is None or contract_spec.is_cash_settled:
171-
return TimeBasedRoll(days_before_expiration=5).select_contracts(data, contract_spec)
203+
return self._time_roll.select_contracts(data, contract_spec)
172204
days = (contract_spec.first_notice_days or 25) + self.days_before_first_notice
173-
return TimeBasedRoll(days_before_expiration=days).select_contracts(data, contract_spec)
205+
return TimeBasedRoll(
206+
days_before_expiration=days,
207+
calendar=self._time_roll.calendar,
208+
feed_spec=self._time_roll.feed_spec,
209+
).select_contracts(data, contract_spec)
174210

175211

176212
class CalendarRoll(RollStrategy):

0 commit comments

Comments
 (0)