Skip to content

Commit 721adee

Browse files
fix: preserve daily timestamp declarations through storage and gaps
Closes ml4t#103. Share native daily-session projection across gap utilities, range planning and provider-row backfill. Preserve declared timezone labels after canonical UTC storage and honor explicit instant semantics. Actual provider/storage/backfill regressions and all applicable exact-head gates passed. AI-assisted source/tests/prose reviewed and verified under maintainer direction.
1 parent 1501f00 commit 721adee

6 files changed

Lines changed: 251 additions & 48 deletions

File tree

‎docs/user-guide/incremental-updates.md‎

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -77,15 +77,21 @@ applies, then NYSE. `calendar="24/7"` expects a daily observation on weekends an
7777
feed declaration selects a calendar. Version-1 feed metadata with `calendar=None` still means
7878
unspecified; version-2 metadata preserves an explicit null as disabled after serialization.
7979

80-
Daily inputs are session labels (Polars Date, or the date component of Datetime values in their
81-
existing timezone). Non-midnight labels remain supported. Exchange holidays and weekends do not
80+
Undeclared daily timestamps and explicit `session_label` inputs are labels (Polars Date, or the
81+
date component of Datetime values). Aware labels restore a declared `timezone` before extracting
82+
dates, including after storage's canonical UTC read. Naive labels remain civil identifiers rather
83+
than localized event clocks. Non-midnight labels remain supported. Exchange holidays and weekends do not
8284
count as missing NYSE sessions; a shortened session still requires one daily observation. Gap bounds
8385
remain the two observed timestamps, while `missing_periods` counts only scheduled dates strictly
8486
between them. Inspect each asset separately; observed dates from other assets must not define the
8587
schedule. This operation does not validate off-session observations or discover gaps outside the
8688
observed range.
8789

88-
Explicit event-time/bar-close metadata is rejected rather than reinterpreted as session labels.
90+
Explicit `event_time`, `bar_close` or bar-side metadata instead uses native authoritative session
91+
mapping, even with daily frequency or midnight timestamps. Right-labeled continuous midnight belongs
92+
to the preceding session; event time belongs to the new session. Out-of-session instants fail clearly.
93+
Gap classification does not add a new off-session rejection policy for legacy daily labels. The
94+
requested daily frequency cannot silently override a declared intraday frequency.
8995
Named calendar/feed declarations for intraday detection are not supported yet; legacy intraday
9096
calls retain their existing behavior. Unknown calendars and malformed daily timestamps fail without
9197
modifying the input. `fill_gaps()` remains a separate elapsed-grid operation, not calendar-aware
@@ -111,10 +117,11 @@ or feed declaration is present. `exclude_weekends=True` now follows NYSE holiday
111117
Omission follows NYSE, so callers requiring the old elapsed-day counts should pass `calendar="24/7"`.
112118
Intraday/weekly range planning retains its legacy elapsed-time behavior; named calendar/feed
113119
declarations for those frequencies are not supported by this planner yet. `calendar=None` disables
114-
classification. Session-label dates retain their existing timezone's civil date. Explicit event-time
115-
or bar-close declarations are rejected for daily planning rather than silently relabeled. Declare
116-
the feed's `timezone` when a non-UTC label must survive storage's canonical UTC reads; the planner
117-
restores that declared timezone before interpreting dates.
120+
classification. Session-label dates use their declared timezone, or their existing timezone when
121+
undeclared. Explicit event-time/bar-close/bar-side declarations map to authoritative session labels
122+
before planning, including for provider-row selection during backfill. Out-of-session instants fail
123+
clearly; original provider timestamps are retained in storage. Declare the feed's `timezone` when a
124+
non-UTC label must survive storage's canonical UTC reads.
118125

119126
If a provider's history limit cannot reach the stored dataset, the updater records
120127
`provider_history_limited` and skips gap filling across the unavailable interval.

‎src/ml4t/data/sessions/assigner.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,45 @@ def _session_intervals(row: Any) -> list[tuple[datetime, datetime]]:
6666
}
6767

6868

69+
def _daily_session_labels(
70+
df: pl.DataFrame, assigner: SessionAssigner, timestamp_col: str = "timestamp"
71+
) -> pl.Series:
72+
if timestamp_col not in df.columns:
73+
raise ValueError(f"DataFrame must have '{timestamp_col}' column")
74+
feed = assigner.feed_spec
75+
if feed.data_frequency is not None and str(feed.data_frequency).lower() not in {
76+
"daily",
77+
"1day",
78+
"1d",
79+
}:
80+
raise ValueError("daily session projection requires a declared daily frequency")
81+
if df.is_empty():
82+
return pl.Series(timestamp_col, [], dtype=pl.Date)
83+
dtype = df.schema[timestamp_col]
84+
if dtype != pl.Date and not isinstance(dtype, pl.Datetime):
85+
raise TypeError(f"'{timestamp_col}' must contain Date or Datetime values")
86+
if df[timestamp_col].null_count():
87+
raise ValueError(f"'{timestamp_col}' must not contain null timestamps")
88+
if feed.timestamp_semantics in (
89+
TimestampSemantics.EVENT_TIME,
90+
TimestampSemantics.BAR_CLOSE,
91+
) or (
92+
feed.timestamp_semantics != TimestampSemantics.SESSION_LABEL and feed.bar_label is not None
93+
):
94+
assigned = assigner.assign_sessions(
95+
df.select(pl.col(timestamp_col).alias("timestamp")),
96+
outside_session="raise",
97+
bar_frequency="daily",
98+
)
99+
return assigned["session_date"].rename(timestamp_col)
100+
labels = df[timestamp_col]
101+
if isinstance(dtype, pl.Datetime):
102+
if dtype.time_zone is not None and feed.timezone is not None:
103+
labels = labels.dt.convert_time_zone(feed.timezone)
104+
labels = labels.dt.date()
105+
return labels
106+
107+
69108
class SessionAssigner:
70109
"""Assigns session dates to trading data based on exchange calendars.
71110

‎src/ml4t/data/update_manager.py‎

Lines changed: 10 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,10 @@
1313

1414
from ml4t.data.core.schemas import align_frames_for_concat
1515
from ml4t.data.providers.base import BaseProvider
16-
from ml4t.data.sessions.assigner import SessionAssigner
16+
from ml4t.data.sessions.assigner import SessionAssigner, _daily_session_labels
1717
from ml4t.data.storage.backend import StorageBackend, normalize_storage_metadata
1818
from ml4t.data.storage.metadata_tracker import MetadataTracker, UpdateRecord
19-
from ml4t.specs import UNSET_CALENDAR, CalendarUnset, FeedSpec, TimestampSemantics
19+
from ml4t.specs import UNSET_CALENDAR, CalendarUnset, FeedSpec
2020

2121
logger = structlog.get_logger()
2222

@@ -136,34 +136,20 @@ def _feed_calendar(
136136
return feed, feed.resolve_calendar(calendar=calendar)
137137

138138
@staticmethod
139-
def _label_frame(df: pl.DataFrame, feed: FeedSpec) -> pl.DataFrame:
140-
dtype = df.schema["timestamp"]
141-
if dtype != pl.Date and not isinstance(dtype, pl.Datetime):
142-
raise TypeError("'timestamp' must contain Date or Datetime values")
143-
if df["timestamp"].null_count():
144-
raise ValueError("'timestamp' must not contain null timestamps")
145-
if isinstance(dtype, pl.Datetime) and feed.timezone is not None:
146-
timestamp = pl.col("timestamp")
147-
if dtype.time_zone is None:
148-
timestamp = timestamp.dt.replace_time_zone(feed.timezone)
149-
else:
150-
timestamp = timestamp.dt.convert_time_zone(feed.timezone)
151-
return df.with_columns(timestamp)
152-
return df
139+
def _label_frame(df: pl.DataFrame, feed: FeedSpec, calendar: str | None) -> pl.DataFrame:
140+
labels = _daily_session_labels(df, SessionAssigner(calendar, feed_spec=feed))
141+
return df.with_columns(labels.cast(pl.Datetime("us")).dt.replace_time_zone("UTC"))
153142

154143
@staticmethod
155144
def _daily_ranges(
156145
df: pl.DataFrame,
157146
start: datetime,
158147
end: datetime,
159-
feed: FeedSpec,
160148
calendar: str,
161149
tolerance_days: int = 0,
162150
) -> list[dict[str, Any]]:
163151
if start.date() > end.date():
164152
raise ValueError("start_date must not be after end_date")
165-
if feed.timestamp_semantics not in (None, TimestampSemantics.SESSION_LABEL):
166-
raise NotImplementedError("daily range planning requires session_label timestamps")
167153
assigner = SessionAssigner(calendar)
168154
observed = {
169155
value.date() if isinstance(value, datetime) else value
@@ -222,12 +208,11 @@ def detect_gaps(
222208
if df.is_empty() or "timestamp" not in df.columns:
223209
return []
224210
if frequency.lower() in {"daily", "1day"}:
225-
df = self._label_frame(df, feed)
211+
df = self._label_frame(df, feed, selected)
226212
return self._daily_ranges(
227213
df,
228214
_session_label(df["timestamp"].min()),
229215
_session_label(df["timestamp"].max()),
230-
feed,
231216
selected,
232217
tolerance_days,
233218
)
@@ -312,7 +297,7 @@ def detect_gaps_in_storage(
312297
except KeyError:
313298
df = pl.DataFrame(schema={"timestamp": pl.Datetime("us", "UTC")})
314299
return self._daily_ranges(
315-
self._label_frame(df, feed), start_date, end_date, feed, selected
300+
self._label_frame(df, feed, selected), start_date, end_date, selected
316301
)
317302
start_date = _ensure_datetime(start_date)
318303
end_date = _ensure_datetime(end_date)
@@ -677,7 +662,9 @@ def apply_strategy(
677662
start_date=gap["start"].date(), end_date=gap["end"].date()
678663
)
679664
gap_rows = new_data.filter(
680-
self.gap_detector._label_frame(new_data, feed)["timestamp"]
665+
self.gap_detector._label_frame(new_data, feed, selected_calendar)[
666+
"timestamp"
667+
]
681668
.dt.date()
682669
.is_in([label.date() for label in expected.index])
683670
)

‎src/ml4t/data/utils/gaps.py‎

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,8 @@
1010
import polars as pl
1111
import structlog
1212

13-
from ml4t.data.sessions.assigner import SessionAssigner
14-
from ml4t.specs import UNSET_CALENDAR, CalendarUnset, FeedSpec, TimestampSemantics
13+
from ml4t.data.sessions.assigner import SessionAssigner, _daily_session_labels
14+
from ml4t.specs import UNSET_CALENDAR, CalendarUnset, FeedSpec
1515

1616
logger = structlog.get_logger()
1717

@@ -185,21 +185,12 @@ def _detect_daily_gaps(
185185
calendar: str,
186186
feed: FeedSpec,
187187
) -> list[DataGap]:
188-
assigner = SessionAssigner(calendar)
189-
if timestamp_col not in df.columns:
190-
raise ValueError(f"DataFrame must have '{timestamp_col}' column")
188+
assigner = SessionAssigner(calendar, feed_spec=feed)
189+
ordered = df.sort(timestamp_col) if timestamp_col in df.columns else df
190+
labels = _daily_session_labels(ordered, assigner, timestamp_col).to_list()
191191
if df.is_empty():
192192
return []
193-
dtype = df.schema[timestamp_col]
194-
if dtype != pl.Date and not isinstance(dtype, pl.Datetime):
195-
raise TypeError(f"'{timestamp_col}' must contain Date or Datetime values")
196-
if df[timestamp_col].null_count():
197-
raise ValueError(f"'{timestamp_col}' must not contain null timestamps")
198-
if feed.timestamp_semantics not in (None, TimestampSemantics.SESSION_LABEL):
199-
raise NotImplementedError("daily gap detection requires session_label timestamps")
200-
201-
timestamps = df.sort(timestamp_col)[timestamp_col].to_list()
202-
labels = [value.date() if isinstance(value, datetime) else value for value in timestamps]
193+
timestamps = ordered[timestamp_col].to_list()
203194
sessions = [
204195
label.date()
205196
for label in assigner.calendar.schedule(start_date=labels[0], end_date=labels[-1]).index

‎tests/test_daily_session_gaps.py‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -142,10 +142,9 @@ def test_date_session_labels_have_the_same_schedule_as_datetime_labels() -> None
142142

143143

144144
@pytest.mark.parametrize("semantics", ["event_time", "bar_close"])
145-
def test_event_semantics_are_not_silently_reinterpreted_as_daily_labels(semantics) -> None:
145+
def test_explicit_daily_instants_are_mapped_without_creating_a_weekend_gap(semantics) -> None:
146146
frame = provider_frame("2025-07-11", "2025-07-14")
147-
with pytest.raises(NotImplementedError, match="session_label"):
148-
GapDetector().detect_gaps(frame, feed_spec={"timestamp_semantics": semantics})
147+
assert GapDetector().detect_gaps(frame, feed_spec={"timestamp_semantics": semantics}) == []
149148

150149

151150
def test_null_timestamps_fail_without_input_mutation() -> None:

0 commit comments

Comments
 (0)