Skip to content

Commit dc9344d

Browse files
author
Stefan Jansen
committed
fix: count daily missing observations against declared sessions
1 parent 486f5c6 commit dc9344d

6 files changed

Lines changed: 184 additions & 10 deletions

File tree

‎docs/user-guide/incremental-updates.md‎

Lines changed: 39 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -48,10 +48,45 @@ remain explicit:
4848
ml4t-data update --symbol AAPL --no-fill-gaps --storage-path ./data
4949
```
5050

51-
The detector uses the requested frequency and distinguishes crypto, which trades continuously, from
52-
other asset classes. It treats common overnight and weekend intervals as expected for non-crypto
53-
intraday data. This is a general interval check, not an exchange-calendar reconstruction. Use
54-
[trading-session completion](sessions.md) when exact exchange schedules are required.
51+
Daily detection counts missing sessions from the selected exchange schedule, not elapsed days.
52+
The updater currently selects NYSE for non-crypto data and 24/7 for crypto. Intraday detection still
53+
uses approximate US overnight/weekend rules. It is not an exchange-calendar reconstruction. Use
54+
[trading-session completion](sessions.md) when exact intraday schedules are required.
55+
56+
Detection does not itself fill or modify observations. The updater's separately enabled forward
57+
fill uses a regular elapsed-time grid, not the selected exchange schedule, and can insert closed
58+
dates when a real gap is present. Disable filling when those synthetic rows are inappropriate.
59+
60+
### Inspect Daily Gaps Without Filling
61+
62+
```python
63+
from ml4t.data.utils.gaps import GapDetector
64+
from ml4t.specs import FeedSpec
65+
66+
feed = FeedSpec(calendar="NYSE", time_contract_version=2)
67+
gaps = GapDetector().detect_gaps(df, frequency="daily", feed_spec=feed)
68+
```
69+
70+
The canonical `calendar` keyword overrides the feed's declaration. Otherwise the declared calendar
71+
applies, then NYSE. `calendar="24/7"` expects a daily observation on weekends and holidays;
72+
`calendar=None` disables schedule-based gap classification, regardless of timestamp spacing.
73+
`is_crypto=True` remains a compatibility hint for 24/7 only when neither an explicit calendar nor a
74+
feed declaration selects a calendar. Version-1 feed metadata with `calendar=None` still means
75+
unspecified; version-2 metadata preserves an explicit null as disabled after serialization.
76+
77+
Daily inputs are session labels (Polars Date, or the date component of Datetime values in their
78+
existing timezone). Non-midnight labels remain supported. Exchange holidays and weekends do not
79+
count as missing NYSE sessions; a shortened session still requires one daily observation. Gap bounds
80+
remain the two observed timestamps, while `missing_periods` counts only scheduled dates strictly
81+
between them. Inspect each asset separately; observed dates from other assets must not define the
82+
schedule. This operation does not validate off-session observations or discover gaps outside the
83+
observed range.
84+
85+
Explicit event-time/bar-close metadata is rejected rather than reinterpreted as session labels.
86+
Named calendar/feed declarations for intraday detection are not supported yet; legacy intraday
87+
calls retain their existing behavior. Unknown calendars and malformed daily timestamps fail without
88+
modifying the input. `fill_gaps()` remains a separate elapsed-grid operation, not calendar-aware
89+
imputation.
5590

5691
If a provider's history limit cannot reach the stored dataset, the updater records
5792
`provider_history_limited` and skips gap filling across the unavailable interval.

‎pyproject.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,9 @@ build-backend = "hatchling.build"
88
[tool.hatch.version]
99
source = "vcs"
1010

11+
[tool.hatch.metadata]
12+
allow-direct-references = true
13+
1114
[tool.hatch.build.hooks.vcs]
1215
version-file = "src/ml4t/data/_version.py"
1316

@@ -74,6 +77,7 @@ requires-python = ">=3.12,<3.15"
7477

7578
# Core dependencies
7679
dependencies = [
80+
"ml4t-specs @ git+https://github.com/ml4t/specs.git@701bde68744035050ede386efce498234c661c21",
7781
# Data processing
7882
"polars>=0.20.0",
7983
"pandas>=2.0.0",

‎src/ml4t/data/utils/gaps.py‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,17 @@
22

33
from __future__ import annotations
44

5+
from bisect import bisect_left, bisect_right
56
from dataclasses import dataclass
67
from datetime import UTC, date, datetime, timedelta
78
from typing import Any, ClassVar
89

910
import polars as pl
1011
import structlog
1112

13+
from ml4t.data.sessions.assigner import SessionAssigner
14+
from ml4t.specs import UNSET_CALENDAR, CalendarUnset, FeedSpec, TimestampSemantics
15+
1216
logger = structlog.get_logger()
1317

1418

@@ -69,6 +73,9 @@ def detect_gaps(
6973
frequency: str = "daily",
7074
timestamp_col: str = "timestamp",
7175
is_crypto: bool = False,
76+
*,
77+
calendar: str | None | CalendarUnset = UNSET_CALENDAR,
78+
feed_spec: Any | None = None,
7279
) -> list[DataGap]:
7380
"""
7481
Detect gaps in time series data.
@@ -78,10 +85,33 @@ def detect_gaps(
7885
frequency: Data frequency (minute, hourly, daily, etc.)
7986
timestamp_col: Name of timestamp column
8087
is_crypto: If True, expect 24/7 data; if False, market hours only
88+
calendar: Daily schedule override; None disables schedule-based detection.
89+
feed_spec: Shared feed declaration, resolved before the NYSE default.
8190
8291
Returns:
8392
List of detected gaps
8493
"""
94+
feed = FeedSpec.from_any(feed_spec)
95+
# Preserve the old crypto hint only when the feed has no declaration.
96+
if (
97+
frequency.lower() in {"daily", "1day"}
98+
and calendar is UNSET_CALENDAR
99+
and is_crypto
100+
and (
101+
feed.calendar is UNSET_CALENDAR
102+
or (feed.time_contract_version == 1 and feed.calendar is None)
103+
)
104+
):
105+
calendar = "24/7"
106+
selected_calendar = feed.resolve_calendar(calendar=calendar)
107+
if selected_calendar is None:
108+
return []
109+
if frequency.lower() in {"daily", "1day"}:
110+
return self._detect_daily_gaps(df, timestamp_col, frequency, selected_calendar, feed)
111+
if calendar is not UNSET_CALENDAR or feed_spec is not None:
112+
raise NotImplementedError(
113+
"calendar/feed declarations currently support daily gaps only"
114+
)
85115
if df.is_empty() or len(df) < 2:
86116
return []
87117

@@ -147,6 +177,50 @@ def detect_gaps(
147177

148178
return gaps
149179

180+
def _detect_daily_gaps(
181+
self,
182+
df: pl.DataFrame,
183+
timestamp_col: str,
184+
frequency: str,
185+
calendar: str,
186+
feed: FeedSpec,
187+
) -> list[DataGap]:
188+
assigner = SessionAssigner(calendar)
189+
if timestamp_col not in df.columns:
190+
raise ValueError(f"DataFrame must have '{timestamp_col}' column")
191+
if df.is_empty():
192+
return []
193+
dtype = df.schema[timestamp_col]
194+
if dtype != pl.Date and not isinstance(dtype, pl.Datetime):
195+
raise TypeError(f"'{timestamp_col}' must contain Date or Datetime values")
196+
if df[timestamp_col].null_count():
197+
raise ValueError(f"'{timestamp_col}' must not contain null timestamps")
198+
if feed.timestamp_semantics not in (None, TimestampSemantics.SESSION_LABEL):
199+
raise NotImplementedError("daily gap detection requires session_label timestamps")
200+
201+
timestamps = df.sort(timestamp_col)[timestamp_col].to_list()
202+
labels = [value.date() if isinstance(value, datetime) else value for value in timestamps]
203+
sessions = [
204+
label.date()
205+
for label in assigner.calendar.schedule(start_date=labels[0], end_date=labels[-1]).index
206+
]
207+
gaps = []
208+
for current, following, start, end in zip(timestamps, timestamps[1:], labels, labels[1:]):
209+
missing = bisect_left(sessions, end) - bisect_right(sessions, start)
210+
if missing > 0:
211+
current = (
212+
current
213+
if isinstance(current, datetime)
214+
else datetime.combine(current, datetime.min.time())
215+
)
216+
following = (
217+
following
218+
if isinstance(following, datetime)
219+
else datetime.combine(following, datetime.min.time())
220+
)
221+
gaps.append(DataGap(current, following, missing, following - current, frequency))
222+
return gaps
223+
150224
def _get_expected_interval(self, frequency: str, is_crypto: bool = False) -> timedelta:
151225
"""
152226
Get expected time interval between data points.

‎tests/test_daily_session_gaps.py‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,3 +102,57 @@ def test_bad_calendar_fails_without_mutating_provider_input() -> None:
102102
with pytest.raises(ValueError, match="Unknown calendar"):
103103
GapDetector().detect_gaps(frame, calendar="not-an-exchange")
104104
assert_frame_equal(frame, original)
105+
106+
107+
@pytest.mark.parametrize(
108+
"metadata, expected",
109+
[({"calendar": None}, 1), ({"time_contract_version": 2, "calendar": None}, 0)],
110+
)
111+
def test_legacy_null_inherits_but_declared_null_disables_missing_session_checks(
112+
metadata, expected
113+
) -> None:
114+
from ml4t.specs import FeedSpec
115+
116+
frame = provider_frame("2025-07-15", "2025-07-17")
117+
frame = frame.filter(pl.col("timestamp").dt.date() != date(2025, 7, 16))
118+
feed = FeedSpec.from_mapping(FeedSpec.from_mapping(metadata).to_dict())
119+
gaps = GapDetector().detect_gaps(frame, feed_spec=feed)
120+
assert sum(g.missing_periods for g in gaps) == expected
121+
122+
123+
def test_missing_early_close_session_is_still_one_missing_observation() -> None:
124+
frame = provider_frame("2025-07-02", "2025-07-07")
125+
frame = frame.filter(~pl.col("timestamp").dt.date().is_in([date(2025, 7, 3), date(2025, 7, 4)]))
126+
gaps = GapDetector().detect_gaps(frame)
127+
assert len(gaps) == 1
128+
assert gaps[0].missing_periods == 1
129+
130+
131+
def test_non_us_calendar_uses_its_own_holidays() -> None:
132+
frame = provider_frame("2025-07-03", "2025-07-07")
133+
frame = frame.filter(pl.col("timestamp").dt.date() != date(2025, 7, 4))
134+
assert GapDetector().detect_gaps(frame, calendar="NYSE") == []
135+
assert sum(g.missing_periods for g in GapDetector().detect_gaps(frame, calendar="LSE")) == 1
136+
137+
138+
def test_date_session_labels_have_the_same_schedule_as_datetime_labels() -> None:
139+
frame = provider_frame("2025-07-11", "2025-07-14").with_columns(pl.col("timestamp").dt.date())
140+
assert GapDetector().detect_gaps(frame) == []
141+
assert sum(g.missing_periods for g in GapDetector().detect_gaps(frame, calendar="24/7")) == 2
142+
143+
144+
@pytest.mark.parametrize("semantics", ["event_time", "bar_close"])
145+
def test_event_semantics_are_not_silently_reinterpreted_as_daily_labels(semantics) -> None:
146+
frame = provider_frame("2025-07-11", "2025-07-14")
147+
with pytest.raises(NotImplementedError, match="session_label"):
148+
GapDetector().detect_gaps(frame, feed_spec={"timestamp_semantics": semantics})
149+
150+
151+
def test_null_timestamps_fail_without_input_mutation() -> None:
152+
frame = provider_frame("2025-07-11", "2025-07-14").with_columns(
153+
pl.lit(None).cast(pl.Datetime("us", "UTC")).alias("timestamp")
154+
)
155+
original = frame.clone()
156+
with pytest.raises(ValueError, match="null timestamps"):
157+
GapDetector().detect_gaps(frame)
158+
assert_frame_equal(frame, original)

‎tests/test_gaps.py‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ def test_detect_multiple_gaps(self) -> None:
7979
)
8080

8181
detector = GapDetector()
82-
gaps = detector.detect_gaps(df, frequency="daily")
82+
gaps = detector.detect_gaps(df, frequency="daily", calendar="24/7")
8383

8484
assert len(gaps) == 2
8585
assert gaps[0].missing_periods == 2
@@ -150,10 +150,7 @@ def test_stock_weekend_not_gap(self) -> None:
150150
detector = GapDetector()
151151
gaps = detector.detect_gaps(df, frequency="daily", is_crypto=False)
152152

153-
# Weekend should not be a gap for stocks
154-
# Note: Basic implementation might still detect it
155-
# This test documents expected behavior
156-
assert len(gaps) <= 1 # May detect as gap in simple implementation
153+
assert gaps == []
157154

158155
def test_gap_summary(self) -> None:
159156
"""Test gap summarization."""
@@ -171,7 +168,7 @@ def test_gap_summary(self) -> None:
171168
)
172169

173170
detector = GapDetector()
174-
gaps = detector.detect_gaps(df, frequency="daily")
171+
gaps = detector.detect_gaps(df, frequency="daily", calendar="24/7")
175172
summary = detector.summarize_gaps(gaps)
176173

177174
assert summary["count"] == 2

‎uv.lock‎

Lines changed: 10 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)