Skip to content

Commit b714ac9

Browse files
author
Stefan Jansen
committed
fix: recover calendar gaps from exact Databento source ranges
1 parent 60f6591 commit b714ac9

6 files changed

Lines changed: 862 additions & 128 deletions

File tree

‎docs/providers/databento.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,37 @@ df = provider.fetch_continuous_futures(
146146

147147
---
148148

149+
## Exact Timestamp Ranges
150+
151+
`fetch_ohlcv_range()` requests one series between timezone-aware instants, including `start` and
152+
excluding `end`. Unlike the date-based API, it never rounds to dates or applies
153+
`adjust_session_dates`. The wrapper supports native minute, hourly and daily OHLCV schemas and
154+
rejects unsupported frequencies, parent expansion and `ALL_SYMBOLS` before a request.
155+
156+
```python
157+
from datetime import UTC, datetime
158+
159+
with DataBentoProvider(stype_in="continuous") as provider:
160+
rows = provider.fetch_ohlcv_range(
161+
"BTC.c.0",
162+
datetime(2026, 9, 5, 12, 1, tzinfo=UTC),
163+
datetime(2026, 9, 5, 12, 2, tzinfo=UTC),
164+
"minute",
165+
)
166+
```
167+
168+
The returned `symbol` identifies the requested series; `contract` retains the resolved contract
169+
when the SDK supplies it. Select the same dataset, continuous roll rule and price convention used
170+
for the stored series. This method does not reconstruct adjusted continuous prices or select another
171+
contract to fill an absence. Databento timestamps identify interval starts, and no-trade intervals
172+
can have no OHLCV record. A download may therefore leave gaps unresolved.
173+
174+
For a stored report, see [targeted source recovery](../user-guide/incremental-updates.md#download-real-observations-for-a-selected-report).
175+
For raw Parquet files, this method returns a frame for explicit review and merging; it does not
176+
rewrite those files. Requests are metered. Use the native client's cost estimate before broad
177+
requests. The endpoint and bar semantics follow the [historical range API](https://databento.com/docs/api-reference-historical/timeseries/timeseries-get-range)
178+
and [OHLCV specification](https://databento.com/docs/schemas-and-data-formats/ohlcv).
179+
149180
## API Key Setup
150181

151182
```bash

‎docs/user-guide/incremental-updates.md‎

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -136,8 +136,8 @@ Explicit `calendar=None` disables classification, while `"24/7"` still expects c
136136

137137
This replaces the old fixed-US-clock approximation. Unqualified legacy intraday calls now require
138138
a complete feed declaration or explicit `calendar=None`; the detector does not invent an anchor,
139-
label side or partial-bar policy. Daily classification is unchanged. The separate intraday/weekly
140-
backfill range planner below still has its documented limitations.
139+
label side or partial-bar policy. Daily classification is unchanged. Use the bounded range report
140+
below to inspect leading/trailing absences; weekly grids are not supported.
141141

142142
### Inspect Missing Observations on Disk
143143

@@ -221,6 +221,51 @@ row does not justify creating a replacement candle. Targeted source downloads ar
221221
not an effect of inspecting the dataset. Values inside present rows are checked by validation, not
222222
by this timestamp-completeness report.
223223

224+
### Download Real Observations for a Selected Report
225+
226+
This is an explicit, potentially billable operation. For an existing one-instrument CME minute
227+
dataset, first inspect the desired interval. The storage key and vendor symbol are separate:
228+
229+
```python
230+
from ml4t.data.providers import DataBentoProvider
231+
from ml4t.data.storage.metadata_tracker import MetadataTracker
232+
from ml4t.data.update_manager import BackfillManager
233+
234+
cme_feed = feed.with_overrides(calendar="CME Globex Crypto")
235+
key = "futures/minute/BTC"
236+
gaps = GapDetector(feed_spec=cme_feed).detect_gaps_in_storage(
237+
storage,
238+
key,
239+
datetime(2026, 9, 5, 12, 0, tzinfo=UTC),
240+
datetime(2026, 9, 5, 12, 2, tzinfo=UTC),
241+
)
242+
with DataBentoProvider(stype_in="continuous") as provider:
243+
result = BackfillManager(storage, MetadataTracker(storage.base_path)).execute_backfill(
244+
key, gaps, provider, source_symbol="BTC.c.0", feed_spec=cme_feed
245+
)
246+
print(result.rows_added, result.remaining_gaps)
247+
```
248+
249+
The consumer rechecks the selected ranges against disk before requesting data. It downloads only
250+
currently missing ranges, preserves real stored observations and identity/calendar metadata, and replaces a marked
251+
null placeholder only when an actual source observation is returned. Intraday requests use the
252+
optional `TimestampRangeProvider` capability; unsupported providers fail without falling back to a
253+
full-day download. `calendar=None` disables calendar-based recovery. The first supported adapter is
254+
Databento, with a left-labeled grid and one instrument per storage key. Multi-asset inspection is
255+
supported, but targeted recovery of a panel requires separate per-instrument datasets.
256+
257+
`result.success` means processing completed without errors, not that every observation was recovered.
258+
Inspect `errors` and `remaining_gaps`. Empty no-trade responses do not change storage or trigger
259+
unbounded retries. Partial responses retain their residual gaps; `gaps_filled` counts only completely
260+
recovered ranges. Existing observed null values are a separate validation concern, not permission
261+
to overwrite a source observation. `identify_candidates(min_gap_bars=...)` uses bar counts for
262+
intraday data; its daily `min_gap_days` remains a session count.
263+
264+
Coin Metrics is not currently a Data provider. Its candle and metric endpoints use different
265+
identities and schemas; this Databento workflow does not claim to download them. See the
266+
[Databento range contract](../providers/databento.md#exact-timestamp-ranges) before selecting a
267+
continuous symbol or price convention.
268+
224269
### Daily Range Compatibility
225270

226271
Daily plans include missing sessions at either boundary and within the inclusive requested date

‎src/ml4t/data/providers/databento.py‎

Lines changed: 62 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
NetworkError,
2626
RateLimitError,
2727
)
28-
from ml4t.data.providers.base import BaseProvider
28+
from ml4t.data.providers.base import BaseProvider, provider_retry
2929

3030
logger = structlog.get_logger()
3131

@@ -91,6 +91,8 @@ def __init__(
9191
rate_limit: tuple[int, float] | None = None,
9292
adjust_session_dates: bool = False,
9393
session_start_hour_utc: int = 0,
94+
*,
95+
stype_in: str = "raw_symbol",
9496
):
9597
"""Initialize Databento provider.
9698
@@ -100,6 +102,7 @@ def __init__(
100102
rate_limit: Optional custom rate limit (calls, period_seconds)
101103
adjust_session_dates: Whether to adjust dates for CME session logic
102104
session_start_hour_utc: Hour in UTC when trading session starts (for futures)
105+
stype_in: Input symbology for OHLCV requests, such as continuous for BTC.c.0.
103106
"""
104107
super().__init__(rate_limit=rate_limit or self.DEFAULT_RATE_LIMIT)
105108

@@ -123,6 +126,7 @@ def __init__(
123126
self.default_schema = "ohlcv-1m"
124127
self.adjust_session_dates = adjust_session_dates
125128
self.session_start_hour_utc = session_start_hour_utc
129+
self.stype_in = stype_in
126130

127131
self.logger.info(
128132
"Initialized Databento provider",
@@ -149,15 +153,15 @@ def _create_empty_dataframe(self) -> pl.DataFrame:
149153
}
150154
)
151155

152-
def _map_frequency_to_schema(self, frequency: str) -> str:
156+
def _map_frequency_to_schema(self, frequency: str, *, strict: bool = False) -> str:
153157
"""Map frequency parameter to Databento schema."""
154158
freq_lower = frequency.lower()
155159

156-
if freq_lower in ["daily", "day", "1d", "d"]:
160+
if freq_lower in ["daily", "day", "1day", "1d", "d"]:
157161
return "ohlcv-1d"
158-
if freq_lower in ["hourly", "hour", "1h", "h"]:
162+
if freq_lower in ["hourly", "hour", "1hour", "1h", "h"]:
159163
return "ohlcv-1h"
160-
if freq_lower in ["minute", "min", "1m", "m"]:
164+
if freq_lower in ["minute", "min", "1minute", "1min", "1m", "m"]:
161165
return "ohlcv-1m"
162166
if freq_lower in ["tick", "trades"]:
163167
return "trades"
@@ -166,6 +170,8 @@ def _map_frequency_to_schema(self, frequency: str) -> str:
166170
if freq_lower in ["mbo"]:
167171
return "mbo"
168172

173+
if strict:
174+
raise ValueError(f"Unsupported targeted OHLCV frequency: {frequency}")
169175
# Default to daily
170176
return "ohlcv-1d"
171177

@@ -303,7 +309,7 @@ def _fetch_raw_data(
303309
end=end_dt,
304310
symbols=symbol,
305311
schema=schema,
306-
stype_in="raw_symbol",
312+
stype_in=self.stype_in,
307313
)
308314

309315
return response
@@ -320,6 +326,56 @@ def _fetch_raw_data(
320326
message=f"Failed to fetch data from Databento: {e}",
321327
)
322328

329+
@provider_retry
330+
def fetch_ohlcv_range(
331+
self,
332+
symbol: str,
333+
start: datetime,
334+
end: datetime,
335+
frequency: str = "minute",
336+
*,
337+
stype_in: str | None = None,
338+
) -> pl.DataFrame:
339+
"""Fetch one series within exact UTC instants, including start and excluding end.
340+
341+
Session-date adjustment is never applied. Only native one-minute, one-hour and
342+
daily OHLCV schemas are supported; no reaggregation or synthetic bars are produced.
343+
"""
344+
if (
345+
not isinstance(start, datetime)
346+
or not isinstance(end, datetime)
347+
or start.utcoffset() is None
348+
or end.utcoffset() is None
349+
):
350+
raise ValueError("start and end must be timezone-aware datetimes")
351+
start, end = start.astimezone(UTC), end.astimezone(UTC)
352+
if start >= end:
353+
raise ValueError("start must be before end for a half-open range")
354+
schema = self._map_frequency_to_schema(frequency, strict=True)
355+
if schema not in {"ohlcv-1m", "ohlcv-1h", "ohlcv-1d"}:
356+
raise ValueError(f"Unsupported targeted OHLCV frequency: {frequency}")
357+
selected_stype = self.stype_in if stype_in is None else stype_in
358+
if selected_stype not in {"raw_symbol", "continuous", "instrument_id"}:
359+
raise ValueError("stype_in must select a single series, not a parent or universe")
360+
if not isinstance(symbol, str) or symbol.upper() == "ALL_SYMBOLS":
361+
raise ValueError("symbol must identify a single instrument, not a universe")
362+
self._validate_inputs(symbol, start.date().isoformat(), end.date().isoformat(), frequency)
363+
self._acquire_rate_limit()
364+
365+
def fetch() -> pl.DataFrame:
366+
raw = self._fetch_timeseries(
367+
dataset=self.dataset,
368+
symbols=symbol,
369+
schema=schema,
370+
stype_in=selected_stype,
371+
start=start,
372+
end=end,
373+
)
374+
frame = self._validate_ohlcv(self._transform_data(raw, symbol), self.name, symbol)
375+
return frame.filter((pl.col("timestamp") >= start) & (pl.col("timestamp") < end))
376+
377+
return self._with_circuit_breaker(fetch)
378+
323379
def _transform_data(self, raw_data: Any, symbol: str) -> pl.DataFrame:
324380
"""Transform Databento data to standard schema."""
325381
try:

‎src/ml4t/data/providers/protocols.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ def fetch_data(provider: OHLCVProvider, symbol: str) -> pl.DataFrame:
2626
from typing import TYPE_CHECKING, Protocol, runtime_checkable
2727

2828
if TYPE_CHECKING:
29+
from datetime import datetime
30+
2931
import polars as pl
3032

3133

@@ -105,6 +107,21 @@ def capabilities(self) -> ProviderCapabilities:
105107
...
106108

107109

110+
@runtime_checkable
111+
class TimestampRangeProvider(Protocol):
112+
"""Optional single-series OHLCV capability with half-open aware timestamp bounds."""
113+
114+
def fetch_ohlcv_range(
115+
self,
116+
symbol: str,
117+
start: datetime,
118+
end: datetime,
119+
frequency: str = "minute",
120+
) -> pl.DataFrame:
121+
"""Return actual source observations with start <= timestamp < end."""
122+
...
123+
124+
108125
@runtime_checkable
109126
class FactorProvider(Protocol):
110127
"""Protocol for academic factor data providers.

0 commit comments

Comments
 (0)