Skip to content

Commit d6076e5

Browse files
author
Stefan Jansen
committed
fix: consolidate calendar gap inspection and source-backed recovery
Keep read-only reports, removal of synthetic filling, and exact Databento recovery in one maintainer-directed workflow. Preserve each reviewed source commit and its regression evidence. Include the source guide batching policy without creating a policy-only PR.
2 parents 5bd368f + 0878003 commit d6076e5

32 files changed

Lines changed: 1305 additions & 496 deletions

‎AGENTS.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,3 +71,23 @@ pre-commit run --all-files
7171
For packaging or release changes, also run the verification scripts used by
7272
`.github/workflows/release.yml`. Never bypass hooks or weaken provider-specific checks to make a
7373
shared gate pass.
74+
75+
## Maintainer-directed change batching
76+
77+
For our own fixes and agent work directed by the maintainer, use a bounded, coherent PR rather
78+
than one PR per small defect or implementation step. One PR may close several related issues
79+
when the changes serve the same user workflow, subsystem repair or compatibility objective and
80+
can be reviewed and reverted together. Keep implementation, regressions, documentation and
81+
follow-up repairs together; use focused commits and retain evidence for each defect. Do not
82+
create separate issues or PRs merely for individual files, tests, assertions or platform variants.
83+
84+
Split unrelated changes, independently risky or breaking changes, separate ownership boundaries,
85+
and work that needs independent release or rollback decisions. A general maintenance label alone
86+
does not make a batch coherent. Reuse or safely consolidate existing owned submissions instead of
87+
opening overlapping PRs; preserve contributor attribution, recoverable history and other owners'
88+
active work. Do not rewrite or close someone else's work merely to reduce the queue count.
89+
90+
Run focused checks while iterating, then qualify the complete final revision with every applicable
91+
required gate. Group related work before expensive CI; do not weaken checks, bypass review or
92+
claim old-head checks qualify changed code. This maintainer-directed batching rule takes precedence
93+
over a literal one-issue-per-PR reading of contributor intake guidance.

‎docs/providers/databento.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,37 @@ df = provider.fetch_continuous_futures(
146146

147147
---
148148

149+
## Exact Timestamp Ranges
150+
151+
`fetch_ohlcv_range()` requests one series between timezone-aware instants, including `start` and
152+
excluding `end`. Unlike the date-based API, it never rounds to dates or applies
153+
`adjust_session_dates`. The wrapper supports native minute, hourly and daily OHLCV schemas and
154+
rejects unsupported frequencies, parent expansion and `ALL_SYMBOLS` before a request.
155+
156+
```python
157+
from datetime import UTC, datetime
158+
159+
with DataBentoProvider(stype_in="continuous") as provider:
160+
rows = provider.fetch_ohlcv_range(
161+
"BTC.c.0",
162+
datetime(2026, 9, 5, 12, 1, tzinfo=UTC),
163+
datetime(2026, 9, 5, 12, 2, tzinfo=UTC),
164+
"minute",
165+
)
166+
```
167+
168+
The returned `symbol` identifies the requested series; `contract` retains the resolved contract
169+
when the SDK supplies it. Select the same dataset, continuous roll rule and price convention used
170+
for the stored series. This method does not reconstruct adjusted continuous prices or select another
171+
contract to fill an absence. Databento timestamps identify interval starts, and no-trade intervals
172+
can have no OHLCV record. A download may therefore leave gaps unresolved.
173+
174+
For a stored report, see [targeted source recovery](../user-guide/incremental-updates.md#download-real-observations-for-a-selected-report).
175+
For raw Parquet files, this method returns a frame for explicit review and merging; it does not
176+
rewrite those files. Requests are metered. Use the native client's cost estimate before broad
177+
requests. The endpoint and bar semantics follow the [historical range API](https://databento.com/docs/api-reference-historical/timeseries/timeseries-get-range)
178+
and [OHLCV specification](https://databento.com/docs/schemas-and-data-formats/ohlcv).
179+
149180
## API Key Setup
150181

151182
```bash

‎docs/tutorials/03_incremental_updates.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -283,7 +283,10 @@ Today: Jan 15
283283
Missing: [Jan 11] [Jan 12] [Jan 13] [Jan 14]
284284
```
285285

286-
**Solution**: Incremental updates automatically detect and fill gaps!
286+
Incremental updates request actual observations from the configured source. Missing periods remain
287+
missing when the source supplies no row; they are never replaced with synthetic market values.
288+
Use the [stored calendar report](../user-guide/incremental-updates.md#inspect-missing-observations-on-disk)
289+
to inspect internal gaps independently of acquisition.
287290

288291
```python
289292
# Even after 5 days without updates, incremental mode works
@@ -296,7 +299,7 @@ print(result)
296299
# 'records_added': 5,
297300
# 'gap_detected': True,
298301
# 'gap_days': 5,
299-
# 'message': 'Filled 5-day gap'
302+
# 'message': 'Downloaded available source observations'
300303
# }
301304
```
302305

@@ -410,7 +413,7 @@ def catch_up_all_symbols():
410413

411414
if result.get('gap_detected'):
412415
logger.warning(
413-
f"{symbol}: Filled {result['gap_days']}-day gap "
416+
f"{symbol}: Downloaded source observations for {result['gap_days']} days "
414417
f"({result['records_added']} records)"
415418
)
416419
elif result.get('skip_reason') == 'already_up_to_date':

‎docs/tutorials/04_data_quality.md‎

Lines changed: 6 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -274,23 +274,13 @@ data = provider.fetch_ohlcv("AAPL", start, end)
274274
data = clean_ohlcv_data(data)
275275
```
276276

277-
### Option 3: Fill Forward (Last Resort)
277+
### Preserve and Investigate Missing Observations
278278

279-
```python
280-
def fill_missing_data(df):
281-
"""Fill missing values with forward fill (use sparingly)."""
282-
# Fill nulls with previous valid value
283-
df = df.with_columns([
284-
pl.col("open").fill_null(strategy="forward"),
285-
pl.col("high").fill_null(strategy="forward"),
286-
pl.col("low").fill_null(strategy="forward"),
287-
pl.col("close").fill_null(strategy="forward"),
288-
pl.col("volume").fill_null(0), # Volume = 0 for missing days
289-
])
290-
291-
logger.warning("Applied forward fill to missing data")
292-
return df
293-
```
279+
Do not replace absent OHLCV values with copied prices or zero volume. Preserve the missing data,
280+
use the [calendar-based disk report](../user-guide/incremental-updates.md#inspect-missing-observations-on-disk)
281+
to identify absent scheduled observations, and request actual rows from the original source when
282+
its API and coverage permit. Intervals without trades may legitimately have no candle. A null price
283+
inside a present row remains a validation finding, not permission to invent a replacement.
294284

295285
## Provider-Specific Validation
296286

‎docs/user-guide/cli-reference.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,8 @@ ml4t-data update -s AAPL --config ml4t-data.yaml
7575
| `--frequency` | `daily`, `hourly`, or `weekly` |
7676
| `--asset-class` | `equities`, `crypto`, `forex`, or `futures` |
7777
| `--lookback-days` | Existing-data overlap used for revisions |
78-
| `--fill-gaps`, `--no-fill-gaps` | Enable or disable gap filling |
78+
| `--check-gaps`, `--no-check-gaps` | Enable or disable read-only calendar gap checks |
79+
| `--fill-gaps`, `--no-fill-gaps` | Legacy: filling is rejected; `--no-fill-gaps` disables checks |
7980
| `--provider`, `-p` | Provider to use for fetching |
8081
| `--start`, `--end` | Date bounds used only for a first load |
8182
| `--initial-load-days` | First-load history when `--start` is omitted |

‎docs/user-guide/configuration.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -166,7 +166,8 @@ provider, frequency, asset class, and bootstrap range.
166166
| `frequency` | `daily` | Bar frequency passed to `DataManager.update()` |
167167
| `asset_class` | `equity` | Storage key prefix and validation context |
168168
| `lookback_days` | `7` | Overlap fetched for incremental updates |
169-
| `fill_gaps` | `true` | Enable gap detection and filling after merge |
169+
| `check_gaps` | `true` | Report calendar gaps without changing observations |
170+
| `fill_gaps` | unset | Legacy `false` disables checks; `true` is rejected |
170171
| `start` / `start_date` | unset | Explicit first-load start date when storage is empty |
171172
| `end` / `end_date` | today | Explicit first-load end date when storage is empty |
172173
| `initial_load_days` | `365` | First-load history length when no start date is set |

‎docs/user-guide/incremental-updates.md‎

Lines changed: 69 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -41,26 +41,27 @@ metadata. Supplying the option overrides that value.
4141

4242
## Gap Handling
4343

44-
Gap detection and forward filling are enabled by default. Disable them when missing periods must
45-
remain explicit:
44+
Updates preserve absent observations and null market values. Calendar gap checks are enabled by
45+
default and never create rows or substitute prices or volumes. To skip those checks:
4646

4747
```bash
48-
ml4t-data update --symbol AAPL --no-fill-gaps --storage-path ./data
48+
ml4t-data update --symbol AAPL --no-check-gaps --storage-path ./data
4949
```
5050

5151
Daily detection counts missing sessions from the selected exchange schedule, not elapsed days.
5252
The Python updater resolves its explicit `calendar`, then supplied feed or stored metadata, then
5353
NYSE. Asset class and descriptive exchange metadata do not select a schedule. Continuous feeds
54-
must declare `calendar="24/7"`; `calendar=None` disables schedule-derived detection and filling.
54+
must declare `calendar="24/7"`; `calendar=None` disables schedule-derived detection.
5555
Intraday detection requires the complete version-2 grid declaration described below, supplied
5656
through `feed_spec` or retained in stored metadata. Undeclared intraday datasets must migrate their
57-
metadata or use `--no-fill-gaps` / `fill_gaps=False` for updates.
58-
[Trading-session completion](sessions.md) is a separate one-minute operation. The acquisition-range
59-
planner's intraday limitations are described separately below.
57+
metadata or use `--no-check-gaps` / `check_gaps=False` for updates.
58+
[Minute-grid expansion](sessions.md) is a separate operation that exposes absent positions as nulls.
6059

61-
Detection does not itself fill or modify observations. The updater's separately enabled forward
62-
fill uses a regular elapsed-time grid, not the selected exchange schedule, and can insert closed
63-
dates when a real gap is present. Disable filling when those synthetic rows are inappropriate.
60+
Synthetic imputation is unsupported. An explicit `fill_gaps=True` or `--fill-gaps` request fails
61+
before acquisition or a dataset write. The legacy `fill_gaps=False` / `--no-fill-gaps` controls remain
62+
compatible aliases for disabling checks. Configurations should use `check_gaps` instead.
63+
The updater records whether checking ran and the detected missing-observation count; disabled
64+
checks are not reported as proof of zero gaps. Source-history limits do not suppress detection.
6465

6566
### Inspect Daily Gaps Without Filling
6667

@@ -94,8 +95,8 @@ mapping, even with daily frequency or midnight timestamps. Right-labeled continu
9495
to the preceding session; event time belongs to the new session. Out-of-session instants fail clearly.
9596
Gap classification does not add a new off-session rejection policy for legacy daily labels. The
9697
requested daily frequency cannot silently override a declared intraday frequency.
97-
Unknown calendars and malformed daily timestamps fail without modifying the input. `fill_gaps()`
98-
remains a separate elapsed-grid operation, not calendar-aware imputation.
98+
Unknown calendars and malformed daily timestamps fail without modifying the input.
99+
The retired `fill_gaps()` helper rejects nonempty gap lists rather than manufacturing observations.
99100

100101
### Inspect Intraday Gaps Against a Declared Grid
101102

@@ -135,8 +136,8 @@ Explicit `calendar=None` disables classification, while `"24/7"` still expects c
135136

136137
This replaces the old fixed-US-clock approximation. Unqualified legacy intraday calls now require
137138
a complete feed declaration or explicit `calendar=None`; the detector does not invent an anchor,
138-
label side or partial-bar policy. Daily classification is unchanged. The separate intraday/weekly
139-
backfill range planner below still has its documented limitations.
139+
label side or partial-bar policy. Daily classification is unchanged. Use the bounded range report
140+
below to inspect leading/trailing absences; weekly grids are not supported.
140141

141142
### Inspect Missing Observations on Disk
142143

@@ -220,6 +221,55 @@ row does not justify creating a replacement candle. Targeted source downloads ar
220221
not an effect of inspecting the dataset. Values inside present rows are checked by validation, not
221222
by this timestamp-completeness report.
222223

224+
### Download Real Observations for a Selected Report
225+
226+
This is an explicit, potentially billable operation. For an existing one-instrument CME minute
227+
dataset, first inspect the desired interval. The storage key and vendor symbol are separate:
228+
229+
```python
230+
from ml4t.data.providers import DataBentoProvider
231+
from ml4t.data.storage.metadata_tracker import MetadataTracker
232+
from ml4t.data.update_manager import BackfillManager
233+
234+
cme_feed = feed.with_overrides(calendar="CME Globex Crypto")
235+
key = "futures/minute/BTC"
236+
gaps = GapDetector(feed_spec=cme_feed).detect_gaps_in_storage(
237+
storage,
238+
key,
239+
datetime(2026, 9, 5, 12, 0, tzinfo=UTC),
240+
datetime(2026, 9, 5, 12, 2, tzinfo=UTC),
241+
)
242+
with DataBentoProvider(stype_in="continuous") as provider:
243+
result = BackfillManager(storage, MetadataTracker(storage.base_path)).execute_backfill(
244+
key, gaps, provider, source_symbol="BTC.c.0", feed_spec=cme_feed
245+
)
246+
print(result.rows_added, result.remaining_gaps)
247+
```
248+
249+
The consumer rechecks the selected ranges against disk before requesting data. It downloads only
250+
currently missing ranges, preserves real stored observations and identity/calendar metadata, and replaces a marked
251+
null placeholder only when an actual source observation is returned. Intraday requests use the
252+
optional `TimestampRangeProvider` capability; unsupported providers fail without falling back to a
253+
full-day download. `calendar=None` disables calendar-based recovery. The first supported adapter is
254+
Databento, with a left-labeled grid and one instrument per storage key. Multi-asset inspection is
255+
supported, but targeted recovery of a panel requires separate per-instrument datasets.
256+
257+
`result.success` means processing completed without errors, not that every observation was recovered.
258+
An empty input report succeeds without downloading or writing; use `rows_added` and `gaps_filled`
259+
to distinguish recovered data from a no-op. Previously an empty report returned `success=False`.
260+
Inspect `errors` and `remaining_gaps`. Empty no-trade responses do not change storage or trigger
261+
unbounded retries. Partial responses retain their residual gaps; `gaps_filled` counts only completely
262+
recovered ranges. A history-recording failure reports an error even if observations were stored;
263+
the residual report still reflects disk. A failed operation that cannot establish coverage returns
264+
`remaining_gaps=None`, not an unverified clean report. Existing observed null values are a separate
265+
validation concern, not permission to overwrite a source observation. `identify_candidates(min_gap_bars=...)` uses bar counts for
266+
intraday data; its daily `min_gap_days` remains a session count.
267+
268+
Coin Metrics is not currently a Data provider. Its candle and metric endpoints use different
269+
identities and schemas; this Databento workflow does not claim to download them. See the
270+
[Databento range contract](../providers/databento.md#exact-timestamp-ranges) before selecting a
271+
continuous symbol or price convention.
272+
223273
### Daily Range Compatibility
224274

225275
Daily plans include missing sessions at either boundary and within the inclusive requested date
@@ -241,7 +291,7 @@ clearly; original provider timestamps are retained in storage. Declare the feed'
241291
non-UTC label must survive storage's canonical UTC reads.
242292

243293
If a provider's history limit cannot reach the stored dataset, the updater records
244-
`provider_history_limited` and skips gap filling across the unavailable interval.
294+
`provider_history_limited`. Missing observations remain detectable even when they cannot be retrieved.
245295

246296
## Python API
247297

@@ -260,7 +310,7 @@ key = manager.update(
260310
frequency="daily",
261311
asset_class="equities",
262312
lookback_days=7,
263-
fill_gaps=True,
313+
check_gaps=True,
264314
initial_start="2023-01-01",
265315
initial_end="2024-01-01",
266316
)
@@ -280,7 +330,7 @@ Feed timezone, timestamp semantics and bar-grid declarations survive ingestion.
280330
`session_start_time` override is not supported by the native schedule adapter and is rejected.
281331
The requested frequency and bar type must agree with declared feed metadata. These declarations
282332
do not themselves validate OHLCV completeness or transform timestamps. Gap detection remains
283-
separate from the updater's optional elapsed-grid imputation described above.
333+
separate from acquisition; neither validation nor gap checking imputes missing market values.
284334

285335
## Inspect Stored State
286336

@@ -321,7 +371,7 @@ fetching or writing data.
321371
- Run an update only after the provider is expected to have finalized the latest requested period.
322372
- Reuse one storage directory and the same asset-class and frequency values so updates address the
323373
intended key.
324-
- Treat forward-filled rows as synthetic observations. For minute-session completion, the
325-
`is_imputed` column identifies inserted rows; the general updater does not add that marker.
374+
- Preserve missing market data. Optional minute-grid expansion marks null placeholder rows with
375+
the legacy `is_imputed` column; those rows are not recovered source observations.
326376
- Use `ml4t-data validate` after an update when schema, OHLC relationships, duplicates, or anomaly
327377
checks are part of the ingestion contract.

‎docs/user-guide/sessions.md‎

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -135,24 +135,30 @@ Ordinary spot crypto uses `calendar="24/7"`, with civil-day labels and no CME ma
135135
`calendar=None` disables session handling instead. Select the schedule from the feed's actual
136136
market, not from its symbol or asset-class name.
137137

138-
## Complete Minute Sessions
138+
<a id="complete-minute-sessions"></a>
139139

140-
`SessionCompleter` expands minute bars to the exchange schedule. It requires `timestamp`, `open`,
141-
`high`, `low`, `close`, and `volume`. The result adds `session_date` and `is_imputed`.
140+
## Inspect a Nullable Minute Grid
141+
142+
`SessionCompleter` optionally expands a timestamped series to scheduled minutes for inspection.
143+
Missing positions have null market values; they are not recovered observations. It requires a
144+
Datetime `timestamp` column and retains supplied value columns. The result adds `session_date` and
145+
the legacy `is_imputed` marker, which identifies null placeholder rows.
142146

143147
```python
144148
from ml4t.data.sessions import SessionCompleter
145149

146150
completer = SessionCompleter("NYSE")
147151
complete = completer.complete_sessions(
148152
bars,
149-
fill_method="forward",
150-
zero_volume=True,
153+
fill_method="none",
154+
zero_volume=False,
151155
)
152156
```
153157

154-
The supported fill methods are `"forward"`, `"backward"`, and `"none"`. Forward and backward
155-
filling apply only within a session. `zero_volume=True` sets volume to zero for inserted rows.
158+
Only `fill_method="none"` and `zero_volume=False` are supported. Requests to copy prices or invent
159+
zero volume fail clearly. Observed nulls and all other original values remain unchanged.
160+
Use the [disk gap report](incremental-updates.md#inspect-missing-observations-on-disk) when only a list
161+
of missing observations is needed; it does not expand the data.
156162

157163
Session completion rejects duplicate or non-minute-aligned timestamps and observations outside the
158164
selected schedule. Lunch breaks and declared calendar interruptions are excluded from the template;

0 commit comments

Comments
 (0)