Skip to content

Commit a9a9084

Browse files
fix: unify calendar validation and real-observation recovery (#113)
Apply authoritative shared calendars throughout validation, read-only gap reports, incremental planning and explicit Databento recovery. Retire automatic fabricated market values. Preserve real observations, identity and residual reports; block SDK-expanded universe requests before acquisition. Qualified the exact reviewed head with full coverage and ResourceWarning suites, isolated artifacts, documentation and all applicable supported-platform checks. Closes #108. Closes #109. Refs #110; Coin Metrics dataset-specific work remains open.
1 parent 06c0fa9 commit a9a9084

46 files changed

Lines changed: 3007 additions & 749 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎AGENTS.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,3 +71,23 @@ pre-commit run --all-files
7171
For packaging or release changes, also run the verification scripts used by
7272
`.github/workflows/release.yml`. Never bypass hooks or weaken provider-specific checks to make a
7373
shared gate pass.
74+
75+
## Maintainer-directed change batching
76+
77+
For our own fixes and agent work directed by the maintainer, use a bounded, coherent PR rather
78+
than one PR per small defect or implementation step. One PR may close several related issues
79+
when the changes serve the same user workflow, subsystem repair or compatibility objective and
80+
can be reviewed and reverted together. Keep implementation, regressions, documentation and
81+
follow-up repairs together; use focused commits and retain evidence for each defect. Do not
82+
create separate issues or PRs merely for individual files, tests, assertions or platform variants.
83+
84+
Split unrelated changes, independently risky or breaking changes, separate ownership boundaries,
85+
and work that needs independent release or rollback decisions. A general maintenance label alone
86+
does not make a batch coherent. Reuse or safely consolidate existing owned submissions instead of
87+
opening overlapping PRs; preserve contributor attribution, recoverable history and other owners'
88+
active work. Do not rewrite or close someone else's work merely to reduce the queue count.
89+
90+
Run focused checks while iterating, then qualify the complete final revision with every applicable
91+
required gate. Group related work before expensive CI; do not weaken checks, bypass review or
92+
claim old-head checks qualify changed code. This maintainer-directed batching rule takes precedence
93+
over a literal one-issue-per-PR reading of contributor intake guidance.

‎docs/providers/databento.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,37 @@ df = provider.fetch_continuous_futures(
146146

147147
---
148148

149+
## Exact Timestamp Ranges
150+
151+
`fetch_ohlcv_range()` requests one series between timezone-aware instants, including `start` and
152+
excluding `end`. Unlike the date-based API, it never rounds to dates or applies
153+
`adjust_session_dates`. The wrapper supports native minute, hourly and daily OHLCV schemas and
154+
rejects unsupported frequencies, parent expansion and `ALL_SYMBOLS` before a request.
155+
156+
```python
157+
from datetime import UTC, datetime
158+
159+
with DataBentoProvider(stype_in="continuous") as provider:
160+
rows = provider.fetch_ohlcv_range(
161+
"BTC.c.0",
162+
datetime(2026, 9, 5, 12, 1, tzinfo=UTC),
163+
datetime(2026, 9, 5, 12, 2, tzinfo=UTC),
164+
"minute",
165+
)
166+
```
167+
168+
The returned `symbol` identifies the requested series; `contract` retains the resolved contract
169+
when the SDK supplies it. Select the same dataset, continuous roll rule and price convention used
170+
for the stored series. This method does not reconstruct adjusted continuous prices or select another
171+
contract to fill an absence. Databento timestamps identify interval starts, and no-trade intervals
172+
can have no OHLCV record. A download may therefore leave gaps unresolved.
173+
174+
For a stored report, see [targeted source recovery](../user-guide/incremental-updates.md#download-real-observations-for-a-selected-report).
175+
For raw Parquet files, this method returns a frame for explicit review and merging; it does not
176+
rewrite those files. Requests are metered. Use the native client's cost estimate before broad
177+
requests. The endpoint and bar semantics follow the [historical range API](https://databento.com/docs/api-reference-historical/timeseries/timeseries-get-range)
178+
and [OHLCV specification](https://databento.com/docs/schemas-and-data-formats/ohlcv).
179+
149180
## API Key Setup
150181

151182
```bash

‎docs/tutorials/03_incremental_updates.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -283,7 +283,10 @@ Today: Jan 15
283283
Missing: [Jan 11] [Jan 12] [Jan 13] [Jan 14]
284284
```
285285

286-
**Solution**: Incremental updates automatically detect and fill gaps!
286+
Incremental updates request actual observations from the configured source. Missing periods remain
287+
missing when the source supplies no row; they are never replaced with synthetic market values.
288+
Use the [stored calendar report](../user-guide/incremental-updates.md#inspect-missing-observations-on-disk)
289+
to inspect internal gaps independently of acquisition.
287290

288291
```python
289292
# Even after 5 days without updates, incremental mode works
@@ -296,7 +299,7 @@ print(result)
296299
# 'records_added': 5,
297300
# 'gap_detected': True,
298301
# 'gap_days': 5,
299-
# 'message': 'Filled 5-day gap'
302+
# 'message': 'Downloaded available source observations'
300303
# }
301304
```
302305

@@ -410,7 +413,7 @@ def catch_up_all_symbols():
410413

411414
if result.get('gap_detected'):
412415
logger.warning(
413-
f"{symbol}: Filled {result['gap_days']}-day gap "
416+
f"{symbol}: Downloaded source observations for {result['gap_days']} days "
414417
f"({result['records_added']} records)"
415418
)
416419
elif result.get('skip_reason') == 'already_up_to_date':

‎docs/tutorials/04_data_quality.md‎

Lines changed: 6 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -274,23 +274,13 @@ data = provider.fetch_ohlcv("AAPL", start, end)
274274
data = clean_ohlcv_data(data)
275275
```
276276

277-
### Option 3: Fill Forward (Last Resort)
277+
### Preserve and Investigate Missing Observations
278278

279-
```python
280-
def fill_missing_data(df):
281-
"""Fill missing values with forward fill (use sparingly)."""
282-
# Fill nulls with previous valid value
283-
df = df.with_columns([
284-
pl.col("open").fill_null(strategy="forward"),
285-
pl.col("high").fill_null(strategy="forward"),
286-
pl.col("low").fill_null(strategy="forward"),
287-
pl.col("close").fill_null(strategy="forward"),
288-
pl.col("volume").fill_null(0), # Volume = 0 for missing days
289-
])
290-
291-
logger.warning("Applied forward fill to missing data")
292-
return df
293-
```
279+
Do not replace absent OHLCV values with copied prices or zero volume. Preserve the missing data,
280+
use the [calendar-based disk report](../user-guide/incremental-updates.md#inspect-missing-observations-on-disk)
281+
to identify absent scheduled observations, and request actual rows from the original source when
282+
its API and coverage permit. Intervals without trades may legitimately have no candle. A null price
283+
inside a present row remains a validation finding, not permission to invent a replacement.
294284

295285
## Provider-Specific Validation
296286

‎docs/user-guide/cli-reference.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,8 @@ ml4t-data update -s AAPL --config ml4t-data.yaml
7575
| `--frequency` | `daily`, `hourly`, or `weekly` |
7676
| `--asset-class` | `equities`, `crypto`, `forex`, or `futures` |
7777
| `--lookback-days` | Existing-data overlap used for revisions |
78-
| `--fill-gaps`, `--no-fill-gaps` | Enable or disable gap filling |
78+
| `--check-gaps`, `--no-check-gaps` | Enable or disable read-only calendar gap checks |
79+
| `--fill-gaps`, `--no-fill-gaps` | Legacy: filling is rejected; `--no-fill-gaps` disables checks |
7980
| `--provider`, `-p` | Provider to use for fetching |
8081
| `--start`, `--end` | Date bounds used only for a first load |
8182
| `--initial-load-days` | First-load history when `--start` is omitted |

‎docs/user-guide/configuration.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -166,7 +166,8 @@ provider, frequency, asset class, and bootstrap range.
166166
| `frequency` | `daily` | Bar frequency passed to `DataManager.update()` |
167167
| `asset_class` | `equity` | Storage key prefix and validation context |
168168
| `lookback_days` | `7` | Overlap fetched for incremental updates |
169-
| `fill_gaps` | `true` | Enable gap detection and filling after merge |
169+
| `check_gaps` | `true` | Report calendar gaps without changing observations |
170+
| `fill_gaps` | unset | Legacy `false` disables checks; `true` is rejected |
170171
| `start` / `start_date` | unset | Explicit first-load start date when storage is empty |
171172
| `end` / `end_date` | today | Explicit first-load end date when storage is empty |
172173
| `initial_load_days` | `365` | First-load history length when no start date is set |

‎docs/user-guide/data-quality.md‎

Lines changed: 53 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,11 +59,11 @@ from ml4t.data.validation.rules import ValidationRulePresets
5959

6060
# Crypto is more volatile - wider thresholds
6161
crypto_rules = ValidationRulePresets.crypto_rules()
62-
# max_return_threshold=0.5, staleness_threshold=3, no market hours check
62+
# max_return_threshold=0.5, staleness_threshold=3
6363

6464
# Forex is less volatile - tighter thresholds
6565
forex_rules = ValidationRulePresets.forex_rules()
66-
# max_return_threshold=0.1, staleness_threshold=10, 24/5 market
66+
# max_return_threshold=0.1, staleness_threshold=10
6767

6868
# Strict mode for production data
6969
strict_rules = ValidationRulePresets.strict_rules()
@@ -74,6 +74,57 @@ Available presets: `equity_rules()`, `crypto_rules()`, `forex_rules()`,
7474
`commodity_rules()`, `futures_rules()`, `strict_rules()`, `relaxed_rules()`.
7575
Commodity and futures presets report negative prices as warnings. The
7676
`negative_price_policy` setting accepts `"forbid"`, `"warn"`, or `"allow"`.
77+
Presets adjust numerical thresholds, not calendar selection. Crypto and forex presets no longer
78+
disable schedule checks implicitly; declare the market's actual schedule. Explicit per-check
79+
opt-outs remain available.
80+
81+
### Validate Calendar Membership and Continuity
82+
83+
`CrossValidator`, `AssetValidator` and `CryptoSessionValidator` accept the canonical `calendar`
84+
keyword and a shared `feed_spec` on construction and validation. A call's explicit calendar
85+
overrides its constructor setting; the effective explicit setting overrides feed metadata, then
86+
NYSE applies. Version-2 metadata can also explicitly disable calendars. Neither a descriptive
87+
asset class nor an exchange name selects a schedule.
88+
89+
```python
90+
from ml4t.data.providers import SyntheticProvider
91+
from ml4t.data.validation.cross_validation import CrossValidator
92+
from ml4t.specs import FeedSpec
93+
94+
with SyntheticProvider(calendar_mode="continuous") as provider:
95+
spot = provider.fetch_ohlcv("BTCUSD", "2025-07-12", "2025-07-12", "minute")
96+
97+
feed = FeedSpec(calendar="24/7", data_frequency="minute", timestamp_semantics="event_time")
98+
report = CrossValidator(feed_spec=feed).validate(spot)
99+
row_only = CrossValidator(calendar=None).validate(spot)
100+
```
101+
102+
Calendar membership uses the same assignment as [native sessions](sessions.md), including DST,
103+
holidays, early closes, overnight dates, breaks and registered custom schedules. Nanosecond instants
104+
retain their precision in membership comparisons. Daily session
105+
labels are not wall-clock observations; explicit event or close semantics override the daily-label
106+
heuristic. Invalid calendars and incompatible temporal declarations fail clearly. Reported sample
107+
indices refer to the original input positions. `calendar=None` disables temporal checks without
108+
disabling price, volume, schema or reference-price checks.
109+
110+
`CrossValidator` checks membership, not missing-bar completeness. `CryptoSessionValidator` checks
111+
continuity through the shared gap detector: explicit `"24/7"` still detects real outages, whereas
112+
`None` performs no schedule-derived gap check. Intraday continuity requires the complete version-2
113+
bar declaration described in [incremental updates](incremental-updates.md). An explicitly supplied
114+
legacy `CryptoCalendar` object retains its elapsed continuous-gap and maintenance behavior; omission
115+
no longer constructs that object automatically. `CrossValidator(is_crypto=True)` remains a legacy
116+
24/7 hint only when neither an explicit calendar nor a feed calendar is declared. It cannot override
117+
a canonical selection.
118+
119+
`DataManager.load()` and `import_data()` pass their effective declaration and acquisition frequency
120+
to validation before storage. For spot data, declare `"24/7"`; use the selected exchange calendar
121+
for futures. Explicit calendar disable does not bypass OHLCV validation.
122+
123+
`ValidationRuleConfig(calendar=...)` preserves omitted, named and explicitly disabled selections
124+
through model copying, JSON and `ValidationRuleSet` YAML. New rule configurations use version 2;
125+
version-1 payloads with a null calendar still inherit the default. Pass a restored configuration
126+
as `CrossValidator(feed_spec=rules)` to apply its declaration, or override it with `calendar=...`.
127+
Use ordinary `model_dump_json()` or `model_dump(mode="json")` without excluding explicit nulls.
77128

78129
### Persistent Rule Sets
79130

0 commit comments

Comments
 (0)