Skip to content

fix: recover calendar gaps from exact Databento source ranges - #115

Merged
stefan-jansen merged 5 commits into
fix/stored-calendar-gap-reportfrom
fix/source-backed-gap-recovery
Oct 18, 2026
Merged

stefan-jansen merged 5 commits into
fix/stored-calendar-gap-reportfrom
fix/source-backed-gap-recovery

Conversation

@stefan-jansen

Copy link
Copy Markdown
Contributor

Outcome

Consume a selected calendar gap report in a separate, explicit source-backed download operation. Refs #110. No synthetic candles, prices or volume. This is the Databento stage, not a claim of universal source support or completed Coin Metrics integration.

Databento adds exact aware half-open timestamp requests with shared rate limiting, retries, circuit breaking and OHLCV normalization. Single-series symbology is explicit; session-date shifting, parent/universe expansion and unsupported-frequency daily fallback are rejected. Preserve the resolved contract separately from the requested series.

Backfill rechecks disk before fetching, separates storage keys from vendor instruments, preserves real observations and identity/calendar metadata, replaces only marked placeholders with returned observations, and reports residual gaps. Empty/no-trade responses are not filled or retried indefinitely. Partial recovery is distinct from successful processing. Unsupported timestamp capabilities, source mismatches, disabled calendars and multi-instrument write-back fail explicitly. Generic backfill shares row selection and identity-safe merging; candidate scans use bar counts for intraday gaps.

Reproduction and checks

  • Configured public workflow at base60f65912802582484e152b812403e950f8eafa50:18 failures2.12s, clean exit1. Actual SyntheticProvider producer, Databento response transformation/normalization, authoritative schedule, Flat/Hive storage, disk report and consumer. Only the external SDK is substituted; no paid or client production-data requests.
  • Additional review regressions before fixes:4 nullable-grid/foreign-instrument failures;3 universe/candidate failures with28 controls;2 native-minute alias failures. A private-path exploratory run exited139 at teardown and is not accepted as a clean baseline; ordinary configured public tests supersede it.
  • Head b714ac9:274 focused passes30.03s, fatal ResourceWarning. Covers original-row/null preservation, partial/no-trade/stale/boundary/placeholder recovery, source aliasing and invalid capability handling. Both maintained guide recipes execute against actual provider/storage paths with only the SDK substituted.
  • Ruff check/format100,ty,normal hooks,all-file precommit,diff check and strict MkDocs passed. Final full coverage/resource suite and isolated wheel/sdist verification are running; this draft is not qualified for merge yet.

Dependencies and remaining scope

Draft stacked on no-imputation PR114, which depends on report113 and Specs112. Includes the report's latest bounded-frame follow-up through a normal merge, not a history rewrite. Retarget/integrate only the next selected dependent after its prerequisite lands. Stacked main-base workflow exclusions do not establish qualification; all applicable current-head supported checks remain required before ordinary merge.

Coin Metrics has no current Data adapter. Market candles and asset metrics/reference rates need different identities and schemas; the maintainer's dataset type remains to be confirmed. Keep issue110 open until accepted source-specific work is resolved. Raw-file inspection is generic; native timestamp retrieval returns real frames for separately reviewed raw-file merging, not automatic production-data migration.

AI-assisted implementation and owner review under maintainer direction. No independent reviewer claimed. Data0.2.1 remains immutable; no new release or deployment.

Base automatically changed from fix/preserve-market-absences to fix/stored-calendar-gap-report October 18, 2026 00:22
@stefan-jansen
stefan-jansen merged commit d6076e5 into fix/stored-calendar-gap-report Oct 18, 2026
8 of 11 checks passed
@stefan-jansen
stefan-jansen deleted the fix/source-backed-gap-recovery branch October 18, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant