Repository navigation
Preserve observation fidelity through storage and pandas handoffs #117
Description
Saved market data must preserve source observation identity, missingness and temporal declarations before analytical conversion. Current Flat/Hive reads normalize datetime timestamps to UTC microseconds; reproduce adjacent-nanosecond and session-label behavior through public disk workflows before repairing any demonstrated defect.
Public contract and ownership
Extend existing storage read paths and utils.conversion with key-safe to_price_panel/from_price_panel. Reuse FeedSpec v2 and declared timestamp_col/entity_col mappings; new examples use timestamp/asset. Keep calendar precedence explicit argument > declared feed > NYSE, explicit None row mode and named24/7 scheduled mode.
Data owns preparation/acquisition facts only. Supplied tables work independently of other libraries; shared types reuse Specs. No other owner's source changes are part of this issue.
Acceptance
- Actual write/read retains adjacent ns instants, complete keys and source values; declared session labels retain their civil-label meaning across UTC/local representations. Legacy undeclared inputs retain documented compatibility.
- Round-trip omitted/null/named calendar and availability/basis/source metadata; test Flat and Hive, projected columns, bounded reads, empty frames and malformed declarations.
- Long/wide round trip preserves complete price clock, null cells, ragged universes, stable IDs and order. Duplicate keys or conflicting mappings fail before mutation; no fills, cleaning or returns calculations.
- Independent fixture: two assets with prices [100,101,98,103] and [50,52,null,53] over July2/3/7/8; missing B quote remains null. Test CME overnight, NYSE holiday/early close, DST, crypto weekend, 30m/1ns/composite declarations.
- Installed wheel/sdist Data output consumed by actual supported Engineer/Diagnostic public APIs; no mandatory runtime dependency on those packages. Runnable offline migration example and strict docs.
Run focused public API/regression tests first, then full final-revision native/source,ResourceWarning,coverage,type,docs,wheel/sdist,security and applicable hosted platform gates. Retain failing-base/passing-fix evidence for any bug; new capability tests use independent expected values. Include implementation,tests and executable migration docs in the same bounded PR. AI assistance must be disclosed and maintainer-reviewed.
Dependencies
No successor implementation prerequisite. Reuse landed calendar/storage fixes; do not reopen completed calendar issues.
Premises
Input: observation-fidelity
| Question | Answer | Kind |
|---|---|---|
| record | One declared observation keyed by timestamp and stable entity; null value and absent clock row are distinct. | definitional |
| absent-record | Absence means missing source coverage,not automatically zero,no-trade,holiday or delisting. | definitional |
| ordering-spacing | Use declared timestamp/session semantics and grid/window unit; preserve observed keys rather than compressing gaps. | definitional |
| entity-identity | Stable asset identity is separate from source symbol,sid and display name; explicit mappings are required. | definitional |
| known-at | Observation timestamps and source availability are distinct fields; storage/conversion preserves both without inferring availability. | definitional |
| institution | Source/exchange facts and selected authoritative schedule govern; this issue does not simulate trading. | definitional |
Claims
| Id | Claim | How it could be false |
|---|---|---|
| observation-fidelity-workflow | The declared public producer and conversion workflow retains the contract and fails atomically on invalid input. | A real produced input loses keys,precision,source units,availability,missingness or revision identity; a reference-compatible route changes values/shapes/endpoints; or an error publishes partial output. |
Closes: observation-fidelity.record, observation-fidelity.absent-record, observation-fidelity.ordering-spacing, observation-fidelity.entity-identity, observation-fidelity.known-at, observation-fidelity.institution, observation-fidelity-workflow
Advances: none
Plan reference: accepted Data successor implementation plan, Issue 1 in milestone Successor migration - Data inputs and provenance. Harmonized ownership: Data prepares inputs; Engineer owns features,Backtest execution,Diagnostic analytics. No publication/release authorization.
Activity
docs(data): qualify successor price-table migration inputs
fix(data): retain declared source clocks and native missing-value dis…
fix(data): preserve panel provenance through pandas persistence
feat(history): prepare dated identities and known-at adjusted inputs (#…