Skip to content

Commit da7d259

Browse files
fix: preserve calendar declarations and authoritative session intervals
Closes ml4t#93
1 parent c7b9dfa commit da7d259

8 files changed

Lines changed: 728 additions & 136 deletions

File tree

‎docs/user-guide/sessions.md‎

Lines changed: 49 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,9 @@ dependency is included with the package.
1010

1111
## Assign Session Dates
1212

13-
`SessionAssigner` requires a Polars `Datetime` column named `timestamp`. Naive timestamps are
14-
interpreted as UTC; timezone-aware timestamps are converted to UTC before they are matched to a
15-
schedule.
13+
`SessionAssigner` accepts Polars `Datetime` instants or `Date` session labels in `timestamp`.
14+
Naive instants use UTC unless the feed declares a timezone; timezone-aware instants are matched in
15+
UTC. Ambiguous or nonexistent local times fail rather than being guessed.
1616

1717
```python
1818
from datetime import datetime
@@ -55,6 +55,40 @@ fall within an open interval:
5555
Set `bar_frequency="daily"` when timestamps are daily labels. The default, `"auto"`, treats a
5656
series as daily when every non-null timestamp is midnight.
5757

58+
Explicit `FeedSpec.timestamp_semantics` overrides these heuristics: `session_label` preserves the
59+
civil label date, `event_time` matches an instant, and `bar_close` matches the ending bar boundary.
60+
Event intervals include the open and exclude the close; close-labeled bars exclude the open and
61+
include the close. The corresponding boundaries apply to lunch breaks and declared calendar
62+
interruptions. An explicit `bar_label` also selects the declared side; `event_time` alone does not
63+
infer a bar side. This assignment does
64+
not validate intraday grid completeness.
65+
66+
`session_start_time` is retained in metadata, but these schedule adapters reject a non-null override.
67+
Register and select an authoritative calendar with the required intervals instead of silently
68+
replacing a named exchange's session boundaries.
69+
70+
```python
71+
from ml4t.specs import FeedSpec
72+
73+
feed = FeedSpec(calendar="24/7", timestamp_semantics="event_time", time_contract_version=2)
74+
assigned = SessionAssigner(feed_spec=feed).assign_sessions(bars)
75+
unchanged = SessionAssigner(calendar=None).assign_sessions(bars)
76+
```
77+
78+
Use the canonical `calendar` keyword on constructors and facade methods. Explicit selection wins
79+
over feed metadata, then NYSE applies. Explicit `None` returns observations unchanged, without
80+
adding a session column or reconstructing a continuous grid. `calendar="24/7"` remains a real
81+
schedule, including an event exactly at midnight in the new day's session. Unknown names fail.
82+
The positional calendar argument, `calendar_name` constructor keyword and explicit `exchange`
83+
selector remain compatibility forms; contradictory explicit selectors raise an error. Descriptive
84+
exchange identity in feed metadata does not select a calendar.
85+
86+
Data's `Metadata` supports the same versioned temporal declaration. Version 1 retains legacy null
87+
inheritance; version 2 preserves omitted calendar versus explicit null through JSON, copying and
88+
shared FeedSpec projection. Use `model_dump(mode="json")` before a YAML writer, and ordinary
89+
`model_dump_json()` for JSON; do not deliberately exclude null fields when preserving a disable
90+
declaration. Declared timezone, timestamp semantics and fixed-grid fields survive these bridges.
91+
5892
## Select an Exchange Calendar
5993

6094
`SessionAssigner.from_exchange()` recognizes these exchange codes:
@@ -100,8 +134,18 @@ The supported fill methods are `"forward"`, `"backward"`, and `"none"`. Forward
100134
filling apply only within a session. `zero_volume=True` sets volume to zero for inserted rows.
101135

102136
Session completion rejects duplicate or non-minute-aligned timestamps and observations outside the
103-
selected schedule. Use session assignment first when you need to inspect how individual timestamps
104-
map to the exchange calendar.
137+
selected schedule. Lunch breaks and declared calendar interruptions are excluded from the template;
138+
completion never inserts rows for those closures. Use session assignment first to inspect how
139+
timestamps map to the exchange calendar.
140+
141+
Completion remains a one-minute, open-labeled operation, not a general bar-grid validator. Explicit
142+
close-labeled or session-label metadata is rejected. `calendar=None` returns the input unchanged.
143+
Declared event-count bar types and frequencies other than one minute are also rejected.
144+
General bar-grid anchor/partial-bar policies are preserved as metadata but explicitly rejected by
145+
this native minute completer; they are not silently approximated by its legacy minute template.
146+
`DataManager.complete_sessions(fill_gaps=False)` performs assignment only and never expands rows,
147+
whether or not storage is configured. Feed declarations are accepted through the facade's
148+
`feed_spec` keyword with the same precedence as the low-level classes.
105149

106150
## Use Session Dates in Validation
107151

‎src/ml4t/data/core/models.py‎

Lines changed: 62 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,18 @@
77
from typing import Any
88

99
import polars as pl
10-
from pydantic import BaseModel, Field, model_validator
10+
from pydantic import BaseModel, Field, field_validator, model_serializer, model_validator
1111

1212
from ml4t.data.assets.asset_class import AssetClass # noqa: F401 — re-exported
13+
from ml4t.specs import (
14+
UNSET_CALENDAR,
15+
BarGridAnchor,
16+
BarLabel,
17+
CalendarUnset,
18+
FeedSpec,
19+
PartialBarPolicy,
20+
TimestampSemantics,
21+
)
1322

1423

1524
class SchemaVersion(StrEnum):
@@ -73,7 +82,14 @@ class Metadata(BaseModel):
7382

7483
# Exchange and calendar (for session-aware operations)
7584
exchange: str = "UNKNOWN"
76-
calendar: str | None = None # pandas_market_calendars name (e.g., "CME_Globex_Crypto")
85+
calendar: str | None | CalendarUnset = UNSET_CALENDAR
86+
time_contract_version: int = 1
87+
timezone: str | None = None
88+
timestamp_semantics: TimestampSemantics | str | None = None
89+
session_start_time: str | None = None
90+
bar_grid_anchor: BarGridAnchor | str | None = None
91+
bar_label: BarLabel | str | None = None
92+
partial_bar_policy: PartialBarPolicy | str | None = None
7793

7894
# Tracking
7995
start_date: datetime | None = None
@@ -89,6 +105,50 @@ class Metadata(BaseModel):
89105

90106
model_config = {"use_enum_values": True}
91107

108+
@field_validator("calendar", mode="wrap")
109+
@classmethod
110+
def preserve_calendar_presence(cls, value: Any, handler: Any) -> Any:
111+
return value if value is UNSET_CALENDAR else handler(value)
112+
113+
@field_validator("time_contract_version", mode="before")
114+
@classmethod
115+
def validate_time_contract_version(cls, value: Any) -> int:
116+
return FeedSpec(time_contract_version=value).time_contract_version
117+
118+
def _time_declaration(self) -> FeedSpec:
119+
return FeedSpec(
120+
calendar=self.calendar,
121+
time_contract_version=self.time_contract_version,
122+
timezone=self.timezone,
123+
timestamp_semantics=self.timestamp_semantics,
124+
session_start_time=self.session_start_time,
125+
bar_grid_anchor=self.bar_grid_anchor,
126+
bar_label=self.bar_label,
127+
partial_bar_policy=self.partial_bar_policy,
128+
)
129+
130+
@model_validator(mode="after")
131+
def normalize_time_declaration(self) -> Metadata:
132+
object.__setattr__(self, "calendar", self._time_declaration().calendar)
133+
return self
134+
135+
@model_serializer(mode="wrap")
136+
def serialize_time_declaration(self, handler: Any) -> dict[str, Any]:
137+
data = handler(self)
138+
declaration = self._time_declaration().to_dict()
139+
for name in (
140+
"calendar",
141+
"time_contract_version",
142+
"bar_grid_anchor",
143+
"bar_label",
144+
"partial_bar_policy",
145+
):
146+
if name in declaration and name in data:
147+
data[name] = declaration[name]
148+
elif name not in declaration:
149+
data.pop(name, None)
150+
return data
151+
92152
@property
93153
def frequency(self) -> str:
94154
"""Get frequency string for backward compatibility.

‎src/ml4t/data/data_manager.py‎

Lines changed: 12 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
from ml4t.data.managers.metadata_manager import MetadataManager
3030
from ml4t.data.managers.provider_manager import ProviderManager, ProviderRouter
3131
from ml4t.data.managers.storage_manager import StorageManager
32+
from ml4t.specs import UNSET_CALENDAR, CalendarUnset
3233

3334
logger = structlog.get_logger()
3435

@@ -453,51 +454,30 @@ def assign_sessions(
453454
self,
454455
df: pl.DataFrame,
455456
exchange: str | None = None,
456-
calendar: str | None = None,
457+
calendar: str | None | CalendarUnset = UNSET_CALENDAR,
457458
bar_frequency: Literal["auto", "daily", "intraday"] = "auto",
459+
*,
460+
feed_spec: Any | None = None,
458461
) -> pl.DataFrame:
459462
"""Assign sessions to daily period labels or intraday timestamps."""
460-
if not self._metadata_manager:
461-
# Fall back to direct implementation
462-
from ml4t.data.sessions import SessionAssigner
463-
464-
if calendar is None:
465-
if exchange is None:
466-
raise ValueError("Must provide either exchange or calendar parameter")
467-
assigner = SessionAssigner.from_exchange(exchange)
468-
else:
469-
assigner = SessionAssigner(calendar)
470-
return assigner.assign_sessions(df, bar_frequency=bar_frequency)
471-
472-
return self._metadata_manager.assign_sessions(df, exchange, calendar, bar_frequency)
463+
manager = self._metadata_manager or MetadataManager(None)
464+
return manager.assign_sessions(df, exchange, calendar, bar_frequency, feed_spec=feed_spec)
473465

474466
def complete_sessions(
475467
self,
476468
df: pl.DataFrame,
477469
exchange: str | None = None,
478-
calendar: str | None = None,
470+
calendar: str | None | CalendarUnset = UNSET_CALENDAR,
479471
fill_gaps: bool = True,
480472
fill_method: str = "forward",
481473
zero_volume: bool = True,
474+
*,
475+
feed_spec: Any | None = None,
482476
) -> pl.DataFrame:
483477
"""Complete sessions by filling gaps."""
484-
if not self._metadata_manager:
485-
# Fall back to direct implementation
486-
from ml4t.data.sessions import SessionCompleter
487-
from ml4t.data.sessions.assigner import EXCHANGE_CALENDARS
488-
489-
if calendar is None:
490-
if exchange is None:
491-
raise ValueError("Must provide either exchange or calendar parameter")
492-
calendar = EXCHANGE_CALENDARS.get(exchange.upper())
493-
if not calendar:
494-
raise ValueError(f"Unknown exchange '{exchange}'")
495-
496-
completer = SessionCompleter(calendar)
497-
return completer.complete_sessions(df, fill_method=fill_method, zero_volume=zero_volume)
498-
499-
return self._metadata_manager.complete_sessions(
500-
df, exchange, calendar, fill_gaps, fill_method, zero_volume
478+
manager = self._metadata_manager or MetadataManager(None)
479+
return manager.complete_sessions(
480+
df, exchange, calendar, fill_gaps, fill_method, zero_volume, feed_spec=feed_spec
501481
)
502482

503483
# ========================================================================

‎src/ml4t/data/managers/metadata_manager.py‎

Lines changed: 10 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616

1717
from ml4t.data.storage.backend import normalize_storage_metadata
1818
from ml4t.data.storage.keys import storage_key_path
19+
from ml4t.specs import UNSET_CALENDAR, CalendarUnset
1920

2021
logger = structlog.get_logger()
2122

@@ -156,8 +157,10 @@ def assign_sessions(
156157
self,
157158
df: pl.DataFrame,
158159
exchange: str | None = None,
159-
calendar: str | None = None,
160+
calendar: str | None | CalendarUnset = UNSET_CALENDAR,
160161
bar_frequency: Literal["auto", "daily", "intraday"] = "auto",
162+
*,
163+
feed_spec: Any | None = None,
161164
) -> pl.DataFrame:
162165
"""Assign session_date column to DataFrame based on exchange calendar.
163166
@@ -175,23 +178,20 @@ def assign_sessions(
175178
"""
176179
from ml4t.data.sessions import SessionAssigner
177180

178-
if calendar is None:
179-
if exchange is None:
180-
raise ValueError("Must provide either exchange or calendar parameter")
181-
assigner = SessionAssigner.from_exchange(exchange)
182-
else:
183-
assigner = SessionAssigner(calendar)
181+
assigner = SessionAssigner(calendar, exchange=exchange, feed_spec=feed_spec)
184182

185183
return assigner.assign_sessions(df, bar_frequency=bar_frequency)
186184

187185
def complete_sessions(
188186
self,
189187
df: pl.DataFrame,
190188
exchange: str | None = None,
191-
calendar: str | None = None,
189+
calendar: str | None | CalendarUnset = UNSET_CALENDAR,
192190
fill_gaps: bool = True,
193191
fill_method: str = "forward",
194192
zero_volume: bool = True,
193+
*,
194+
feed_spec: Any | None = None,
195195
) -> pl.DataFrame:
196196
"""Complete sessions by filling gaps.
197197
@@ -213,24 +213,13 @@ def complete_sessions(
213213
"""
214214
from ml4t.data.sessions import SessionCompleter
215215

216-
if calendar is None:
217-
if exchange is None:
218-
raise ValueError("Must provide either exchange or calendar parameter")
219-
220-
from ml4t.data.sessions.assigner import EXCHANGE_CALENDARS
221-
222-
calendar = EXCHANGE_CALENDARS.get(exchange.upper())
223-
if not calendar:
224-
raise ValueError(f"Unknown exchange '{exchange}'")
225-
226-
completer = SessionCompleter(calendar)
227-
228216
if not fill_gaps:
229217
from ml4t.data.sessions import SessionAssigner
230218

231-
assigner = SessionAssigner(calendar)
219+
assigner = SessionAssigner(calendar, exchange=exchange, feed_spec=feed_spec)
232220
return assigner.assign_sessions(df)
233221

222+
completer = SessionCompleter(calendar, exchange=exchange, feed_spec=feed_spec)
234223
return completer.complete_sessions(
235224
df,
236225
fill_method=fill_method,

‎src/ml4t/data/managers/storage_manager.py‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -624,7 +624,11 @@ def update(
624624
existing_model_values = {
625625
name: value
626626
for name in Metadata.model_fields
627-
if (value := existing_metadata.get(name)) is not None
627+
if name in existing_metadata
628+
and (
629+
(value := existing_metadata[name]) is not None
630+
or (name == "calendar" and existing_metadata.get("time_contract_version") == 2)
631+
)
628632
}
629633
resolved_provider = (
630634
provider
@@ -662,7 +666,18 @@ def update(
662666
for detail in error.errors()
663667
if (location := detail.get("loc")) and isinstance(location[0], str)
664668
}
665-
removable_fields = invalid_fields - {"provider", "symbol", "asset_class"}
669+
removable_fields = invalid_fields - {
670+
"provider",
671+
"symbol",
672+
"asset_class",
673+
"calendar",
674+
"time_contract_version",
675+
"timezone",
676+
"timestamp_semantics",
677+
"bar_grid_anchor",
678+
"bar_label",
679+
"partial_bar_policy",
680+
}
666681
if not removable_fields:
667682
raise
668683
logger.warning(

0 commit comments

Comments
 (0)