Skip to content

Commit b0d2027

Browse files
fix(temporal)!: qualify shared calendar grids, feature windows and labels (#98)
* fix: preserve shared calendar intent in session-aware labeling (#94) * docs: align feature task guidance with declared calendar modes (#94) * fix: retain calendar declaration presence in labeling config equality (#94) * fix(api): accept timestamp-free observation rows in explicit row mode (#76) * fix: use canonical calendar selector on the shared schedule adapter (#94) * fix: preserve timestamp precision and sparse-event label schemas (#94) * fix: keep row label duration metadata portable (#94) * fix: honor explicit labeling serialization projections (#94) * ci: run one full wheel suite and prove failure propagation (#99) * ci: verify installed identity and separate native fresh-process stress (#99) * ci: bound native build and use explicit fail-closed shells (#99) * test(ci): reject real installed version and package drift (#99) * build: qualify published Specs 0.2 temporal contract (#76) * fix(grid): validate daily session labels before feature returns (#76) * fix(ci): execute Git Bash rather than Windows WSL launcher (#99) * feat(labeling): apply scheduled-session horizons through existing barrier engine (#94) * feat(api): distinguish causal session and elapsed feature windows (#76) * test(api): retain independent annualization consumer regressions (#76) * fix(labeling): preserve portable schemas when price filters remove all rows (#94) * fix(labeling): keep session deadlines before excluded short-session gaps (#94) * docs(labeling): execute session horizons and preserve the previous guide anchor (#94) * test(labeling): qualify session horizons across DST and overnight venues (#94) * test(api): retain scheduled sessions with no bars under exclusion policy (#76) * fix(labeling): keep timestamp-free legacy durations portable and unset (#94) * docs: remove superseded daily session-label rejection (#94) * docs: apply maintainer-directed batching to temporal contract maintenance * fix(labeling): distinguish session identifiers from physical elapsed time * docs(calendars): reconcile helper guidance with daily crypto sessions
1 parent 3507071 commit b0d2027

25 files changed

Lines changed: 2802 additions & 133 deletions

‎AGENTS.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,3 +61,23 @@ uv build
6161
uv run python -c "import ml4t.engineer"
6262
uv run mkdocs build --strict
6363
```
64+
65+
## Maintainer-directed change batching
66+
67+
For our own fixes and agent work directed by the maintainer, use a bounded, coherent PR rather
68+
than one PR per small defect or implementation step. One PR may close several related issues
69+
when the changes serve the same user workflow, subsystem repair or compatibility objective and
70+
can be reviewed and reverted together. Keep implementation, regressions, documentation and
71+
follow-up repairs together; use focused commits and retain evidence for each defect. Do not
72+
create separate issues or PRs merely for individual files, tests, assertions or platform variants.
73+
74+
Split unrelated changes, independently risky or breaking changes, separate ownership boundaries,
75+
and work that needs independent release or rollback decisions. A general maintenance label alone
76+
does not make a batch coherent. Reuse or safely consolidate existing owned submissions instead of
77+
opening overlapping PRs; preserve contributor attribution, recoverable history and other owners'
78+
active work. Do not rewrite or close someone else's work merely to reduce the queue count.
79+
80+
Run focused checks while iterating, then qualify the complete final revision with every applicable
81+
required gate. Group related work before expensive CI; do not weaken checks, bypass review or
82+
claim old-head checks qualify changed code. This maintainer-directed batching rule takes precedence
83+
over a literal one-issue-per-PR reading of contributor intake guidance.

‎docs/user-guide/features.md‎

Lines changed: 94 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -69,8 +69,9 @@ Common asset columns such as `asset_id`, `symbol`, and `ticker` are detected aut
6969
and every rolling or lagged feature is evaluated independently for each asset.
7070
The examples explicitly select `calendar=None`: their periods count observations and
7171
input order is preserved, even when timestamps are irregular. Sort prices yourself before
72-
deriving returns if chronological order is intended. Without timestamps, also pass
73-
`assume_sorted=True` to assert that the supplied row order is correct.
72+
deriving returns if chronological order is intended. Row mode does not require timestamps
73+
or an additional `assume_sorted=True` assertion. A disabled calendar in shared feed metadata
74+
has the same behavior, even when the feed's timestamp column is absent.
7475

7576
Use `output` when requesting more than one configuration of the same feature:
7677

@@ -87,23 +88,52 @@ raise `ValueError` before feature execution.
8788
### Calendar and bar grid
8889

8990
The calendar-aware default targets the next minor version (0.2), separate from the 0.1.7 patch.
90-
Development and qualification currently pin the merged Specs v2 implementation at
91-
`701bde68744035050ede386efce498234c661c21`. The same immutable Git dependency is declared
92-
in built package metadata; published Specs 0.1.5 does not provide this API. Before a PyPI
93-
release, replace that source dependency with the verified published Specs version containing
94-
the v2 contract and repeat installed-package qualification.
91+
The shared temporal contract requires published `ml4t-specs>=0.2.0,<0.3.0`, also declared
92+
in built package metadata. Specs 0.1.5 does not provide this API. Engineer's development
93+
branch uses the published dependency; the temporal migration is not part of the 0.1.7 patch.
9594

9695
Calendar selection
9796
uses the explicit `calendar` argument, then declared feed metadata, then NYSE. Explicit
9897
`calendar=None` disables schedule interpretation. A named `"24/7"` calendar instead defines
9998
expected bars and detects outages, including on weekends.
10099

101-
Calendar-aware execution requires a shared `FeedSpec(time_contract_version=2)` with an
102-
explicit fixed intraday frequency and bar-grid policy. The initial implementation supports
103-
time bars anchored at each tradable interval's open. It uses real holidays, early closes and
104-
breaks from the selected schedule. Daily/session-label grids and session or elapsed-duration
105-
windows are not supported by this validator; they fail rather than being treated as row counts.
106-
Naive timestamps need a declared timezone. Aware timestamps already declare their timezone.
100+
Calendar-aware execution requires a shared `FeedSpec(time_contract_version=2)` with declared
101+
frequency and timestamp meaning. Fixed intraday time bars require an explicit bar-grid policy,
102+
anchored at each tradable interval's open. The selected schedule supplies holidays, early closes
103+
and breaks. Naive instants need a declared timezone; aware instants declare their timezone.
104+
105+
Daily bars instead declare `timestamp_semantics="session_label"`, `data_frequency="1d"` and
106+
`bar_type="time"`. Do not supply intraday anchor, label-side or partial-bar fields for session
107+
labels. Date values and datetime representations identify the displayed session date, without
108+
timezone conversion. Validation checks every scheduled session between each asset's first and
109+
last label, including completely missing sessions. A named `24/7` schedule includes weekends;
110+
an NYSE schedule does not. No prices are filled or rows inserted. A daily frequency alone does
111+
not turn `event_time` instants into session labels. Temporal feature windows below use the
112+
same completeness checks; choosing a unit does not permit missing scheduled observations.
113+
114+
<!-- ml4t-exec -->
115+
```python
116+
from datetime import date
117+
118+
import polars as pl
119+
from ml4t.engineer import compute_features
120+
from ml4t.specs import FeedSpec
121+
122+
close = [100.0, 103.0, 106.0, 109.0]
123+
daily_prices = pl.DataFrame({
124+
"timestamp": [date(2025, 7, day) for day in (3, 7, 8, 9)],
125+
"open": close, "high": close, "low": close, "close": close, "volume": 1.0,
126+
})
127+
daily_feed = FeedSpec(
128+
time_contract_version=2, calendar="NYSE", data_frequency="1d",
129+
timestamp_semantics="session_label", bar_type="time",
130+
)
131+
checked = compute_features(daily_prices, [], feed_spec=daily_feed)
132+
returns = checked.with_columns(pl.col("close").log().diff().alias("returns"))
133+
features = compute_features(returns, [{"name": "sma", "params": {"period": 2}}], feed_spec=daily_feed)
134+
assert features["sma"].to_list() == [None, 101.5, 104.5, 107.5]
135+
assert features["timestamp"].equals(daily_prices["timestamp"])
136+
```
107137

108138
`CryptoCalendar` also changes in the next minor version. Install the `calendars` extra
109139
to use its authoritative `24/7` schedule. It enumerates UTC daily session opens, not
@@ -174,6 +204,56 @@ This does not change legacy Specs bridges or the numerical feature kernels.
174204
Neither means sessions or elapsed hours. A shortened final bar still counts as one expected
175205
bar when `partial_bar_policy="include"`. Annualization remains a separate convention: for
176206
example, `realized_volatility(trading_periods=252)` retains its documented numerical default.
207+
208+
### Session and elapsed feature windows
209+
210+
`sma`, `stddev` and `realized_volatility` accept `window_unit="session"` or `"elapsed"`.
211+
They retain original bars and row counts. They do not first aggregate to session closes.
212+
For sessions, `params.period=N` includes the current session's observations through the
213+
current row and the preceding N-1 scheduled sessions. Earlier rows of the first N-1
214+
observed sessions are null. July 4 holidays and lunch breaks do not create extra sessions.
215+
A one-session average of intraday prices100/110 is therefore100 at the first row and105
216+
at the second row, not the session's eventual closing price110 at both rows.
217+
218+
Elapsed periods use a positive duration such as `params.period="2h"` and the physical
219+
UTC interval `(timestamp - duration, timestamp]`, including through DST changes.
220+
Outputs remain null until observed coverage spans the requested duration. A half-hour
221+
final bar can put three observations inside two elapsed hours. Daily `session_label`
222+
identifiers cannot define elapsed windows; they do support session-count windows.
223+
Windows with insufficient samples for the numerical statistic also remain null.
224+
Annualization parameters and all other supported feature parameters retain their
225+
numerical definitions. Recursive EMA/RSI and other unqualified window definitions
226+
reject temporal units explicitly rather than silently resetting or reseeding.
227+
228+
<!-- ml4t-exec -->
229+
```python
230+
from datetime import date, datetime, timedelta, timezone
231+
232+
import polars as pl
233+
from ml4t.engineer import compute_features
234+
from ml4t.specs import FeedSpec
235+
236+
daily = FeedSpec(time_contract_version=2, calendar="NYSE", data_frequency="1d",
237+
timestamp_semantics="session_label", bar_type="time")
238+
values = [100.0, 101.0, 104.0]
239+
prices = pl.DataFrame({
240+
"timestamp": [date(2025, 7, day) for day in (2, 3, 7)],
241+
"open": values, "high": values, "low": values, "close": values, "volume": 1.0,
242+
})
243+
result = compute_features(prices, [{"name": "sma", "params": {"period": 2}}],
244+
feed_spec=daily, window_unit="session")
245+
assert result["sma"].to_list() == [None, 100.5, 102.5]
246+
247+
hourly = FeedSpec(time_contract_version=2, calendar="24/7", data_frequency="1h",
248+
timestamp_semantics="event_time", timezone="UTC", bar_type="time",
249+
bar_grid_anchor="session_open", bar_label="left", partial_bar_policy="include")
250+
prices = prices.with_columns(pl.Series("timestamp", [
251+
datetime(2025, 7, 5, tzinfo=timezone.utc) + timedelta(hours=i) for i in range(3)
252+
]))
253+
result = compute_features(prices, [{"name": "sma", "params": {"period": "2h"}}],
254+
feed_spec=hourly, window_unit="elapsed")
255+
assert result["sma"].to_list() == [None, None, 102.5]
256+
```
177257
No calendar-derived annualization factor is silently substituted.
178258

179259
### Equity schedule helpers
@@ -501,7 +581,7 @@ Most features expect a DataFrame with standardized column names (lowercase):
501581

502582
| Column | Type | Required By |
503583
|--------|------|-------------|
504-
| `timestamp` | datetime | Temporal ordering, unless `assume_sorted=True` |
584+
| `timestamp` | datetime | Required for calendar-aware grids; optional in observation-row mode |
505585
| `asset_id` | string | Optional panel grouping, auto-detected when present |
506586
| `open` | float | OHLCV, OHLC features |
507587
| `high` | float | OHLCV, OHLC, HLC, HL features |

‎docs/user-guide/labeling.md‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -414,6 +414,12 @@ result = apply_meta_model(
414414

415415
For futures and other instruments with defined trading sessions, calendar-aware labeling respects session boundaries. A label that would span an overnight gap is handled correctly.
416416

417+
In the next minor version, `calendar_aware_labels` uses the shared temporal contract.
418+
An explicit `calendar` argument wins over a declared `LabelingConfig.calendar`, then
419+
`FeedSpec` metadata, then NYSE. Omitting a selector differs from explicit `None`.
420+
Config JSON/YAML output preserves that distinction and emits only explicitly declared
421+
column overrides, so a round trip does not replace feed column mappings with defaults.
422+
417423
```python
418424
from ml4t.engineer.labeling import calendar_aware_labels
419425

@@ -433,6 +439,112 @@ former pause because trading is continuous until 4:00 PM.
433439
See the [CME exchange notice](https://www.cmegroup.com/notices/electronic-trading/2021/06/20210621.html)
434440
for the affected equity-index contracts.
435441

442+
<a id="observation-rows-expected-bars-and-elapsed-durations"></a>
443+
444+
### Observation rows, expected bars, sessions and elapsed durations
445+
446+
Use `calendar=None` for observation-row labeling, including timestamp-indexed data:
447+
448+
```python
449+
row_config = LabelingConfig.triple_barrier(
450+
upper_barrier=0.02,
451+
lower_barrier=0.02,
452+
max_holding_period=10,
453+
calendar=None,
454+
window_unit="observation",
455+
)
456+
labels = calendar_aware_labels(df, row_config)
457+
```
458+
459+
Row mode retains input order, processes each asset independently, does not require
460+
timestamps and never loads a schedule provider. Its `label_bars` and `label_time` are
461+
observation counts and per-asset row offsets; `label_duration` is unset. Timestamp
462+
spacing and weekend placement do not alter its numerical labels. Duration holding
463+
periods are incompatible with this mode.
464+
465+
Named calendars segment authoritative sessions and breaks. New shared-feed calls and
466+
omitted selectors default count windows to `"expected_bar"`; an undeclared or incomplete
467+
grid fails before labeling. Supply the complete shared grid
468+
declarations described in the [feature guide](features.md#calendar-and-bar-grid) to reject
469+
missing scheduled bars before labeling. Named `"24/7"` still has daily schedule boundaries
470+
and is not equivalent to row mode. Duration holding periods select `"elapsed"` and remain truncated at session
471+
boundaries, rather than counting sessions or observed dates.
472+
473+
Pass `feed_spec=FeedSpec(...)` directly or through `LabelingConfig.feed_spec`. Shared
474+
`timestamp_semantics` must distinguish intraday instants from `session_label` identifiers.
475+
Naive feed timestamps require an explicit timezone, and ambiguous/nonexistent local
476+
times fail. Right-labeled/bar-close observations at a break boundary end the preceding
477+
tradable interval. Input timestamp representations are retained in the output.
478+
Instant horizons and `label_duration` use physical elapsed time, including across
479+
daylight-saving transitions. Shared `session_label` identifiers do not identify instants;
480+
their `label_time` retains the session identifier and `label_duration` is unset.
481+
482+
Legacy calls without FeedSpec keep naive-UTC handling. Custom protocol objects and
483+
explicit `calendar="auto"` remain compatibility modes without shared feed metadata.
484+
An explicit legacy calendar selector without FeedSpec retains observation-count horizons.
485+
New calls can also select `window_unit="observation"` explicitly to count received rows
486+
within each authoritative segment, without claiming scheduled-bar completeness.
487+
Auto-gap segmentation does not establish market holidays, session completeness or a
488+
bar grid. Standalone labeling kernels retain their documented row-count or elapsed-time
489+
definitions and chronological sorting; they do not acquire calendar interpretation from
490+
timestamps. Use this wrapper when selecting the shared calendar/row contract.
491+
Standalone kernels reject configs declaring calendar/feed/window semantics instead
492+
of silently ignoring those declarations.
493+
Standalone kernels retain date-valued label times and calendar-day differences in
494+
`label_duration` for legacy `Date` inputs. These standalone calls do not validate
495+
the dates against an exchange schedule.
496+
497+
For `window_unit="session"`, an integer `max_holding_period=N` expires at the final
498+
bar of the Nth scheduled session after the entry session. A positive integral count
499+
column can select N per event. July 3, 2025 on NYSE therefore reaches July 7 for N=1,
500+
not the July 4 holiday. Intraday entries reach the following session's last bar,
501+
including an early close; lunch and overnight breaks do not truncate this horizon.
502+
Profit, stop and trailing barriers still use the existing triple-barrier calculation.
503+
The full declared grid is checked before any labels are computed, including whole-session
504+
outages. No prices are filled or rows inserted.
505+
506+
With `partial_bar_policy="exclude"`, a shortened session can have no bars. It still
507+
counts as a scheduled session. The exit uses the last available bar no later than the
508+
deadline, even if that is the entry itself with zero holding bars. The calculation
509+
does not extend the deadline into a later session to obtain another observation.
510+
511+
Daily input uses `FeedSpec(time_contract_version=2, calendar="NYSE",
512+
timestamp_semantics="session_label", data_frequency="1d", bar_type="time")`.
513+
Do not supply intraday anchor/label/partial-bar fields. Date-valued and midnight datetime
514+
identifiers keep their displayed session date; they are not converted to UTC instants.
515+
Daily observation and expected-bar horizons scan across session rows rather than
516+
truncating every label to its entry row. Elapsed horizons reject these identifiers.
517+
518+
As with existing count horizons, a deadline beyond the available dataset is censored
519+
at the last available bar. `label_bars` reports the actual observed holding length and
520+
`label_time` the actual exit, not an inferred future observation. These changes target
521+
the next 0.2 development release, not the historical 0.1.7 patch.
522+
523+
<!-- ml4t-exec -->
524+
```python
525+
from datetime import date
526+
from io import BytesIO
527+
528+
import polars as pl
529+
from ml4t.specs import FeedSpec
530+
from ml4t.engineer.config import LabelingConfig
531+
from ml4t.engineer.labeling import calendar_aware_labels
532+
533+
prices = pl.DataFrame({
534+
"timestamp": [date(2025, 7, day) for day in (3, 7, 8)],
535+
"close": [100.0, 101.0, 103.0],
536+
})
537+
config = LabelingConfig.triple_barrier(
538+
upper_barrier=None, lower_barrier=None, max_holding_period=1, window_unit="session",
539+
feed_spec=FeedSpec(time_contract_version=2, calendar="NYSE", data_frequency="1d",
540+
timestamp_semantics="session_label", bar_type="time"),
541+
)
542+
labels = calendar_aware_labels(prices, config)
543+
assert labels["label_time"].to_list() == [date(2025, 7, 7), date(2025, 7, 8), date(2025, 7, 8)]
544+
assert labels["label_bars"].to_list() == [1, 1, 0]
545+
labels.write_parquet(BytesIO())
546+
labels.head(0).write_parquet(BytesIO())
547+
```
436548
## Sample Weighting
437549

438550
AFML Chapter 4 addresses the problem of overlapping labels creating correlated samples. The library provides the full toolkit:

‎pyproject.toml‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,6 @@ build-backend = "hatchling.build"
88
[tool.hatch.version]
99
source = "vcs"
1010

11-
[tool.hatch.metadata]
12-
# Qualification pin until the shared time-contract v2 API is published on PyPI.
13-
allow-direct-references = true
14-
1511
[tool.hatch.build.targets.wheel]
1612
packages = ["src/ml4t"]
1713
namespaces = true
@@ -76,7 +72,7 @@ requires-python = ">=3.12,<3.15"
7672

7773
# Core dependencies
7874
dependencies = [
79-
"ml4t-specs @ git+https://github.com/ml4t/specs.git@701bde68744035050ede386efce498234c661c21",
75+
"ml4t-specs>=0.2.0,<0.3.0",
8076

8177
# Data processing (Polars-first)
8278
"polars>=0.20.0",

0 commit comments

Comments
 (0)