Skip to content

Commit cc07c82

Browse files
patelchaitanypatelchaitany
authored
fix: Use UTC end bound and correct docs for disable_event_timestamp (#6956)
* fix: Use UTC end bound and correct docs for disable_event_timestamp disable_event_timestamp was documented as stamping rows with the current datetime, but no materialization engine reads the flag; rows keep their source event timestamps. Update docstrings, CLI help and docs so the flag only promises full-window materialization. Also compute the full-window bounds in UTC. The CLI and feature server used naive datetime.now(), which make_tzaware() treats as UTC, so hosts west of UTC skipped the newest rows. Fixes #6936 Signed-off-by: Chaitany Patel <[email protected]> * docs: State the UTC upper bound for disable_event_timestamp Signed-off-by: Chaitany Patel <[email protected]> --------- Signed-off-by: Chaitany Patel <[email protected]>
1 parent ac47f30 commit cc07c82

13 files changed

Lines changed: 35 additions & 23 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@ feast materialize 2021-04-12T00:00:00 $CURRENT_TIME
127127
feast materialize --disable-event-timestamp
128128
```
129129

130-
The `--disable-event-timestamp` flag allows you to materialize all available feature data using the current datetime as the event timestamp, without needing to specify start and end timestamps. This is useful when your source data lacks proper event timestamp columns.
130+
The `--disable-event-timestamp` flag allows you to materialize all available data (from 1970-01-01 up to the current UTC time) without specifying start and end timestamps. Rows keep the event timestamps from the source.
131131

132132
```commandline
133133
Materializing feature view driver_hourly_stats from 2021-04-14 to 2021-04-15 done!

‎docs/getting-started/components/online-store.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Online store
22

33
Feast uses online stores to serve features at low latency.
4-
Feature values are loaded from data sources into the online store through _materialization_, which can be triggered through the `materialize` command (either with specific timestamps or using `--disable-event-timestamp` to materialize all data with current timestamps).
4+
Feature values are loaded from data sources into the online store through _materialization_, which can be triggered through the `materialize` command (either with specific timestamps or using `--disable-event-timestamp` to materialize all available data).
55

66
The storage schema of features within the online store mirrors that of the original data source.
77
One key difference is that for each [entity key](../concepts/entity.md), only the latest feature values are stored.

‎docs/getting-started/components/overview.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
* **Create Batch Features:** ELT/ETL systems like Spark and SQL are used to transform data in the batch store.
88
* **Create Stream Features:** Stream features are created from streaming services such as Kafka or Kinesis, and can be pushed directly into Feast via the [Push API](../../reference/data-sources/push.md).
99
* **Feast Apply:** The user (or CI) publishes versioned controlled feature definitions using `feast apply`. This CLI command updates infrastructure and persists definitions in the object store registry.
10-
* **Feast Materialize:** The user (or scheduler) executes `feast materialize` (with timestamps or `--disable-event-timestamp` to materialize all data with current timestamps) which loads features from the offline store into the online store.
10+
* **Feast Materialize:** The user (or scheduler) executes `feast materialize` (with timestamps or `--disable-event-timestamp` to materialize all available data) which loads features from the offline store into the online store.
1111
* **Model Training:** A model training pipeline is launched. It uses the Feast Python SDK to retrieve a training dataset that can be used for training models.
1212
* **Get Historical Features:** Feast exports a point-in-time correct training dataset based on the list of features and entity dataframe provided by the model training pipeline.
1313
* **Deploy Model:** The trained model binary (and list of features) are deployed into a model serving system. This step is not executed by Feast.

‎docs/getting-started/concepts/data-ingestion.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
7070
feast materialize-incremental $CURRENT_TIME
7171
```
7272

73-
**Simple materialization (for data without event timestamps):**
73+
**Simple materialization (all available data, no start/end timestamps):**
7474
```bash
7575
feast materialize --disable-event-timestamp
7676
```

‎docs/getting-started/quickstart.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -513,7 +513,7 @@ feast materialize-incremental $CURRENT_TIME
513513
{% endtab %}
514514
{% tab title="Bash (simple)" %}
515515
```bash
516-
# Alternative: Materialize all data using current timestamp (for data without event timestamps)
516+
# Alternative: Materialize all available data without specifying start/end timestamps
517517
feast materialize --disable-event-timestamp
518518
```
519519
{% endtab %}

‎docs/reference/feast-cli-commands.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -272,7 +272,7 @@ Load data from feature views into the online store.
272272
feast materialize 2020-01-01T00:00:00 2022-01-01T00:00:00
273273
```
274274

275-
**Without timestamps (uses current datetime):**
275+
**Without timestamps (materializes all available data):**
276276
```bash
277277
feast materialize --disable-event-timestamp
278278
```
@@ -287,7 +287,7 @@ feast materialize -v driver_hourly_stats 2020-01-01T00:00:00 2022-01-01T00:00:00
287287
feast materialize --disable-event-timestamp -v driver_hourly_stats
288288
```
289289

290-
The `--disable-event-timestamp` flag is useful when your source data lacks event timestamp columns, allowing you to materialize all available data using the current datetime as the event timestamp.
290+
The `--disable-event-timestamp` flag lets you materialize all available data (from 1970-01-01 up to the current UTC time) without specifying start and end timestamps. Rows keep the event timestamps from the source.
291291

292292
```text
293293
Materializing 1 feature views from 2020-01-01 to 2022-01-01

‎docs/reference/feature-servers/python-feature-server.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -286,7 +286,7 @@ curl -X POST "http://localhost:6566/materialize" -d '{
286286
}' | jq
287287
```
288288

289-
When `disable_event_timestamp` is set to `true`, the `start_ts` and `end_ts` parameters are not required, and all available data is materialized using the current datetime as the event timestamp. This is useful when your source data lacks proper event timestamp columns.
289+
When `disable_event_timestamp` is set to `true`, the `start_ts` and `end_ts` parameters are not required, and all available data (from 1970-01-01 up to the current UTC time) is materialized. Rows keep the event timestamps from the source.
290290

291291
Or from Python:
292292
```python

‎infra/templates/README.md.jinja2‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ feast materialize 2021-04-12T00:00:00 $CURRENT_TIME
125125
feast materialize --disable-event-timestamp
126126
```
127127

128-
The `--disable-event-timestamp` flag allows you to materialize all available feature data using the current datetime as the event timestamp, without needing to specify start and end timestamps. This is useful when your source data lacks proper event timestamp columns.
128+
The `--disable-event-timestamp` flag allows you to materialize all available data (from 1970-01-01 up to the current UTC time) without specifying start and end timestamps. Rows keep the event timestamps from the source.
129129

130130
```commandline
131131
Materializing feature view driver_hourly_stats from 2021-04-14 to 2021-04-15 done!

‎sdk/python/feast/cli/cli.py‎

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
# limitations under the License.
1414
import json
1515
import logging
16-
from datetime import datetime
16+
from datetime import datetime, timezone
1717
from importlib.metadata import version as importlib_version
1818
from pathlib import Path
1919
from typing import List, Optional
@@ -363,7 +363,7 @@ def registry_dump_command(ctx: click.Context):
363363
@click.option(
364364
"--disable-event-timestamp",
365365
is_flag=True,
366-
help="Materialize all available data using current datetime as event timestamp (useful when source data lacks event timestamps)",
366+
help="Materialize all available data, from 1970-01-01 up to the current UTC time, without specifying START_TS and END_TS (rows keep their source event timestamps)",
367367
)
368368
@click.option(
369369
"--version",
@@ -388,7 +388,7 @@ def materialize_command(
388388
389389
START_TS and END_TS should be in ISO 8601 format, e.g. '2021-07-16T19:20:01'
390390
391-
If --disable-event-timestamp is used, timestamps are not required and all available data will be materialized using the current datetime as the event timestamp.
391+
If --disable-event-timestamp is used, timestamps are not required and all available data (from 1970-01-01 up to the current UTC time) will be materialized. Rows keep their source event timestamps.
392392
"""
393393
store = create_feature_store(ctx)
394394

@@ -397,12 +397,9 @@ def materialize_command(
397397
raise click.UsageError(
398398
"Cannot specify START_TS or END_TS when --disable-event-timestamp is used"
399399
)
400-
now = datetime.now()
401-
# Query all available data and use current datetime as event timestamp
402-
start_date = datetime(
403-
1970, 1, 1
404-
) # Beginning of time to capture all historical data
405-
end_date = now
400+
# Query all available data, from the beginning of time up to now (UTC)
401+
start_date = datetime(1970, 1, 1, tzinfo=timezone.utc)
402+
end_date = utils._utc_now()
406403
else:
407404
if not start_ts or not end_ts:
408405
raise click.UsageError(

‎sdk/python/feast/feature_server.py‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222
from collections import defaultdict
2323
from concurrent.futures import ThreadPoolExecutor
2424
from contextlib import asynccontextmanager
25-
from datetime import datetime
25+
from datetime import datetime, timezone
2626
from importlib import resources as importlib_resources
2727
from types import SimpleNamespace
2828
from typing import Any, DefaultDict, Dict, List, NamedTuple, Optional, Set, Union
@@ -478,8 +478,7 @@ def _parse_materialize_timestamps(
478478
) -> tuple:
479479
"""Parse and validate start/end timestamps from a MaterializeRequest."""
480480
if request.disable_event_timestamp:
481-
now = datetime.now()
482-
return datetime(1970, 1, 1), now
481+
return datetime(1970, 1, 1, tzinfo=timezone.utc), utils._utc_now()
483482

484483
if not request.start_ts or not request.end_ts:
485484
raise ValueError(

0 commit comments

Comments
 (0)