Skip to content

Latest commit

 

History

History
121 lines (92 loc) · 6.01 KB

chronon.md

File metadata and controls

121 lines (92 loc) · 6.01 KB

Chronon offline store (contrib)

Description

The Chronon offline store provides support for reading ChrononSources from Chronon's Parquet materialization output.

Chronon remains the system of record for feature computation and materialization. Feast reads Chronon-produced data for historical retrieval, feature reuse, and registry-driven training workflows.

Getting started

Chronon-backed feature repos use the Chronon offline store with Chronon sources:

{% code title="feature_store.yaml" %}

project: my_project
registry: data/registry.db
provider: chronon
offline_store:
  type: chronon
online_store:
  type: chronon
  path: http://localhost:8080

{% endcode %}

Example feature view:

from feast import Entity, FeatureView, Field
from feast.infra.offline_stores.contrib.chronon_offline_store.chronon_source import (
    ChrononSource,
)
from feast.types import Float32

driver = Entity(name="driver", join_keys=["driver_id"])

driver_stats = FeatureView(
    name="driver_stats",
    entities=[driver],
    schema=[Field(name="rating", dtype=Float32)],
    source=ChrononSource(
        name="driver_stats_source",
        materialization_path="data/chronon/driver_stats",
        chronon_join="team/driver_stats.v1",
        timestamp_field="event_timestamp",
        created_timestamp_column="created_timestamp",
    ),
)

Historical retrieval

get_historical_features performs point-in-time joins against Chronon's materialized Parquet data. For each entity row, Feast selects the latest Chronon row with an event timestamp at or before the entity timestamp. If created_timestamp_column is configured and duplicate event timestamps exist, the latest created row wins.

Reads select only the required feature, entity-key, event-time, and (when needed) created-time columns from Parquet, translating field mappings to physical column names before loading. Joins still execute locally in pandas; column selection reduces I/O and memory use but does not provide distributed execution.

Python on-demand feature views can transform the retrieved features locally through Feast's standard retrieval job. Put required request-time inputs in the pandas entity_df. to_df(), to_arrow(), and saved datasets include the requested transformed outputs. Time-range retrieval without an entity_df cannot supply request-time inputs.

This is intended for training and validation workflows that want Feast's registry and retrieval APIs while using Chronon as the feature computation engine.

Save retrieval results

Use provider: chronon and SavedDatasetFileStorage to persist a retrieval result as Parquet and load it again through Feast:

from feast.infra.offline_stores.file_source import SavedDatasetFileStorage

job = store.get_historical_features(
    entity_df=entity_df,
    features=["driver_stats:rating"],
)
training = store.create_saved_dataset(
    from_=job,
    name="training_v1",
    storage=SavedDatasetFileStorage(path="data/training_v1.parquet"),
)
training_df = store.get_saved_dataset("training_v1").to_df()

Relative paths resolve against the feature repository. File storage can also use a Parquet directory or supported PyArrow filesystem URI, including S3 with the existing file-storage credentials/endpoint configuration. Existing destinations are rejected unless allow_overwrite=True; overwriting a directory replaces its contents. Use a dedicated output location. Writes are synchronous and are not an atomic publication mechanism.

The saved result retains request columns, custom entity timestamp names, full feature names, and already-computed on-demand features. This saves training results; it does not write feature values into Chronon's computation or online storage.

Configuration reference

Parameter Required Default Description
type yes - Must be set to chronon.
path no - Reserved for future offline store configuration. Source-level materialization_path controls where data is read from.

Functionality Matrix

The set of functionality supported by offline stores is described in detail here. Below is a matrix indicating which functionality is supported by the Chronon offline store.

Chronon
get_historical_features (point-in-time correct join) yes
pull_latest_from_table_or_query (retrieve latest feature values) yes
pull_all_from_table_or_query (retrieve a saved dataset) yes
offline_write_batch (persist dataframes to offline store) no
write_logged_features (persist logged features to offline store) no

Below is a matrix indicating which functionality is supported by ChrononRetrievalJob.

Chronon
export to dataframe yes
export to arrow table yes
export to arrow batches no
export to SQL no
export to data lake (S3, GCS, etc.) no
export to data warehouse no
export as Spark dataframe no
local execution of Python-based on-demand transforms yes
remote execution of Python-based on-demand transforms no
persist results in the offline store yes (Parquet saved datasets)
preview the query plan before execution no
read partitioned data yes

To compare this set of functionality against other offline stores, please see the full functionality matrix.