Skip to content

Commit e1c6870

Browse files
Zhuoxi2000Zhuoxi2000
committed
Merge branch 'master' into fix-rest-nonfinite-500
Signed-off-by: Edson <[email protected]>
2 parents 95890f7 + acadb17 commit e1c6870

64 files changed

Lines changed: 4881 additions & 62 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/pr_chronon_integration_tests.yml‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
name: pr-chronon-integration-tests
2+
3+
on:
4+
pull_request:
5+
types:
6+
- opened
7+
- synchronize
8+
- labeled
9+
paths:
10+
- "sdk/python/feast/infra/online_stores/chronon_online_store/**"
11+
- "sdk/python/feast/infra/offline_stores/contrib/chronon_offline_store/**"
12+
- "sdk/python/feast/infra/chronon_provider.py"
13+
- "sdk/python/tests/unit/infra/online_stores/chronon_online_store/**"
14+
- "sdk/python/tests/unit/infra/offline_stores/contrib/chronon_offline_store/**"
15+
- "sdk/python/tests/unit/examples/test_chronon_demo.py"
16+
- "sdk/python/tests/unit/infra/scripts/test_chronon_local_service_script.py"
17+
- "examples/chronon/**"
18+
- "sdk/python/tests/integration/online_store/test_chronon_online_store.py"
19+
- "sdk/python/tests/integration/online_store/test_chronon_online_store_real_service.py"
20+
- "sdk/python/tests/integration/offline_store/test_chronon_offline_store.py"
21+
- "infra/scripts/chronon/**"
22+
- "chronon/**"
23+
- ".github/workflows/pr_chronon_integration_tests.yml"
24+
25+
concurrency:
26+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
27+
cancel-in-progress: true
28+
29+
jobs:
30+
chronon-python-tests:
31+
if:
32+
((github.event.action == 'labeled' && (github.event.label.name == 'approved' || github.event.label.name == 'lgtm' || github.event.label.name == 'ok-to-test')) ||
33+
(github.event.action != 'labeled' && (contains(github.event.pull_request.labels.*.name, 'ok-to-test') || contains(github.event.pull_request.labels.*.name, 'approved') || contains(github.event.pull_request.labels.*.name, 'lgtm')))) &&
34+
github.event.pull_request.base.repo.full_name == 'feast-dev/feast'
35+
# The Chronon quickstart image is published for ARM64 only.
36+
runs-on: ubuntu-24.04-arm
37+
steps:
38+
- uses: actions/checkout@v4
39+
with:
40+
repository: ${{ github.event.repository.full_name }}
41+
ref: ${{ github.ref }}
42+
token: ${{ secrets.GITHUB_TOKEN }}
43+
- name: Setup Python
44+
uses: actions/setup-python@v5
45+
with:
46+
python-version: "3.11"
47+
architecture: arm64
48+
- name: Install uv
49+
uses: astral-sh/setup-uv@v5
50+
with:
51+
enable-cache: true
52+
- name: Setup Java
53+
uses: actions/setup-java@v4
54+
with:
55+
distribution: temurin
56+
java-version: "8"
57+
- name: Setup sbt
58+
uses: sbt/setup-sbt@v1
59+
- name: Install dependencies
60+
run: make install-python-dependencies-ci
61+
- name: Run Chronon unit tests
62+
run: |
63+
uv run pytest -c sdk/python/pytest.ini \
64+
sdk/python/tests/unit/infra/online_stores/chronon_online_store \
65+
sdk/python/tests/unit/infra/offline_stores/contrib/chronon_offline_store \
66+
sdk/python/tests/unit/examples/test_chronon_demo.py \
67+
sdk/python/tests/unit/infra/scripts/test_chronon_local_service_script.py
68+
- name: Run Chronon stub integration tests
69+
run: |
70+
uv run pytest -c sdk/python/pytest.ini \
71+
sdk/python/tests/integration/online_store/test_chronon_online_store.py \
72+
sdk/python/tests/integration/offline_store/test_chronon_offline_store.py \
73+
--integration
74+
- name: Checkout Chronon
75+
env:
76+
CHRONON_REPO: ${{ github.workspace }}/.chronon
77+
CHRONON_REPO_URL: https://github.com/airbnb/chronon.git
78+
CHRONON_REF: 6c0b8de9f0301521baf61a46ff3083c566fb4052 # pragma: allowlist secret
79+
run: |
80+
git clone "${CHRONON_REPO_URL}" "${CHRONON_REPO}"
81+
git -C "${CHRONON_REPO}" checkout "${CHRONON_REF}"
82+
- name: Install Chronon build dependencies
83+
run: |
84+
sudo apt-get update
85+
sudo apt-get install -y \
86+
apt-transport-https \
87+
autoconf \
88+
automake \
89+
bison \
90+
build-essential \
91+
ca-certificates \
92+
curl \
93+
flex \
94+
g++ \
95+
gnupg \
96+
libtool \
97+
pkg-config \
98+
python3-pip \
99+
python3-venv
100+
curl -fsSL --retry 3 --connect-timeout 15 \
101+
"https://github.com/apache/thrift/archive/refs/tags/v0.13.0.tar.gz" \
102+
-o /tmp/thrift.tar.gz
103+
sudo rm -rf /usr/src/thrift
104+
sudo mkdir -p /usr/src/thrift
105+
sudo tar zxf /tmp/thrift.tar.gz -C /usr/src/thrift --strip-components=1
106+
cd /usr/src/thrift
107+
sudo ./bootstrap.sh
108+
sudo ./configure --disable-libs --disable-tests --disable-tutorial
109+
sudo make -j2
110+
sudo make install
111+
python3 -m pip install --break-system-packages build
112+
thrift -version
113+
- name: Build Chronon quickstart and service jars
114+
env:
115+
CHRONON_REPO: ${{ github.workspace }}/.chronon
116+
run: |
117+
cd "${CHRONON_REPO}/quickstart/mongo-online-impl"
118+
sbt assembly
119+
cd "${CHRONON_REPO}"
120+
sbt "project service" assembly
121+
- name: Start live Chronon service
122+
env:
123+
CHRONON_REPO: ${{ github.workspace }}/.chronon
124+
run: infra/scripts/chronon/start-local-chronon-service.sh
125+
- name: Run Chronon live-service integration test
126+
env:
127+
CHRONON_SERVICE_URL: http://127.0.0.1:9000
128+
run: |
129+
uv run pytest -c sdk/python/pytest.ini \
130+
sdk/python/tests/integration/online_store/test_chronon_online_store_real_service.py \
131+
--integration
132+
- name: Stop live Chronon service
133+
if: always()
134+
run: infra/scripts/chronon/stop-local-chronon-service.sh

‎.secrets.baseline‎

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎AGENTS.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,9 @@ Architecture & design intent: `docs/getting-started/architecture/` (overview, wr
9292

9393
- Use type hints on all Python function signatures
9494
- Follow existing patterns in the module you are modifying
95+
- Do not put `pip install` commands in docstrings or code comments. Security scanners flag them as runtime
96+
package installs, which fail in air-gapped or mirror-only environments. Name the extra instead
97+
(e.g. "Requires the ``glide`` extra (``feast[glide]``).") and keep install commands in `docs/`.
9598
- PR titles must follow conventional commit conventions with a lowercase type and a capitalized subject after the colon: `feat: Add ...`, `fix: Correct ...`, `ci: Update ...`, `chore: Refresh ...`, `docs: Add ...`
9699
- Sign off commits with `git commit -s` (DCO requirement)
97100
- Uses `ruff` for Python linting and formatting; Go uses standard `gofmt`

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,7 @@ lock-python-dependencies-all: ## Recompile and lock all Python dependency sets f
138138
"uv pip compile -p $(ver) --universal --no-strip-extras pyproject.toml --extra ci \
139139
--torch-backend cpu \
140140
--generate-hashes --output-file sdk/python/requirements/py$(ver)-ci-requirements.txt" && \
141+
python3 infra/scripts/add_cpu_torch_hashes.py --python-version $(ver) sdk/python/requirements/py$(ver)-ci-requirements.txt && \
141142
pixi run --environment $(call get_env_name,$(ver)) --manifest-path infra/scripts/pixi/pixi.toml \
142143
"uv pip compile -p $(ver) --no-strip-extras pyproject.toml \
143144
--generate-hashes --output-file sdk/python/requirements/py$(ver)-requirements.txt" && \

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,7 @@ The list below contains the functionality that contributors are planning to deve
173173
* [x] Vector Search (Alpha release. See [RFC](https://docs.google.com/document/d/18IWzLEA9i2lDWnbfbwXnMCg3StlqaLVI-uRpQjr_Vos/edit#heading=h.9gaqqtox9jg6))
174174
* [ ] [Enhanced Feature Server and SDK for native support for NLP](https://github.com/feast-dev/feast/issues/4964)
175175
* **Data Sources**
176+
* [x] [Chronon source (contrib)](https://docs.feast.dev/reference/data-sources/chronon)
176177
* [x] [Snowflake source](https://docs.feast.dev/reference/data-sources/snowflake)
177178
* [x] [Redshift source](https://docs.feast.dev/reference/data-sources/redshift)
178179
* [x] [BigQuery source](https://docs.feast.dev/reference/data-sources/bigquery)
@@ -189,6 +190,7 @@ The list below contains the functionality that contributors are planning to deve
189190
* [x] [Ray source (contrib plugin)](https://docs.feast.dev/reference/data-sources/ray)
190191
* [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push))
191192
* **Offline Stores**
193+
* [x] [Chronon (contrib)](https://docs.feast.dev/reference/offline-stores/chronon)
192194
* [x] [Snowflake](https://docs.feast.dev/reference/offline-stores/snowflake)
193195
* [x] [Redshift](https://docs.feast.dev/reference/offline-stores/redshift)
194196
* [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery)
@@ -209,6 +211,7 @@ The list below contains the functionality that contributors are planning to deve
209211
* [x] [Hybrid](https://docs.feast.dev/reference/offline-stores/hybrid)
210212
* [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/customizing-feast/adding-a-new-offline-store)
211213
* **Online Stores**
214+
* [x] [Chronon](https://docs.feast.dev/reference/online-stores/chronon)
212215
* [x] [Snowflake](https://docs.feast.dev/reference/online-stores/snowflake)
213216
* [x] [DynamoDB](https://docs.feast.dev/reference/online-stores/dynamodb)
214217
* [x] [Redis](https://docs.feast.dev/reference/online-stores/redis)

‎docs/SUMMARY.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,7 @@
126126
* [Clickhouse (contrib)](reference/data-sources/clickhouse.md)
127127
* [Ray (contrib)](reference/data-sources/ray.md)
128128
* [MongoDB (contrib)](reference/data-sources/mongodb.md)
129+
* [Chronon (contrib)](reference/data-sources/chronon.md)
129130
* [Offline stores](reference/offline-stores/README.md)
130131
* [Overview](reference/offline-stores/overview.md)
131132
* [Dask](reference/offline-stores/dask.md)
@@ -143,6 +144,7 @@
143144
* [Oracle (contrib)](reference/offline-stores/oracle.md)
144145
* [Athena (contrib)](reference/offline-stores/athena.md)
145146
* [MongoDB (contrib)](reference/offline-stores/mongodb.md)
147+
* [Chronon (contrib)](reference/offline-stores/chronon.md)
146148
* [Remote Offline](reference/offline-stores/remote-offline-store.md)
147149
* [Hybrid](reference/offline-stores/hybrid.md)
148150
* [Online stores](reference/online-stores/README.md)
@@ -166,6 +168,7 @@
166168
* [SingleStore](reference/online-stores/singlestore.md)
167169
* [Milvus](reference/online-stores/milvus.md)
168170
* [MongoDB](reference/online-stores/mongodb.md)
171+
* [Chronon (contrib)](reference/online-stores/chronon.md)
169172
* [Aerospike](reference/online-stores/aerospike.md)
170173
* [Elasticsearch](reference/online-stores/elasticsearch.md)
171174
* [Qdrant](reference/online-stores/qdrant.md)

‎docs/reference/data-sources/chronon.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Chronon source (contrib)
2+
3+
## Description
4+
5+
Chronon sources describe feature data produced by [Chronon](https://chronon.ai/) and consumed by Feast.
6+
They point Feast at Chronon's offline materialization output and, when online reads are needed, identify the Chronon Join or GroupBy that should be queried from Chronon's online service.
7+
8+
Feast does not compute or materialize Chronon features. Chronon owns feature computation, backfills, consistency, and online serving. Feast uses the source metadata for registry, discovery, historical retrieval, and online lookup.
9+
10+
## Examples
11+
12+
Defining a Chronon source for a Chronon Join:
13+
14+
```python
15+
from feast.infra.offline_stores.contrib.chronon_offline_store.chronon_source import (
16+
ChrononSource,
17+
)
18+
19+
driver_stats_source = ChrononSource(
20+
name="driver_stats",
21+
materialization_path="data/chronon/driver_stats",
22+
chronon_join="team/driver_stats.v1",
23+
online_endpoint="http://localhost:8080",
24+
timestamp_field="event_timestamp",
25+
created_timestamp_column="created_timestamp",
26+
)
27+
```
28+
29+
Defining a Chronon source for a Chronon GroupBy:
30+
31+
```python
32+
driver_profile_source = ChrononSource(
33+
name="driver_profile",
34+
materialization_path="data/chronon/driver_profile",
35+
chronon_group_by="team/driver_profile.v1",
36+
timestamp_field="event_timestamp",
37+
)
38+
```
39+
40+
## Configuration reference
41+
42+
| Parameter | Required | Description |
43+
| :------------------------- | :------- | :---------- |
44+
| `materialization_path` | yes | Local or repository-relative path to Chronon's Parquet materialization output. |
45+
| `chronon_join` | no | Chronon Join name used for online reads, for example `team/training_set.v1`. |
46+
| `chronon_group_by` | no | Chronon GroupBy name used for online reads, for example `team/user_features.v1`. |
47+
| `online_endpoint` | no | Chronon online service base URL for this source. If omitted, Feast uses the Chronon online store `path`. |
48+
| `timestamp_field` | yes | Event timestamp column in the materialized Chronon data. |
49+
| `created_timestamp_column` | no | Optional created timestamp column used to select the latest row when duplicate event timestamps exist. |
50+
| `field_mapping` | no | Standard Feast field mapping applied before retrieval. |
51+
52+
Set at most one of `chronon_join` and `chronon_group_by`. Offline-only sources may omit both, but `online_endpoint` requires one of them so Feast can build the Chronon request URL.
53+
54+
## Supported Types
55+
56+
Chronon sources read Parquet data through PyArrow and use Feast's standard PyArrow type mapping.
57+
For a comparison against other batch data sources, please see [here](overview.md#functionality-matrix).

‎docs/reference/offline-stores/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@ Please see [Offline Store](../../getting-started/components/offline-store.md) fo
66
[overview.md](overview.md)
77
{% endcontent-ref %}
88

9+
{% content-ref url="chronon.md" %}
10+
[chronon.md](chronon.md)
11+
{% endcontent-ref %}
12+
913
{% content-ref url="dask.md" %}
1014
[dask.md](dask.md)
1115
{% endcontent-ref %}

0 commit comments

Comments
 (0)