Skip to content

Commit 7a84f2a

Browse files
authored
feat: Add Pinecone online store integration (feast-dev#6585)
* feat: Add Pinecone online store integration Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Align Pinecone store with filters API and operator types Signed-off-by: Siddhesh Khairnar <[email protected]> * test: Expand Pinecone online store unit test coverage Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Address Pinecone online store review feedback Signed-off-by: Siddhesh Khairnar <[email protected]> * ci: Retrigger CI after flaky macOS dependency download Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Resolve rebase conflicts keeping chronon and pinecone Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Prevent protobuf downgrade breaking grpc health checks Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Revert protobuf pin that broke flink and pixi lock Keep uv --no-sync for CI so hashed protobuf 7.x is preserved without conflicting with feast[flink]. Signed-off-by: Siddhesh Khairnar <[email protected]> --------- Signed-off-by: Siddhesh Khairnar <[email protected]>
1 parent b4444cc commit 7a84f2a

20 files changed

Lines changed: 2003 additions & 13 deletions

File tree

‎.github/workflows/unit_tests.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,9 @@ jobs:
7171
export PATH="${{ github.workspace }}/.venv/bin:$PATH"
7272
7373
# Dynamically detect site-packages for uv env
74-
SITE_PACKAGES=$(uv run python -c "import site; print(site.getsitepackages()[0])")
74+
# --no-sync: env was already installed via hashed uv pip sync; a fresh
75+
# uv sync can downgrade protobuf and break grpcio-health-checking.
76+
SITE_PACKAGES=$(uv run --no-sync python -c "import site; print(site.getsitepackages()[0])")
7577
7678
# Preserve any existing PYTHONPATH and add repo + site-packages
7779
export PYTHONPATH="${{ github.workspace }}/sdk/python:${PYTHONPATH}:$SITE_PACKAGES"

‎.secrets.baseline‎

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Makefile‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -169,7 +169,8 @@ benchmark-python-local: ## Run integration + benchmark tests for Python (local d
169169
##@ Tests
170170

171171
test-python-unit: ## Run Python unit tests (use pattern=<pattern> to filter tests, e.g., pattern=milvus, pattern=test_online_retrieval.py, pattern=test_online_retrieval.py::test_get_online_features_milvus)
172-
uv run python -m pytest -n 8 --color=yes $(if $(pattern),-k "$(pattern)") \
172+
# --no-sync: CI installs via hashed uv pip sync first; re-sync can downgrade protobuf.
173+
uv run --no-sync python -m pytest -n 8 --color=yes $(if $(pattern),-k "$(pattern)") \
173174
--cov=feast \
174175
--cov-report=xml \
175176
--cov-report=term-missing \
@@ -199,7 +200,7 @@ test-python-smoke: ## Quick smoke test for development
199200
-n 4 --tb=short
200201

201202
test-python-integration: ## Run Python integration tests (CI)
202-
uv run python -m pytest --tb=short -v -n 8 --integration --color=yes --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
203+
uv run --no-sync python -m pytest --tb=short -v -n 8 --integration --color=yes --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
203204
-k "not snowflake" \
204205
-m "not rbac_remote_integration_test and not ray_offline_stores_only" \
205206
--ignore=sdk/python/tests/integration/registration \
@@ -217,7 +218,7 @@ test-python-integration-parallel: ## Run integration tests with enhanced paralle
217218
test-python-integration-local: ## Run Python integration tests (local dev mode)
218219
FEAST_IS_LOCAL_TEST=True \
219220
FEAST_LOCAL_ONLINE_CONTAINER=True \
220-
uv run python -m pytest --tb=short -v -n auto --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
221+
uv run --no-sync python -m pytest --tb=short -v -n auto --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
221222
-k "not test_lambda_materialization and not test_snowflake_materialization" \
222223
-m "not rbac_remote_integration_test and not ray_offline_stores_only" \
223224
--ignore=sdk/python/tests/component/ray \
@@ -228,7 +229,7 @@ test-python-integration-local: ## Run Python integration tests (local dev mode)
228229
test-python-integration-rbac-remote: ## Run Python remote RBAC integration tests
229230
FEAST_IS_LOCAL_TEST=True \
230231
FEAST_LOCAL_ONLINE_CONTAINER=True \
231-
uv run python -m pytest --tb=short -v -n 8 --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
232+
uv run --no-sync python -m pytest --tb=short -v -n 8 --color=yes --integration --durations=10 --timeout=1200 --timeout_method=thread --dist loadgroup \
232233
-k "not test_lambda_materialization and not test_snowflake_materialization" \
233234
-m "rbac_remote_integration_test and not ray_offline_stores_only" \
234235
--log-cli-level=INFO -s \

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,6 +173,7 @@
173173
* [Aerospike](reference/online-stores/aerospike.md)
174174
* [Elasticsearch](reference/online-stores/elasticsearch.md)
175175
* [Qdrant](reference/online-stores/qdrant.md)
176+
* [Pinecone](reference/online-stores/pinecone.md)
176177
* [Faiss](reference/online-stores/faiss.md)
177178
* [Hybrid](reference/online-stores/hybrid.md)
178179
* [Registries](reference/registries/README.md)

‎docs/reference/alpha-vector-database.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Below are supported vector databases and implemented features:
1616
| SQLite | [x] | [ ] | [x] | [x] |
1717
| Qdrant | [x] | [x] | [] | [] |
1818
| ScyllaDB | [x] | [x] | [x] | [x] |
19+
| Pinecone | [x] | [x] | [x] | [x] |
1920

2021
*Note: V2 Support means the SDK supports retrieval of features along with vector embeddings from vector similarity search.
2122

@@ -424,7 +425,7 @@ print('\n'.join([c.message.content for c in response.choices]))
424425

425426
### Configuration and Installation
426427

427-
We offer [Milvus](https://milvus.io/), [PGVector](https://github.com/pgvector/pgvector), [SQLite](https://github.com/asg017/sqlite-vec), [Elasticsearch](https://www.elastic.co) and [Qdrant](https://qdrant.tech/) as Online Store options for Vector Databases.
428+
We offer [Milvus](https://milvus.io/), [PGVector](https://github.com/pgvector/pgvector), [SQLite](https://github.com/asg017/sqlite-vec), [Elasticsearch](https://www.elastic.co), [Qdrant](https://qdrant.tech/) and [Pinecone](https://www.pinecone.io/) as Online Store options for Vector Databases.
428429

429430
Milvus offers a convenient local implementation for vector similarity search. To use Milvus, you can install the Feast package with the Milvus extra.
430431

@@ -444,6 +445,12 @@ pip install feast[elasticsearch]
444445
```bash
445446
pip install feast[qdrant]
446447
```
448+
#### Installation with Pinecone
449+
450+
```bash
451+
pip install feast[pinecone]
452+
```
453+
447454
#### Installation with SQLite
448455

449456
If you are using `pyenv` to manage your Python versions, you can install the SQLite extension with the following command:

‎docs/reference/online-stores/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,10 @@ Please see [Online Store](../../getting-started/components/online-store.md) for
9898
[milvus.md](milvus.md)
9999
{% endcontent-ref %}
100100

101+
{% content-ref url="pinecone.md" %}
102+
[pinecone.md](pinecone.md)
103+
{% endcontent-ref %}
104+
101105
{% content-ref url="faiss.md" %}
102106
[faiss.md](faiss.md)
103107
{% endcontent-ref %}
Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Pinecone online store
2+
3+
## Description
4+
5+
The [Pinecone](https://www.pinecone.io/) online store provides support for materializing feature values into a Pinecone vector database. It is particularly suited for AI workloads that require both low-latency feature serving and vector similarity search for RAG (Retrieval-Augmented Generation) applications.
6+
7+
## Getting started
8+
9+
In order to use this online store, you'll need to install the Pinecone extra (along with the dependency needed for the offline store of choice). E.g.
10+
11+
`pip install 'feast[pinecone]'`
12+
13+
You will also need a Pinecone account and API key. Set your API key via the `PINECONE_API_KEY` environment variable or directly in `feature_store.yaml`.
14+
15+
## Examples
16+
17+
{% code title="feature_store.yaml" %}
18+
```yaml
19+
project: my_feature_repo
20+
registry: data/registry.db
21+
provider: local
22+
online_store:
23+
type: pinecone
24+
api_key: ${PINECONE_API_KEY}
25+
index_name: feast-online
26+
embedding_dim: 384
27+
metric: cosine
28+
cloud: aws
29+
region: us-east-1
30+
vector_enabled: true
31+
```
32+
{% endcode %}
33+
34+
### Using a custom namespace
35+
36+
By default, Feast maps each feature view to a Pinecone namespace (`{project}_{feature_view_name}`). You can override this with a fixed namespace:
37+
38+
{% code title="feature_store.yaml" %}
39+
```yaml
40+
online_store:
41+
type: pinecone
42+
api_key: ${PINECONE_API_KEY}
43+
index_name: feast-online
44+
namespace: my-custom-namespace
45+
embedding_dim: 384
46+
```
47+
{% endcode %}
48+
49+
### Vector similarity search
50+
51+
Pinecone supports the `retrieve_online_documents_v2` API for vector similarity search:
52+
53+
```python
54+
from feast import FeatureStore
55+
56+
store = FeatureStore(repo_path=".")
57+
58+
results = store.retrieve_online_documents_v2(
59+
features=[
60+
"city_embeddings:vector",
61+
"city_embeddings:sentence_chunks",
62+
],
63+
query=query_embedding,
64+
top_k=5,
65+
distance_metric="cosine",
66+
).to_df()
67+
```
68+
69+
The full set of configuration options is available in [PineconeOnlineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.online_stores.pinecone_online_store.pinecone.PineconeOnlineStoreConfig).
70+
71+
## Configuration options
72+
73+
| Option | Type | Default | Description |
74+
|:---|:---|:---|:---|
75+
| `type` | string | `"pinecone"` | Must be `pinecone` |
76+
| `api_key` | string | `None` | Pinecone API key (falls back to `PINECONE_API_KEY` env var) |
77+
| `index_name` | string | `"feast-online"` | Name of the Pinecone index |
78+
| `namespace` | string | `None` | Override namespace (default: `{project}_{fv_name}`) |
79+
| `embedding_dim` | int | `128` | Vector dimension |
80+
| `metric` | string | `"cosine"` | Distance metric: `cosine`, `euclidean`, or `dotproduct` |
81+
| `cloud` | string | `"aws"` | Cloud provider for serverless indexes |
82+
| `region` | string | `"us-east-1"` | Cloud region for serverless indexes |
83+
| `vector_enabled` | bool | `true` | Enable vector similarity search |
84+
| `batch_size` | int | `100` | Number of vectors per upsert batch |
85+
86+
## Functionality Matrix
87+
88+
The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
89+
Below is a matrix indicating which functionality is supported by the Pinecone online store.
90+
91+
| | Pinecone |
92+
|:----------------------------------------------------------|:---------|
93+
| write feature values to the online store | yes |
94+
| read feature values from the online store | yes |
95+
| update infrastructure (e.g. tables) in the online store | yes |
96+
| teardown infrastructure (e.g. tables) in the online store | yes |
97+
| generate a plan of infrastructure changes | no |
98+
| support for on-demand transforms | yes |
99+
| readable by Python SDK | yes |
100+
| readable by Java | no |
101+
| readable by Go | no |
102+
| support for entityless feature views | yes |
103+
| support for concurrent writing to the same key | yes |
104+
| support for ttl (time to live) at retrieval | no |
105+
| support for deleting expired data | no |
106+
| collocated by feature view | no |
107+
| collocated by feature service | no |
108+
| collocated by entity key | no |
109+
| vector similarity search | yes |

‎infra/feast-operator/api/v1/featurestore_types.go‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -785,7 +785,7 @@ type OnlineStoreFilePersistence struct {
785785
// OnlineStoreDBStorePersistence configures the DB store persistence for the online store service
786786
type OnlineStoreDBStorePersistence struct {
787787
// Type of the persistence type you want to use.
788-
// +kubebuilder:validation:Enum=snowflake.online;redis;datastore;dynamodb;bigtable;postgres;cassandra;mysql;hazelcast;singlestore;hbase;elasticsearch;qdrant;couchbase.online;milvus;hybrid;mongodb;aerospike;scylladb;chronon
788+
// +kubebuilder:validation:Enum=snowflake.online;redis;datastore;dynamodb;bigtable;postgres;cassandra;mysql;hazelcast;singlestore;hbase;elasticsearch;qdrant;couchbase.online;milvus;hybrid;mongodb;aerospike;scylladb;chronon;pinecone
789789
Type string `json:"type"`
790790
// Data store parameters should be placed as-is from the "feature_store.yaml" under the secret key. "registry_type" & "type" fields should be removed.
791791
SecretRef corev1.LocalObjectReference `json:"secretRef"`
@@ -814,6 +814,7 @@ var ValidOnlineStoreDBStorePersistenceTypes = []string{
814814
"aerospike",
815815
"scylladb",
816816
"chronon",
817+
"pinecone",
817818
}
818819

819820
// LocalRegistryConfig configures the registry service

‎infra/feast-operator/api/v1alpha1/featurestore_types.go‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -396,7 +396,7 @@ type OnlineStoreFilePersistence struct {
396396
// OnlineStoreDBStorePersistence configures the DB store persistence for the online store service
397397
type OnlineStoreDBStorePersistence struct {
398398
// Type of the persistence type you want to use.
399-
// +kubebuilder:validation:Enum=snowflake.online;redis;datastore;dynamodb;bigtable;postgres;cassandra;mysql;hazelcast;singlestore;hbase;elasticsearch;qdrant;couchbase.online;milvus;hybrid;mongodb;aerospike;scylladb;chronon
399+
// +kubebuilder:validation:Enum=snowflake.online;redis;datastore;dynamodb;bigtable;postgres;cassandra;mysql;hazelcast;singlestore;hbase;elasticsearch;qdrant;couchbase.online;milvus;hybrid;mongodb;aerospike;scylladb;chronon;pinecone
400400
Type string `json:"type"`
401401
// Data store parameters should be placed as-is from the "feature_store.yaml" under the secret key. "registry_type" & "type" fields should be removed.
402402
SecretRef corev1.LocalObjectReference `json:"secretRef"`
@@ -425,6 +425,7 @@ var ValidOnlineStoreDBStorePersistenceTypes = []string{
425425
"aerospike",
426426
"scylladb",
427427
"chronon",
428+
"pinecone",
428429
}
429430

430431
// LocalRegistryConfig configures the registry service

‎infra/feast-operator/bundle/manifests/feast.dev_featurestores.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3379,6 +3379,7 @@ spec:
33793379
- aerospike
33803380
- scylladb
33813381
- chronon
3382+
- pinecone
33823383
type: string
33833384
required:
33843385
- secretRef
@@ -10727,6 +10728,7 @@ spec:
1072710728
- aerospike
1072810729
- scylladb
1072910730
- chronon
10731+
- pinecone
1073010732
type: string
1073110733
required:
1073210734
- secretRef
@@ -16345,6 +16347,7 @@ spec:
1634516347
- aerospike
1634616348
- scylladb
1634716349
- chronon
16350+
- pinecone
1634816351
type: string
1634916352
required:
1635016353
- secretRef
@@ -20923,6 +20926,7 @@ spec:
2092320926
- aerospike
2092420927
- scylladb
2092520928
- chronon
20929+
- pinecone
2092620930
type: string
2092720931
required:
2092820932
- secretRef

0 commit comments

Comments
 (0)