Repository navigation
feat: Add Pinecone online store integration - #6585
Conversation
804ff38 to
020a0e6
Compare
a9607d7 to
0fa28bc
Compare
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6585 +/- ##
==========================================
+ Coverage 51.01% 51.27% +0.25%
==========================================
Files 455 458 +3
Lines 57015 57409 +394
Branches 8327 8415 +88
==========================================
+ Hits 29089 29436 +347
- Misses 25939 25965 +26
- Partials 1987 2008 +21
... and 2 files with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
5db79c2 to
134f76d
Compare
jyejare
left a comment
There was a problem hiding this comment.
This PR adds a comprehensive Pinecone online store integration for Feast, including vector similarity search capabilities. The implementation is well-structured with proper error handling, comprehensive tests, and good documentation. However, there are several areas that need improvement including error handling gaps, potential security concerns, and some code quality issues that should be addressed.
|
Thanks for the thorough review, @jyejare. I’ve addressed all of the comments in the latest commits (softened the API-key error, upsert/query error handling, metadata size checks, returning match vector values, |
|
Hey @SIDDHESH1564 , could you please rebase and resolve conflicts. |
Signed-off-by: Siddhesh Khairnar <[email protected]>
Signed-off-by: Siddhesh Khairnar <[email protected]>
Signed-off-by: Siddhesh Khairnar <[email protected]>
Signed-off-by: Siddhesh Khairnar <[email protected]>
Signed-off-by: Siddhesh Khairnar <[email protected]>
Signed-off-by: Siddhesh Khairnar <[email protected]>
aff2085 to
3dfadb2
Compare
Signed-off-by: Siddhesh Khairnar <[email protected]>
c0487cc to
3b87daf
Compare
Keep uv --no-sync for CI so hashed protobuf 7.x is preserved without conflicting with feast[flink]. Signed-off-by: Siddhesh Khairnar <[email protected]>
fa36ecd to
929474c
Compare
|
Hey @jyejare merge conflict is resolved. Please take a look! |
* feat: Add Pinecone online store integration (feast-dev#6585) * feat: Add Pinecone online store integration Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Align Pinecone store with filters API and operator types Signed-off-by: Siddhesh Khairnar <[email protected]> * test: Expand Pinecone online store unit test coverage Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Address Pinecone online store review feedback Signed-off-by: Siddhesh Khairnar <[email protected]> * ci: Retrigger CI after flaky macOS dependency download Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Resolve rebase conflicts keeping chronon and pinecone Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Prevent protobuf downgrade breaking grpc health checks Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Revert protobuf pin that broke flink and pixi lock Keep uv --no-sync for CI so hashed protobuf 7.x is preserved without conflicting with feast[flink]. Signed-off-by: Siddhesh Khairnar <[email protected]> --------- Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Install patched libbrotli1 to address DSA-6582-1 (feast-dev#7096) Signed-off-by: Joystyxx <[email protected]> Co-authored-by: Shuchu Han <[email protected]> * docs: Fix docstring argument names that do not match the signature (feast-dev#7098) Several Args entries named a parameter the function does not have. Rename them to the actual parameter or drop them, and update the DecisionEvaluator example, which passed a DecisionStrategy that the constructor does not accept. Signed-off-by: mmporong <[email protected]> Co-authored-by: Shuchu Han <[email protected]> * chore(deps-dev): bump minimatch from 10.2.6 to 11.0.1 in /ui (feast-dev#7102) Bumps [minimatch](https://github.com/isaacs/minimatch) from 10.2.6 to 11.0.1. - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md) - [Commits](isaacs/minimatch@v10.2.6...v11.0.1) --- updated-dependencies: - dependency-name: minimatch dependency-version: 11.0.1 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: Siddhesh Khairnar <[email protected]> Signed-off-by: Joystyxx <[email protected]> Signed-off-by: mmporong <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Chaitany Patel <[email protected]> Co-authored-by: Siddhesh Khairnar <[email protected]> Co-authored-by: JoyStyx <[email protected]> Co-authored-by: Shuchu Han <[email protected]> Co-authored-by: Lim Jooyoung <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ges) (#281) * feat: Add Pinecone online store integration (feast-dev#6585) * feat: Add Pinecone online store integration Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Align Pinecone store with filters API and operator types Signed-off-by: Siddhesh Khairnar <[email protected]> * test: Expand Pinecone online store unit test coverage Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Address Pinecone online store review feedback Signed-off-by: Siddhesh Khairnar <[email protected]> * ci: Retrigger CI after flaky macOS dependency download Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Resolve rebase conflicts keeping chronon and pinecone Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Prevent protobuf downgrade breaking grpc health checks Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Revert protobuf pin that broke flink and pixi lock Keep uv --no-sync for CI so hashed protobuf 7.x is preserved without conflicting with feast[flink]. Signed-off-by: Siddhesh Khairnar <[email protected]> --------- Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Install patched libbrotli1 to address DSA-6582-1 (feast-dev#7096) Signed-off-by: Joystyxx <[email protected]> Co-authored-by: Shuchu Han <[email protected]> * docs: Fix docstring argument names that do not match the signature (feast-dev#7098) Several Args entries named a parameter the function does not have. Rename them to the actual parameter or drop them, and update the DecisionEvaluator example, which passed a DecisionStrategy that the constructor does not accept. Signed-off-by: mmporong <[email protected]> Co-authored-by: Shuchu Han <[email protected]> * chore(deps-dev): bump minimatch from 10.2.6 to 11.0.1 in /ui (feast-dev#7102) Bumps [minimatch](https://github.com/isaacs/minimatch) from 10.2.6 to 11.0.1. - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md) - [Commits](isaacs/minimatch@v10.2.6...v11.0.1) --- updated-dependencies: - dependency-name: minimatch dependency-version: 11.0.1 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: Siddhesh Khairnar <[email protected]> Signed-off-by: Joystyxx <[email protected]> Signed-off-by: mmporong <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Chaitany Patel <[email protected]> Co-authored-by: Siddhesh Khairnar <[email protected]> Co-authored-by: JoyStyx <[email protected]> Co-authored-by: Shuchu Han <[email protected]> Co-authored-by: Lim Jooyoung <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… only, do not squash) (#283) * feat: Add Pinecone online store integration (feast-dev#6585) * feat: Add Pinecone online store integration Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Align Pinecone store with filters API and operator types Signed-off-by: Siddhesh Khairnar <[email protected]> * test: Expand Pinecone online store unit test coverage Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Address Pinecone online store review feedback Signed-off-by: Siddhesh Khairnar <[email protected]> * ci: Retrigger CI after flaky macOS dependency download Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Resolve rebase conflicts keeping chronon and pinecone Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Prevent protobuf downgrade breaking grpc health checks Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Revert protobuf pin that broke flink and pixi lock Keep uv --no-sync for CI so hashed protobuf 7.x is preserved without conflicting with feast[flink]. Signed-off-by: Siddhesh Khairnar <[email protected]> --------- Signed-off-by: Siddhesh Khairnar <[email protected]> * fix: Install patched libbrotli1 to address DSA-6582-1 (feast-dev#7096) Signed-off-by: Joystyxx <[email protected]> Co-authored-by: Shuchu Han <[email protected]> * docs: Fix docstring argument names that do not match the signature (feast-dev#7098) Several Args entries named a parameter the function does not have. Rename them to the actual parameter or drop them, and update the DecisionEvaluator example, which passed a DecisionStrategy that the constructor does not accept. Signed-off-by: mmporong <[email protected]> Co-authored-by: Shuchu Han <[email protected]> * chore(deps-dev): bump minimatch from 10.2.6 to 11.0.1 in /ui (feast-dev#7102) Bumps [minimatch](https://github.com/isaacs/minimatch) from 10.2.6 to 11.0.1. - [Changelog](https://github.com/isaacs/minimatch/blob/main/changelog.md) - [Commits](isaacs/minimatch@v10.2.6...v11.0.1) --- updated-dependencies: - dependency-name: minimatch dependency-version: 11.0.1 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: Siddhesh Khairnar <[email protected]> Signed-off-by: Joystyxx <[email protected]> Signed-off-by: mmporong <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Chaitany Patel <[email protected]> Co-authored-by: Siddhesh Khairnar <[email protected]> Co-authored-by: JoyStyx <[email protected]> Co-authored-by: Shuchu Han <[email protected]> Co-authored-by: Lim Jooyoung <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
# [0.68.0](v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060) * Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584)) * Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011) * Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad)) * Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7)) * Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84)) * Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88)) * Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7)) * Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe)) * Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105) * Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1)) * Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024) * **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054) * Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168)) * Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876) * Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c)) ### Features * Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940) * Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c)) * Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041)) * Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))
# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060) * Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584)) * Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011) * Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad)) * Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7)) * Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84)) * Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88)) * Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7)) * Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe)) * Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105) * Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1)) * Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024) * **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054) * Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168)) * Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876) * Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c)) ### Features * Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940) * Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c)) * Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041)) * Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e)) Signed-off-by: Shuchu Han <[email protected]>
What this PR does / why we need it:
This PR adds Pinecone as a new online store integration for Feast. Pinecone is a managed vector database that enables fast similarity search and retrieval, making it ideal for ML feature serving workloads that involve embeddings and vector-based lookups.
Key changes:
PineconeOnlineStoreclass with full CRUD operations (online_write_batch,online_read,teardown)PineconeOnlineStoreConfigWhich issue(s) this PR fixes:
Fixes #6582
Checks
git commit -s)Testing Strategy
Misc
pinecone-clientas an optional dependency (pip install feast[pinecone])docs/reference/online-stores/pinecone.md