Skip to content

Commit bf4e3fa

Browse files
caseyclementsdevin-ai-integration[bot]ntkathole
authored
feat: Add OnlineStore for MongoDB (#6025)
* Initial commit on INTPYTHON-297-MongoDB-Feast-Integration Signed-off-by: Casey Clements <[email protected]> * Added mongodb to project.optional-dependencies in pyproject.toml. Now pymongo is found as extra Signed-off-by: Casey Clements <[email protected]> * Checkpoint. Passing tests.unit.online_store.test_online_writes.TestOnlineWrites with default args of MongoDBOnlineStoreConfig. Lots to do. Signed-off-by: Casey Clements <[email protected]> * Handle Nan in dfs for test_online_writes.py. Now all tests in the module pass Signed-off-by: Casey Clements <[email protected]> * Removed suffix of implementation: mongodb_openai -> mongodb Signed-off-by: Casey Clements <[email protected]> * Moved MongoDBOnlineStore to feast.infra.online_store.contrib Signed-off-by: Casey Clements <[email protected]> * Formatting Signed-off-by: Casey Clements <[email protected]> * Refactor online_read that converts bson to proto. Left two methods for now. simply and transformming Signed-off-by: Casey Clements <[email protected]> * Remove file added early during discovery Signed-off-by: Casey Clements <[email protected]> * Formatting Signed-off-by: Casey Clements <[email protected]> * Added version of test that uses testcontainers mongodb instead of assuming one is running Signed-off-by: Casey Clements <[email protected]> * Create Make target for universal tests Signed-off-by: Casey Clements <[email protected]> * Cleanup Signed-off-by: Casey Clements <[email protected]> * Removed temporary integration tests requiring one to spin up own mongodb server. Signed-off-by: Casey Clements <[email protected]> * Format Signed-off-by: Casey Clements <[email protected]> * Typing Signed-off-by: Casey Clements <[email protected]> * Implemented ASync API and Tests Signed-off-by: Casey Clements <[email protected]> * Removed offline store stubs. The first PR will only contain the OnlineStore Signed-off-by: Casey Clements <[email protected]> * Moved mongodb_online_store out of cobtrib package. Signed-off-by: Casey Clements <[email protected]> * Add documentation. Signed-off-by: Casey Clements <[email protected]> * Cleanups and docstrings Signed-off-by: Casey Clements <[email protected]> * Fixed another reference to contrib dir Signed-off-by: Casey Clements <[email protected]> * Typos Signed-off-by: Casey Clements <[email protected]> * Made _convert_raw_docs_to_proto staticmethods private Signed-off-by: Casey Clements <[email protected]> * After benchmarking two alogithm for conevrting read results from bson to proto, removed the naive one. It was outcompeted 3X across dimensions Signed-off-by: Casey Clements <[email protected]> * Add extra unit tests Signed-off-by: Casey Clements <[email protected]> * Formatting Signed-off-by: Casey Clements <[email protected]> * Fixes in pyproject.toml Signed-off-by: Casey Clements <[email protected]> * Fixed Detect secrets false positives. Signed-off-by: Casey Clements <[email protected]> * Update sdk/python/feast/infra/online_stores/mongodb_online_store/mongodb.py Set _client and _collection to None upon synchronous teardown. Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: Casey Clements <[email protected]> * Fix CI: guard pymongo imports and skip test module when pymongo unavailable In the unit-test-python CI job, pymongo is not installed (it is an optional extra). This caused ModuleNotFoundError during pytest collection. Two changes: 1. mongodb.py: wrap pymongo imports in try/except -> FeastExtrasDependencyImportError (consistent with redis.py, dynamodb.py, datastore.py pattern) 2. test_mongodb_online_retrieval.py: add pytest.importorskip(\pymongo\) so the entire test module is skipped gracefully when pymongo is absent Signed-off-by: Casey Clements <[email protected]> * Fix: return (None, None) when entity doc exists but feature view was never written MongoDB stores all feature views for an entity in a single document. If FV 'driver_stats' is written, an entity doc exists for driver_1. A subsequent read for FV 'pricing' (never written) was previously returning (None, {all_features: ValueProto()}) - a truthy feature dict with all-empty protos - instead of the correct (None, None) sentinel. Feast and downstream callers use (None, None) to signal entity absence. A non-None feature dict means 'entity found, values are null', which causes different behaviour in the feature retrieval pipeline. Fix: after confirming the entity doc exists, also check that the feature view key is present in doc['features']. If absent, return (None, None) rather than a dict of empty protos. Adds test_convert_raw_docs_entity_exists_but_fv_not_written to cover the multi-feature-view colocation scenario identified in code review. Signed-off-by: Casey Clements <[email protected]> * Update pixi.lock after adding mongodb optional dependency to pyproject.toml Any change to pyproject.toml invalidates the pixi.lock manifest hash, causing 'pixi install --locked' to fail in CI even when the changed section (mongodb optional extras) is not used by any pixi environment. Regenerated with: pixi install (v0.63.2) Signed-off-by: Casey Clements <[email protected]> * fix: catch FeastExtrasDependencyImportError in doctest runner Stores like mongodb, redis, and dynamodb raise FeastExtrasDependencyImportError at import time when their optional Python extras are not installed. test_all.py only caught ModuleNotFoundError, so any such import caused the entire test_docstrings() function to abort rather than gracefully skipping the module. Extend the except clause to also catch FeastExtrasDependencyImportError so the doctest run completes for all other modules when an optional extra is absent in the test environment. Signed-off-by: Casey Clements <[email protected]> * fix: update PYTEST_PLUGINS path for mongodb online store tests Directory was renamed from tests/integration/feature_repos to tests/universal/feature_repos in upstream/master. Update the PYTEST_PLUGINS module path in test-python-universal-mongodb-online to match the new location. Signed-off-by: Casey Clements <[email protected]> * fix: broaden import exception handling in doctest runner to catch TypeError and other third-party import errors qdrant_client raises TypeError at import time when a gRPC EnumTypeWrapper is used with the | union operator on Python < 3.12. The previous fix only caught ModuleNotFoundError and FeastExtrasDependencyImportError, leaving the test runner vulnerable to any other exception raised by third-party libraries during pkgutil.walk_packages. Changes: - Catch bare Exception in the import try/except block so any import-time error from an optional dependency causes a graceful skip rather than an abort of the entire test run. - Initialize temp_module = None before the try block and add a continue guard so that a failed import never leaves a stale module reference to be used in the subsequent doctest execution block. Signed-off-by: Casey Clements <[email protected]> * fix: pass onerror to pkgutil.walk_packages to suppress non-ImportError package import failures FeastExtrasDependencyImportError inherits from FeastError -> Exception, not from ImportError. pkgutil.walk_packages calls __import__ internally when recursing into sub-packages, and only silently swallows ImportError subclasses; any other exception is re-raised, crashing the entire test run. Passing onerror=lambda _: None tells walk_packages to skip any package that fails to import during the walk phase, regardless of the exception type. The inner importlib.import_module try/except already handles the same errors for the explicit import step used to collect doctests. Signed-off-by: Casey Clements <[email protected]> * fix: update stale tests.integration.feature_repos imports to tests.universal.feature_repos The upstream directory tests/integration/feature_repos was renamed to tests/universal/feature_repos. Three files still referenced the old path: - mongodb_repo_configuration.py: IntegrationTestRepoConfig and MongoDBOnlineStoreCreator imports - tests/.../online_store/mongodb.py: OnlineStoreCreator import - tests/unit/online_store/test_mongodb_online_retrieval.py: TAGS import Updating all three unblocks test collection and allows make test-python-universal-mongodb-online to run locally. Signed-off-by: Casey Clements <[email protected]> * feat: add mongodb to ValidOnlineStoreDBStorePersistenceTypes in feast-operator The Python ONLINE_STORE_CLASS_FOR_TYPE now includes 'mongodb', so the operator's parity check (test-datasources) fails unless the Go API also lists it. Add 'mongodb' to ValidOnlineStoreDBStorePersistenceTypes in both api/v1 and api/v1alpha1. Signed-off-by: Casey Clements <[email protected]> * feat: add Feast driver metadata to MongoDB client instantiations Add DriverInfo with the Feast name and version to both the sync MongoClient and async AsyncMongoClient so that MongoDB can identify traffic originating from a Feast AI integration. Signed-off-by: Casey Clements <[email protected]> * docs: update MongoDB online store status from alpha to preview Signed-off-by: Casey Clements <[email protected]> * fix: add mongodb to kubebuilder Enum annotations for OnlineStoreDBStorePersistence The +kubebuilder:validation:Enum annotation on the Type field of OnlineStoreDBStorePersistence was not updated when mongodb was added to ValidOnlineStoreDBStorePersistenceTypes. This annotation drives CRD OpenAPI schema validation at Kubernetes admission time, so any FeatureStore CR specifying type: mongodb would be rejected by the API server. Updated both api/v1 and api/v1alpha1. Signed-off-by: Casey Clements <[email protected]> * Update +GOLANGCI_LINT_VERSION to fix upstream issue golang/go#74462 Signed-off-by: Casey Clements <[email protected]> * fix: raise ValueError in _get_client_async for invalid config type, consistent with sync counterpart Signed-off-by: Casey Clements <[email protected]> * fix: Added mongodb to operator yamls Signed-off-by: ntkathole <[email protected]> * Small change suggested by ntkathole Signed-off-by: Casey Clements <[email protected]> * Factor out write logic into utility function making sync/async essentially identical. Signed-off-by: Casey Clements <[email protected]> --------- Signed-off-by: Casey Clements <[email protected]> Signed-off-by: ntkathole <[email protected]> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ntkathole <[email protected]>
1 parent 1858daf commit bf4e3fa

25 files changed

Lines changed: 8296 additions & 89 deletions

File tree

‎.secrets.baseline‎

Lines changed: 78 additions & 78 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Makefile‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -535,6 +535,25 @@ test-python-universal-elasticsearch-online: ## Run Python Elasticsearch online s
535535
not test_snowflake" \
536536
sdk/python/tests
537537

538+
test-python-universal-mongodb-online: ## Run Python MongoDB online store integration tests
539+
PYTHONPATH='.' \
540+
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.mongodb_online_store.mongodb_repo_configuration \
541+
PYTEST_PLUGINS=sdk.python.tests.universal.feature_repos.universal.online_store.mongodb \
542+
python -m pytest -n 8 --integration \
543+
-k "not test_universal_cli and \
544+
not test_go_feature_server and \
545+
not test_feature_logging and \
546+
not test_reorder_columns and \
547+
not test_logged_features_validation and \
548+
not test_lambda_materialization_consistency and \
549+
not test_offline_write and \
550+
not test_push_features_to_offline_store and \
551+
not gcs_registry and \
552+
not s3_registry and \
553+
not test_universal_types and \
554+
not test_snowflake" \
555+
sdk/python/tests
556+
538557
test-python-universal-milvus-online: ## Run Python Milvus online store integration tests
539558
PYTHONPATH='.' \
540559
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.milvus_online_store.milvus_repo_configuration \

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,7 @@
131131
* [ScyllaDB](reference/online-stores/scylladb.md)
132132
* [SingleStore](reference/online-stores/singlestore.md)
133133
* [Milvus](reference/online-stores/milvus.md)
134+
* [MongoDB](reference/online-stores/mongodb.md)
134135
* [Registries](reference/registries/README.md)
135136
* [Local](reference/registries/local.md)
136137
* [S3](reference/registries/s3.md)

‎docs/reference/online-stores/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,10 @@ Please see [Online Store](../../getting-started/components/online-store.md) for
5050
[mysql.md](mysql.md)
5151
{% endcontent-ref %}
5252

53+
{% content-ref url="mongodb.md" %}
54+
[mongodb.md](mongodb.md)
55+
{% endcontent-ref %}
56+
5357
{% content-ref url="hazelcast.md" %}
5458
[hazelcast.md](hazelcast.md)
5559
{% endcontent-ref %}

0 commit comments

Comments
 (0)