Repository navigation
feast[snowflake] demo workflow fails to get historical features through sql. get_table_query_string not implemented #3301
Description
I'm trying to run 'feast[snowflake]' demo workflow ( through a trial snowflake account ) from here: Running on Snowflake/AWS/GCP by executing workflow file as python3 test_workflow.py.
Most of the steps work fine, but the execution fails at this function : fetch_historical_features_entity_sql(store, for_batch_scoring=False).
Expected Behavior
fetch_historical_features_entity_sql(store, for_batch_scoring=False) should work fine and display a pandas data frame with 5 rows.
Current Behavior
All Steps Work Fine, starting from feast apply
--- Historical features for training (all entities in a window using SQL entity dataframe) --- Generating training data for all entities in a time window Traceback (most recent call last): File "/Users/aadir00/Desktop/Amith/Feast_Demo/able_parakeet/test_workflow.py", line 187, in <module> run_demo() File "/Users/aadir00/Desktop/Amith/Feast_Demo/able_parakeet/test_workflow.py", line 29, in run_demo fetch_historical_features_entity_sql(store, for_batch_scoring=False) File "/Users/aadir00/Desktop/Amith/Feast_Demo/able_parakeet/test_workflow.py", line 102, in fetch_historical_features_entity_sql FROM {store.list_data_sources()[-1].get_table_query_string()} File "/Users/aadir00/Albvenv_feast/lib/python3.9/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) File "/Users/aadir00/Albvenv_feast/lib/python3.9/site-packages/feast/data_source.py", line 605, in get_table_query_string raise NotImplementedError NotImplementedError
Steps to reproduce
pip3 install 'feast[snowflake]'
feast init -t snowflake .... Snowflake creds
'cd project_folder python3 test_workflow.py`
Specifications
-
Version:
Python: 3.9.6 and 3.8.10
Pip: 21.2.4 and 21.0.1
Feast: 0.26.0 -
Platform: MacBook Pro 16, M1 Chip
-
Subsystem: NA
Possible Solution
Update source code at feast/data_source.py to implement loading data from sql string.
P.S: When I comment out the two functions fetch_historical_features_entity_sql(store, for_batch_scoring=False) and fetch_historical_features_entity_sql(store, for_batch_scoring=True) remaining steps in the test_workflow.py works fine and as expected.
Activity
amithadiraju1694 commentedon Oct 19, 2022
I do see some data sources from
store.list_data_sources()which outputs :[<feast.data_source.PushSource object at 0x1578d64c0>, <feast.infra.offline_stores.snowflake_source.SnowflakeSource object at 0x1578d6280>, <feast.data_source.RequestSource object at 0x1578ead00>]My feature_store.yaml looks like this:
project: able_parakeet registry: registry.db provider: local offline_store: type: snowflake.offline account: *********.azure user: ********* password: ********* role: ACCOUNTADMIN warehouse: ********* database: ********* batch_engine: type: snowflake.engine account: *********.azure user: ********* password: ********* role: ACCOUNTADMIN warehouse: ********* database: ********* online_store: type: snowflake.online account: *********.azure user: ********* password: ********* role: ACCOUNTADMIN warehouse: ********* database: ********* entity_key_serialization_version: 2amithadiraju1694 commentedon Oct 23, 2022
I figured out the issue, was able to make some changes to make it work.
Issue:
It was more of trying to implement
get_table_query_stringfrom the wrong source ( RequestSource ) than the source code not implemented for the right source.In my case
store.list_data_sources()prints the following sources in this order:[<feast.data_source.PushSource object at 0x159c95670>, <feast.infra.offline_stores.snowflake_source.SnowflakeSource object at 0x159caad60>, <feast.data_source.RequestSource object at 0x159cafbb0>]Obviously, RequestSource doesn't implement
get_table_query_stringmethod, hence the error.Solution:
Change the
test_workflow.py -> fetch_historical_features_entity_sql -> entity_sql ( in both conditions )from
{store.list_all_data_sources()[-1].get_table_query_string()}. to{store.get_data_source(data_source_name).get_table_query_string()}data_source_namecan be read like follows:# Reading data source name, which will be used ot execute this query project_name = yaml.safe_load(open("./feature_repo/feature_store.yaml"))["project"]data_source_name = f"{project_name}_feast_driver_hourly_stats"P.S: Don't provide a separate name to the Snowflake source at
driver_repo.pyas incremental materialization fails ( couldn't figure out why yet ). @NhuanTDBK Please look into this, will be happy to contribute these changes , if this is a broader issue.