Repository navigation
Remote apply #4529
Description
Expected Behavior
Running feast apply from a client connected using remote proxies should apply the python definitions.
Current Behavior
The apply fails. With a project using the postgres template and a deployment on kind using #4528 it throws the following error:
% feast feature-views list
NAME ENTITIES TYPE
driver_hourly_stats_fresh {'driver'} FeatureView
driver_hourly_stats {'driver'} FeatureView
transformed_conv_rate_fresh {'driver'} OnDemandFeatureView
transformed_conv_rate {'driver'} OnDemandFeatureView
% feast apply
...
File "/Users/dmartino/.pyenv/versions/3.11.9/bin/feast", line 8, in <module>
sys.exit(cli())
^^^^^
...
File "/Users/dmartino/.pyenv/versions/3.11.9/lib/python3.11/site-packages/feast/repo_operations.py", line 355, in apply_total
apply_total_with_repo_instance(
File "/Users/dmartino/.pyenv/versions/3.11.9/lib/python3.11/site-packages/feast/repo_operations.py", line 313, in apply_total_with_repo_instance
store.apply(all_to_apply, objects_to_delete=all_to_delete, partial=False)
File "/Users/dmartino/.pyenv/versions/3.11.9/lib/python3.11/site-packages/feast/feature_store.py", line 903, in apply
self._make_inferences(
File "/Users/dmartino/.pyenv/versions/3.11.9/lib/python3.11/site-packages/feast/feature_store.py", line 627, in _make_inferences
update_feature_views_with_inferred_features_and_entities(
File "/Users/dmartino/.pyenv/versions/3.11.9/lib/python3.11/site-packages/feast/inference.py", line 174, in update_feature_views_with_inferred_features_and_entities
_infer_features_and_entities(
File "/Users/dmartino/.pyenv/versions/3.11.9/lib/python3.11/site-packages/feast/inference.py", line 212, in _infer_features_and_entities
table_column_names_and_types = fv.batch_source.get_table_column_names_and_types(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dmartino/.pyenv/versions/3.11.9/lib/python3.11/site-packages/feast/infra/offline_stores/contrib/postgres_offline_store/postgres_source.py", line 113, in get_table_column_names_and_types
with _get_conn(config.offline_store) as conn, conn.cursor() as cur:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/dmartino/.pyenv/versions/3.11.9/lib/python3.11/site-packages/feast/infra/utils/postgres/connection_utils.py", line 18, in _get_conn
conninfo=_get_conninfo(config),
^^^^^^^^^^^^^^^^^^^^^
File "/Users/dmartino/.pyenv/versions/3.11.9/lib/python3.11/site-packages/feast/infra/utils/postgres/connection_utils.py", line 60, in _get_conninfo
"user": config.user,
^^^^^^^^^^^
File "/Users/dmartino/.pyenv/versions/3.11.9/lib/python3.11/site-packages/pydantic/main.py", line 811, in __getattr__
raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'RemoteOfflineStoreConfig' object has no attribute 'user'Reference to local feature_store.yaml:
project: sample
registry:
path: localhost:8001
registry_type: remote
offline_store:
host: localhost
port: 8002
type: remote
online_store:
path: http://localhost:8003
type: remote
entity_key_serialization_version: 2
auth:
type: no_authForwarded ports for all remote services to local ports 8001-8003 (see working example of feast feature-views list).
Steps to reproduce
Follow steps of PR #4528 and then run feast apply from the client folder (after copying the example_repo.py)
Activity
tokoko commentedon Sep 17, 2024
right, makes sense. this is similar to #4186. Basically, we need to move all
DataSourcemethods that actually need to touch the datasets fromDataSourcetoOfflineStore. I did this only forvaliadatebefore, now we need to do the same forget_table_column_names_and_types.dmartinol commentedon Sep 18, 2024
I see that
validate_data_sourceis a static method inOfflineStoreinterface. I assume we want to make it an instance method and re-implement if in the OfflineServer class, right? together with the other new methodget_table_column_names_and_types, of course.This should not require a big effort, even if the Arrow Flight server was not exactly design to serve such purposes but rather suited for "efficient data transport". Shouldn't we make it a multi-protocol application instead and delegate some endpoints unrelated to data-transport to a traditional REST or grpc server?
tokoko commentedon Sep 18, 2024
imho that would end up being more complicated than adding a couple of non-data "endpoints" to a flight server,
dmartinol commentedon Sep 18, 2024
I will start evaluating this one.
tokoko commentedon Sep 18, 2024
I think we can start by requiring data source read permissions for both.
dmartinol commentedon Sep 23, 2024
@tokoko While exploring this option I had one big doubt:
XYZDataSource.get_table_column_names_and_typesmust execute from an instance where theoffline_storeconfig is the actual data store, so it can run the metadata collection query likeSELECT * FROM (SELECT * FROM feast_driver_hourly_stats) AS sub LIMIT 0. The Offline Server seems the perfect place to run this query, BUT:At least, this is the psql implementation, but I had a look at other DS and I think they are also raising errors in case the "table" is not there at the time we execute the method. I debugged and the method is called before the tables being created, so it fails.
Am I missing something?