You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
DataSource abstract interface contains methods like validate and get_table_column_names_and_types for validation and schema extraction respectively. This doesn't make too much sense when you consider the separation of concerns between DataSource and OfflineStore. DataSource is supposed to be a static description of source dataset, while OfflineStore is an engine that knows how to read one or more data source types. Having these methods in DataSource classes means data sources should also be able to somehow access the underlying sources.
I propose to move validate method to OfflineStore abstract class as a validate_data_source. This also makes sense for scenarios when a single source can be read by multiple offline stores. For example, FileSource (which can be read by dask, duckdb and probably spark in the future) is right now validated with pyarrow instead of leaving it up to the OfflineStores to choose how to validate the sources).
I think it has nothing to do with dqm. validation here refers to validating whether the data source exists or not. Here is the abstract method I'm referring to. The only place where it's called from is this code segment in repo_operations.py (meaning it's called during feast apply).
Yeah, you're right. I just took a quick looks. It looks like it's referenced in feature logging but doesn't appear to be used and is mostly used for the offline store. I agree with your take here. 👍
Activity
franciscojavierarceo commentedon May 9, 2024
Can you point me to the code you're referring to?
I want to confirm if it's used for Data Quality Monitoring or not.
tokoko commentedon May 9, 2024
I think it has nothing to do with dqm. validation here refers to validating whether the data source exists or not. Here is the abstract method I'm referring to. The only place where it's called from is this code segment in
repo_operations.py(meaning it's called duringfeast apply).franciscojavierarceo commentedon May 9, 2024
Yeah, you're right. I just took a quick looks. It looks like it's referenced in feature logging but doesn't appear to be used and is mostly used for the offline store. I agree with your take here. 👍