Repository navigation
File source get_histroical_features get wrong result #3360
Description
Expected Behavior
- if Feast has data with same keys, then run
get_historical_features' get latest data bycreated time' - but it doesn't work expected way
Current Behavior
- if duplicated data on same keys are over 3 copies (diffrent 'created time') , feast cannot give right result
Steps to reproduce
data = pd.read_parquet("data/driver_stats.parquet")
new_data = data.copy()
new_data['avg_daily_trips'] = new_data['avg_daily_trips'] + 1000
new_data['event_timestamp'] = new_data['event_timestamp']
new_data['created'] = pd.Timestamp.now()
merged_df = pd.concat([data, new_data]).reset_index(drop=True)
merged_df.to_parquet("./data/driver_stats.parquet")
store = FeatureStore(".")
entity_df = pd.DataFrame.from_dict(
{
# entity's join key -> entity values
"driver_id": [1001, 1002, 1003],
# "event_timestamp" (reserved key) -> timestamps
"event_timestamp": [
datetime(2022, 11, 20, 23, 00, 42),
datetime(2022, 11, 20, 23, 00, 42),
datetime(2022, 11, 20, 23, 00, 42),
]
}
)
training_df = store.get_historical_features(
entity_df=entity_df,
features=[
"driver_hourly_stats:conv_rate",
"driver_hourly_stats:acc_rate",
"driver_hourly_stats:avg_daily_trips"
],
).to_df()
training_df # it is right, result has (plus+1000) datas
new_data = data.copy()
new_data['avg_daily_trips'] = new_data['avg_daily_trips'] + 10000
new_data['event_timestamp'] = new_data['event_timestamp']
new_data['created'] = pd.Timestamp.now()
# this "merged_df" which is merged by pd.concat is above one.
merged_df = pd.concat([merged_df, new_data]).reset_index(drop=True)
merged_df.to_parquet("./data/driver_stats.parquet")
store = FeatureStore(".")
entity_df = pd.DataFrame.from_dict(
{
# entity's join key -> entity values
"driver_id": [1001, 1002, 1003],
# "event_timestamp" (reserved key) -> timestamps
"event_timestamp": [
datetime(2022, 11, 20, 23, 00, 42),
datetime(2022, 11, 20, 23, 00, 42),
datetime(2022, 11, 20, 23, 00, 42),
]
}
)
training_df = store.get_historical_features(
entity_df=entity_df,
features=[
"driver_hourly_stats:conv_rate",
"driver_hourly_stats:acc_rate",
"driver_hourly_stats:avg_daily_trips"
],
).to_df()
training_df
above is wrong, in my pc result is
Specifications
- Version: Feast SDK Version: "feast 0.26.0"
- Platform: M1 mac
- Subsystem: local
Possible Solution
- This problem seems to have arisen because of pandas sorting & drop_duplicates. but i'm not sure.
- if it is not a but, let me know how to fix that.
Reacted by Seunggabi Kim
Activity
seunggabi commentedon Jan 30, 2023
https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/offline_stores/file.py#L205-L208
Isn't it because you sorted only with event_timestamp?
stale commentedon Jun 10, 2023
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.