Repository navigation
On demand feature view not supporting objects #2220
Description
Expected Behavior
@on_demand_feature_view(
inputs={
'driver_hourly_stats': driver_hourly_stats_view
},
features=[
Feature(name='conv_rate_plus_val1', dtype=ValueType.DOUBLE),
Feature(name='conv_rate_plus_val2', dtype=ValueType.DOUBLE),
Feature(name='conv_rate_plus_val3', dtype=ValueType.STRING)
]
)
def transformed_conv_rate(inputs: pd.DataFrame) -> pd.DataFrame:
df = pd.DataFrame()
df['conv_rate_plus_val1'] = (inputs['conv_rate'] + 1)
df['conv_rate_plus_val2'] = (inputs['conv_rate'] + 1)
df['conv_rate_plus_val3'] = str(inputs['conv_rate'])
return df
Should process fine when applying.
Current Behavior
Traceback (most recent call last):
File "/Users/user/anaconda3/envs/feast/bin/feast", line 8, in <module>
sys.exit(cli())
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/cli.py", line 390, in apply_total_command
apply_total(repo_config, repo, skip_source_validation)
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/usage.py", line 269, in wrapper
return func(*args, **kwargs)
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/repo_operations.py", line 260, in apply_total
diff = store.apply(all_to_apply, objects_to_delete=all_to_delete, partial=False)
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/usage.py", line 280, in wrapper
raise exc.with_traceback(traceback)
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/usage.py", line 269, in wrapper
return func(*args, **kwargs)
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/feature_store.py", line 579, in apply
odfv.infer_features()
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/on_demand_feature_view.py", line 226, in infer_features
name=f, dtype=python_type_to_feast_value_type(f, type_name=str(dt))
File "/Users/user/anaconda3/envs/feast/lib/python3.10/site-packages/feast/type_map.py", line 162, in python_type_to_feast_value_type
assert value
AssertionError
Steps to reproduce
Get the on demand feature view demo from: https://github.com/feast-dev/feast-demo/blob/main/feature_repo/features.py
Add the 2 lines as noted in the snippet above for the new STRING feature: conv_rate_plus_val3
Specifications
- Version: 0.17.0
- Platform: Mac
- Subsystem: 12.0.1
Possible Solution
I might be doing something wrong or expecting something which shouldn't be supported. Would love to hear feedback.
Incase this turns out to be a bug, this might be a possible solution --
The type map defined at:
feast/sdk/python/feast/type_map.py
Line 94 in 9dc9e60
Should possibly include an entry for pandas
obj dtype?
Activity
pyalex commentedon Jan 19, 2022
Hi @roy651. Thanks for reporting this. I understand that exception information is very confusing. We should add some details to the assertion or better raise proper exceptions.
That being said, this line is not correct
If you need to convert column to string the correct way would be:
Otherwise, you create a column with dtype object, which is intentionally not supported by Feast.
[-]On demand feature view not supporting STRINGs?[/-][+]On demand feature view not supporting objects[/+]roy651 commentedon Jan 19, 2022
Thanks @pyalex for the guidance to use
astypeinstead of the plain casting, however, in practice it yields the same result.Using this code:
Crashes similarly with:
Note that I've added a couple of debug prints, the last of which comes from within
type_map.py.A quick search reveals that Pandas relies upon Numpy for typing and, in short, because Numpy treats strings as arrays they become objects. There is an alternative approach, starting from Pandas 1.0.0, but it supports the use of
stringtype and notstras explained in the Pandas documentation.I saw that the
type_map.pyalready has a newer version (than 0.17.0) with a different flow for the type check but I believe it will fail in a similar manner as the type map still doesn't containstringorobj.roy651 commentedon Jan 19, 2022
@pyalex Just to clarify the point - This is obviously just an example, but as it stands, I couldn't get any strings to work in the transformed ODFV.
Hence the (previous) title which I gave the issue.
pyalex commentedon Jan 19, 2022
@roy651, you're right. This is my mistake. It always will be dtype
objectunless we use specialpandas.StringType. But we currently are not converting either, because evenpandas.StringTypewill have the namestring, which is not in the listI will increase priority and return the bug label. We're gonna fix it ASAP.