Repository navigation
TTL for entities in the Online stores other Than Redis #1988
Description
Is your feature request related to a problem? Please describe.
(for context: we are using the Redis online store)
At the moment, once an entity is written to the Online store, it's there until it's manually cleaned up.
In most of our business cases, the Entity will have a natural expiration time (when it's no longer relevant for the use case).
It would be nice to be able to expire the entity in the online store, with some kind of TTL setting.
Describe the solution you'd like
- in our local setup we end up expiring the key bin in Redis (combination of entity + project)
def online_write_batch(...):
redis_key_bin = _redis_key(project, entity_key)
...
if table.ttl:
redis_client_pipeline.expire(name=redis_key_bin, time=table.ttl)
the issue with this is that the ttl param in feature_view/table is overloaded (initially designed for a use case in the offline store)
Also, in this type of setup and use case, the TTL should be on the entity level not feature_view because the Redis key bin is based on entity (feature_view ttl is designed for something else also)
so the proposal:
add online_store_ttl param to the Entity object
example:
driver_entity = Entity(
name="driver",
description="Driver Identifier",
value_type=ValueType.INT64,
online_store_ttl=Duration(seconds=86400)
)
def online_write_batch(...):
redis_key_bin = _redis_key(project, entity_key)
...
if entity.online_store_ttl:
redis_client_pipeline.expire(name=redis_key_bin, time=entity.online_store_ttl)
Note: implementation details may vary a bit based on the Online store type
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
felixwang9817 commentedon Nov 1, 2021
Somewhat related to #1802 (this issue is about entity TTL in online stores, #1802 is about FV TTL)
r-nagaraj commentedon Dec 9, 2021
Hi Team,
I have also observed online database (Redis online store) is not sync with latest Materializing feature views. In short Redis online store is not truncating and reloading with latest data. Thanks
felixwang9817 commentedon Dec 9, 2021
hey @r-nagaraj, could you elaborate on the exact problem you're seeing, and how we might reproduce it? thanks!
adchia commentedon Jan 5, 2022
See some initial discussion happening in https://docs.google.com/document/d/1K8k756lgsG-YoIiEyHMOOBEejQerurhfMIABlu8Cir4/edit
[-]TTL for entities in the Online store[/-][+]TTL for entities in the Online stores other Than Redis[/+]stale commentedon Dec 20, 2022
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.