Feast BigQuery offline store gets wrong argument types for event_timestamp #2532

Activity

1. KarolisKont

   added

   kind/bug

   priority/p2

   on Apr 13, 2022
2. achals commented on Apr 13, 2022

   

   achals

   on Apr 13, 2022

   Member

   More actions

   @KarolisKont could you also share your feature view definition and the schema of the offline store bigquery table?
3. KarolisKont commented on Apr 13, 2022

   

   KarolisKont

   on Apr 13, 2022

   Author

   More actions

   @KarolisKont could you also share your feature view definition and the schema of the offline store bigquery table?

   Of course, I can @achals, but have one issue this Bigquery table has 109 fields 😅 . We don't use all fields in the feature view. So perhaps for you will be enough just those fields that are used feature view only?

   Also perhaps forgot to mention that when I dropped the google-cloud-bigquery version to under 3.0.0 Python SDK can returns offline data and jobs in BigQuery don't have errors.
4. achals commented on Apr 13, 2022

   

   achals

   on Apr 13, 2022

   Member

   More actions

   That's very interesting, yeah I just need the fields used in the feature view. Can you also provide the output of the table schema, after getting the table using the python API? Looking at this code snippet an example: https://googleapis.dev/python/bigquery/latest/usage/tables.html#getting-a-table

   👍React with 👍1Reacted by Karolis Kontrimas
5. KarolisKont commented on Apr 15, 2022

   

   KarolisKont

   on Apr 15, 2022

   Author

   More actions

   Sorry for the late response 😊 , but here is the info:
   feature view definition:

   from datetime import timedelta
   
   from feast import BigQuerySource, Feature, FeatureView, ValueType
   
   from marketplace_sandbox.config import BQ_POC_RERANKER_TABLE_REFERENCE
   from marketplace_sandbox.entities import item_user
   from marketplace_sandbox.utils import generate_data_bq_query
   
   FEATURE_VIEW_NAME = "item_user_stats_v1"
   
   entities = [item_user]
   
   features = [
       Feature(
           name="total_items_sold",
           dtype=ValueType.INT32,
           labels={"owner": "VMIP"},
       ),
       Feature(
           name="is_high_risk_seller",
           dtype=ValueType.INT32,
           labels={"owner": "VMIP"},
       ),
   ]
   
   batch_source = BigQuerySource(
       name=f"{BQ_POC_RERANKER_TABLE_REFERENCE}.{FEATURE_VIEW_NAME}",
       query=generate_data_bq_query(
           features=features,
           entities=entities,
           tables=[BQ_POC_RERANKER_TABLE_REFERENCE],
       ),
       event_timestamp_column="event_time",
   )
   
   item_user_stats_v1_fv = FeatureView(
       name=FEATURE_VIEW_NAME,
       entities=[entity.name for entity in entities],
       ttl=timedelta(weeks=4),
       features=features,
       batch_source=batch_source,
       owner="VMIP",
   )

   In this feature view used entity definition:

   from feast import Entity, ValueType
   
   item_user = Entity(
       name="item_user",
       join_key="item_user_id",
       value_type=ValueType.INT64,
       owner="VMIP",
   )

   In this feature view helper function to construct query:

   from typing import Iterable
   
   from feast import Entity, Feature
   
   
   def generate_data_bq_query(
       features: Iterable[Feature],
       entities: Iterable[Entity],
       tables: Iterable[str],
   ) -> str:
       """BigQuery SQL query to execute to generate data for feature view.
   
       Args:
           features (Iterable[Feature]): all used features in feature view.
           entities (Iterable[Entity]): all used entities in feature view.
           tables (Iterable[str]): all BigQuery tables that contain used features
               and enities.
   
       Returns:
           str: BigQuery SQL query.
       """
       return (
           f"SELECT event_time, {', '.join(f.name for f in features)}, "
           f"{', '.join(entity.join_key for entity in entities)} "
           f"FROM `{', '.join(table for table in tables)}`"
       )

   This feature view schema from BigQuery UI:

   Field name | Type | Mode
   
   item_user_id | INTEGER | NULLABLE
   total_items_sold | INTEGER | NULLABLE
   is_high_risk_seller | INTEGER | NULLABLE
   event_time | TIMESTAMP | NULLABLE

   The output of table schema using python API (google-cloud-bigquery==3.0.1):

   Table schema: [SchemaField('item_user_id', 'INTEGER', 'NULLABLE', None, (), None), SchemaField('event_time', 'TIMESTAMP', 'NULLABLE', None, (), None), SchemaField('total_items_sold', 'INTEGER', 'NULLABLE', None, (), None), SchemaField('is_high_risk_seller', 'INTEGER', 'NULLABLE', None, (), None)]

   The output of table schema using python API (google-cloud-bigquery==2.34.2):

   able schema: [SchemaField('item_user_id', 'INTEGER', 'NULLABLE', None, (), None), SchemaField('event_time', 'TIMESTAMP', 'NULLABLE', None, (), None), SchemaField('total_items_sold', 'INTEGER', 'NULLABLE', None, (), None), SchemaField('is_high_risk_seller', 'INTEGER', 'NULLABLE', None, (), None)]
6. KarolisKont commented on Apr 16, 2022

   

   KarolisKont

   on Apr 16, 2022

   Author

   More actions

   OK seems that my proposed solution was valid for this particular issue. Thanks @achals 🙇
7. KarolisKont

   closed this as completedon Apr 16, 2022