Repository navigation
Snowflake support does not include list features #2280
Description
Expected Behavior
Snowflake offline store works with list features
Current Behavior
Right now Snowflake produces errors when using list features. This isn't caught by tests because of this skip with the aws provider: https://github.com/feast-dev/feast/blob/master/sdk/python/tests/integration/registration/test_universal_types.py#L39
Activity
adchia commentedon Feb 3, 2022
cc @sfc-gh-madkins
sfc-gh-madkins commentedon Feb 19, 2022
@adchia @woop here is the docs related to semi-structured data in Snowflake: https://docs.snowflake.com/en/sql-reference/data-types-semistructured.html#array
Arrays for list feature types are possible in Snowflake, but there is no enforcement of data type within the array, its stored as type VARIANT, where VARIANT, can store values of any other type, including OBJECT and ARRAY, up to a maximum size of 16 MB. Snowflake does not currently support fixed-size arrays or arrays of elements of a specific non-VARIANT type.
Im curious to your initial thoughts on this. We can certainly test an array if it complies with a certain data type that is supported or throw an error. If it does then execellent, if not we throw an error to remove the feature
stale commentedon Jun 19, 2022
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
achals commentedon Jul 13, 2022
I think we can take the approach described by @sfc-gh-madkins
4 remaining items
sfc-gh-madkins commentedon Dec 16, 2022
bump
stale commentedon Apr 26, 2023
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
sfc-gh-madkins commentedon Apr 26, 2023
JohnLemmonMedely commentedon Sep 5, 2023
Looks like we'll need this for our implementation so I'll be taking a look at implementing this in our sprint starting tomorrow.
sfc-gh-madkins commentedon Sep 5, 2023
@JohnLemmonMedely I am afraid snowflake does not support structured types on the server side. All semistructured types are currently returns as strings. We are working to support this in the next couple of quarters.
JohnLemmonMedely commentedon Sep 5, 2023
@sfc-gh-madkins I was planning on using the method you suggested earlier and attempt to cast them to what they need to be and throwing errors when we can't. It'll be better to guarantee data types via Snowflake but until that's possible I can enforce those guarantees myself via the data pipeline.
sfc-gh-madkins commentedon Sep 5, 2023
JohnLemmonMedely commentedon Dec 8, 2023
@sfc-gh-madkins I had to pause my work on the PR but we were able to use my fork in production. I'm trying to see if it makes sense at this point to get the PR ready to merge or if I should just wait on Snowflake adding structured types instead. Is that still on the Roadmap to add early next year?
sfc-gh-madkins commentedon Dec 8, 2023
JohnLemmonMedely commentedon Dec 15, 2023
@sfc-gh-madkins just following up on if Snowflake still plans on supporting structured types soon. I just need to know if work will start on it over the next quarter or not.
sfc-gh-madkins commentedon Dec 16, 2023