Repository navigation
Test loading S3 parquet folder in dataset format #3235
Description
Test that a parquet dataset saved in dataset format can be loaded and accessed using get_historical_features.
Steps:
- upload a parquet dataset to the test s3 folder
- point a source to the parquet dataset s3 folder (not the file)
- load the source features using
get_historical_features
Ideally, there'd be a test for this running, but you'd be blocked on making a file within the Feast integration tests S3 bucket. Mind making an issue for that separately? @mzwiessele
Originally posted by @adchia in #3217 (comment)
Activity
[-]Ideally, there'd be a test for this running, but you'd be blocked on making a file within the Feast integration tests S3 bucket. Mind making an issue for that separately? @mzwiessele[/-][+]Test loading S3 parquet folder in dataset format[/+]felixwang9817 commentedon Sep 24, 2022
I think there are probably two ways to do this:
S3FileDataSourceCreatorinsdk/python/tests/integration/universal/feature_repos/universal/data_sources/file.py, which usesminio- it hasn't been used in a while, but it could be updated and usedminio, we could also just create a dataset and upload data directly to anS3bucket - we could just use thefeast-integration-testsS3 bucket, which is used inuniversal/data_sources/redshift.py@mzwiessele if you want to take this on, I'd be happy to help you and review any PRs!