Repository navigation
FileSource.offline_write_batch doesn't work with S3 #3490
Description
Expected Behavior
If FileSource.offline_write_batch, write correctly on S3 storage if use s3:// path.
Current Behavior
file_options = feature_view.batch_source.file_options
filesystem, path = FileSource.create_filesystem_and_path(
file_options.uri, file_options.s3_endpoint_override
)
prev_table = pyarrow.parquet.read_table(path, memory_map=True)When read prev_table, it doens't use "filesystem" kwargs
Steps to reproduce
Always happened
Specifications
- Version: 0.29.0
- Platform:
- Subsystem:
Possible Solution
# default filesystem is None, so it works local & s3fs both
prev_table = pyarrow.parquet.read_table(path, filesystem=filesystem, memory_map=True)
Activity
fix: Add filesystem kwargs when read prev_table on FileRetrievalJob (… (
chore(release): release 0.30.0
chore(release): release 0.30.0
stale commentedon Aug 10, 2023
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.