Skip to content

Add Parquet as an Export format #2591

@philrz

Description

@philrz

At the time of the opening of this issue, Brim is at commit 9d0a742.

Since the backend already supports it, like we did for ZSON with #1486, it should be trivial to add Parquet as an Export format. However, as I was about to impulsively do this, I remembered that the backend still has this limitation:

$ zq -version
Version: v1.2.0-130-g8dd27d98

$ zq -f parquet 'fuse' ./src/test/shared/data/sample.tsv 
union type unsupported

Since union types often show up in Zed formats (particularly as a side effect of things like applying fuse) my inclination is to wait on an improvement in the backend before we add it as an Export option.

For now, I've got a branch https://github.com/brimdata/brim/tree/parquet-export where I've got it working for the happy path. Separately I opened up brimdata/super#4215 to track what I'm observing in the API interactions that are currently preventing the app from reporting the failure scenarios to the user.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions