Repository navigation
Support CSV files with more than 512 columns #3825
Description
Summary
When parsing wide CSV files (e.g., 1000+ columns), Univocity throws an ArrayIndexOutOfBoundsException due to a hard limit of 512 columns. This prevents the ingestion of valid datasets with more columns.
How to Reproduce
- Create a CSV file with >512 columns (e.g., 1000 columns × 100 rows).
- Run the workflow using the CSV Scan Operator.
- Observe the error in the result panel (shown below)
Activity
chenlica commentedon Oct 4, 2025
@Yicong-Huang How to follow the new "type protocol" for this issue?
Yicong-Huang commentedon Oct 4, 2025
@Ma77Ball can you try use the bug template to create it?
Ma77Ball commentedon Oct 4, 2025
@Yicong-Huang, I can recreate it using the bug template. I was wondering if this would be considered a bug or a new feature, as I think the goal is to allow the admin to set the maximum number of columns allowed to prevent memory issues, which is similar to why Univocity caps it at 512 by default.
Yicong-Huang commentedon Oct 4, 2025
I see. It is indeed more of a feature request. Then let's keep it as it is. I will later introduce a feature template. You can ignore the comments regarding bug template.
Ma77Ball commentedon Oct 4, 2025
Currently, the engine can process that many columns when the limit is increased on Univocity (as shown below).
However, if the output is not passed to an operator that reduces the number of columns, the frontend will crash. Or if the view results are put on an operator that has not reduced the number of columns, it also crashes (as shown below)
I'm examining two key potential issues: the result panel is not optimized for this task, and the schema propagation step currently uses 100 rows to evaluate the schema, which may be excessive, especially with larger datasets (1 million columns).
feat(ui): add pagination and search for wide-column tables (apache#4086)
feat(ui): add pagination and search for wide-column tables (apache#4086)
fix: admin-adjustable max columns with a clear overflow message (#4669)
feat(ui): add pagination and search for wide-column tables (apache#4086)
feat(ui): add pagination and search for wide-column tables (apache#4086)
fix: admin-adjustable max columns with a clear overflow message (apac…