Skip to content

fix: Don't materialise Ibis table to PyArrow if using vegafusion data transformer#3566

Merged
jonmmease merged 1 commit intovega:mainfrom
MarcoGorelli:delay-materialisation
Sep 3, 2024
Merged

fix: Don't materialise Ibis table to PyArrow if using vegafusion data transformer#3566
jonmmease merged 1 commit intovega:mainfrom
MarcoGorelli:delay-materialisation

Conversation

@MarcoGorelli
Copy link
Copy Markdown
Contributor

@MarcoGorelli MarcoGorelli commented Sep 3, 2024

Background

In #3452 (review), @jonmmease pointed out that

To avoid regressing performance in the VegaFusion+Ibis case, I think we'll need to avoid loading the full DataFrame into memory as pyarrow at this point, and instead use the dataframe protocol directly to introspect the column types.

I'd originally done that in the linked PR, but then in a subsequent refactor (in the later commit from the same PR) I must have accidentally regressed it, because currently, arrow_table_from_dfi_dataframe does get called, even with the vegafusion transformer

This PR fixes it, and adds a test to ensure this doesn't happen again


I noticed this whilst looking into supporting DuckDB, which is now also supported (for now, it requires PyArrow, as it goes down the same path in Altair that Ibis does, but we're aiming to get DuckDB supported here with zero extra dependencies - it's going to take some more work, but we're not that far off):

image

@MarcoGorelli MarcoGorelli changed the title fix: dont materialise Ibis table to PyArrow if using vegafusion data transformer fix: Don't materialise Ibis table to PyArrow if using vegafusion data transformer Sep 3, 2024
@MarcoGorelli MarcoGorelli marked this pull request as ready for review September 3, 2024 14:29
Copy link
Copy Markdown
Contributor

@jonmmease jonmmease left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @MarcoGorelli, lgtm!

@jonmmease jonmmease merged commit db97a0b into vega:main Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants