Skip to content

Add TableProviderFactory and test for SQL to register tables dynamically at runtime - #892

Merged
wjones127 merged 5 commits into
delta-io:mainfrom
spaceandtimefdn:bg_register_table
Nov 17, 2022
Merged

wjones127 merged 5 commits into
delta-io:mainfrom
spaceandtimefdn:bg_register_table

Conversation

@avantgardnerio

Copy link
Copy Markdown
Contributor

Description

Use (hopefully soon-to-be-merged) Datafusion functionality to allow dynamic (i.e. SQL-based) registration of deltalake tables at runtime.

Related Issue(s)

Documentation

  • TBD

Comment thread rust/tests/datafusion_test.rs Outdated

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the goal of the PR

Comment thread rust/Cargo.toml Outdated

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to upgrade arrow to match Datafusion master

Comment thread rust/Cargo.toml Outdated

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will point back at apache/arrow-datafusion once that PR (#3867) is merged. Then probably v14.0.0 once it is released.

@avantgardnerio

Copy link
Copy Markdown
Contributor Author

This is working in Ballista from DataGrip through FlightSQL! 🥳

Screenshot from 2022-10-23 12-44-02

@alamb that's the end goal for the related datafusion PR

@avantgardnerio
avantgardnerio force-pushed the bg_register_table branch 2 times, most recently from d96c625 to 3795587 Compare October 24, 2022 18:40
@avantgardnerio
avantgardnerio marked this pull request as ready for review October 24, 2022 18:40
@avantgardnerio

Copy link
Copy Markdown
Contributor Author

@houqp I was finally able to get Ballista working running deltalake queries. I'd appreciate a review when you get the chance (and a CI kick would be appreciated as well).

@avantgardnerio

Copy link
Copy Markdown
Contributor Author

I fixed the CI checks I could, and fixed datafusion to not require delta-rs to mess with the build. If someone could fire off CI, I'd appreciate it 🙌

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very cool!

I have an option suggestion and a question on how we are handling storage options.

Comment thread rust/src/delta_datafusion.rs Resolved
Comment thread rust/src/delta_datafusion.rs Resolved
Comment thread rust/src/delta_datafusion.rs Outdated
Comment on lines 312 to 316

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, any URI passed with the same scheme and bucket name will match to this object store? And the object stores options will be preserved? I'm worried about the cases where storage options were directly passed into the DeltaTable instance.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the object store url we get from table.storage.object_store_url is just for internal purposes and should not collide with common urls. It uses delta-rs:// as a scheme and does a little bit of pruning required for widows paths.

Comment thread rust/src/delta_datafusion.rs Outdated
Comment on lines 858 to 842

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is perhaps a place where storage_options will get lost, if we care about that. If we only want to support passing credentials via env variables for systems that use this (just distributed, right?), that might be sensible.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we only want to support passing credentials via env variables for systems that use this (just distributed, right?), that might be sensible.

For now, that is all I am attempting to do with this PR. I can possibly see someone wanting to file another PR to extend it in the future, but if they are passing storage_options, I think that means they are registering tables in Rust (not SQL), and therefor already are writing a fork of Ballista, in which case they can write their own serde?

It's a worthwhile thought experiment, but I wouldn't recommend coding for it quite yet...

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fair. I just wanted to make sure you were aware of that limitation.

@roeap

roeap commented Oct 25, 2022

Copy link
Copy Markdown
Collaborator

I just marked #852 as ready for review. That should fix the python builds which are breaking due to the arrow version bump as well as handle the parquet2 build that needed some changes as parquet-format moved into the parquet crate.

@roeap roeap left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work! really excited to see this land.

Comment thread rust/src/delta_datafusion.rs Outdated
Comment on lines 312 to 316

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the object store url we get from table.storage.object_store_url is just for internal purposes and should not collide with common urls. It uses delta-rs:// as a scheme and does a little bit of pruning required for widows paths.

Comment thread rust/src/delta_datafusion.rs Resolved
Comment thread rust/src/delta_datafusion.rs Outdated
Comment on lines 444 to 450

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be completely off, but can a scan operation even have children? ParquetScan itself just returns Ok(self) .

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can a scan operation even have children?

I would think not, but the goal of this implementation was merely to proxy every call to the wrapped ParquetScan with the one additional behavior of registering the ObjectStore prior to proxying the execute() request... so I just followed that pattern here. I think it's probably best to leave it this way just in case there is some future change where a ParquetScan might have children for some reason - possibly because someone wants to register their own proxy in the same way I just did :)

@avantgardnerio

Copy link
Copy Markdown
Contributor Author

I did some fixes, so there's a reasonable chance CI should pass now.

@avantgardnerio

Copy link
Copy Markdown
Contributor Author

It looks like the remaining errors are in src/python, so I assume they will be resolved by #852 .

wjones127
wjones127 previously approved these changes Oct 26, 2022
@avantgardnerio

Copy link
Copy Markdown
Contributor Author

I see #852 was merged. I think we can merge this now. If someone wouldn't mind kicking off CI, I'd appreciate it!

@avantgardnerio

Copy link
Copy Markdown
Contributor Author

I don't understand the python test failure... it looks like a timeout, is it an intermittent test? Or should I try to dig into it more?

@wjones127

Copy link
Copy Markdown
Collaborator

Details

Yeah I think that is unrelated. I'll need to look into that.

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks!

@wjones127
wjones127 merged commit a8f2ff2 into delta-io:main Nov 17, 2022
@avantgardnerio
avantgardnerio deleted the bg_register_table branch November 17, 2022 18:14
@avantgardnerio

Copy link
Copy Markdown
Contributor Author

Thanks everyone!

@roeap

roeap commented Nov 17, 2022

Copy link
Copy Markdown
Collaborator

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support CREATE EXTERNAL TABLE dt STORED AS DELTATABLE syntax with datafusion

3 participants