Update to apache/arrow-datafusion#2578#48
Conversation
tustvold
changed the title
| let df = ctx.read_csv(table_name, options).await?; | ||
|
|
||
| let uri = format!("file:///{}.csv", table_name); | ||
| ctx.register_csv(table_name, &uri, options).await?; |
There was a problem hiding this comment.
Changing to using this method is necessary as otherwise it infers the ?table? as the table name, which seems to result in an additional projection appearing, not entirely sure why though...
alamb
left a comment
alamb
left a comment
There was a problem hiding this comment.
The changes make sense to me
| clap = { version = "3", features = ["derive", "cargo"] } | ||
| datafusion = { git = "https://github.com/apache/arrow-datafusion", rev = "b6fb0dd52c2abd0f8e134aa46cc1571cc6a0971b" } | ||
| datafusion-proto = { git = "https://github.com/apache/arrow-datafusion", rev = "b6fb0dd52c2abd0f8e134aa46cc1571cc6a0971b" } | ||
| datafusion = { git = "https://github.com/apache/arrow-datafusion", rev = "fdb8fecf0ab475ba07dc0d15f7b53e25ccf30ee7" } |
There was a problem hiding this comment.
@andygrove / @thinkharderdev -- what would you think about writing some script that automatically updates this pin rather than requiring datafusion developers manually make PRs?
We basically upgrade the DataFusion pin manually for IOx and while it takes times it seems to work for us -- for example https://github.com/influxdata/influxdb_iox/pull/4619
| let df = ctx.read_csv(table_name, options).await?; | ||
|
|
||
| let uri = format!("file:///{}.csv", table_name); | ||
| ctx.register_csv(table_name, &uri, options).await?; |
|
I am going to merge this in because:
|
3 participants
Which issue does this PR close?
Rationale for this change
apache/datafusion#2578 contains breaking changes (introduces ListingTableUri)
What changes are included in this PR?
Updates Ballista with breaking changes
Are there any user-facing changes?
There are breaking changes to DataFusion