You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add a hint about expected extension in error message in register_csv, register_parquet, register_json #14144
When attempting to register an existing file with different format using the register_csv, register_json, register_parquet, instead of receiving an error, the operation silently returns an empty dataframe.
To Reproduce
let file = "foo.parquet";// foo.parquet does existlet ctx = SessionContext::new();
ctx.register_csv("foo", file,CsvReadOptions::default()).await?;// trying to register parquet file with register_csv// ctx.register_json("foo", file, NdJsonReadOptions::default()).await?; // or same using register_jsonlet res = ctx.sql("select * from foo").await?;
res.show().await?;// Observe that no error is raised, but an empty DataFrame is returned.// ++// ++
Expected behavior
Introduce a validation step in register_csv, register_json, register_parquet to check the file extension or validate the file's format. If the file does not match the expected format, return a clear error message:
Error:Execution("File path 'path/to/file' does not match the expected extension '.extension'")
@devhprl hi, thanks for the comment! When I was working on this, I took in consideration how read_xyz methods work. And they check if the file extension matches the expected extension. So it was one of the reason why I chose this approach for register_xyz methods as well. Actually, I don't see any good reason not to relay on file extension in this regard. For example, if I have some foo.parquet file, I expect to read (or register) it into dataframe with read_parquet method (but not read_csv).
@cj-zhukov@alamb hi, I've just hit this one. judging file content by an extension doesn't feel fully right to me (as you can put there basically anything). yes if there is an content mismatch (expected vs actual) then there will be an undesired result (although I would expect some parsing error or something). nevertheless enforcing untrustworthy extensions doesn't seem to bring much value in that regard. in our case we intentionally omit the extensions completely in the storage and have other ways how the content info gets propagated (to the point where register_xyz gets picked). thanks for the great lib
I agree that ensuring there is some way to override the format used to interpret files (e.g. reading a .dat file as parquet) is a good feature. @devhprl if there is something preventing you from doing that, please do file an issue with a reproducer and we can help make sure the APIs support your usecase
Activity
cj-zhukov commentedon Jan 16, 2025
take
Add a hint about expected extension in error message in register_csv,…
Add a hint about expected extension in error message in register_csv,… (
cj-zhukov commentedon Feb 20, 2025
@devhprl hi, thanks for the comment! When I was working on this, I took in consideration how read_xyz methods work. And they check if the file extension matches the expected extension. So it was one of the reason why I chose this approach for register_xyz methods as well. Actually, I don't see any good reason not to relay on file extension in this regard. For example, if I have some foo.parquet file, I expect to read (or register) it into dataframe with read_parquet method (but not read_csv).
alamb commentedon Feb 23, 2025
I agree that ensuring there is some way to override the format used to interpret files (e.g. reading a
.datfile as parquet) is a good feature. @devhprl if there is something preventing you from doing that, please do file an issue with a reproducer and we can help make sure the APIs support your usecase