Repository navigation
[datafusion] Fix "wrong results" 🙄 in DataFusion due toto_char dialect differences - #7
Merged
Merged
Conversation
to_char dialect differencesto_char dialect differences
This was referenced Oct 23, 2026
toschmidt
force-pushed
the
alamb/fix_date_formatting
branch
from
October 23, 2026 19:03
86d016d to
b9601b1
Compare
Contributor
|
/benchmark |
Contributor
|
@toschmidt datafusion on sf1, sf10, sf100, sf1000 on commit The results of each system: datafusion. The results are the pull request's last commit, |
Contributor
|
datafusion is done, system 1 of 1 of the run on commit
Not ok (1501) |
Run by the benchmark bot for pull request dbolympics#7, approved by @toschmidt: dbolympics#7 (comment) DB-Olympics-Results: datafusion Benchmarked-System: datafusion d5b86d5a7f52801be1d0560eda5737dfbe75d275 Approved-By: toschmidt
Contributor
|
Thanks for looking into this! We will also review all wrong results again before the official announcement. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DataFusion's
to_charformat string is different than used in this benchmark which results in "wrong results"DuckDB has an override here for this. DataFusion should have one too.
This brings the errors from "34 wrong results" to "12 wrong results" for each config
Details
Before this fix:
After this fix:
Background
The benchmark currently uses a postgres style formatting string
YY-MMMMwhich returns a constant in DataFusion rather than formatting the fields of a timestamp.The correct syntax, per the DataFusion to_char documentation is:
This accounts for a substantial number of the 'wrong results' for DataFusion (xx of the 34)
Standard
to_charIn case anyone was curious,
to_charis not part of the SQL Standard (at least not in my copy of "NCITS/ISO/IEC 9075-2:2011 [2012] -- (ISO/IEC 9075-2:2011, IDT)", which I will fully admit is a nightmare to read). I thinkto_charwas introduced by postgres (or maybe Oracle back in the day)Therefore this is not a DataFusion violation of the SQL standard, but a dialect incompatibility.