Skip to content

Commit ce0e829

Browse files
authored
fix: Fix flaky sort fuzzer test (#26983)
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123. --> - Closes #. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. Please explain the problem you are trying to solve in terms of the user-visible behavior, rather than the implementation. For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the implementation. "COUNT(DISTINCT) returns wrong results when the column contains nulls" is the user-visible problem. --> Sort fuzzer fails in https://github.com/apache/datafusion/actions/runs/41196052207/job/124522323666 It's a false positive due to fuzzer implementation itself, not the actual execution logic. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR. --> 1. add regression test for above failure 2. Fix fuzzer implementation (see comment for detailed explanation) ## What is the testing strategy for this PR? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 3. Serve as another way to document the expected behavior of the code Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'. If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it. You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised. --> Added one regression UT ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. -->
1 parent d4c2a32 commit ce0e829

1 file changed

Lines changed: 58 additions & 8 deletions

File tree

‎datafusion/core/tests/fuzz_cases/sort_query_fuzz.rs‎

Lines changed: 58 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ use std::sync::Arc;
1010

1111
use arrow::array::RecordBatch;
1212
use arrow::compute::concat_batches;
13+
use arrow::util::pretty::pretty_format_batches;
1314
use arrow_schema::SchemaRef;
1415
use datafusion::datasource::MemTable;
1516
use datafusion::prelude::{SessionConfig, SessionContext};
@@ -210,9 +211,7 @@ impl SortQueryFuzzer {
210211
if expected_results.is_none() {
211212
expected_results = Some(results);
212213
} else if let Some(ref expected) = expected_results {
213-
// Every projected column is an ORDER BY key, so tied
214-
// rows have identical output and can be compared in order.
215-
assert_eq!(expected, &results);
214+
assert_batches_equal_in_order(expected, &results, "");
216215
} else {
217216
unreachable!();
218217
}
@@ -401,6 +400,9 @@ impl SortFuzzerTestGenerator {
401400

402401
/// Generates a random SQL query string and an optional limit value.
403402
/// Returns a tuple containing the query string and an optional limit.
403+
///
404+
/// Every projected column is an ORDER BY key, so tied rows have identical
405+
/// output and results can be compared in order, including with LIMIT.
404406
pub fn generate_random_query(&self, rng_seed: u64) -> (String, Option<usize>) {
405407
let mut rng = StdRng::seed_from_u64(rng_seed);
406408

@@ -593,10 +595,10 @@ impl SortFuzzerTestGenerator {
593595
let ctx =
594596
self.generate_random_config(config_seed, false, Some(threshold))?;
595597
let actual = ctx.sql(&query_str).await?.collect().await?;
596-
assert_eq!(
597-
results,
598-
concat_batches(&schema, &actual)?,
599-
"sort_in_place_threshold_bytes={threshold}"
598+
assert_batches_equal_in_order(
599+
&results,
600+
&concat_batches(&schema, &actual)?,
601+
&format!("sort_in_place_threshold_bytes={threshold}"),
600602
);
601603
}
602604
}
@@ -605,10 +607,58 @@ impl SortFuzzerTestGenerator {
605607
}
606608
}
607609

610+
/// Compares rows in order by their displayed values, so logically equal
611+
/// batches with different physical encodings (e.g. dictionaries) still match.
612+
#[track_caller]
613+
fn assert_batches_equal_in_order(
614+
expected: &RecordBatch,
615+
actual: &RecordBatch,
616+
msg: &str,
617+
) {
618+
let expected = pretty_format_batches(std::slice::from_ref(expected))
619+
.unwrap()
620+
.to_string();
621+
let actual = pretty_format_batches(std::slice::from_ref(actual))
622+
.unwrap()
623+
.to_string();
624+
assert_eq!(expected, actual, "{msg}");
625+
}
626+
608627
#[cfg(test)]
609628
mod test {
610629
use super::*;
611630

631+
/// The sort query fuzzer uses the current time as its seed for each run.
632+
/// This test replays seeds from previously failing cases.
633+
///
634+
/// Regressions:
635+
/// https://github.com/apache/datafusion/actions/runs/41196052207/job/124522323666
636+
#[tokio::test(flavor = "multi_thread")]
637+
async fn test_regression_sort_query_fuzz() {
638+
// Regression seeds
639+
let gen_seed = 1_793_445_468;
640+
641+
// Reproducing
642+
let mut test_generator = SortFuzzerTestGenerator::new(
643+
2000,
644+
3,
645+
"sort_fuzz_table".to_string(),
646+
get_supported_types_columns(gen_seed),
647+
false,
648+
gen_seed,
649+
);
650+
651+
test_generator
652+
.fuzzer_run(
653+
17_320_197_313_368_524_478,
654+
16_774_755_103_655_697_964,
655+
1_568_822_825_954_225_201,
656+
true,
657+
)
658+
.await
659+
.unwrap();
660+
}
661+
612662
/// Given the same seed, the result should be the same
613663
#[tokio::test]
614664
async fn test_sort_query_fuzzer_deterministic() {
@@ -624,6 +674,6 @@ mod test {
624674

625675
let res1 = test_generator.fuzzer_run(1, 2, 3, false).await.unwrap();
626676
let res2 = test_generator.fuzzer_run(1, 2, 3, false).await.unwrap();
627-
assert_eq!(res1, res2);
677+
assert_batches_equal_in_order(&res1, &res2, "");
628678
}
629679
}

0 commit comments

Comments
 (0)