Skip to content

quickCheckWithResult with low maxSuccess fails unexpectedly  #338

@robx

Description

@robx

Contrary to expectation, lowering the maxSuccess parameter may cause tests to fail:

> let p k k' = k /= k' ==> (k :: Int) /= k'
> quickCheckWithResult stdArgs p
+++ OK, passed 100 tests; 11 discarded.
> quickCheckWithResult stdArgs{maxSuccess=2} p
+++ OK, passed 2 tests; 11 discarded.
> quickCheckWithResult stdArgs{maxSuccess=1} p
*** Gave up! Passed only 0 tests; 10 discarded tests.

The explanation seems to be due to a combination of:

  • discard ratio defaults to 10, which means we allow only 10 discarded results in total for the case maxSuccess=1 (maxSuccess=2 gets 20 discards upfront)
  • computeSize doesn't ramp up the size quickly enough, it starts out [0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,2,2,2... regardless of maxSuccess parameter

Those computed sizes in particular seem a bit weird to me, but I can't say I have any kind of understanding of most of what's going on here. It does seem a bit at odds with the comment here

-- 0, 1, 2, ..., 99, 0, 1, 2, ..., 99, 0, 2, 4, ..., 98.
, which would make me expect [0,1,2,....

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions