Skip to content

tests: disabled "quota exceeded" test#2511

Merged
georglauterbach merged 3 commits intomasterfrom
tests/disable-quota-test
Apr 2, 2022
Merged

tests: disabled "quota exceeded" test#2511
georglauterbach merged 3 commits intomasterfrom
tests/disable-quota-test

Conversation

@georglauterbach
Copy link
Copy Markdown
Member

@georglauterbach georglauterbach commented Mar 27, 2022

Description

The "quota exceeded" test is unreliable and failed too often lately for my taste. Therefore, I'd like to disable it because there is no use in having such a test.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (README.md or the documentation under docs/)
  • If necessary I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@georglauterbach
disabled unreliable test
82969a5 
The "quota exceeded" test is unreliable and failed too often lately for
my taste. Therefore, I'd like to disable it because there is no use in
having such a test.
@georglauterbach georglauterbach added this to the v11.0.0 milestone Mar 27, 2022
@georglauterbach georglauterbach self-assigned this Mar 27, 2022
wernerfred
wernerfred reviewed Apr 1, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@wernerfred wernerfred left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't removing a test the wrong way to deal with an unreliable test? Why is it flaky? How can we make sure it will produce the correct output repeatedly?

@georglauterbach
Copy link
Copy Markdown
Member Author

georglauterbach commented Apr 1, 2022

Isn't removing a test the wrong way to deal with an unreliable test? Why is it flaky? How can we make sure it will produce the correct output repeatedly?

Sure, this is not the most optimal way, but I cannot put more time into investigating the issue at the moment. Therefore, not runner the test (i.e. skipping it) is better than having an unreliable test IMHO...

@polarathene
Copy link
Copy Markdown
Member

polarathene commented Apr 1, 2022 •
edited
Loading

  • We have several tests disabled presently because they're not deterministic.
  • Fixing them would be ideal, but we're all short for time to try debug these due to their difficulty to reproduce consistently.
  • As they're not reliable for sustained periods, we don't consider them useful to have enabled, but retain by skipping so future maintenance can try enabling in future (may resolve itself, or maintainers have more time to investigate).
  • With a bit of luck, a bug report would get raised that is a real issue which the test covered but is reproducible 😅

And of course, once I start contributing again, I will try to refactor the test suite + some changes like reverting file locking to flock. These tests may be better resolved at that point :)

wernerfred
wernerfred approved these changes Apr 2, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@wernerfred wernerfred left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx for explanation

@georglauterbach georglauterbach merged commit 04e452a into master Apr 2, 2022
@georglauterbach georglauterbach deleted the tests/disable-quota-test branch April 2, 2022 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@polarathene polarathene polarathene approved these changes

@wernerfred wernerfred wernerfred approved these changes

@casperklein casperklein Awaiting requested review from casperklein

Assignees

@georglauterbach georglauterbach

Labels

area/ci

Projects

None yet

Milestone

v11.0.0

Development

Successfully merging this pull request may close these issues.

3 participants

@georglauterbach @polarathene @wernerfred