Describe the bug

We have become accustomed to some tests failing every now and then.
Making sweeping changes is harder than necessary because we can't trust the outcomes of tests.

Flaky tests are a nuisance to everyone who invests good work to get changes through and have the side-effect of wearing down motivation to painstaking.

Steps To Reproduce

Steps to reproduce the behavior:

1. Make a mass-rebuild change or a change to the test driver
2. Some tests fail inexplicably. What now?

Expected behavior

Flaky tests can be marked as such, so that they can still be fixed and don't bog anyone down.

See also earlier short discussion:

• nixos/doc: Add Developing the Test Driver #216660

Some requirements for the marking solution:

• hydra puts the flaky tests in a separate attrset for easy recognition
• marking / unmarking is a one-line change; no file moving churn
• evaluation does not suffer: no attribute set filtering, but rather mapAttrs to set some tests to null (and only on Hydra)
• pkgs.nixosTests attrset works the same regardless of flakiness status
• pkgs.nixosTests prints a warning when evaluation or running a known-flaky test
• update the manual; change what was added in nixos/doc: Add Developing the Test Driver #216660

Additional context

• nixos/doc: Add Developing the Test Driver #216660

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here