Skip to content

Run the functional tests in a NixOS environment#10922

Merged
Ericson2314 merged 15 commits intoNixOS:masterfrom
hercules-ci:functional-tests-on-nixos
Jun 24, 2024
Merged

Run the functional tests in a NixOS environment#10922
Ericson2314 merged 15 commits intoNixOS:masterfrom
hercules-ci:functional-tests-on-nixos

Conversation

@roberth
Copy link
Member

@roberth roberth commented Jun 16, 2024 •
edited
Loading

Motivation

By running the functional tests also in a mostly normal NixOS environment, we increase our test coverage.

This reproduces

It also allows for testing

Context

  • Rewrite of most of Run test suite on NixOS #7778

  • Possible follow-ups:

    • A VM test with many features enabled
    • Some sudo action in the functional tests
    • Maybe move some stand-alone VM test scripts into the functional tests, if that's better?

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

@github-actions github-actions bot added documentation contributor-experience Developer experience for Nix contributors with-tests Issues related to testing. PRs with tests have some priority labels Jun 16, 2024
@roberth roberth mentioned this pull request Jun 16, 2024
Closed
14 tasks
@fricklerhandwerk
Copy link
Contributor

fricklerhandwerk commented Jun 17, 2024

Indeed I discussed just this sort of change with @tfc recently, and I'd take a slightly different architectural approach: instead of including environment-dependent setup/teardown, source the script itself from each environment. WDYT?

@roberth
Copy link
Member Author

roberth commented Jun 17, 2024

instead of including environment-dependent setup/teardown, source the script itself from each environment. WDYT?

The implementation and benefit of this idea are unclear to me.

I have not changed the architecture to keep the diff manageable, so I'd like to proceed with this and refactor later.

@roberth roberth mentioned this pull request Jun 17, 2024
Open
@fricklerhandwerk
Copy link
Contributor

fricklerhandwerk commented Jun 17, 2024

Discussed in Nix maintainer meeting:

  • looks good generally
  • have to figure out better patterns to set up environments as we go
  • eventually we'd also want the noGC etc. outputs separately as well, currently they're only in Hydra jobs

edolstra
edolstra reviewed Jun 17, 2024
View reviewed changes
@Ericson2314 Ericson2314 enabled auto-merge June 17, 2024 12:18
@nixos-discourse
Copy link

nixos-discourse commented Jun 18, 2024

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-06-17-nix-team-meeting-minutes-153/47186/1

@roberth roberth force-pushed the functional-tests-on-nixos branch from 3b4c254 to 18f602d Compare June 19, 2024 19:47
fricklerhandwerk
fricklerhandwerk approved these changes Jun 19, 2024
View reviewed changes
roberth and others added 10 commits June 20, 2024 14:54
@roberth
flake.nix: Factor pkgs.nix_noTests out of buildNoTests
dc720f8 
This is useful when iterating on the functional tests when trying
to run them in a VM test, for example.
@roberth
tests: Add hydraJobs.tests.functional_*
439022c
@roberth
tests/functional/timeout.sh: Find missing test case
211aec4 
This reproduces an instance of
NixOS#4813
@roberth
tests/functional: Skip tests that don't work in NixOS environment yet
8557d79
@roberth
tests: Add quickBuild to all VM tests
d208e9d
@roberth
doc/contributing/testing: Describe functional VM tests and quickBuild
fca160f
@roberth
ci: Build tests.functional_user for PRs
f0abe4d
@roberth
tests/functional: Enable more tests in NixOS VM
648302b
@roberth @edolstra
tests/functional/common/init.sh: Use parentheses around negation
7c9f3ee 
roberth: Not strictly necessary, but probably a good habit

Co-authored-by: Eelco Dolstra <[email protected]>
@roberth @fricklerhandwerk
Apply suggestions from code review
dcee46a 
Co-authored-by: Valentin Gagarin <[email protected]>
@roberth roberth force-pushed the functional-tests-on-nixos branch from 83871c8 to dcee46a Compare June 20, 2024 13:00
@fricklerhandwerk
Copy link
Contributor

fricklerhandwerk commented Jun 20, 2024

@roberth IMO you can self-merge once it's all green.

@roberth
Revert "tests/functional/common/init.sh: Use parentheses around negat…
d968466 
…ion"

ShellCheck doesn't want us to add extra parentheses for show.

This reverts commit 7c9f3ee.
@roberth roberth force-pushed the functional-tests-on-nixos branch from cb96b87 to d968466 Compare June 20, 2024 20:32
Ericson2314
Ericson2314 reviewed Jun 23, 2024
View reviewed changes
tests/functional/common/vars-and-functions.sh
Comment on lines 13 to 16
die() {
echo "fatal error: $*" >&2
exit 1
}
Copy link
Member

@Ericson2314 Ericson2314 Jun 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this like fail?

Copy link
Member Author

@roberth roberth Jun 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't aware, but it is similar. fail doesn't prefix anything.
They do seem to differ a bit in their use cases, which suggests that their behavior could be differentiated more. Perhaps:

  • die: echo "fatal error in test setup: $*"
  • fail: echo "test failed: $*"

@roberth roberth force-pushed the functional-tests-on-nixos branch from d70a198 to bacd9fc Compare June 24, 2024 17:05
@roberth roberth force-pushed the functional-tests-on-nixos branch from bacd9fc to 445a4a0 Compare June 24, 2024 17:05
@Ericson2314 Ericson2314 merged commit 5c497a9 into NixOS:master Jun 24, 2024
@roberth
Copy link
Member Author

roberth commented Jun 24, 2024

The extra swap appears to handle the extra evaluations with no significant slowdown 🎉

@nixos-discourse
Copy link

nixos-discourse commented Jun 26, 2024

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-06-24-nix-team-meeting-minutes-155/47739/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Ericson2314 Ericson2314 Ericson2314 approved these changes

@edolstra edolstra edolstra left review comments

+1 more reviewer

@fricklerhandwerk fricklerhandwerk fricklerhandwerk approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

contributor-experience Developer experience for Nix contributors documentation with-tests Issues related to testing. PRs with tests have some priority

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@roberth @fricklerhandwerk @nixos-discourse @Ericson2314 @edolstra