Skip to content

Comments

WIP: Try using bats for running lib/tests/modules.sh#185022

Draft
infinisil wants to merge 3 commits intoNixOS:masterfrom
infinisil:bats-testing
Draft

WIP: Try using bats for running lib/tests/modules.sh#185022
infinisil wants to merge 3 commits intoNixOS:masterfrom
infinisil:bats-testing

Conversation

@infinisil
Copy link
Member

Description of changes

This is an initial attempt at using bats to run lib/tests/modules.sh (but hopefully more in the future). By using bats we don't have to hand-roll our testing setup, it can take care of things like capturing exit status, command output, counting and reporting test failures. It also allows easily restructuring tests into a directory structure instead of having them all in a single file. It can also give names to tests.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.11 Release Notes (or backporting 22.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Aug 3, 2022
- init bats-assert 2.0.0
- init bats-support 0.3.0
- init bats-file 0.3.0
Adds a `bats.withLibraries (p: [ ... ])` function, which creates a
`bats` wrapper where the `BATS_LIB_PATH` environment variable contains
fallbacks for the given list of libraries.

This allows to e.g. use the `bats-assert` library (which itself requires the
`bats-support` library) with

    bats.withLibraries (p: [ p.bats-support p.bats-assert ])

In a `.bats` file you can then call `bats_load_library` [1] to load the
libraries in the `setup()` function:

    setup() {
      bats_load_library bats-support
      bats_load_library bats-assert
    }

[1]: https://bats-core.readthedocs.io/en/stable/writing-tests.html?highlight=library#bats-load-library-load-system-wide-libraries
@abathur
Copy link
Member

abathur commented Aug 9, 2022

I have a little custom code in resholve's bats helpers to write test cases like these:

@test "allow (but do not parse) --fake 'source:path'" {
  require <({
    status 5
    line 3 ends "Unexpected absolute source path."
  })
} <<CASES
resholve --interpreter $INTERP --fake 'source:/not/a/real/script' < absolute_source.sh
CASES

@test "allow (*and* do not parse) --keep 'source:path' + --fake 'source:path'" {
  require <({
    status 0
    line 2 equals "source /not/a/real/script"
  })
} <<CASES
resholve --interpreter $INTERP --keep 'source:/not/a/real/script' --fake 'source:/not/a/real/script' < absolute_source.sh
resholve --interpreter $INTERP --keep '.:/not/a/real/script' --fake 'source:/not/a/real/script' < absolute_source.sh
CASES

@test "objects to unexempted tilde executable paths" {
  require <({
    status 7
    line -1 equals "[ stdinNone ]:2: Can't resolve dynamic command"
  })
} <<CASES
resholve --interpreter $INTERP < $PWD/tests/tilde_dynamic_pipeline.sh
CASES

I don't see any material improvements over what you're already getting from the assert library so far (and I haven't tried to support regex), but I could look at splitting it out if it's meaningful.

@infinisil infinisil added the 6.topic: module system About "NixOS" module system internals label Sep 28, 2023
@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: module system About "NixOS" module system internals 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux.

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants