Skip to content

Some questions about tests #2350

@casperklein

Description

@casperklein
  1. I see a lot of these lines ok 240 first # skip This version natively supports setup/teardown_file. Does this mean, the dummy tests can be removed?

  2. When running the tests locally, some of them are always failing:

Details
 ✗ checking tls: cipher list - rsa intermediate [25151]
   (from function `assert_success' in file test/test_helper/bats-assert/src/assert.bash, line 114,
    from function `compare_cipherlist' in file test/security_tls_cipherlists.bats, line 164,
    from function `check_cipherlists' in file test/security_tls_cipherlists.bats, line 176,
    from function `check_ports' in file test/security_tls_cipherlists.bats, line 90,
    in test file test/security_tls_cipherlists.bats, line 47)
     `check_ports 'rsa' 'intermediate'' failed with status 5
   [ TASKLOG ]  mail.example.test is up and running

   -- command failed --
   status : 5
   output : jq: error (at /tmp/results/rsa/intermediate/port_25.json:14): Cannot iterate over null (null)
   --

   tls_test_cipherlists
 ✗ checking tls: cipher list - rsa modern [22107]
   (from function `assert_success' in file test/test_helper/bats-assert/src/assert.bash, line 114,
    from function `compare_cipherlist' in file test/security_tls_cipherlists.bats, line 164,
    from function `check_cipherlists' in file test/security_tls_cipherlists.bats, line 179,
    from function `check_ports' in file test/security_tls_cipherlists.bats, line 90,
    in test file test/security_tls_cipherlists.bats, line 51)
     `check_ports 'rsa' 'modern'' failed with status 5
   [ TASKLOG ]  mail.example.test is up and running

   -- command failed --
   status : 5
   output : jq: error (at /tmp/results/rsa/modern/port_25.json:14): Cannot iterate over null (null)
   --

   tls_test_cipherlists
 ✗ checking tls: cipher list - ecdsa intermediate [22167]
   (from function `assert_success' in file test/test_helper/bats-assert/src/assert.bash, line 114,
    from function `compare_cipherlist' in file test/security_tls_cipherlists.bats, line 164,
    from function `check_cipherlists' in file test/security_tls_cipherlists.bats, line 176,
    from function `check_ports' in file test/security_tls_cipherlists.bats, line 90,
    in test file test/security_tls_cipherlists.bats, line 55)
     `check_ports 'ecdsa' 'intermediate'' failed with status 5
   [ TASKLOG ]  mail.example.test is up and running

   -- command failed --
   status : 5
   output : jq: error (at /tmp/results/ecdsa/intermediate/port_25.json:14): Cannot iterate over null (null)
   --

   tls_test_cipherlists
 ✓ checking tls: cipher list - ecdsa modern [24607]
 ✗ checking tls: cipher list - ecdsa intermediate, with rsa fallback [29217]
   (from function `assert_success' in file test/test_helper/bats-assert/src/assert.bash, line 114,
    from function `compare_cipherlist' in file test/security_tls_cipherlists.bats, line 164,
    from function `check_cipherlists' in file test/security_tls_cipherlists.bats, line 176,
    from function `check_ports' in file test/security_tls_cipherlists.bats, line 90,
    in test file test/security_tls_cipherlists.bats, line 66)
     `check_ports 'ecdsa' 'intermediate' 'rsa'' failed with status 5
   [ TASKLOG ]  mail.example.test is up and running

   -- command failed --
   status : 5
   output : jq: error (at /tmp/results/ecdsa_rsa/intermediate/port_25.json:14): Cannot iterate over null (null)
   --

   tls_test_cipherlists

I use this to run the tests on my debian 11 host:

git clone --recurse-submodules https://github.com/docker-mailserver/docker-mailserver dms
cd dms
make all

Any idea, why these tests fail locally?

  1. I noticed, that this tests can take some seconds to gracefully stop the containers. But if they are not being reused, it should be safe to stop them "faster" by using docker stop -t 0 or docker rm -f:

@test "checking that the container stops cleanly: mail_override_hostname" {
run docker stop -t 60 mail_override_hostname
assert_success
}
@test "checking that the container stops cleanly: mail_non_subdomain_hostname" {
run docker stop -t 60 mail_non_subdomain_hostname
assert_success
}
@test "checking that the container stops cleanly: mail_srs_domainname" {
run docker stop -t 60 mail_srs_domainname
assert_success
}
@test "checking that the container stops cleanly: mail_domainname" {
run docker stop -t 60 mail_domainname
assert_success
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/questionSomeone asked a question - feel free to answermeta/help wantedThe OP requests help from others - chime in! :Dmeta/needs triageThis issue / PR needs checks and verification from maintainerspriority/low

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions