Skip to content

Make TLS tests more reliable#2354

Merged
casperklein merged 2 commits intodocker-mailserver:masterfrom
casperklein:tls-tests
Jan 1, 2022
Merged

Make TLS tests more reliable#2354
casperklein merged 2 commits intodocker-mailserver:masterfrom
casperklein:tls-tests

Conversation

@casperklein
Copy link
Copy Markdown
Member

@casperklein casperklein commented Jan 1, 2022

Description

The TLS tests can fail, if testssl starts scanning, while the container is not fully started yet:

Failed tests
 ✗ checking tls: cipher list - rsa intermediate [25151]
   (from function `assert_success' in file test/test_helper/bats-assert/src/assert.bash, line 114,
    from function `compare_cipherlist' in file test/security_tls_cipherlists.bats, line 164,
    from function `check_cipherlists' in file test/security_tls_cipherlists.bats, line 176,
    from function `check_ports' in file test/security_tls_cipherlists.bats, line 90,
    in test file test/security_tls_cipherlists.bats, line 47)
     `check_ports 'rsa' 'intermediate'' failed with status 5
   [ TASKLOG ]  mail.example.test is up and running

   -- command failed --
   status : 5
   output : jq: error (at /tmp/results/rsa/intermediate/port_25.json:14): Cannot iterate over null (null)
   --

   tls_test_cipherlists
 ✗ checking tls: cipher list - rsa modern [22107]
   (from function `assert_success' in file test/test_helper/bats-assert/src/assert.bash, line 114,
    from function `compare_cipherlist' in file test/security_tls_cipherlists.bats, line 164,
    from function `check_cipherlists' in file test/security_tls_cipherlists.bats, line 179,
    from function `check_ports' in file test/security_tls_cipherlists.bats, line 90,
    in test file test/security_tls_cipherlists.bats, line 51)
     `check_ports 'rsa' 'modern'' failed with status 5
   [ TASKLOG ]  mail.example.test is up and running

   -- command failed --
   status : 5
   output : jq: error (at /tmp/results/rsa/modern/port_25.json:14): Cannot iterate over null (null)
   --

   tls_test_cipherlists
 ✗ checking tls: cipher list - ecdsa intermediate [22167]
   (from function `assert_success' in file test/test_helper/bats-assert/src/assert.bash, line 114,
    from function `compare_cipherlist' in file test/security_tls_cipherlists.bats, line 164,
    from function `check_cipherlists' in file test/security_tls_cipherlists.bats, line 176,
    from function `check_ports' in file test/security_tls_cipherlists.bats, line 90,
    in test file test/security_tls_cipherlists.bats, line 55)
     `check_ports 'ecdsa' 'intermediate'' failed with status 5
   [ TASKLOG ]  mail.example.test is up and running

   -- command failed --
   status : 5
   output : jq: error (at /tmp/results/ecdsa/intermediate/port_25.json:14): Cannot iterate over null (null)
   --

   tls_test_cipherlists
 ✓ checking tls: cipher list - ecdsa modern [24607]
 ✗ checking tls: cipher list - ecdsa intermediate, with rsa fallback [29217]
   (from function `assert_success' in file test/test_helper/bats-assert/src/assert.bash, line 114,
    from function `compare_cipherlist' in file test/security_tls_cipherlists.bats, line 164,
    from function `check_cipherlists' in file test/security_tls_cipherlists.bats, line 176,
    from function `check_ports' in file test/security_tls_cipherlists.bats, line 90,
    in test file test/security_tls_cipherlists.bats, line 66)
     `check_ports 'ecdsa' 'intermediate' 'rsa'' failed with status 5
   [ TASKLOG ]  mail.example.test is up and running

   -- command failed --
   status : 5
   output : jq: error (at /tmp/results/ecdsa_rsa/intermediate/port_25.json:14): Cannot iterate over null (null)
   --

   tls_test_cipherlists
/tmp/results/rsa/intermediate/port_25.json
{
          "Invocation"  : "testssl.sh --quiet --warnings=batch --mode parallel --overwrite --preference --jsonfile-pretty port_25.json --starttls smtp example.test:25",
          "at"          : "b9aba4dd4e0c:/home/testssl/bin/openssl.Linux.x86_64",
          "version"     : "3.1dev ",
          "openssl"     : "OpenSSL 1.0.2-chacha from Jan 18 17:12:17 2019",
          "startTime"   : "1640945987",
          "scanResult"  : [
                            {
                                "id"           : "scanProblem",
                                "severity"     : "FATAL",
                                "finding"      : "Can't connect to '172.21.0.2:25' Make sure a firewall is not between you and your scanning target!"
                           }          ],
                    "scanTime"  : "Scan interrupted"
}

wait_for_finished_setup_in_container does not guarantee, that postfix is running and listening on port 25.

wait_for_tcp_port_in_container is the better choise.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Improvement (non-breaking change that does improve existing functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (README.md or the documentation under docs/)
  • If necessary I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@casperklein casperklein requested a review from a team January 1, 2022 11:10
@casperklein casperklein self-assigned this Jan 1, 2022
@casperklein casperklein added this to the v10.5.0 milestone Jan 1, 2022
@casperklein casperklein marked this pull request as ready for review January 1, 2022 11:11
@casperklein casperklein changed the title Make tls tests more reliable Make TLS tests more reliable Jan 1, 2022
@casperklein casperklein linked an issue Jan 1, 2022 that may be closed by this pull request
@casperklein casperklein merged commit 9bcc3df into docker-mailserver:master Jan 1, 2022
@casperklein casperklein deleted the tls-tests branch January 1, 2022 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Some questions about tests

3 participants