GHA/macos: enable HTTPS tests with stunnel by AkiSakurai · Pull Request #14486 · curl/curl

AkiSakurai · 2024-08-11T02:43:54Z

Install Stunnel.
Regenerate Certificates (as SecureTransport requires a validity period less than 398 days).
Restart Server if It Is Unresponsive.
Do Not Hardcode the SHA-256 Base64 Public Pinned Key.
Ignore test 313 as SecureTransport does not support crl file.
Ignore tests 1631 and 1632 as SecureTransport is not yet able to shut down FTP over HTTPS gracefully.
Add a CMake Target for generating certificates.

.github/workflows/macos.yml

tests/certs/CMakeLists.txt

tests/certs/Makefile.inc

tests/CMakeLists.txt

.github/workflows/macos.yml

tests/certs/Makefile.am

.github/workflows/macos.yml

tests/certs/CMakeLists.txt

vszakats · 2024-08-12T17:35:18Z

Thank you @AkiSakurai for your updates!

I left one change suggestion regarding Makefile.inc and two tiny ones.

There were these test failures, then gone after a restart:
FAIL 2041: 'simple HTTPS GET with base64-sha256 public key pinning' HTTPS, HTTP GET, PEM certificate
https://github.com/curl/curl/actions/runs/10355416534/job/28663015869?pr=14486#step:15:4340

FAIL 515: 'make a POSTFIELDS set to NULL with POSTFIELDSIZE set to zero' HTTP, HTTP POST
https://github.com/curl/curl/actions/runs/10355416534/job/28663047488?pr=14486

FAIL 2037: 'simple HTTPS GET with PEM public key pinning' HTTPS, HTTP GET, PEM certificate
https://github.com/curl/curl/actions/runs/10355416534/job/28663025351?pr=14486#step:12:3828

FAIL 1429: 'HTTP GET with 999 response code' HTTP, HTTP GET, HTTP/0.9
https://github.com/curl/curl/actions/runs/10355416534/job/28663040268?pr=14486#step:12:3647

dfandrich · 2024-08-12T18:33:57Z

• Regenerate Certificates (as SecureTransport requires a validity period less than 398 days).

This is going to be a problem. This implies that someone is going to have to manually regenerate the certificates in the curl repo at least once a year? Which, if not done, will cause not just macOS but ALL targets to fail? Which will furthermore cause anyone running a curl test suite from a release that's more than a year old (at most) also to fail? This just isn't going to fly. Especially since SecureTransport seems to be on its way out.

vszakats · 2024-08-12T19:18:37Z

We can do the regeneration on each run, for Secure Transport jobs.
And keep the original long-running certs in repo.

tests/certs/CMakeLists.txt

.github/workflows/macos.yml

- Install Stunnel. - Regenerate Certificates (as SecureTransport requires a validity period less than 398 days). - Restart Server if It Is Unresponsive. - Do Not Hardcode the SHA-256 Base64 Public Pinned Key. - Ignore test 313 as SecureTransport does not support crl file. - Ignore tests 1631 and 1632 as SecureTransport is not yet able to shut down FTP over HTTPS gracefully. - Add a CMake Target for generating certificates.

Co-authored-by: Viktor Szakats <[email protected]>

vszakats · 2024-08-23T09:07:55Z

Thank you @AkiSakurai, merged!

- FAIL 2037: 'simple HTTPS GET with PEM public key pinning' HTTPS, HTTP GET, PEM certificate https://github.com/curl/curl/actions/runs/10529314166/job/29176801513?pr=14658 - FAIL 2041: 'simple HTTPS GET with base64-sha256 public key pinning' HTTPS, HTTP GET, PEM certificate https://github.com/curl/curl/actions/runs/10529181645/job/29176408976#step:16:4363 https://github.com/curl/curl/actions/runs/10529019749/job/29175882809#step:13:5235 https://github.com/curl/curl/actions/runs/10528920874/job/29175559070#step:16:4363 https://github.com/curl/curl/actions/runs/10528759817/job/29175043048#step:16:4274 https://github.com/curl/curl/actions/runs/10525250422/job/29163662307#step:16:4366 https://github.com/curl/curl/actions/runs/10525250422/job/29163662602#step:13:3569 Happened with cmake, autotools, gcc, clang and various TLS backends. Follow-up to fa461b4 curl#14486

- FAIL 2037: 'simple HTTPS GET with PEM public key pinning' HTTPS, HTTP GET, PEM certificate https://github.com/curl/curl/actions/runs/10529314166/job/29176801513?pr=14658 - FAIL 2041: 'simple HTTPS GET with base64-sha256 public key pinning' HTTPS, HTTP GET, PEM certificate https://github.com/curl/curl/actions/runs/10529181645/job/29176408976#step:16:4363 https://github.com/curl/curl/actions/runs/10529019749/job/29175882809#step:13:5235 https://github.com/curl/curl/actions/runs/10528920874/job/29175559070#step:16:4363 https://github.com/curl/curl/actions/runs/10528759817/job/29175043048#step:16:4274 https://github.com/curl/curl/actions/runs/10525250422/job/29163662307#step:16:4366 https://github.com/curl/curl/actions/runs/10525250422/job/29163662602#step:13:3569 Happened with cmake, autotools, gcc, clang and various TLS backends. Follow-up to fa461b4 #14486 Closes #14667

Fixes intermittend CI failure because the shell-based random generator generated the same number twice: ``` $ openssl ca -config EdelCurlRoot-ca.cnf -revoke Server-localhost0h-sv.crt Using configuration from EdelCurlRoot-ca.cnf ERROR:Already revoked, serial number 66FDB23A make: *** [../../../tests/certs/Server-localhost0h-sv.pem] Error 1 ``` https://github.com/curl/curl/actions/runs/11151401083/job/30994755798?pr=15128#step:10:498 Follow-up to fa461b4 curl#14486 Follow-up to fa69b41 curl#13307

Generate the certificate serial numbers automatically instead of doing from shell (or Perl earlier). Fixes intermittent CI failures due to the shell-based random generator generating the same serial number twice: ``` $ openssl ca -config EdelCurlRoot-ca.cnf -revoke Server-localhost0h-sv.crt Using configuration from EdelCurlRoot-ca.cnf ERROR:Already revoked, serial number 66FDB23A make: *** [../../../tests/certs/Server-localhost0h-sv.pem] Error 1 ``` https://github.com/curl/curl/actions/runs/11151401083/job/30994755798?pr=15128#step:10:498 Follow-up to fa461b4 #14486 Follow-up to fa69b41 #13307 Closes #15129

https://github.com/curl/curl/actions/runs/13683996410/job/38262956317 with LibreSSL https://github.com/curl/curl/actions/runs/13680682695/job/38252047077 with wolfSSL Following GHA/macos. Ref: fa461b4 curl#14486

We already marked them flaky in GHA/macos CI. They are also flaky in other CI jobs, in other OSes, with multiple TLS backends: - MSVC/LibreSSL: https://github.com/curl/curl/actions/runs/13683996410/job/38262956317 - MSVC/wolfSSL: https://github.com/curl/curl/actions/runs/13680682695/job/38252047077 - FreeBSD/OpenSSL3: https://github.com/curl/curl/actions/runs/13690910863/job/38283867721#step:3:1 Ref: fa461b4 #14486 Closes #16584

Before this patch the curl repository and source tarball distribution contained test certificates as binary blobs. Used by runtests. Drop these certificates in favor of generating them dynamically as part of the build process. Both via autotools and CMake. As part of this, improve certificates, the generator script and process, file layout, and fix any issue to make it work fast and smooth both in CI and local builds. Note, cert generator scripts require OpenSSL >=1.0.2 (or LibreSSL >=3.1.0). Generation requires POSIX shell, also with CMake. Without a POSIX shell tests relying on TLS (and stunnel) will fail. Details: - build: generate certs as part of the test run process. - build, tests: generate certs in the build directory. - binarycheck: drop concept of known binary files with hashes. - binarycheck: move binary check logic into spacecheck and drop this separate checker tool. - build: fix to clean all cert files. - autotools: fix to not run leaf cert generators in parallel. To avoid confusion when updating the revocation database and counter. - scripts: drop `scripts` subdir, merge two scripts into one, auto-generate root cert, allow generating multiple leafs at once. - scripts: switch to EC-256 keys (was: RSA-2048). For key size and perf. - scripts: drop `-x` echo, text dumps, most other output. To avoid log noise and make it quicker in CI. - scripts: make it non-RSA-specific. - scripts: delete unused code. - scripts: use POSIX shell shebang. Some envs don't have bash (Alpine). - scripts: pass test pseudo-secrets via the command-line. To avoid: ``` + openssl genrsa -out test-ca.key -passout fd:0 2048 Invalid password argument, starting with "fd:" ``` - cmake: fix to launch generator scripts via the detected POSIX shell. - cmake: fix `build-certs` rule to not depend on `SRPFILES` (`srp-verifier-*`). - cmake: drop `EXCLUDE_FROM_ALL` for the cert subdir. It makes the Visual Studio generator miss to create the `clean-certs`, `build-certs` targets. No target depend on them, so they don't execute implicitly anyway. Fixes: ``` MSBUILD : error MSB1009: Project file does not exist. Switch: clean-certs.vcxproj ``` - cmake: add `VERBATIM USES_TERMINAL` to `build-certs` target. - GHA/linux: install openssl on Alpine, for the cert generator scripts. Follow-up to 556f722 #16593 Follow-up to fa461b4 #14486 Closes #16824

Generate the certificate serial numbers automatically instead of doing from shell (or Perl earlier). Fixes intermittent CI failures due to the shell-based random generator generating the same serial number twice: ``` $ openssl ca -config EdelCurlRoot-ca.cnf -revoke Server-localhost0h-sv.crt Using configuration from EdelCurlRoot-ca.cnf ERROR:Already revoked, serial number 66FDB23A make: *** [../../../tests/certs/Server-localhost0h-sv.pem] Error 1 ``` https://github.com/curl/curl/actions/runs/11151401083/job/30994755798?pr=15128#step:10:498 Follow-up to fa461b4 curl#14486 Follow-up to fa69b41 curl#13307 Closes curl#15129

We already marked them flaky in GHA/macos CI. They are also flaky in other CI jobs, in other OSes, with multiple TLS backends: - MSVC/LibreSSL: https://github.com/curl/curl/actions/runs/13683996410/job/38262956317 - MSVC/wolfSSL: https://github.com/curl/curl/actions/runs/13680682695/job/38252047077 - FreeBSD/OpenSSL3: https://github.com/curl/curl/actions/runs/13690910863/job/38283867721#step:3:1 Ref: fa461b4 curl#14486 Closes curl#16584

Before this patch the curl repository and source tarball distribution contained test certificates as binary blobs. Used by runtests. Drop these certificates in favor of generating them dynamically as part of the build process. Both via autotools and CMake. As part of this, improve certificates, the generator script and process, file layout, and fix any issue to make it work fast and smooth both in CI and local builds. Note, cert generator scripts require OpenSSL >=1.0.2 (or LibreSSL >=3.1.0). Generation requires POSIX shell, also with CMake. Without a POSIX shell tests relying on TLS (and stunnel) will fail. Details: - build: generate certs as part of the test run process. - build, tests: generate certs in the build directory. - binarycheck: drop concept of known binary files with hashes. - binarycheck: move binary check logic into spacecheck and drop this separate checker tool. - build: fix to clean all cert files. - autotools: fix to not run leaf cert generators in parallel. To avoid confusion when updating the revocation database and counter. - scripts: drop `scripts` subdir, merge two scripts into one, auto-generate root cert, allow generating multiple leafs at once. - scripts: switch to EC-256 keys (was: RSA-2048). For key size and perf. - scripts: drop `-x` echo, text dumps, most other output. To avoid log noise and make it quicker in CI. - scripts: make it non-RSA-specific. - scripts: delete unused code. - scripts: use POSIX shell shebang. Some envs don't have bash (Alpine). - scripts: pass test pseudo-secrets via the command-line. To avoid: ``` + openssl genrsa -out test-ca.key -passout fd:0 2048 Invalid password argument, starting with "fd:" ``` - cmake: fix to launch generator scripts via the detected POSIX shell. - cmake: fix `build-certs` rule to not depend on `SRPFILES` (`srp-verifier-*`). - cmake: drop `EXCLUDE_FROM_ALL` for the cert subdir. It makes the Visual Studio generator miss to create the `clean-certs`, `build-certs` targets. No target depend on them, so they don't execute implicitly anyway. Fixes: ``` MSBUILD : error MSB1009: Project file does not exist. Switch: clean-certs.vcxproj ``` - cmake: add `VERBATIM USES_TERMINAL` to `build-certs` target. - GHA/linux: install openssl on Alpine, for the cert generator scripts. Follow-up to 556f722 curl#16593 Follow-up to fa461b4 curl#14486 Closes curl#16824

github-actions bot added tests CI Continuous Integration labels Aug 11, 2024

vszakats requested changes Aug 11, 2024

View reviewed changes

vszakats reviewed Aug 12, 2024

View reviewed changes

tests/certs/Makefile.am Outdated Show resolved Hide resolved

vszakats reviewed Aug 12, 2024

View reviewed changes

.github/workflows/macos.yml Outdated Show resolved Hide resolved

vszakats reviewed Aug 12, 2024

View reviewed changes

tests/certs/CMakeLists.txt Outdated Show resolved Hide resolved

AkiSakurai changed the title ~~tests: Enable HTTPS CI Test on macOS~~ GHA/macos: enable HTTPS tests with stunnel Aug 14, 2024

vszakats reviewed Aug 14, 2024

View reviewed changes

tests/certs/CMakeLists.txt Outdated Show resolved Hide resolved

vszakats reviewed Aug 14, 2024

View reviewed changes

.github/workflows/macos.yml Outdated Show resolved Hide resolved

vszakats reviewed Aug 14, 2024

View reviewed changes

.github/workflows/macos.yml Outdated Show resolved Hide resolved

AkiSakurai and others added 5 commits August 17, 2024 13:35

Update tests/certs/CMakeLists.txt

734cab0

Co-authored-by: Viktor Szakats <[email protected]>

Apply suggestions from code review

97c57de

Co-authored-by: Viktor Szakats <[email protected]>

make sure no locks left for responsive test

599d374

handle windows style newline

8db9a27

vszakats approved these changes Aug 19, 2024

View reviewed changes

vszakats added the appleOS specific to an Apple operating system label Aug 20, 2024

vszakats closed this in fa461b4 Aug 23, 2024

vszakats mentioned this pull request Aug 23, 2024

GHA/macos: ignore flaky tests 2041 and 2037 #14667

Closed

vszakats mentioned this pull request Oct 2, 2024

tests: let openssl generate random cert serials #15129

Closed

vszakats mentioned this pull request Mar 6, 2025

tests: mark tests 1631, 1632 flaky #16584

Closed

1 task

vszakats mentioned this pull request Mar 25, 2025

runtests: generate certs dynamically, bump to EC-256, tidy up #16824

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

GHA/macos: enable HTTPS tests with stunnel #14486

GHA/macos: enable HTTPS tests with stunnel #14486
AkiSakurai wants to merge 5 commits intocurl:masterfrom
AkiSakurai:macos

AkiSakurai commented Aug 11, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vszakats commented Aug 12, 2024

Uh oh!

dfandrich commented Aug 12, 2024 via email

Uh oh!

vszakats commented Aug 12, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vszakats commented Aug 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

AkiSakurai commented Aug 11, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vszakats commented Aug 12, 2024

Uh oh!

dfandrich commented Aug 12, 2024 via email

Uh oh!

vszakats commented Aug 12, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vszakats commented Aug 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants