CA certificate trust stores

Secure network connections using TLS (or SSL), e.g. https:// URLs, are based on certificates issued to server operators by certificate authorities (CAs). Each program which makes these secure connections ultimately needs to have a list of CAs that are to be trusted and be able to trace the issuance of a server’s certificate back to a trusted CA’s root certificate.

This set of trusted CA root certificates is called a trust store. Every computer system has its own OS-level trust store and usually also has several other application-specific trust stores on the system. Typically these are automatically managed and do not need configuration or modification.

When your network environment includes a proxy, egress firewall, or institutional CA used internally, however, it may be necessary to configure the trust store used by an application or modify the trust store contents (e.g. adding a institutional CA certificate).

This document focuses on configuration and modification of trust stores used by software that is in turn used by Nextstrain’s software. It focuses only on changes that can be made without modification of source or program code, e.g. thru environment variables or config files.

Operating system

Linux uses OpenSSL for its system-wide trust store. macOS and Windows use their own platform-specific stores, Keychain (managed via Keychain.app) and Certificate Store (managed via certmgr.msc), respectively. OpenSSL and some other software on macOS and Windows systems (e.g. curl) is sometimes configured by default to use those platform-specific stores, but not always.

If you need to include an additional CA certificate in the trust store and that certificate applies to all software on the computer (e.g. your web browser), not just Nextstrain’s, then consider adding the CA certificate to the system-wide trust store and configuring Nextstrain’s software to use the system’s trust store instead of any application-specific ones when necessary. This blanket approach is generally more reliable over time than application-by-application configuration.

Application software

Most software provides a way to override the trust store by providing a path to a file containing the the entire set of CA certificates to trust. Thus, if you only need to include an additional CA certificate in the trust store, you should ensure that the file you provide contains a standard set of CA certificates as well as your addition. An exception here is if you know for certain that all connections will use your CA certificate and only yours (e.g. a mandatory egress proxy that removes (and adds back) TLS for inspection and policy purposes).

OpenSSL library

OpenSSL is the most common library used to provide TLS/SSL support in application software. Its default locations of trusted CA certificates can be overridden by setting the SSL_CERT_FILE and/or SSL_CERT_DIR environment variables. Filenames in the latter must be hashed with OpenSSL’s c_rehash utility.

Its final trust store is built from certificates in all default locations, so to comprehensively override the defaults, all locations must be overridden. This is typically unnecessary unless you need to ensure some CAs in the defaults aren’t trusted.

Note

Each Conda environment or Docker container has its own isolated copy of OpenSSL and its own isolated default CA certificate trust store separate from any OS-level OpenSSL trust store.

Node.js

Used by nextstrain.org and Auspice’s standalone server component.

Node.js may use either its own bundled snapshot of Mozilla’s CA trust store or the OpenSSL default CA trust store, with the default choice depending on how it was compiled.

This typically means that node from your official OS package repositories defaults to the OpenSSL default CA trust store while node from nodejs.org (e.g. via nvm), conda-forge, and other third-party places defaults to the bundled CA trust store. You can be explicit about the preferred CA trust store by including --use-openssl-ca or --use-bundled-ca in the NODE_OPTIONS environment variable or by passing those options directly in your node invocation.

To override the trust store, set the NODE_OPTIONS environment variable to include --use-openssl-ca and then set OpenSSL’s SSL_CERT_FILE or SSL_CERT_DIR environment variables.

To add CA certificates to the trust store, set the NODE_EXTRA_CA_CERTS environment variable.

Python

Uses the OpenSSL library and its defaults. See ssl.SSLContext.load_default_certs() and ssl.SSLContext.set_default_verify_paths().

requests module

Used by Nextstrain CLI and some pathogen workflows.

Uses a snapshot of Mozilla’s CA trust store via the certifi Python package.

Set the REQUESTS_CA_BUNDLE environment variable to override.

Note

If a requests-specific CA bundle isn’t configured, requests falls back to the CURL_CA_BUNDLE environment variable (if set).

Nextclade CLI

Applies to Nextclade v3.9.0 and onwards.

Uses CA certificates extracted from the OS-level trust store via the rustls-native-certs Rust crate plus its own bundled snapshot of Mozilla’s CA trust store via the webpki-roots Rust crate (by way of the reqwest crate’s rustls-tls-webpki-roots feature).

Set the OpenSSL-style SSL_CERT_FILE or SSL_CERT_DIR environment variables to override the OS-level trust store (on all platforms, not just those using OpenSSL). The bundled trust store is always included and cannot be overridden or disabled.

Set the NEXTCLADE_EXTRA_CA_CERTS environment variable to add CA certificates to the default trust store.

Note

Nextclade v3.8.2 and earlier provides no way to configure or modify the trust store.

AWS CLI

Used by some pathogen workflows.

The AWS CLI (v1 and v2), via Botocore, uses a snapshot of Mozilla’s CA trust store via the certifi Python package when it’s available, otherwise it falls back to its own bundled CA trust store.

Set the AWS_CA_BUNDLE environment variable or ca_bundle profile configuration to override.

Note

If an AWS-specific CA bundle isn’t configured, the AWS CLI falls back to the REQUESTS_CA_BUNDLE environment variable (if set).

AWS SDKs

Used by Nextstrain CLI and nextstrain.org.

The AWS SDKs for JavaScript (v2 and v3) default to the Node.js trust store.

Set the AWS_CA_BUNDLE environment variable or ca_bundle profile configuration to override.

curl

Used by many pathogen workflows.

Curl can come with support for many different TLS implementations, which means how to configure or modify the CA trust store in use can vary between copies of curl, even when they’re the same version. Refer to the curl book and curl docs for more details about which trust stores are used when and configuration basics.

Set the CURL_CA_BUNDLE environment variable to override.

Note

Curl also respects the SSL_CERT_FILE and SSL_CERT_DIR environment variables, even if the TLS implementation in use is not OpenSSL (with one exception). This is handy if you’re setting those already, as you can avoid setting CURL_CA_BUNDLE too.