I wonder if its worth a tracking issue to make a GHA cron job that opens a PR updating these every month we can just merge periodically.

First, I'd just like to get this working... Not sure what I'm missing here. Definitely a pain in the ass.

I checked out the branch locally and all the tests pass on macOS and Windows (I noticed CI hadn't tried those yet) so the failures seem odd to just be an issue on Android and Linux 🤔.

I'm on mobile and can't verify at the moment, but is it possible the intermediates need updating too?

IIRC the update script is only grabbing the leaf, and at least for the LE testdata they rolled out a change to rotate intermediates that is likely making this more painful. The platform differences might be related to AIA chasing (?)

I'm on mobile and can't verify at the moment, but is it possible the intermediates need updating too?

That does appear to be the case for the letsencrypt_org_1.crt update. That cert is issued by E5 but we have vendored E6 as letsencrypt_org_valid_2.crt :

❯ git rev-parse HEAD
12fb2ff26312aef292c7c81d7447ac64c39a5f6d

❯ openssl x509 -in rustls-platform-verifier/src/tests/verification_real_world/letsencrypt_org_valid_1.crt -noout -issuer
issuer=C=US, O=Let's Encrypt, CN=E5

❯ openssl x509 -in rustls-platform-verifier/src/tests/verification_real_world/letsencrypt_org_valid_2.crt -noout -subject
subject=C=US, O=Let's Encrypt, CN=E6

The issuer of _1 and the subject of _2 look like they match for the 1password_com_valid cases.

I rewrote the update script to be a lot simpler. However, it runs into some issues:

connecting to 1password.com...
wrote certificate to /Users/djc/src/rustls-platform-verifier/rustls-platform-verifier/src/tests/verification_real_world/1password_com_valid_1.crt
wrote certificate to /Users/djc/src/rustls-platform-verifier/rustls-platform-verifier/src/tests/verification_real_world/1password_com_valid_2.crt
connecting to agilebits.com...
failed to write to agilebits.com: received fatal alert: HandshakeFailure
no certificates received for agilebits.com
connecting to letsencrypt.org...
wrote certificate to /Users/djc/src/rustls-platform-verifier/rustls-platform-verifier/src/tests/verification_real_world/letsencrypt_org_valid_1.crt
wrote certificate to /Users/djc/src/rustls-platform-verifier/rustls-platform-verifier/src/tests/verification_real_world/letsencrypt_org_valid_2.crt

The old script connected to agilebits.com and seemed to work -- and it connected to my.1password.com which got a different cert with a wildcard, it looks like? It all seems like a bit of a mess.

We can just ditch the agilebits.com host IMO. There wasn't a specific reason it was used internally AFAIK. It seems like infrastructure changes are just making it not worth its weight. Should we switch it to something with a Google CA, maybe? That'd give us coverage of LE, AWS, and Google issued certificates.