Skip to content

SSH key priority / failover on CircleCI+GitHub #10280

Description

@crepererum

Problem

On 2022-01-11 https://github.blog/2021-09-01-improving-git-protocol-security-github/ was activated (brownout). On CircleCI the builtin ~/.ssh/known_hosts looks something like this:

github.com ssh-rsa <stripped>
github.com ecdsa-sha2-nistp256 <stripped>
github.com ssh-ed25519 <stripped>
...

On top it also provides ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub. Funnily, this is NOT the same key as in known_hosts:

$ ssh-keygen -l -f ~/.ssh/id_rsa
4096 SHA256:Yju3nzv/JuTvvHNUD... /home/rust/.ssh/id_rsa.pub (RSA)
$ ssh-keygen -l -f ~/.ssh/known_hosts
2048 SHA256:nThbg6kXUpJ... github.com (RSA)
256 SHA256:p2QAMXNIC1T... github.com (ECDSA)
256 SHA256:+DiY3wvvV6T... github.com (ED25519)
...

Basically every cargo command (e.g. cargo build) now fails with:

$ cargo build
    Updating crates.io index
error: failed to get `async-trait` as a dependency of package `minikafka v0.1.0 (/home/rust/project)`

Caused by:
  failed to load source for dependency `async-trait`

Caused by:
  Unable to update registry `crates-io`

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to authenticate when downloading repository: ssh://[email protected]/rust-lang/crates.io-index

  * attempted ssh-agent authentication, but no usernames succeeded: `git`

  if the git CLI succeeds then `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
  ERROR: You're using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type.
  Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.

  ; class=Ssh (23); code=Eof (-20)

The CLI-driven SSH works just fine:

$ CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build
   Updating crates.io index
 Downloading crates ...
  Downloaded syn v1.0.85
   Compiling libc v0.2.112
   Compiling proc-macro2 v1.0.36
...

I'm not sure if this is a cargo or libgit2 issue though.

Steps

No response

Possible Solution(s)

As a workaround set CARGO_NET_GIT_FETCH_WITH_CLI=true, see https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli.

Notes

It seems that CLI SSH uses ~/.ssh/id_rsa which is 4096 bits and NOT affected by the brownout:

Details
$ ssh -v [email protected]
OpenSSH_8.4p1 Debian-5, OpenSSL 1.1.1k  25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to github.com [140.82.112.4] port 22.
debug1: Connection established.
debug1: identity file /home/rust/.ssh/id_rsa type 0
debug1: identity file /home/rust/.ssh/id_rsa-cert type -1
debug1: identity file /home/rust/.ssh/id_dsa type -1
debug1: identity file /home/rust/.ssh/id_dsa-cert type -1
debug1: identity file /home/rust/.ssh/id_ecdsa type -1
debug1: identity file /home/rust/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/rust/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/rust/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/rust/.ssh/id_ed25519 type -1
debug1: identity file /home/rust/.ssh/id_ed25519-cert type -1
debug1: identity file /home/rust/.ssh/id_ed25519_sk type -1
debug1: identity file /home/rust/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/rust/.ssh/id_xmss type -1
debug1: identity file /home/rust/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Debian-5
debug1: Remote protocol version 2.0, remote software version babeld-d2b829ee
debug1: no match: babeld-d2b829ee
debug1: Authenticating to github.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:p2QAMXNIC...
debug1: Host 'github.com' is known and matches the ECDSA host key.
debug1: Found key in /home/rust/.ssh/known_hosts:2
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/rust/.ssh/id_rsa RSA SHA256:Yju3nzv/JuTvvHNUDXeFF88TuhpDqDKEKsTxPnSqfIg agent
debug1: Will attempt key: /home/rust/.ssh/id_dsa
debug1: Will attempt key: /home/rust/.ssh/id_ecdsa
debug1: Will attempt key: /home/rust/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/rust/.ssh/id_ed25519
debug1: Will attempt key: /home/rust/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/rust/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<[email protected],[email protected],[email protected],[email protected],[email protected],sk-ecdsa-sha2-nistp
[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-
nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/rust/.ssh/id_rsa RSA SHA256:Yju3nzv/JuTvvHN... agent
debug1: Server accepts key: /home/rust/.ssh/id_rsa RSA SHA256:Yju3nzv/JuTv... agent
debug1: Authentication succeeded (publickey).
Authenticated to github.com ([140.82.112.4]:22).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = C.UTF-8
PTY allocation request failed on channel 0
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
Hi influxdata/minikafka-rs! You've successfully authenticated, but GitHub does not provide shell access.
debug1: channel 0: free: client-session, nchannels 1
Connection to github.com closed.
Transferred: sent 3888, received 2940 bytes, in 0.1 seconds
Bytes per second: sent 40708.4, received 30782.6
debug1: Exit status 1

Version

cargo 1.57.0 (b2e52d7ca 2021-10-21)
release: 1.57.0
commit-hash: b2e52d7cab0a286ee9fcc0c17510b1e72fcb53eb
commit-date: 2021-10-21
host: x86_64-unknown-linux-gnu
libgit2: 1.3.0 (sys:0.13.23 vendored)
libcurl: 7.79.1-DEV (sys:0.4.49+curl-7.79.1 vendored ssl:OpenSSL/1.1.1l)
os: OracleLinux 11.0.0 [64-bit]

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-gitArea: anything dealing with gitC-bugCategory: bugS-needs-infoStatus: Needs more info, such as a reproduction or more background for a feature request.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions