Repository navigation
ssh-keyscan failing intermittently #17
Description
Hello, I'm observing what appears to be intermittent failure depending on which particular runner gets the job. This is in parallel with runs for the standard aarch64 and amd64 GitHub Actions runners which do not exhibit the failures, so... I think it's something ssh adjacent.
Broad strokes description is that the workflow builds python wheels and uploads them (ssh rsync) to a web server which will later serve as the python wheels index for some other workflows. The private key is in a secret and the print of a pubkey is simply proof that the key entered is valid before continuing; however one of those validation commands fails sometimes on the RISE RISC-V runners, but not always:
if [[ "false" =~ false|False ]] && [ -z "" ]; then
umask g-rwx,o-rwx
chmod -f g-rwx,o-rwx .ssh .ssh/id_rsa .ssh/known_hosts .ssh/wheels-key || true
mkdir -p .ssh
# Write valid wheels-key or wrapped with PEM header-footer
echo "***
...snip...
***" > .ssh/wheels-key
ssh-keygen -y -e -f .ssh/wheels-key 2>&1>/dev/null || \
sed -e '1i-----BEGIN RSA PRIVATE KEY-----' \
-e '$a-----END RSA PRIVATE KEY-----' -i .ssh/wheels-key
ssh-keygen -y -e -f .ssh/wheels-key 2>&1>/dev/null && \
cat .ssh/wheels-key >> .ssh/id_rsa && rm -f .ssh/wheels-key
# Validate & update known_hosts
ssh-keygen -y -e -f .ssh/id_rsa
ssh-keyscan -H "ai6fs.net" >> .ssh/known_hosts
fi
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---- BEGIN SSH2 PUBLIC KEY ----
Comment: "3072-bit RSA, converted by runner@riscv-runner-12 from OpenS"
AAAAB3NzaC1yc2EAAAADAQABAAABgQDAFGj/8WmGysWZwN9aXWtwfjVhyy821k7Fg1QQ1k
l4qdQ0GJNyh64oWBK7dNsJLd53gSeijK/Yvz0fFZybp7ZGaR12N/0/baV8IQ0Ga9b+rR2m
+eKI/YinsWJej+iP7yWdZ/VataLDMUtbg2TpufawFbBmeX5m3h/Ffjc7ArD5LAjBqB0NQc
CSldMboNcoESi7bcqAXvUako+69gl3QKryHdEDR1tLykKRPE7ItTt479OkVRUPRdv986l0
dUwXK8Kjf6cqukCAf5eWgoiocyJRm3WdHdm+LfK8AUsWQjk/4yDCqHiWbYxryVW+5ck7Ss
Zem7ennP4VBPxLWFtpd6FYlmLPh7aPNUECDCdY6mKWwwzqFdzaOLj2FmVdF0uFLAxuwnWy
iI13onMyYTHqyq9JOm6uK+pDYXDMETG4ZgfCaQ3XW/xvmy+ev0FPa4sZhDu+dbW+nAdnsb
9hdokl7UdP3ZTlFdnIbj5IVr8XbV1kuZ/a+oGn8UmAFeiW2Q96Ox8=
---- END SSH2 PUBLIC KEY ----
Error: Process completed with exit code 1.
and then, the same exact job re-run is successful:
if [[ "false" =~ false|False ]] && [ -z "" ]; then
umask g-rwx,o-rwx
chmod -f g-rwx,o-rwx .ssh .ssh/id_rsa .ssh/known_hosts .ssh/wheels-key || true
mkdir -p .ssh
# Write valid wheels-key or wrapped with PEM header-footer
echo "***
...snip...
***" > .ssh/wheels-key
ssh-keygen -y -e -f .ssh/wheels-key 2>&1>/dev/null || \
sed -e '1i-----BEGIN RSA PRIVATE KEY-----' \
-e '$a-----END RSA PRIVATE KEY-----' -i .ssh/wheels-key
ssh-keygen -y -e -f .ssh/wheels-key 2>&1>/dev/null && \
cat .ssh/wheels-key >> .ssh/id_rsa && rm -f .ssh/wheels-key
# Validate & update known_hosts
ssh-keygen -y -e -f .ssh/id_rsa
ssh-keyscan -H "ai6fs.net" >> .ssh/known_hosts
fi
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---- BEGIN SSH2 PUBLIC KEY ----
Comment: "3072-bit RSA, converted by runner@riscv-runner-28 from OpenS"
AAAAB3NzaC1yc2EAAAADAQABAAABgQDAFGj/8WmGysWZwN9aXWtwfjVhyy821k7Fg1QQ1k
l4qdQ0GJNyh64oWBK7dNsJLd53gSeijK/Yvz0fFZybp7ZGaR12N/0/baV8IQ0Ga9b+rR2m
+eKI/YinsWJej+iP7yWdZ/VataLDMUtbg2TpufawFbBmeX5m3h/Ffjc7ArD5LAjBqB0NQc
CSldMboNcoESi7bcqAXvUako+69gl3QKryHdEDR1tLykKRPE7ItTt479OkVRUPRdv986l0
dUwXK8Kjf6cqukCAf5eWgoiocyJRm3WdHdm+LfK8AUsWQjk/4yDCqHiWbYxryVW+5ck7Ss
Zem7ennP4VBPxLWFtpd6FYlmLPh7aPNUECDCdY6mKWwwzqFdzaOLj2FmVdF0uFLAxuwnWy
iI13onMyYTHqyq9JOm6uK+pDYXDMETG4ZgfCaQ3XW/xvmy+ev0FPa4sZhDu+dbW+nAdnsb
9hdokl7UdP3ZTlFdnIbj5IVr8XbV1kuZ/a+oGn8UmAFeiW2Q96Ox8=
---- END SSH2 PUBLIC KEY ----
# ai6fs.net:22 SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u1
# ai6fs.net:22 SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u1
# ai6fs.net:22 SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u1
# ai6fs.net:22 SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u1
# ai6fs.net:22 SSH-2.0-OpenSSH_10.0p2 Debian-7+deb13u1
Metadata
Metadata
Assignees
Labels
No labels
Activity
luhenry commentedon May 10, 2026
Hi @eshattow thank you for your report. I'm not sure I understand exactly what's going wrong. Do you know which command exactly fails? The TH1520 chip (which underlines the EM-RV1) has a known hardware bug which triggers with atomic operations where an atomic operation returns success when it hasn't. Could it be that?
eshattow commentedon May 12, 2026
After the second attempt today where it fails I have the sense to set GitHub Actions Secret 'ACTIONS_STEP_DEBUG'=true but now it does not fail.... of course, that is how these "fun" intermittent problems are so anyhow the "good" output with debug:
2026-05-12-0538-debug-success-job-logs.txt
And just previous to this the two attempts (no debug, unfortunately) that fail:
2026-05-12-0448-job-logs.txt
and in the host that is the target of ssh keyscan I read through the journal but it is not clear to me which information corresponds to the github actions activity.
then the other github attempt that failed:
2026-05-12-0502-job-logs.txt
Looking in the log for "SSH2" pubkey validation are the list of commands above it, framed for our understanding as funneling the GitHub Actions Secret 'WHEELS_KEY' (containing ssh id_rsa private key) into file '.ssh/wheels-key' and then follow along from there; this is preparation for rsync the produced python wheels to a webhost with id_rsa privkey as the authentication mechanism. The invocation of 'ssh-keygen' in multiple possible tests is to validate the user input is a valid private key (and to print the corresponding pubkey what we see in the log) prior to appending it to the target file. So... we do see the pubkey is output but then somewhere between that and the assumed completion of the very next command performing the keyscan amending the known_hosts file it fails the workflow. My guesses are one of:
Would it be worthwhile to craft a workflow specifically to try to loop and get a failure from these ssh commands in sequence, could it be expected to re-try this sequence of commands until failure or would that be flawed in some way? - I want to ask before any such thing as it is not my intent to be wasteful of any community resource offering. Also if in a side channel you want the privkey to set up your own such reproducer for this issue and you think it's more than just some user error on my part I'd be sure glad to share if it helps make troubleshooting easier.
eshattow commentedon May 12, 2026
P.S. I should mention what it is, this all, is. I'm readying Home Assistant for RISC-V and the superset work of that is making Home Assistant CI/CD workflows able to be forked and built outside of the home-assistant organization. There's layers of docker images and reusable actions up to the "wheels" builder image, and then after the "docker" image does use the former to build some python wheels and rsync those to a webhost as it becomes the PEP503 index for the build of "core" image (what we think of as the canonical Home Assistant docker image) which also does more of the same building a long list of python wheels and rsync to the webhost. I've proven it's possible to get there and now backtracking to make the whole thing repeatable by anyone that wants to, at: Special Interest Group for Home Assistant on RISC-V
[-]ssh-keygen failing intermittently[/-][+]ssh-keyscan failing intermittently[/+]eshattow commentedon Jul 2, 2026
2026-07-02-sshkeyscan-exit1-job-logs.txt riscv-runner-18 fail
2026-07-02-sshkeyscan-exit1-job-logs2.txt riscv-runner-17 fail
2026-07-02-sshkeyscan-exit1-job-logs3.txt riscv-runner-17 fail
2026-07-02-sshkeyscan-exit1-job-logs4.txt riscv-runner-4 success
debug enabled and this appears to fail once on riscv-runner-18 then twice on riscv-runner-17 before success on riscv-runner-4, possibly a connection issue to my server from some portion of RISE runners (but not GitHub runners)?
luhenry commentedon Jul 15, 2026
I'm not sure exactly what happens. I expect a combination of three sources of flakiness:
I'm sorry I have no better answer... I can only expect it's going to get better once we have better hardware.
Merge provisioning scripts into a single scw.py (riseproject-dev#17)