@rbradford I think the Jenkins pipeline can be deployed to the AArch64 node now. I am not very sure though however I can see AArch64 related building and testing logs. However I think some other issues with musl toolchain on arm64 needs to be fixed first. It seems that I can always reproduce this either in the container or in my development machine or the CI machine.

Error (only occur when building with musl toolchain):

 = note: /usr/bin/ld: /tmp/rustcE1X2Bs/liblibc-1170e40c1719a823.rlib(__stack_chk_fail.lo): undefined reference to symbol '__stack_chk_guard@@GLIBC_2.17'
          /usr/bin/ld: /lib/aarch64-linux-gnu/ld-linux-aarch64.so.1: error adding symbols: DSO missing from command line
          collect2: error: ld returned 1 exit status

This error happens at the last step, i.e. linking the cloud-hypervisor binary. Root cause see here: https://www.openwall.com/lists/musl/2014/11/05/3

Also I noticed that the bionic-arm64 machine is very unstable. It seems that this machine constantly goes offline and then restart. Here I showed the error log when the bionic-arm64 machine is offline:

Connection was broken

java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2738)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3213)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:896)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
	at hudson.remoting.Command.readFrom(Command.java:142)
	at hudson.remoting.Command.readFrom(Command.java:128)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)

Also I noticed that the bionic-arm64 machine is very unstable. It seems that this machine constantly goes offline and then restart. Here I showed the error log when the bionic-arm64 machine is offline:

Connection was broken

java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2738)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3213)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:896)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
	at hudson.remoting.Command.readFrom(Command.java:142)
	at hudson.remoting.Command.readFrom(Command.java:128)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)

I think this is normal with the SSH launcher. It comes back when the node is needed.

I think this is normal with the SSH launcher. It comes back when the node is needed.

If it is normal that is very good, but I am worrying a little bit if you do this in your local development machine: ssh root@<the AArch64 CI machine IP> and leave it there without doing anything. If you leave it there for about an hour (or even less), the connection of SSH will be closed as client_loop: send disconnect: Broken pipe.

I checked the /etc/ssh/sshd_config file and the ClientAliveInterval is indeed commented. Hope it is just me worrying too much, but this behaviour is very abnormal to me.

Can you do something like: https://www.jenkins.io/blog/2019/07/05/jenkins-pipeline-stage-result-visualization-improvements/

try {
  sh('false')
} catch (ex) {
  unstable('Script failed!')
}

So the the Jenkins build doesn't fail. We need the Jenkins builds to be stable as part of our PR merging criteria.

Hi @rbradford, I have struggled for some weeks to fix the musl toolchain on arm, but sadly none of these methods worked...So in order to let the CI online and let the enablement of integration test unblocked, I will firstly get the GNU toolchain working in this PR and propose following PR for the musl toolchain. rust-lang/rust#73493 created to track this musl issue.

try {
  sh('false')
} catch (ex) {
  unstable('Script failed!')
}

So the the Jenkins build doesn't fail. We need the Jenkins builds to be stable as part of our PR

I am not sure if I misunderstood, but I think in this case it seems that this unstable thing is not needed?

Hi @rbradford, I have struggled for some weeks to fix the musl toolchain on arm, but sadly none of these methods worked...So in order to let the CI online and let the enablement of integration test unblocked, I will firstly get the GNU toolchain working in this PR and propose following PR for the musl toolchain. rust-lang/rust#73493 created to track this musl issue.

Okay with me.

try {
  sh('false')
} catch (ex) {
  unstable('Script failed!')
}

So the the Jenkins build doesn't fail. We need the Jenkins builds to be stable as part of our PR

I am not sure if I misunderstood, but I think in this case it seems that this unstable thing is not needed?

This was just if you wanted to keep the musl rules in there and have them fail but not block the build.