tests: AArch64: Enable AArch64 integration tests #1395

MrXinWang · 2020-06-30T02:35:17Z

This PR contains 3 commits in series to enable the integration tests for AArch64:

tests: Preparation of enabling integration tests for AArch64

This commit adds supporting components and code for enabling the AArch64 integration tests, including:
1. A Linux kernel config file to build kernel on AArch64 machines.
2. Refactoring the run_integration_test.sh to support AArch64.
tests: Enable test_aarch64_pe_boot case for AArch64

The compiled AArch64 linux kernel by running make is in PE format instead of vmlinux, vmlinux.pvh and bzImage format. Therefore we need to add integration test for this case.
tests: Add a Jenkins stage for AArch64 integration test

This commit adds a Jenkins stage for AArch64 integration test, and the test is carried out using the GNU toolchain.

MrXinWang · 2020-06-30T03:49:42Z

Hi @rbradford, it seems that in order to run the integration tests, the docker image for arm64 should be updated (This is to install setcap for arm64, and the x86_64 one does not need such update). Would you prefer we keep current naming strategy or separate these two images as cloudhypervisor/dev-$ARCH? Thanks.

I've also already built a cloudhypervisor/dev:v4 arm64 docker image in the arm64 CI machine just in case, but x86_64 v4 image is missing, which caused CI failure.

rbradford · 2020-06-30T11:15:11Z

Hi @rbradford, it seems that in order to run the integration tests, the docker image for arm64 should be updated (This is to install setcap for arm64, and the x86_64 one does not need such update). Would you prefer we keep current naming strategy or separate these two images as cloudhypervisor/dev-$ARCH? Thanks.

I've also already built a cloudhypervisor/dev:v4 arm64 docker image in the arm64 CI machine just in case, but x86_64 v4 image is missing, which caused CI failure.

Please create a single PR updating just the Dockerfile. I'll then create a new version of the container and then we can proceed with the rest of this PR.

MrXinWang · 2020-07-01T09:39:25Z

Hi @rbradford, thanks for updating the docker image. Now I think this PR is ready for being reviewed. Would you mind taking a look again? Thanks!

Also a very kind reminder: I just checked tags in https://hub.docker.com/r/cloudhypervisor/dev/tags and it seems that the v4 image is arm64 only. I think probably a manifest of v4 image is missing in order to make it multi-arch. I think current tag will cause future CI break on newly-added x86_64 CI machines as they cannot pull x86_64 v4 image.

rbradford · 2020-07-01T09:44:50Z

Also a very kind reminder: I just checked tags in https://hub.docker.com/r/cloudhypervisor/dev/tags and it seems that the v4 image is arm64 only. I think probably a manifest of v4 image is missing in order to make it multi-arch. I think current tag will cause future CI break on newly-added x86_64 CI machines as they cannot pull x86_64 v4 image.

I re-uploaded an amd64 version. Seems it doesn't handle multiple architectures very well.

MrXinWang · 2020-07-01T09:54:52Z

@rbradford Yeah I saw that and the arm64 tag is gone...anyway it doesn't matter now as we only have 1 arm64 machine and the right image is already there.

To support multi-arch image in dockerhub, I suggest that we can try methods in the Manifest section in https://cstan.io/?p=11870&lang=en

rbradford

I really would like you to switch to two versions of the scripts. I think it would make things a lot neater. Everything else is good.

scripts/run_integration_tests.sh

MrXinWang · 2020-07-03T03:51:25Z

Hi @rbradford , I have splited the arm64 and x86_64 integration test scripts, but I have two things which I am not really sure:

ci: Move from Clear Linux to Ubuntu Focal Fossa cloud image #1406 changed the naming stragegy of qcow2 and raw images. I am wondering if arm64 should do the same changes. Currently I managed to keep the old name and make everything works (see here), but I personally think the arm64 names are not very elegant and also less readable. If we want to change the names, could you please help (As I don't think I have access to where the images are stored)?
ci: Move from Clear Linux to Ubuntu Focal Fossa cloud image #1406 also made a pretty big change (adding a field) to the struct UbuntuDiskConfig. I believe this is to test the booting of the qcow2 images. However I don't think current arm64 cloud-hypervisor can boot qcow2 images. As when I tried to boot a qcow2 image, following error will occur:
cloud-hypervisor: 342.277382ms: ERROR:virtio-devices/src/block.rs:668 -- Failed to execute request: Read(IOError(Os { code: 95, kind: Other, message: "Operation not supported" }))

The raw image worked well.

For this problem, I made a workaround in here. I have very limited knowledge in this area so I am wondering could you please confirm if the cloud-hypervisor qcow2 support is missing for arm64? or did I miss anything?

@sboeuf Any suggestions?

rbradford · 2020-07-03T07:23:26Z

We can't use the QCOW2 images produced from Ubuntu directly as they have compressed blocks. Convert to raw instead.

The ones used in the ci are converted from QCOW to raw and then back to QCOW which means they do not have compressed blocks.

MrXinWang · 2020-07-03T07:28:20Z

Hi @rbradford Thanks for the explanation, now I understand. Would you prefer we stick to current way or any method that I can upload the right qcow2 image? Thanks!

sboeuf · 2020-07-03T07:29:37Z

Yes I think we could simply take the qcow2 image, convert it into a raw one, and then convert it back to a qcow2. The resulting image won't be compressed.

MrXinWang · 2020-07-06T02:10:20Z

Hi @sboeuf ! That makes sense. Do you prefer I do that conversion in current arm64 script or we just upload the right image in cloudhypervisorstorage.blob.core.windows.net? If the latter is preferred can anyone help in this? Thanks!

sboeuf · 2020-07-07T05:35:19Z

@MrXinWang you can do the conversion from the script, it should be pretty quick. This will keep the image in the storage account pretty small, hence the download time will be kept short.

MrXinWang · 2020-07-07T07:15:48Z

@sboeuf Done and rebased. Didn't think about the downloading time problem before but that absolutely makes sense.

Could you please take a look again? Thanks!

sboeuf

One small comment, otherwise the PR looks good to me.

tests/integration.rs

sboeuf · 2020-07-07T07:38:58Z

@rbradford PTAL

rbradford

Other than the potential kernel staleness issue looks very good.

scripts/run_integration_tests_aarch64.sh

This commit adds supporting components and code for enabling the AArch64 integration tests, including: 1. A Linux kernel config file to build kernel on AArch64 machines. 2. Refactoring the `run_integration_test.sh` to architecture specific scripts for readability. Signed-off-by: Henry Wang <[email protected]>

The compiled AArch64 linux kernel by running `make` is in PE format instead of vmlinux, vmlinux.pvh and bzImage format. Therefore we need to add integration test for this case. Signed-off-by: Henry Wang <[email protected]>

This commit adds a Jenkins stage for AArch64 integration test, and the test is carried out using the GNU toolchain. Signed-off-by: Henry Wang <[email protected]>

MrXinWang mentioned this pull request Jul 1, 2020

resources: Dockerfile: Update Dockerfile for AArch64 #1403

Merged

MrXinWang force-pushed the arm64_integration branch from d724480 to d3ec109 Compare July 1, 2020 09:09

rbradford reviewed Jul 1, 2020

View reviewed changes

scripts/run_integration_tests.sh Outdated Show resolved Hide resolved

MrXinWang force-pushed the arm64_integration branch from d3ec109 to ad32322 Compare July 3, 2020 03:33

MrXinWang force-pushed the arm64_integration branch from ad32322 to 7370a6d Compare July 3, 2020 03:55

MrXinWang requested a review from sboeuf July 3, 2020 03:55

MrXinWang force-pushed the arm64_integration branch 4 times, most recently from 806c263 to d0dc073 Compare July 7, 2020 01:38

MrXinWang force-pushed the arm64_integration branch from d0dc073 to 201eb05 Compare July 7, 2020 07:05

MrXinWang requested a review from rbradford July 7, 2020 07:26

sboeuf reviewed Jul 7, 2020

View reviewed changes

tests/integration.rs Outdated Show resolved Hide resolved

MrXinWang force-pushed the arm64_integration branch from 201eb05 to d0047e5 Compare July 7, 2020 07:28

sboeuf approved these changes Jul 7, 2020

View reviewed changes

rbradford reviewed Jul 7, 2020

View reviewed changes

scripts/run_integration_tests_aarch64.sh Outdated Show resolved Hide resolved

MrXinWang force-pushed the arm64_integration branch from d0047e5 to a43400a Compare July 7, 2020 09:25

MrXinWang added 3 commits July 7, 2020 18:34

tests: Enable test_aarch64_pe_boot case for AArch64

d1d1fdd

The compiled AArch64 linux kernel by running `make` is in PE format instead of vmlinux, vmlinux.pvh and bzImage format. Therefore we need to add integration test for this case. Signed-off-by: Henry Wang <[email protected]>

tests: Add a Jenkins stage for AArch64 integration test

26fbdac

This commit adds a Jenkins stage for AArch64 integration test, and the test is carried out using the GNU toolchain. Signed-off-by: Henry Wang <[email protected]>

MrXinWang force-pushed the arm64_integration branch from a43400a to 26fbdac Compare July 7, 2020 10:36

rbradford approved these changes Jul 7, 2020

View reviewed changes

rbradford merged commit 603445e into cloud-hypervisor:master Jul 7, 2020

MrXinWang mentioned this pull request Jul 8, 2020

AArch64 CI Enablement #1263

Closed

8 tasks

MrXinWang deleted the arm64_integration branch June 4, 2021 07:37

tests: AArch64: Enable AArch64 integration tests #1395

tests: AArch64: Enable AArch64 integration tests #1395

Uh oh!

Conversation

MrXinWang commented Jun 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MrXinWang commented Jun 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford commented Jun 30, 2020

Uh oh!

MrXinWang commented Jul 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford commented Jul 1, 2020

Uh oh!

MrXinWang commented Jul 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MrXinWang commented Jul 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford commented Jul 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MrXinWang commented Jul 3, 2020

Uh oh!

sboeuf commented Jul 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MrXinWang commented Jul 6, 2020

Uh oh!

sboeuf commented Jul 7, 2020

Uh oh!

MrXinWang commented Jul 7, 2020

Uh oh!

sboeuf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sboeuf commented Jul 7, 2020

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MrXinWang commented Jun 30, 2020 •

edited

Loading

MrXinWang commented Jun 30, 2020 •

edited

Loading

MrXinWang commented Jul 1, 2020 •

edited

Loading

MrXinWang commented Jul 1, 2020 •

edited

Loading

MrXinWang commented Jul 3, 2020 •

edited

Loading

rbradford commented Jul 3, 2020 •

edited

Loading

sboeuf commented Jul 3, 2020 •

edited

Loading