AArch64: Adding AArch64 arch-related files #1223

MrXinWang · 2020-05-22T08:28:46Z

This PR adds the AArch64 arch-related files for the Cloud-Hypervisor AArch64 enabling, including:

AArch64 memory layout
Ported Firecracker AArch64 files with some modifications
libfdt dependancy for cross-building in the Github action

The commits in this PR are splited from #1126.

Signed-off-by: Henry Wang <[email protected]>

arch/src/aarch64/fdt.rs

arch/src/aarch64/gic.rs

MrXinWang · 2020-06-03T11:12:40Z

Hi @sboeuf. I have rebased this PR to the master branch and tested each commit can be built independantly. PTAL.

sboeuf

PR looks good, just a few comments.

arch/src/aarch64/gic.rs

arch/src/aarch64/fdt.rs

sboeuf · 2020-06-04T17:58:16Z

arch/src/aarch64/fdt.rs

+    fn fdt_open_into(fdt: *const c_void, buf: *mut c_void, bufsize: c_int) -> c_int;
+    fn fdt_finish(fdt: *const c_void) -> c_int;
+    fn fdt_pack(fdt: *mut c_void) -> c_int;
+}


Oh that's not great we have to rely on some external C library to generate the device tree blob. Any chance there's a Rust native way of doing this?
Also, have you thought about using ACPI instead of DeviceTree? I'm asking because it's already part of Cloud-Hypervisor and that would be nice to reuse this part between x86_64 and aarch64.

Thanks @sboeuf for pointing these out. I actually think the idea of replacing the fdt implementation with Rust native code makes a lot of sense. However I do have a few concerns and would like to discuss them with you:

Speaking of the FDT:

We did think of rewriting everything completely in Rust at the beginning. The reasons why we used the fdt in current way are that there is an existing libfdt-dev which contains everything we need, and rewriting everything might take longer than expected (as we probably need to take care of everything in the library). Existing rust crates related to the DeviceTree (for example: device_tree, dtb and dts_viewer) are crates that either parsing dtb to dts or simply parsing dts files without compiling them to dtb. All of them cannot satisfy the requirements needed for cloud-hypervisor project.

We also have an existing discussion about rewriting fdt in the rust-vmm community about this: vmm-fdt, please kindly leave some comments about this and these comments are greatly appreciated.

As currently it seems that this PR is blocking further PRs from @michael2012z, if the community is not happy with current fdt implementation, I will remove the commit related to the fdt and we can merge the other modules first. Would this be ok with you? Thanks!

Speaking of the ACPI:
Yes of course we want to support ACPI for arm64. Since ACPI also contains a lot of things, currently for the first step of running cloud-hypervisor on arm, we just used the DeviceTree as it is simpler. The supporting of ACPI on arm64 will be investigated and probably submitted as further PRs after we prove the cloud-hypervisor works well on arm64. Hope it is ok :))

Speaking of the FDT

Ok I'm glad to see your actively trying to solve this with a Rust implementation, so I'm okay with using libfdt-dev as a temporary solution. I think extending one of the existing crate mentioned above would make more sense since the generation of a fdt blob is not specific to VMMs or virtualization.

Speaking of the ACPI

Also glad to see you're considering ACPI support, but I understand you need to validate you can make things work with some already existing code before you can go further and refactor/redesign the code.

@sameo @rbradford WDYT?

If there is momentum behind creating a Rust native version for FDT manipulation that is good. As you can see we did the same thing for creating ACPI tables.

As for ACPI boot with ARM I checked the Linux source code and it only works with either getting it from EFI or by scanning the memory. The former will limit the ability to do direct kernel boot but we do locate the ACPI tables within the scannable location.

This commit adds the memory layout design for AArch64 in `arch/src/aarch64/layout.rs` and related changes in `arch/src/lib.rs` to make sure this commit can build. Signed-off-by: Michael Zhao <[email protected]>

This commit adds ported code of Generic Interrupt Controller (GIC) software implementation for AArch64, including both GICv2 and GICv3 devices. These GIC devices are actually emulated by the host kernel through KVM and will be used in the guest VM as the interrupt controller for AArch64. Signed-off-by: Henry Wang <[email protected]>

As on AArch64 systems we need register mpidr to create the flattened device tree, here in this commit we add ported AArch64 register implementation from Firecracker and related changes to make this commit build. Signed-off-by: Henry Wang <[email protected]>

When booting VM on AArch64 machines, we need to construct the flattened device tree before loading kernel. Hence here we add the implementation of the flattened device tree for AArch64. Signed-off-by: Henry Wang <[email protected]>

In AArch64 prototype, there are code to construct the flattened device tree, and to make such code compilable we need to install libfdt-dev. In normal situation, this installation process can be done by either installing libfdt-dev locally or in the development container. Before formal AArch64 CI is setup, we use the workaround in this commit to install libfdt in the github cross-build workflow. Signed-off-by: Henry Wang <[email protected]>

rbradford · 2020-06-05T08:37:07Z

.github/workflows/cross-build.yaml

            target: ${{ matrix.target }}
            override: true
+      - name: Install arm64 libfdt
+        run: wget http://ftp.us.debian.org/debian/pool/main/d/device-tree-compiler/libfdt-dev_1.6.0-1_arm64.deb && dpkg-deb -xv libfdt-dev_1.6.0-1_arm64.deb ./tlibfdtdev && sudo mkdir /tmmmp && mkdir target && mkdir target/debug && mkdir target/debug/deps && sudo cp ./tlibfdtdev/usr/lib/aarch64-linux-gnu/libfdt.a target/debug/deps/libfdt.a && echo "libfdt installed"


This could break if that particular version is removed from Debian. Can you not build it from source?

This is a horrible workaround indeed... In fact this is the main reason of me creating #1263. I am thinking if we can apply an arm64 machine from packet.net and test the build of cloud-hypervisor on real arm64 machines instead of cross-building it on x86_64 machines. Therefore we will not need this libfdt-dev installation anymore. Any comments @rbradford ? If the community does not have existing arm64 machines, I think we can take care of this problem.

(The good thing is that I checked the Debian release page and it seems that they still keep the 1.4.x version of libfdt-dev. So probably this 1.6.0 version will be safe in the near future for some times....but it is really really not ideal...)

One of the benefits of cross-building is that we can results nice and quickly as part of the GitHub CI.

@MrXinWang If you can provide an ARM64 machine we can integrate in into our Jenkins CI via SSH which is something that will be necessary for running the integration tests too.

@rbradford

One of the benefits of cross-building is that we can results nice and quickly as part of the GitHub CI.

I think we can also assign this arm64 machine (or we can even use our own development machine) to the github action?

If you can provide an ARM64 machine we can integrate in into our Jenkins CI via SSH which is something that will be necessary for running the integration tests too.

Talked to my colleague about this yesterday and I think we can (I mean applying from packet), and it will be similar thing in kata-container community. I think I will try to apply one next week (5pm local time here and it is getting hard to find people...).

@rbradford

One of the benefits of cross-building is that we can results nice and quickly as part of the GitHub CI.

I think we can also assign this arm64 machine (or we can even use our own development machine) to the github action?

I don't know about that. If we have a fast and reliable full build CI then we can remove the GitHub action in the future.

If you can provide an ARM64 machine we can integrate in into our Jenkins CI via SSH which is something that will be necessary for running the integration tests too.

Talked to my manager about this yesterday and I think we can (I mean applying from packet), and it will be similar thing in kata-container community. I think I will try to apply one next week (5pm local time here and it is getting hard to find people...).

Okay sounds good, thanks.

@rbradford About the github action, see here about the self-hosted runners, but I think only the owner of this repository can setup it...

Anyway I focus on applying the machine first and will let you know after getting them.

MrXinWang · 2020-06-08T05:59:52Z

Hi @sboeuf @rbradford ! As the CI machine probably will take a few days to apply, I am wondering shall we merge this PR directly or shall I remove fdt-related code in this PR first in order to merge the other modules first. Currently I think the only problem is the libfdt-dev installation workaround in the github action, and this can be addressed after we get CI machine (I have posted my application in here). Some further PRs to enable AArch64 will need the code in this one. Thanks!

MrXinWang force-pushed the arm64_arch_files branch from b5d91bd to 41c4ae1 Compare May 22, 2020 08:55

michael2012z mentioned this pull request Jun 1, 2020

AArch64 support #1125

Closed

15 tasks

sboeuf requested changes Jun 2, 2020

View reviewed changes

arch/src/aarch64/fdt.rs Show resolved Hide resolved

arch/src/aarch64/gic.rs Show resolved Hide resolved

MrXinWang force-pushed the arm64_arch_files branch 6 times, most recently from ce3f91c to 1481327 Compare June 3, 2020 11:06

sboeuf reviewed Jun 4, 2020

View reviewed changes

michael2012z and others added 5 commits June 5, 2020 10:19

aarch64: Add memory layout for AArch64

06618ac

This commit adds the memory layout design for AArch64 in `arch/src/aarch64/layout.rs` and related changes in `arch/src/lib.rs` to make sure this commit can build. Signed-off-by: Michael Zhao <[email protected]>

aarch64: Porting fdt related files from Firecracker

be0a5cf

When booting VM on AArch64 machines, we need to construct the flattened device tree before loading kernel. Hence here we add the implementation of the flattened device tree for AArch64. Signed-off-by: Henry Wang <[email protected]>

MrXinWang force-pushed the arm64_arch_files branch from 1481327 to 7c0ddff Compare June 5, 2020 02:27

sboeuf approved these changes Jun 5, 2020

View reviewed changes

sboeuf requested review from rbradford and sameo June 5, 2020 07:54

rbradford approved these changes Jun 5, 2020

View reviewed changes

MrXinWang mentioned this pull request Jun 5, 2020

AArch64 CI Enablement #1263

Closed

8 tasks

rbradford merged commit e436bbf into cloud-hypervisor:master Jun 8, 2020

MrXinWang deleted the arm64_arch_files branch June 4, 2021 07:38

AArch64: Adding AArch64 arch-related files #1223

AArch64: Adding AArch64 arch-related files #1223

Uh oh!

Conversation

MrXinWang commented May 22, 2020

Uh oh!

Uh oh!

Uh oh!

MrXinWang commented Jun 3, 2020

Uh oh!

sboeuf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sboeuf Jun 4, 2020

Choose a reason for hiding this comment

Uh oh!

MrXinWang Jun 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sboeuf Jun 5, 2020

Choose a reason for hiding this comment

Uh oh!

rbradford Jun 5, 2020

Choose a reason for hiding this comment

Uh oh!

rbradford Jun 5, 2020

Choose a reason for hiding this comment

Uh oh!

MrXinWang Jun 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rbradford Jun 5, 2020

Choose a reason for hiding this comment

Uh oh!

MrXinWang Jun 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rbradford Jun 5, 2020

Choose a reason for hiding this comment

Uh oh!

MrXinWang Jun 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MrXinWang commented Jun 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MrXinWang Jun 5, 2020 •

edited

Loading

MrXinWang Jun 5, 2020 •

edited

Loading

MrXinWang Jun 5, 2020 •

edited

Loading

MrXinWang Jun 5, 2020 •

edited

Loading

MrXinWang commented Jun 8, 2020 •

edited

Loading