Skip to content

Add Build-CI 2.0#24

Merged
dougiesquire merged 38 commits into2025.07from
build-ci
Sep 18, 2025
Merged

Add Build-CI 2.0#24
dougiesquire merged 38 commits into2025.07from
build-ci

Conversation

@dougiesquire
Copy link
Copy Markdown
Collaborator

@dougiesquire dougiesquire commented Aug 11, 2025

This PR adds Build-CI. Because there is already quite extensive CI testing inherited from the upstream repo, I have only added manifests to build access-om3 using oneapi with MOM6 +/~asymmetric_mem.

Closes #20

@codecov
Copy link
Copy Markdown

codecov bot commented Aug 11, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (2025.08@8deb985). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff             @@
##             2025.08      #24   +/-   ##
==========================================
  Coverage           ?   37.49%           
==========================================
  Files              ?      306           
  Lines              ?    92641           
  Branches           ?    17782           
==========================================
  Hits               ?    34733           
  Misses             ?    51356           
  Partials           ?     6552           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dougiesquire dougiesquire marked this pull request as ready for review August 11, 2025 23:11
@anton-seaice
Copy link
Copy Markdown
Collaborator

anton-seaice commented Aug 11, 2025

I'm sure you know this but you'll need to set

find_package(MPI REQUIRED COMPONENTS Fortran) in CMakeLists and make the required / related changes:

(e.g similar to ACCESS-NRI/access3-share@d7d7aac )

(actually maybe that's only for the standalone build)

@dougiesquire dougiesquire marked this pull request as draft August 11, 2025 23:18
@dougiesquire dougiesquire force-pushed the build-ci branch 6 times, most recently from 7e0aec3 to 1356be3 Compare August 12, 2025 05:10
@dougiesquire
Copy link
Copy Markdown
Collaborator Author

dougiesquire commented Aug 12, 2025

Hmm, I pushed an empty commit to test that dependencies are being reused from the build cache, but it appears to have rebuilt everything (?). Ping @CodeGat in case you're interested (not expecting you to do anything here)

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

Latest failure is due to missing language dependencies in the access3-share SPR. Fix is here: ACCESS-NRI/access-spack-packages#303

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

dougiesquire commented Aug 14, 2025

The build-ci on the previous commit did reuse access3_share from the cache - see here

I'll now make a change on the access3_share develop branch and then check whether or not the cache is still used.

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

dougiesquire commented Aug 14, 2025

Hmmm, it appears to have used the cache :'( - see here

It's correctly showing the updated commit in the concretized output

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

dougiesquire commented Aug 14, 2025

In the concretized output the commit variant changes but the spack hash does not:

Pre update to develop branch:

  -   hdbt3sf          ^access3-share@develop~ipo~openmp build_system=cmake build_type=Release commit=f2f35ce5915e82a83899b69560d826deab53b668 generator=make arch=linux-rocky8-x86_64_v3 %c,[email protected]

Post update to develop branch:

  -   hdbt3sf          ^access3-share@develop~ipo~openmp build_system=cmake build_type=Release commit=33c5bd243a7ca2263fabd1330429816de4ff2426 generator=make arch=linux-rocky8-x86_64_v3 %c,[email protected]

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

dougiesquire commented Aug 14, 2025

Oh, it looks like one fix might be to add no_cache=True to our branch-based version() directives - see here (also available in v0.22). I'll test this tomorrow, though it would still be better if we could have caching/reuse that recognised commit hash changes on branch-based versions.

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

Weird, even with no_cache=True it still appears to have used the cache - see here.

@CodeGat, I wonder if this is because there was already an entry in the cache before I set no_cache=True? Is it possible/easy to remove a specific package (hdbt3sf) from the cache? If not, I can try testing with a different package.

@anton-seaice
Copy link
Copy Markdown
Collaborator

Did the spack hash change ?

spack directives (e.g. version()) tend to impact concretisation, rather than install. So I wonder if no_cache means don't use the cache for concretisation ? But if it concretises the same as something in the cache, it will still use the build-cache for a binary?

Now that i've written that, you probably added a commit between installs for testing anyway right ?

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

Did the spack hash change ?

spack directives (e.g. version()) tend to impact concretisation, rather than install. So I wonder if no_cache means don't use the cache for concretisation ? But if it concretises the same as something in the cache, it will still use the build-cache for a binary?

Now that i've written that, you probably added a commit between installs for testing anyway right ?

See this comment above

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

Ok great, that did not use the cache - see here

I'll now make another change on the access3_share develop branch and then check whether or not the cache is still used.

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

Again, the concretization shows the changed commit

 -   hdbt3sf          ^access3-share@develop~ipo~openmp build_system=cmake build_type=Release commit=326f5b611b748d3f8e911d0b9999e22dbc36db9d generator=make arch=linux-rocky8-x86_64_v3 %c,[email protected]

but the cache is still used at install - see here

@aidanheerdegen
Copy link
Copy Markdown
Member

Because there is already quite extensive CI testing inherited from the upstream repo

Are those checks only run when you do PRs back to upstream?

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

Are those checks only run when you do PRs back to upstream?

Nah they're run on push/pull_request. I've just turned them off for now in this PR to reduce noise

@aidanheerdegen
Copy link
Copy Markdown
Member

Did you mean to test with mpich rather than openmpi?

https://github.com/ACCESS-NRI/MOM6/actions/runs/16978072277/job/48131917260#step:13:722

 -   5tr3xy7          ^[email protected]~argobots~cuda+fortran+hwloc+hydra~level_zero+libxml2+pci~rocm+romio~slurm~verbs+wrapperrpath~xpmem build_system=autotools datatype-engine=auto device=ch4 netmod=ofi patches:=66ba8eb,7326028,be529bf,db4173c,de0de41 pmi=default arch=linux-rocky8-x86_64_v3 %c,cxx,[email protected]

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

Did you mean to test with mpich rather than openmpi?

Nope.

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

Apologies @CodeGat, I had to change the base branch and rebase.

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

Hmmm, I tried to force it to use openmpi (previously was using mpich) and got:

==> [2025-09-16-02:01:48.543202] Error: failed to concretize `access-om3@latest ^openmpi` for the following reasons:
     1. cannot satisfy a requirement for package 'perl'.

@CodeGat
Copy link
Copy Markdown
Member

CodeGat commented Sep 17, 2025

Just rerun it and yep, that's an error 😆 I'm not really sure why, though. I've got it down to a minimal example, at least...

[root@77064e1cd29a /]# spack spec openmpi
==> Error: failed to concretize `openmpi` for the following reasons:
     1. cannot satisfy a requirement for package 'perl'.
[root@77064e1cd29a /]# spack spec openmpi%[email protected]
==> Error: failed to concretize `openmpi %[email protected]` for the following reasons:
     1. cannot satisfy a requirement for package 'perl'.
[root@77064e1cd29a /]# spack spec openmpi%[email protected]
==> Error: failed to concretize `openmpi %[email protected]` for the following reasons:
     1. cannot satisfy a requirement for package 'perl'.
[root@77064e1cd29a /]# spack spec openmpi%gcc
==> Error: failed to concretize `openmpi %gcc` for the following reasons:
     1. cannot satisfy a requirement for package 'perl'.

I don't really know what this means though. Something to do with spack-config?

The image is using ACCESS-NRI/access-spack-packages@83407de and ACCESS-NRI/spack-config@e759252

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

I don't know why it can't satisfy the requirement. @harshula may have insights?

@CodeGat
Copy link
Copy Markdown
Member

CodeGat commented Sep 17, 2025

We've found the source of the issue. perl is classed as an external non-buildable package and spack 1.0 can't ascertain what compiler was used to build it. Which means that when we enforce compiler constraints (%oneapi) it doesn't know if that install of perl satisfies that constraint. The way to fix it is to make perl a buildable package so it knows what compiler was used. I'll open a PR in spack-config to fix this, and test that branch in this PR.

@CodeGat
Copy link
Copy Markdown
Member

CodeGat commented Sep 17, 2025

@dougiesquire, depending on how long these verification checks take, you might have to change the parallelism on them...they currently reserve all 20 of the organisations GitHub-hosted runner jobs to do them...

@dougiesquire
Copy link
Copy Markdown
Collaborator Author

dougiesquire commented Sep 17, 2025

Yeah, those are just inherited from upstream. They shouldn't take more than a few mins each but also happy to limit how many can run at once.

@CodeGat
Copy link
Copy Markdown
Member

CodeGat commented Sep 17, 2025

Making perl buildable seems to have gotten it to concretize, which is good! If it works, I'll get the spack-config PR merged and revert the spack-config-ref to main

@harshula
Copy link
Copy Markdown

harshula commented Sep 18, 2025

Hi, When using Spack v1, the compiler constraint is set per language. e.g. https://github.com/ACCESS-NRI/dev-docs/wiki/Spack#spack-v10-caveats , as per ACCESS-NRI/spack-config#74 (comment)

@CodeGat
Copy link
Copy Markdown
Member

CodeGat commented Sep 18, 2025

@dougiesquire, sorry about all the noise. @harshula and myself have been testing out using the spack-v1.0-specific language requirements over just making perl a non-external package (as that is a more robust solution).

Unfortunately, that solution doesn't work when we have a 1.0 runner reading from a 0.22 upstream in build-ci, so we're going to have to go with the perl solution until we have moved allllllllllll the components to spack 1.0 as well, so we can move the upstream to 1.0 too.

I think once the checks complete, this will be good to merge.

@dougiesquire dougiesquire marked this pull request as ready for review September 18, 2025 03:47
@dougiesquire
Copy link
Copy Markdown
Collaborator Author

No worries @CodeGat - thanks for resolving this!

We're in our usual review stalemate. But if you could approve to formally indicate you're happy to merge then I'll bypass the rules.

@dougiesquire dougiesquire merged commit 992deea into 2025.07 Sep 18, 2025
107 checks passed
@dougiesquire dougiesquire deleted the build-ci branch September 18, 2025 06:06
dougiesquire added a commit that referenced this pull request Nov 27, 2025
Builds access-om3 using oneapi with MOM6 +/~asymmetric_mem

Co-authored-by: Tommy Gatti <[email protected]>
dougiesquire added a commit that referenced this pull request Jan 12, 2026
Builds access-om3 using oneapi with MOM6 +/~asymmetric_mem

Co-authored-by: Tommy Gatti <[email protected]>
dougiesquire added a commit that referenced this pull request Jan 12, 2026
Builds access-om3 using oneapi with MOM6 +/~asymmetric_mem

Co-authored-by: Tommy Gatti <[email protected]>
dougiesquire added a commit that referenced this pull request Feb 10, 2026
Builds access-om3 using oneapi with MOM6 +/~asymmetric_mem

Co-authored-by: Tommy Gatti <[email protected]>
dougiesquire added a commit that referenced this pull request Feb 10, 2026
Builds access-om3 using oneapi with MOM6 +/~asymmetric_mem

Co-authored-by: Tommy Gatti <[email protected]>
dougiesquire added a commit that referenced this pull request Feb 17, 2026
Builds access-om3 using oneapi with MOM6 +/~asymmetric_mem

Co-authored-by: Tommy Gatti <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build CI Link Up

5 participants