Depend on openmpi instead of nci-openmpi and add restart reproducibility by harshula · Pull Request #49 · ACCESS-NRI/ACCESS-OM2

harshula · 2024-02-21T22:38:03Z

Requires: libaccessom2: force CMake to use MPI compiler wrappers access-spack-packages#66
Requires: common/gadi/packages.yaml: use system openmpi directly spack-config#26

harshula · 2024-02-22T00:05:18Z

This change was triggered by ACCESS-NRI/access-spack-packages#61 .

CodeGat

We'll need to update the version before merging it - if you rebase onto main it should help out telling what needs to be done.

* Requires: ACCESS-NRI/access-spack-packages#66 * Requires: ACCESS-NRI/spack-config#26

github-actions · 2024-03-19T23:51:54Z

The model version in the spack.yaml has not been updated.
Either update it manually, or comment the following to have it updated and committed automatically:

!bump major for feature releases
!bump minor for bugfixes

aidanheerdegen · 2024-03-20T09:27:12Z

!bump minor

…3.11.24

github-actions · 2024-03-20T09:27:31Z

✅ Version bumped from 2023.11.23 to 2023.11.24 ✅

github-actions · 2024-03-20T09:28:17Z

This ACCESS-NRI/ACCESS-OM2 model will be deployed with the following versions:

2023.11.24 as a Release (when merged).
2023.11.24-2 as a Prerelease (during this PR). This can be accessed on Gadi via spack at /g/data/vk83/prerelease/apps/spack/0.20/spack once deployed.

It will be deployed using:

access-nri/spack-packages version 2023.11.01
access-nri/spack-config version 2023.11.01

If this is not what was expected, commit changes to config/versions.json.

aidanheerdegen · 2024-03-20T10:39:42Z

Pre-release deployment is reporting

/bin/bash: line 5: /g/data/vk83/prerelease/apps/spack/0.20/spack-config/spack-enable.bash: No such file or directory

Despite this change

https://github.com/ACCESS-NRI/build-cd/pull/31/files

Seems that on gadi it is still called spack-start.sh

$ ls /g/data/vk83/prerelease/apps/spack/0.20/spack-config/
LICENSE  README.md  spack-start.bash  tools  v0.20

CodeGat · 2024-03-20T10:47:22Z

Might need to check the version tagged in config/versions.json in the morning - is it spack-enable.bash in that tag? Or spack-start?

github-actions · 2024-03-21T22:44:58Z

The model version in the spack.yaml has not been updated.
Either update it manually, or comment the following to have it updated and committed automatically:

!bump major for feature releases
!bump minor for bugfixes

…4.02.1

github-actions · 2024-03-21T22:47:29Z

This ACCESS-NRI/ACCESS-OM2 model will be deployed with the following versions:

2023.11.24 as a Release (when merged).
2023.11.24-3 as a Prerelease (during this PR). This can be accessed on Gadi via spack at /g/data/vk83/prerelease/apps/spack/0.20/spack once deployed.

It will be deployed using:

access-nri/spack-packages version 2024.03.22
access-nri/spack-config version 2024.02.1

If this is not what was expected, commit changes to config/versions.json.

CodeGat · 2024-03-21T22:48:50Z

!bump major

…4.03.0

github-actions · 2024-03-21T22:49:05Z

✅ Version bumped from 2023.11.24 to 2024.03.0 ✅

github-actions · 2024-03-21T22:49:56Z

This ACCESS-NRI/ACCESS-OM2 model will be deployed with the following versions:

2024.03.0 as a Release (when merged).
2024.03.0-4 as a Prerelease (during this PR). This can be accessed on Gadi via spack at /g/data/vk83/prerelease/apps/spack/0.20/spack once deployed.

It will be deployed using:

access-nri/spack-packages version 2024.03.22
access-nri/spack-config version 2024.02.1

If this is not what was expected, commit changes to config/versions.json.

aidanheerdegen

LGTM

aidanheerdegen · 2024-03-21T23:13:03Z

We shouldn't merge until we have done some performance testing, confirming the impact is minimal

CodeGat · 2024-03-21T23:50:20Z

Good point @aidanheerdegen - I've requested changes so it shouldn't be able to merge

aidanheerdegen · 2024-03-22T03:20:37Z

Seems the pre-release build is still using nci-openmpi

$ grep openmpi /g/data/vk83/prerelease/apps/spack/0.20/spack/var/spack/environments/access-om2-2024_03_0-4/spack.lock                                 
          "name": "nci-openmpi",
          "name": "nci-openmpi",
      "name": "nci-openmpi",
        "path": "/apps/openmpi/4.0.2",
          "openmpi/4.0.2"
          "name": "nci-openmpi",
          "name": "nci-openmpi",
          "name": "nci-openmpi",
          "name": "nci-openmpi",
          "name": "nci-openmpi",

CodeGat · 2024-03-22T03:54:26Z

What does this mean, exactly...?

CodeGat · 2024-03-22T06:15:34Z

@harshula

…i directly instead of nci-openmpi)

github-actions · 2024-03-22T06:24:56Z

This ACCESS-NRI/ACCESS-OM2 model will be deployed with the following versions:

2024.03.0 as a Release (when merged).
2024.03.0-5 as a Prerelease (during this PR). This can be accessed on Gadi via spack at /g/data/vk83/prerelease/apps/spack/0.20/spack once deployed.

It will be deployed using:

access-nri/spack-packages version 2024.03.22
access-nri/spack-config version 2024.03.22

If this is not what was expected, commit changes to config/versions.json.

CodeGat · 2024-03-22T06:25:17Z

@harshula The openmpi thing was fixed - we hadn't tagged the nci-openmpi->openmpi fix in spack-config.

harshula · 2024-03-22T10:37:48Z

Hi @CodeGat , Is there anything else we need to do?

CodeGat · 2024-03-22T11:21:41Z

I assume some kind of performance testing of some kind before proper deployment

access-hive-bot · 2024-03-26T09:04:28Z

This pull request has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/access-om2-bit-repro-testing/1960/1

aidanheerdegen

Looks good. Performance checks suggest no issue with model slowdown.

ACCESS-NRI/access-om2-configs#35

harshula self-assigned this Feb 21, 2024

harshula had a problem deploying to Gadi Prerelease February 21, 2024 22:38 — with GitHub Actions Failure

harshula requested a review from CodeGat March 8, 2024 00:18

harshula added the priority:blocker label Mar 19, 2024

CodeGat requested changes Mar 19, 2024

View reviewed changes

Depend on openmpi instead of nci-openmpi

2ef105b

* Requires: ACCESS-NRI/access-spack-packages#66 * Requires: ACCESS-NRI/spack-config#26

harshula force-pushed the nci-openmpi-to-openmpi branch from 644da04 to 2ef105b Compare March 19, 2024 23:51

spack.yaml: Updated access-om2 package version from 2023.11.23 to 202…

7ce0caa

…3.11.24

CodeGat had a problem deploying to Gadi Prerelease March 20, 2024 09:28 — with GitHub Actions Failure

harshula force-pushed the nci-openmpi-to-openmpi branch from 7ce0caa to 2ef105b Compare March 21, 2024 22:44

spack.yaml: Updated spack-packages to 2024.03.22, spack-config to 202…

4d917ca

…4.02.1

spack.yaml: Updated access-om2 package version from 2023.11.24 to 202…

c80c747

…4.03.0

CodeGat temporarily deployed to Gadi Prerelease March 21, 2024 22:50 — with GitHub Actions Inactive

aidanheerdegen previously approved these changes Mar 21, 2024

View reviewed changes

versions.json: Updated spack-config to 2024.03.22 (uses system openmp…

a871cdd

…i directly instead of nci-openmpi)

CodeGat dismissed aidanheerdegen’s stale review via a871cdd March 22, 2024 06:24

CodeGat temporarily deployed to Gadi Prerelease March 22, 2024 06:25 — with GitHub Actions Inactive

harshula requested review from CodeGat and aidanheerdegen March 22, 2024 10:42

aidanheerdegen changed the title ~~Depend on openmpi instead of nci-openmpi~~ Depend on openmpi instead of nci-openmpi and add restart reproducibility Mar 25, 2024

aidanheerdegen mentioned this pull request Mar 25, 2024

Quantify effects of restart reproducibility on ACCESS-OM2 performance ACCESS-NRI/access-om2-configs#35

Closed

CodeGat approved these changes Mar 27, 2024

View reviewed changes

aidanheerdegen approved these changes Mar 27, 2024

View reviewed changes

aidanheerdegen merged commit bf1f97c into main Mar 27, 2024

aidanheerdegen temporarily deployed to Gadi Prerelease March 27, 2024 05:30 — with GitHub Actions Inactive

harshula deleted the nci-openmpi-to-openmpi branch May 3, 2024 06:45

Conversation

harshula commented Feb 21, 2024

Uh oh!

harshula commented Feb 22, 2024

Uh oh!

CodeGat left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 19, 2024

Uh oh!

aidanheerdegen commented Mar 20, 2024

Uh oh!

github-actions bot commented Mar 20, 2024

Uh oh!

github-actions bot commented Mar 20, 2024

Uh oh!

aidanheerdegen commented Mar 20, 2024

Uh oh!

CodeGat commented Mar 20, 2024

Uh oh!

github-actions bot commented Mar 21, 2024

Uh oh!

github-actions bot commented Mar 21, 2024

Uh oh!

CodeGat commented Mar 21, 2024

Uh oh!

github-actions bot commented Mar 21, 2024

Uh oh!

github-actions bot commented Mar 21, 2024

Uh oh!

aidanheerdegen left a comment

Choose a reason for hiding this comment

Uh oh!

aidanheerdegen commented Mar 21, 2024

Uh oh!

CodeGat commented Mar 21, 2024

Uh oh!

aidanheerdegen commented Mar 22, 2024

Uh oh!

CodeGat commented Mar 22, 2024

Uh oh!

CodeGat commented Mar 22, 2024

Uh oh!

github-actions bot commented Mar 22, 2024

Uh oh!

CodeGat commented Mar 22, 2024

Uh oh!

harshula commented Mar 22, 2024

Uh oh!

CodeGat commented Mar 22, 2024

Uh oh!

access-hive-bot commented Mar 26, 2024

Uh oh!

aidanheerdegen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants