Skip to content

Add system tests workflow#2052

Merged
MakisH merged 28 commits intodevelopfrom
add-systemtest-workflow
Feb 21, 2025
Merged

Add system tests workflow#2052
MakisH merged 28 commits intodevelopfrom
add-systemtest-workflow

Conversation

@uekerman
Copy link
Copy Markdown
Member

@uekerman uekerman commented Jul 10, 2024

Main changes of this PR

Adds a workflow to run the "release" testsuite, which gets triggered by the trigger-system-tests label.

Motivation and additional information

We have the system tests now. Let's use them.

Author's checklist

  • I used the pre-commit hook to prevent dirty commits and used pre-commit run --all to format old commits.
  • I squashed / am about to squash all commits that should be seen as one.

Reviewers' checklist

  • Do you understand the code changes?

@uekerman uekerman requested a review from MakisH July 10, 2024 13:41
MakisH
MakisH reviewed Jul 11, 2024
View reviewed changes
@MakisH
Copy link
Copy Markdown
Member

MakisH commented Jul 11, 2024

Related documentation PR: precice/tutorials#550

@uekerman uekerman mentioned this pull request Jul 16, 2024
Merged
2 tasks
@uekerman
Copy link
Copy Markdown
Member Author

uekerman commented Jul 16, 2024

mmh, seems there is still a problem. @MakisH any quick guess?

@MakisH
Copy link
Copy Markdown
Member

MakisH commented Jul 16, 2024

mmh, seems there is still a problem. @MakisH any quick guess?

You can click on the Summary and from the bottom of that page download the runs.
Knowing that a specific case fails (in this case, all the FEniCS and Nutils cases), you can look into the stdout.log, stderr.log, and the logs of each participant.

The issue is independent of the system tests: it's the familiar NumPy 2.0 incompatibility.

To fix it, we would need a bugfix release in the tutorials (which I cannot prepare this week).

MakisH
MakisH reviewed Jul 16, 2024
View reviewed changes
@uekerman
Copy link
Copy Markdown
Member Author

uekerman commented Jul 18, 2024

To fix it, we would need a bugfix release in the tutorials (which I cannot prepare this week).

I switched to develop for the moment such that we can test if everything else works.

MakisH
MakisH reviewed Jul 18, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@MakisH MakisH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still get numpy 2. From the stdout.log of the flow-over-heated-plate OpenFOAM-FEniCS:

#25 [solid-fenics fenics_adapter 3/4] RUN pip3 install --user fenics-ufl
#25 0.911 Collecting fenics-ufl
#25 0.948   Downloading fenics_ufl-2024.1.0.post1-py3-none-any.whl.metadata (2.6 kB)
#25 0.961 Requirement already satisfied: numpy in /home/precice/.local/lib/python3.10/site-packages (from fenics-ufl) (2.0.0)

I can fix all these issues myself, just not right now.

@MakisH
Copy link
Copy Markdown
Member

MakisH commented Nov 11, 2024

After debugging a bit with @fsimonis, we found out that the issue currently is in preCICE, since the migration of the exporters to samples (e414c76).

Commit a41d1a2 (one before) does not show the segfault.

The issue is not related to Python, as it also happens in OpenFOAM and in the elastic_tube_1d C++. What the failing cases have in common: nearest-neighbor read-consistent mapping and serial-implicit scheme. One case has data initialization, but the other does not.

@MakisH
Copy link
Copy Markdown
Member

MakisH commented Nov 12, 2024

Another system tests run that reproduces the issue, with debug and trace information.

@uekerman
Copy link
Copy Markdown
Member Author

uekerman commented Nov 12, 2024

We are getting closer:

  <participant name="Solid">
    <provide-mesh name="Solid-Nodes-Mesh" />
    <receive-mesh name="Fluid-Nodes-Mesh" from="Fluid" />
    <write-data name="CrossSectionLength" mesh="Solid-Nodes-Mesh" />
    <read-data name="Pressure" mesh="Solid-Nodes-Mesh" />
    <mapping:nearest-neighbor
      direction="read"
      from="Fluid-Nodes-Mesh"
      to="Solid-Nodes-Mesh"
      constraint="consistent" />
      <export:vtk />
  </participant>

We assert here:

precice/src/io/ExportVTK.cpp

Lines 153 to 154 in 736cbf2

for (const mesh::PtrData &data : mesh.data()) { // Plot vertex data
const Eigen::VectorXd &values = data->timeStepsStorage().last().sample.values;

For Fluid-Nodes-Mesh and CrossSectionLength. This is data that we neither write/read nor receive. It actually makes no sense to export this data. We could either write zeros or skip the export.

@MakisH MakisH mentioned this pull request Nov 14, 2024
Merged
9 tasks
@MakisH MakisH mentioned this pull request Jan 27, 2025
Merged
9 tasks
@MakisH
Copy link
Copy Markdown
Member

MakisH commented Jan 27, 2025

self-reminder: Remember to give a green light to the following PRs that will cause some controlled regressions once this is merged:

@MakisH
Copy link
Copy Markdown
Member

MakisH commented Feb 18, 2025 •
edited
Loading

Current state: all elastic-tube-1d cases show regressions, everything else passes.

 +-----------------------------------------------------------+---------+-------------------+-----------------+-----------------------+
| systemtest                                                | success | building time [s] | solver time [s] | fieldcompare time [s] |
+-----------------------------------------------------------+---------+-------------------+-----------------+-----------------------+
| Elastic tube 1D (fluid-cpp, solid-cpp)                    |    0    |      309.83       |      11.88      |         39.42         |
+-----------------------------------------------------------+---------+-------------------+-----------------+-----------------------+
| Elastic tube 1D (fluid-python, solid-python)              |    0    |      206.29       |      49.85      |         4.98          |
+-----------------------------------------------------------+---------+-------------------+-----------------+-----------------------+
| Elastic tube 1D (fluid-cpp, solid-python)                 |    0    |       1.04        |      38.15      |         4.85          |
+-----------------------------------------------------------+---------+-------------------+-----------------+-----------------------+
| Flow over heated plate (fluid-openfoam, solid-nutils)     |    1    |      194.10       |     294.99      |         5.02          |
+-----------------------------------------------------------+---------+-------------------+-----------------+-----------------------+
| Flow over heated plate (fluid-openfoam, solid-fenics)     |    1    |       98.55       |      54.95      |         4.84          |
+-----------------------------------------------------------+---------+-------------------+-----------------+-----------------------+
| Flow over heated plate (fluid-openfoam, solid-openfoam)   |    1    |       1.12        |      29.19      |         4.64          |
+-----------------------------------------------------------+---------+-------------------+-----------------+-----------------------+
| Perpendicular flap (fluid-openfoam, solid-calculix)       |    1    |       71.79       |      74.68      |         15.46         |
+-----------------------------------------------------------+---------+-------------------+-----------------+-----------------------+
| Perpendicular flap (fluid-su2, solid-fenics)              |    1    |      375.97       |     475.54      |         12.62         |
+-----------------------------------------------------------+---------+-------------------+-----------------+-----------------------+

Plot:

tutorials-elastic-tube-1d-all

Triggered with:

 gh workflow run run_testsuite_manual.yml -f suites=release_test -f build_args=PLATFORM:ubuntu_2204,PRECICE_REF:7048bd821c23bec44919d9b356cc2293ea0696ab,PRECICE_PRESET:optimized-debug,PYTHON_BINDINGS_REF:v3.1.2,CALCULIX_VERSION:2.20,CALCULIX_ADAPTER_REF:2a5a73a,FENICS_ADAPTER_REF:3de561d,OPENFOAM_EXECUTABLE:openfoam2312,OPENFOAM_ADAPTER_REF:20b4617,SU2_VERSION:7.5.1,SU2_ADAPTER_REF:64d4aff,TUTORIALS_REF:7314ad4 -f systests_branch=develop --ref=develop

@MakisH MakisH mentioned this pull request Feb 18, 2025
Merged
1 task
MakisH
MakisH approved these changes Feb 21, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@MakisH MakisH left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After several debugging sessions and discussions, I have updated the reference results in precice/tutorials#619 and the tests pass.

I have also fine-tuned several details to make the logs more useful and readable. I think this is now ready to be merged so it does not block other PRs anymore and so that developers can start using it.

@MakisH MakisH merged commit 0ba3783 into develop Feb 21, 2025
3 checks passed
@MakisH MakisH deleted the add-systemtest-workflow branch February 21, 2025 12:55
@MakisH MakisH mentioned this pull request Mar 18, 2025
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MakisH MakisH MakisH approved these changes

Assignees

@uekerman uekerman

@MakisH MakisH

Labels

trigger-system-tests

Projects

None yet

Milestone

Version 3.2.0

Development

Successfully merging this pull request may close these issues.

3 participants

@uekerman @MakisH @BenjaminRodenberg