Skip to content

Docs: Add MPIEvaluator tutorial for multi-node HPC systems, including DelftBlue#308

Merged
EwoutH merged 2 commits intomasterfrom
MPI-tutorial
Nov 16, 2023
Merged

Docs: Add MPIEvaluator tutorial for multi-node HPC systems, including DelftBlue#308
EwoutH merged 2 commits intomasterfrom
MPI-tutorial

Conversation

@EwoutH
Copy link
Copy Markdown
Collaborator

@EwoutH EwoutH commented Nov 3, 2023

This PR introduces a two-part tutorial for the MPIEvaluator introduced in #299.

The first part (d26f54e) focuses on the general setup and usage of MPIEvaluator in multi-node HPC systems:

  • Utilizing the new MPIEvaluator for HPC systems.
  • Instructions on setting up the necessary environment, including the installation of OpenMPI and mpi4py.
  • Detailed step-by-step guidance for defining, testing, and executing a model on an MPI cluster.
  • Demonstrations of saving and loading results using pickle.

The second part (9e04566) extends the tutorial with specific instructions for the DelftBlue supercomputer, which uses the SLURM scheduler:

  • Detailed guide to crafting a SLURM script specifically for the DelftBlue supercomputer.
  • Insights into DelftBlue's capabilities and hardware specifications.
  • Comprehensive steps for environment setup, user authentication, file transfers, job scheduling, and retrieval of results on DelftBlue.

Together, the tutorial provides an end-to-end guide for leveraging the MPIEvaluator on HPC systems.

@EwoutH EwoutH added the docs label Nov 3, 2023
@review-notebook-app
Copy link
Copy Markdown

review-notebook-app bot commented Nov 3, 2023

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@EwoutH EwoutH requested a review from quaquel November 3, 2023 13:37
@EwoutH EwoutH force-pushed the MPI-tutorial branch 2 times, most recently from 5364a18 to b426fb0 Compare November 3, 2023 13:59
@EwoutH
Copy link
Copy Markdown
Collaborator Author

EwoutH commented Nov 3, 2023

I have no idea why the ReadtheDocs build is failing.

nbsphinx.NotebookError: NoSuchKernel in indepth_tutorial/mpi-evaluator.ipynb:
No such kernel named python3

I compared the notebook metadata to general-introduction.ipynb and directed-search.ipynb, and it looks practically identical. directed-search.ipynb:

 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },

mpi-evaluator.ipynb

 "metadata": {
  "kernelspec": {
   "name": "python3",
   "language": "python",
   "display_name": "Python 3 (ipykernel)"
  },

@EwoutH EwoutH force-pushed the MPI-tutorial branch 3 times, most recently from e3bb224 to 7ba1ddc Compare November 3, 2023 14:45
@EwoutH EwoutH removed the request for review from quaquel November 3, 2023 14:49
@EwoutH
Copy link
Copy Markdown
Collaborator Author

EwoutH commented Nov 3, 2023

One of those cases you spend more than an hour to find that one exact line code is a sea of possible configuration options. But it now works.

Ready for review, merge (by fast-forward) after #299.

@EwoutH EwoutH requested a review from quaquel November 3, 2023 15:15
@EwoutH EwoutH mentioned this pull request Nov 3, 2023
Merged
quaquel
quaquel reviewed Nov 15, 2023
View reviewed changes
@coveralls
Copy link
Copy Markdown

coveralls commented Nov 15, 2023

Coverage Status

coverage: 80.585%. remained the same
when pulling 0616fca on MPI-tutorial
into 65e0fc0 on master.

@EwoutH
Docs: Add MPIEvaluator tutorial for multi-node HPC systems
b206fe6 
- Introduced a detailed tutorial on how to use the new MPIEvaluator from EMAworkbench.
- Covered environment setup, including OpenMPI and mpi4py installations.
- Provided step-by-step instructions for defining, testing, and running a model on an MPI cluster.
- Illustrated how to save and load results using pickle.

This is the first part of the tutorial, the second part will follow in the next commit and cover how to run on HPC systems with a SLURM scheduler, like DelftBlue.
@EwoutH
Docs: Update MPI tutorial with DelftBlue / SLURM instructions
1831bd8 
Added a detailed guide on how to run the model on the DelftBlue supercomputer, which utilizes the SLURM scheduler:
  - Steps for creating a SLURM script tailored for DelftBlue.
  - Overview of the DelftBlue supercomputer's capabilities.
  - Instructions on environment setup, logging in, file transfer, job scheduling, and result retrieval for DelftBlue.
@EwoutH
Copy link
Copy Markdown
Collaborator Author

EwoutH commented Nov 15, 2023 •
edited
Loading

@quaquel Thanks for reviewing, I implemented the feedback and cleaned up the commits. If you approve we can fast-forward merge.

Edit: CI doesn't run on docs changes, and currently there is a required check which will prevent merging. Personally I don't think we need the required checks, we won't merge (blindly) when CI fails anyway.

@EwoutH
Copy link
Copy Markdown
Collaborator Author

EwoutH commented Nov 16, 2023

Required statuses must pass before merging

Can you remove the required check? Otherwise I cannot merge.

@EwoutH EwoutH merged commit 1831bd8 into master Nov 16, 2023
@EwoutH EwoutH deleted the MPI-tutorial branch November 16, 2023 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@quaquel quaquel quaquel approved these changes

Assignees

No one assigned

Labels

docs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@EwoutH @coveralls @quaquel