Skip to content

add KernelNanny for remote signaling#514

Merged
minrk merged 2 commits intoipython:mainfrom
minrk:signal-engines
Jul 7, 2021
Merged

add KernelNanny for remote signaling#514
minrk merged 2 commits intoipython:mainfrom
minrk:signal-engines

Conversation

@minrk
Copy link
Copy Markdown
Member

@minrk minrk commented Jul 5, 2021

Nanny proxies control channel, so it can intercept control messages and handle those

Signaling engines directly solves two problems with signaling via e.g. mpiexec and the Cluster API:

  1. subsets of engines can be addressed, useful for interrupting a single task
  2. different mpiexec implementations may not handle important signals, such as SIGINT which openmpi will forward exactly once before forcibly shutting down engines

In the future, this could also enable abort_request to trigger an interrupt if the to-be-aborted message happens to be processing at the moment. The nanny could subscribe to the IOPub channel to watch status messages to determine what task is currently processing

Nanny proxies control channel
- add Client.send_signal for signaling engines
- notify Hub immediately on process exit instead of waiting for heart failure
- exit Engine on any error during completion of registration
- consolidate `_send_control_request` for control-channel requests
- fix some mixed str/bytes uuid/identity logic in the Hub
Send all process output to a file

For easier retrieval of output on failure
@minrk minrk force-pushed the signal-engines branch from 88e1056 to 5762231 Compare July 5, 2021 12:57
@minrk minrk merged commit 7d9f3d1 into ipython:main Jul 7, 2021
@minrk minrk deleted the signal-engines branch July 7, 2021 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

notify Hub of engine stop from launchers Deal with subprocess output without tornado Add registration information to launchers

1 participant