Skip to content

Unhandled error when shutting down from client #270

@rueberger

Description

@rueberger

Example

If you start a cluster manually and then run:

>c = Client(profile='default')
>c.shutdown(hub=True)  # no errors thrown here

The controller throws an error:

> ipcontroller --profile=default
2017-06-15 13:11:05.160 [IPControllerApp] Hub listening on tcp://127.0.0.1:50566 for registration.
2017-06-15 13:11:05.166 [IPControllerApp] Hub using DB backend: 'DictDB'
2017-06-15 13:11:05.419 [IPControllerApp] hub::created hub
2017-06-15 13:11:05.420 [IPControllerApp] writing connection info to /groups/turaga/home/bergera/.ipython/profile_default/security/ipcontroller-client.json
2017-06-15 13:11:05.427 [IPControllerApp] writing connection info to /groups/turaga/home/bergera/.ipython/profile_default/security/ipcontroller-engine.json
2017-06-15 13:11:05.432 [IPControllerApp] task::using Python leastload Task scheduler
2017-06-15 13:11:05.432 [IPControllerApp] Heartmonitor started
2017-06-15 13:11:05.447 [IPControllerApp] Creating pid file: /groups/turaga/home/bergera/.ipython/profile_default/pid/ipcontroller.pid
2017-06-15 13:11:05.465 [scheduler] Scheduler started [leastload]
2017-06-15 13:11:05.467 [IPControllerApp] client::client b'\x00k\x8bEg' requested 'connection_request'
2017-06-15 13:11:05.468 [IPControllerApp] client::client [b'\x00k\x8bEg'] connected
2017-06-15 13:11:19.363 [IPControllerApp] client::client b'aa3e92a5-1e34-4708-b364-abe2a0c6830a' requested 'registration_request'
2017-06-15 13:11:23.435 [IPControllerApp] registration::finished registering engine 0:aa3e92a5-1e34-4708-b364-abe2a0c6830a
2017-06-15 13:11:23.436 [IPControllerApp] engine::Engine Connected: 0
2017-06-15 13:11:31.529 [IPControllerApp] client::client b'\x00k\x8bEh' requested 'connection_request'
2017-06-15 13:11:31.529 [IPControllerApp] client::client [b'\x00k\x8bEh'] connected
2017-06-15 13:11:37.322 [IPControllerApp] ERROR | Uncaught exception in <function Hub.dispatch_monitor_traffic at 0x7fab85862a60>
Traceback (most recent call last):
  File "/groups/turaga/home/bergera/anaconda3/envs/tf_gpu2/lib/python3.5/site-packages/ipyparallel/util.py", line 110, in log_errors
    return f(self, *args, **kwargs)
  File "/groups/turaga/home/bergera/anaconda3/envs/tf_gpu2/lib/python3.5/site-packages/ipyparallel/controller/hub.py", line 513, in dispatch_monitor_traffic
    handler(idents, msg)
  File "/groups/turaga/home/bergera/anaconda3/envs/tf_gpu2/lib/python3.5/site-packages/ipyparallel/controller/hub.py", line 839, in monitor_iopub_message
    uuid = self.engines[eid].uuid
KeyError
2017-06-15 13:11:37.329 [IPControllerApp] client::client b'\x00k\x8bEh' requested 'shutdown_request'
2017-06-15 13:11:37.332 [scheduler] Unhandled message type: 'shutdown_notification'
ERROR:scheduler:Unhandled message type: 'shutdown_notification'
2017-06-15 13:11:38.331 [IPControllerApp] hub::hub shutting down.

Engine session:

>ipengine --profile=default
2017-06-15 13:11:19.349 [IPEngineApp] Loading url_file '/groups/turaga/home/bergera/.ipython/profile_default/security/ipcontroller-engine.json'
2017-06-15 13:11:19.361 [IPEngineApp] Registering with controller at tcp://127.0.0.1:50566
2017-06-15 13:11:19.469 [IPEngineApp] Starting to monitor the heartbeat signal from the hub every 3010 ms.
2017-06-15 13:11:19.476 [IPEngineApp] Completed registration with id 0

Environment info

  • python: 3.5.2
  • ipyparallel: 6.0.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions