[21:04:47] Task [xgboost.dask-0]:tcp://127.0.0.1:40689 got rank 0
[21:04:47] Task [xgboost.dask-1]:tcp://127.0.0.1:34101 got rank 1
[21:05:21] [0] dt-rmse:34901.35366
[21:05:21] [1] dt-rmse:34873.19654
[21:05:22] [2] dt-rmse:34842.21352
[21:05:22] [3] dt-rmse:34808.43900
[21:05:22] [4] dt-rmse:34777.91744
[21:05:22] [5] dt-rmse:34749.22733
[21:05:23] [6] dt-rmse:34715.53136
[21:05:23] [7] dt-rmse:34684.79391
[21:05:23] [8] dt-rmse:34651.51852
[21:05:23] [9] dt-rmse:34617.87699
[21:05:24] [10] dt-rmse:34584.33838
2025-11-11 21:05:26,575 - distributed.scheduler - WARNING - Removing worker 'tcp://127.0.0.1:40689' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'dict-c61bc932-9a43-45c7-8b34-c0a6717a70aa'} (stimulus_id='handle-worker-cleanup-1762895126.574773')
2025-11-11 21:05:26,758 - distributed.nanny - WARNING - Restarting worker
2025-11-11 21:05:27,656 - distributed.scheduler - WARNING - Removing worker 'tcp://127.0.0.1:34101' caused the cluster to lose already computed task(s), which will be recomputed elsewhere: {'dict-1cdb34f6-2459-46a8-a17e-adb372e774e9'} (stimulus_id='handle-worker-cleanup-1762895127.6557338')
2025-11-11 21:05:27,787 - distributed.nanny - WARNING - Restarting worker
2025-11-11 21:05:55,828 - distributed.worker - ERROR - Compute Failed
Key: fn-009ab801-487a-4db6-97a9-5e937b0bb91b
State: executing
Task: <Task 'fn-009ab801-487a-4db6-97a9-5e937b0bb91b' fn(...)>
Exception: "ValueError('Invalid worker address: tcp://127.0.0.1:42895, expecting tcp://127.0.0.1:40689. This is likely caused by one of the workers died and Dask re-scheduled a different one. Resilience is not yet supported.')"
Traceback: ' File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/xgboost/dask/__init__.py", line 536, in fn\n raise ValueError(\n'
2025-11-11 21:05:55,847 - distributed.worker.state_machine - WARNING - Async instruction for <Task cancelled name="execute(('read_parquet-fused-2a3f1d7277bc5387a324b35f580bad87', 0))" coro=<Worker.execute() done, defined at /home/adam/xgb-dask/.venv/lib/python3.12/site-packages/distributed/worker_state_machine.py:3607>> ended with CancelledError
Traceback (most recent call last):
File "/home/adam/xgb-dask/train_xgboost_dask.py", line 110, in <module>
train(client)
File "/home/adam/xgb-dask/train_xgboost_dask.py", line 61, in train
output = dxgb.train(
^^^^^^^^^^^
File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/xgboost/core.py", line 774, in inner_f
return func(**kwargs)
^^^^^^^^^^^^^^
File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/xgboost/dask/__init__.py", line 876, in train
return client.sync(
^^^^^^^^^^^^
File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/distributed/utils.py", line 376, in sync
return sync(
^^^^^
File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/distributed/utils.py", line 452, in sync
raise error
File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/distributed/utils.py", line 426, in f
result = yield future
^^^^^^^^^^^^
File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/tornado/gen.py", line 783, in run
value = future.result()
^^^^^^^^^^^^^^^
File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/xgboost/dask/__init__.py", line 810, in _train_async
result = await map_worker_partitions(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/xgboost/dask/__init__.py", line 578, in map_worker_partitions
result = await client.compute(fut).result()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/distributed/client.py", line 404, in _result
raise exc.with_traceback(tb)
File "/home/adam/xgb-dask/.venv/lib/python3.12/site-packages/xgboost/dask/__init__.py", line 536, in fn
raise ValueError(
ValueError: Invalid worker address: tcp://127.0.0.1:42895, expecting tcp://127.0.0.1:40689. This is likely caused by one of the workers died and Dask re-scheduled a different one. Resilience is not yet supported.
I consistently get a crash after the final training iteration when using Dask. This is reproducible using the example code at https://xgboost.readthedocs.io/en/stable/python/dask-examples/gpu_training.html
When running this, I get:
I also get a similar but slightly different error on a larger training run:
Details
This is from an x86_64 machine running Ubuntu 24.04 with Nvidia H100 GPUs, CUDA 12.7, and Nvidia driver 565.57.01. The full set of dependencies from
uv treeis:Details