Worker and scheduler disagree about key/dep's status #4550
Description
The following was reported to me by a colleague. It seems slightly familiar to work that @gforsyth and @fjetter have been playing with recently
Situation: S thinks W is processing K, but I suspect W's story suggests otherwise.
The key K appears in client.processing()[W]. And the story from W:
k = "('nonan-d6dda2c8a4eb8ab1b66bab78b66cc0ff', 0, 28, 0)"
w = "tcp://worker-1"
client.run(lambda dask_worker: dask_worker.story(k), workers=[w])
gives:
[("('nonan-d6dda2c8a4eb8ab1b66bab78b66cc0ff', 0, 28, 0)", 'new-dep', 'waiting'),
("('nonan-d6dda2c8a4eb8ab1b66bab78b66cc0ff', 0, 28, 0)", 'release-key'),
("('nonan-d6dda2c8a4eb8ab1b66bab78b66cc0ff', 0, 28, 0)", 'new'),
('gather-dependencies', "('nonan-d6dda2c8a4eb8ab1b66bab78b66cc0ff', 0, 28, 0)", []),
("('nonan-d6dda2c8a4eb8ab1b66bab78b66cc0ff', 0, 28, 0)", 'waiting', 'flight'),
('request-dep',
"('rechunk-merge-9a91cf92d9ecd81fb909b9660873466b', 28, 0)",
'tcp://worker-2',
["('rechunk-merge-9a91cf92d9ecd81fb909b9660873466b', 28, 0)", "('nonan-d6dda2c8a4eb8ab1b66bab78b66cc0ff', 0, 28, 0)"]),
("('nonan-d6dda2c8a4eb8ab1b66bab78b66cc0ff', 0, 28, 0)", 'release-key'),
("('nonan-d6dda2c8a4eb8ab1b66bab78b66cc0ff', 0, 28, 0)", 'flight', 'waiting')]
Thoughts? Both of K's depenendencies are in memory on W.