A commonly requested feature is to emit tasks from other tasks. There are a variety of dynamic and data-dependent algorithms that are easy to write down in this way.

There are both API and technical considerations to solving this problem. It's not clear how emitting additional tasks is best accomplished.

Tornado-like example

Here is an example that looks tornado-y where the parent task depends on the children.

@dask.coroutine
def fib(n):
    if n in (0, 1):
        return n 
    else:
        a, b = yield [fib(i - 1), fib(i - 2)]
        return a + b

In this case the worker would start running fib in a thread, stop once it got to the yield line, send back the two tasks to the scheduler to be run somewhere, and then the computation would restart once those two tasks were finished. We can do this yield trick multiple times for the same function.

This requires us to build a significant amount of logic into the Worker (which was previously pretty simple) and also introduces lots of questions about how this plays with cancellation, errors, etc..

Side effects

Another approach is to assume that the parent task does not depend on its children, but instead is just relying on them for side effects.

def f(i):
    if i > 10:
        emit(f, i / 2)
    if i > 5:
        emit(f, i - 1)
    return i

In this case it's not clear who cares about the emitted tasks, so this would only be of use if we were to build the sorts of task-channel machinery necessary for streaming systems.

Option 3?

???

I'm genuinely not certain how people use this sort of thing. I just know that it's frequently requested. I encourage people with experience in these sorts of things to speak up about which abstract models are commonly found in the wild and what their strengths and drawbacks are.

My guess is that people prefer the tornado-like solution. This is non-trivial but feasible with a fair amount of effort. Just to set expectations, this is nowhere on my immediate roadmap.