Skip to content

Frequent CommClosed issues w/ inproc comms on Prefect #4165

New issue
New issue
Closed
Closed
Frequent CommClosed issues w/ inproc comms on Prefect#4165
@jcrist

Description

@jcrist
jcrist
opened on Oct 12, 2020

In Prefect we're seeing issues with distributed >= 2.23.0 when using inproc comms. Everything works fine in 2.22.0, but in 2.23.0 we see comm closed errors and workers marked as dead.

I have a reproducer with this, but unfortunately haven't been able to reproduce without prefect yet (will keep trying):

Reproducer (requires prefect installed):

from prefect import Flow, task
from prefect.engine.executors import DaskExecutor
from time import sleep


@task
def get_data():
    sleep(0.5)
    return list(range(180))


@task
def inc(user):
    sleep(0.5)
    return user + 1


@task
def accum(data):
    return sum(data)


executor = DaskExecutor(cluster_kwargs={"nanny": False, "memory_limit": 0, "processes": False, "n_workers": 20, "threads_per_worker": 1, "silence_logs": 0})

with Flow("test_flow") as flow:
    accum(inc.map(get_data()))


if __name__ == "__main__":
    flow.run(executor=executor)
Logs on dask & distributed `2.30.0`: 
[2020-10-12 22:12:30] INFO - prefect.FlowRunner | Beginning Flow run for 'test_flow'
distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -   Scheduler at: inproc://192.168.1.36/40918/1
distributed.scheduler - INFO -   dashboard at:            localhost:8787
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/3
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65097
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-84m1pdwc
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/4
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65098
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-tghwjywi
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/5
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65099
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-aa13mh4n
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/6
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65100
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-tgp7cb_b
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/7
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65101
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-4ncd59jb
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/8
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65102
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-50ukew9q
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/9
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65103
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-0unm12uq
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/10
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65104
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-xvxakipk
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/11
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65105
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-88ryax75
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/12
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65106
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-ojz616ac
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/13
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65107
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-l97cqvxc
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/14
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65108
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-e22w6nsn
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/15
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65109
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-lgan9zf1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/16
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65110
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-t3r_s91p
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/17
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65111
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-vk8f4abf
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/18
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65112
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-vip55j4t
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/19
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65113
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-antub0bv
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/20
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65114
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-t4x30du5
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/21
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65115
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-h9kq54de
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -       Start worker at: inproc://192.168.1.36/40918/22
distributed.worker - INFO -          Listening to:         inproc192.168.1.36
distributed.worker - INFO -          dashboard at:         192.168.1.36:65116
distributed.worker - INFO - Waiting to connect to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -       Local Directory: /Users/jcristharif/Code/prefect/dask-worker-space/worker-u5ijheja
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/3', name: 12, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/3
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/4', name: 7, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/4
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/5', name: 11, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/5
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/6', name: 6, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/6
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/7', name: 10, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/7
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/8', name: 5, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/8
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/9', name: 9, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/9
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/10', name: 4, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/10
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/11', name: 0, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/11
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/12', name: 3, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/12
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/13', name: 2, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/13
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/14', name: 1, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/14
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/15', name: 19, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/15
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/16', name: 18, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/16
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/17', name: 8, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/17
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/18', name: 17, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/18
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/19', name: 16, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/19
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/20', name: 15, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/20
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/21', name: 14, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/21
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Register worker <Worker 'inproc://192.168.1.36/40918/22', name: 13, memory: 0, processing: 0>
distributed.scheduler - INFO - Starting worker compute stream, inproc://192.168.1.36/40918/22
distributed.core - INFO - Starting established connection
distributed.worker - INFO -         Registered to: inproc://192.168.1.36/40918/1
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Receive client connection: Client-05baa1fe-0cd8-11eb-9fd6-acde48001122
distributed.core - INFO - Starting established connection
[2020-10-12 22:12:31] INFO - prefect.TaskRunner | Task 'get_data': Starting task run...
[2020-10-12 22:12:31] INFO - prefect.TaskRunner | Task 'get_data': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc': finished task run for task with final state: 'Mapped'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[0]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[3]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[1]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[11]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[12]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[2]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[4]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[5]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[6]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[0]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[7]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[20]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[8]': Starting task run...
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[3]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[1]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:32] INFO - prefect.TaskRunner | Task 'inc[12]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[11]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[9]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[2]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[4]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[10]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[5]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[38]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[6]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[50]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[21]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[49]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[39]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[7]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[20]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[8]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[22]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[9]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[23]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[44]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[10]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[45]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[40]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[46]': Starting task run...
distributed.comm.inproc - WARNING - Closing dangling queue in <InProc  local=inproc://192.168.1.36/40918/64 remote=inproc://192.168.1.36/40918/1>
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[38]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[47]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[48]': Starting task run...
[2020-10-12 22:12:33] INFO - prefect.TaskRunner | Task 'inc[50]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[49]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[21]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[39]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[98]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[22]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[23]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[44]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[29]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[99]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[41]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[30]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[40]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[45]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[46]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[47]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[42]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[48]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[43]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[24]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[98]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[27]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[25]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[26]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[100]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[28]': Starting task run...
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[29]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[99]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[41]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[30]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:34] INFO - prefect.TaskRunner | Task 'inc[78]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[42]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[89]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[43]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[24]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[79]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[101]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[90]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[27]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[25]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[100]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[26]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[28]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[102]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[103]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[84]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[78]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[87]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[89]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[85]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[80]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[101]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[88]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[58]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[86]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[79]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[90]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[69]': Starting task run...
[2020-10-12 22:12:35] INFO - prefect.TaskRunner | Task 'inc[81]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[102]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[59]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[84]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[103]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[70]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[87]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[85]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[82]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[67]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[80]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[64]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[83]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[86]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[58]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[88]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[81]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[69]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[65]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[59]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[60]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[70]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[66]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[118]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[61]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[82]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[64]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[83]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[68]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[129]': Starting task run...
[2020-10-12 22:12:36] INFO - prefect.TaskRunner | Task 'inc[67]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[130]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[65]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[119]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[62]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[63]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[127]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[60]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[118]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[124]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[66]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[61]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[125]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[68]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[129]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[120]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[126]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[121]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[158]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[130]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[119]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[128]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[62]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[63]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[109]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[127]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[124]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[110]': Starting task run...
[2020-10-12 22:12:37] INFO - prefect.TaskRunner | Task 'inc[125]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[159]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[122]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[123]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[126]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[107]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[104]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[120]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[121]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[158]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[105]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[128]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[109]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[106]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[110]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[161]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[160]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[138]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[159]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[108]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[122]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[123]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[139]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[104]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[150]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[149]': Starting task run...
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[107]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:38] INFO - prefect.TaskRunner | Task 'inc[105]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[106]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[162]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[138]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[161]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[147]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[108]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[160]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[164]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[163]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[165]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[139]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[149]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[150]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[146]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[148]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[141]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[178]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[140]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[179]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[162]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[169]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[170]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[147]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[142]': Starting task run...
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[164]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[163]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:39] INFO - prefect.TaskRunner | Task 'inc[165]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[145]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[167]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[144]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[146]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[143]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[148]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[141]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[178]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[140]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[179]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[169]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[170]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[168]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[57]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[142]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[166]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[77]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[117]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[145]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[167]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[144]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[131]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[143]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[37]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[75]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[157]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[53]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[97]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[168]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[94]': Starting task run...
[2020-10-12 22:12:40] INFO - prefect.TaskRunner | Task 'inc[57]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[166]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[77]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[117]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[36]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[92]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[131]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[37]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[72]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[111]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[51]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[91]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[75]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[157]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[97]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[53]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[94]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[113]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[153]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[36]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[92]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[74]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[174]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[34]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[134]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[72]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[111]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[51]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[54]': Starting task run...
[2020-10-12 22:12:41] INFO - prefect.TaskRunner | Task 'inc[91]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[114]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[136]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[113]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[153]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[132]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[172]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[52]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[74]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[31]': Starting task run...
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/10
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[174]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[34]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[134]': finished task run for task with final state: 'Success'
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/11
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/12
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/13
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/14
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/15
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/16
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/17
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/18
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/19
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/11', name: 0, memory: 11, processing: 0>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/11
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/10', name: 4, memory: 11, processing: 1>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/10
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/12', name: 3, memory: 11, processing: 2>
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[54]': finished task run for task with final state: 'Success'
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/12
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/13', name: 2, memory: 11, processing: 3>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/13
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/14', name: 1, memory: 11, processing: 4>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/14
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/15', name: 19, memory: 10, processing: 5>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/15
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/16', name: 18, memory: 10, processing: 6>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/16
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/17', name: 8, memory: 11, processing: 7>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/17
distributed.batched - INFO - Batched Comm Closed: 
distributed.batched - INFO - Batched Comm Closed: 
distributed.batched - INFO - Batched Comm Closed: 
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/18', name: 17, memory: 10, processing: 9>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/18
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/19', name: 16, memory: 10, processing: 10>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/19
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/20
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[152]': Starting task run...
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[114]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[136]': finished task run for task with final state: 'Success'
distributed.batched - INFO - Batched Comm Closed: 
distributed.batched - INFO - Batched Comm Closed: 
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/21
distributed.worker - INFO - Stopping worker at inproc://192.168.1.36/40918/22
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[132]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[172]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[52]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:42] INFO - prefect.TaskRunner | Task 'inc[31]': finished task run for task with final state: 'Success'
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.worker - INFO - Comm closed
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/20', name: 15, memory: 10, processing: 13>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/20
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/21', name: 14, memory: 11, processing: 17>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/21
distributed.scheduler - INFO - Remove worker <Worker 'inproc://192.168.1.36/40918/22', name: 13, memory: 11, processing: 21>
distributed.core - INFO - Removing comms to inproc://192.168.1.36/40918/22
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[152]': finished task run for task with final state: 'Success'
distributed.worker - INFO - Comm closed
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[95]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[71]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[55]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[96]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[137]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[32]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[33]': Starting task run...
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[95]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[71]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[55]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[96]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:43] INFO - prefect.TaskRunner | Task 'inc[33]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[137]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[32]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[23]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[7]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[9]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'get_data': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[3]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[10]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[0]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[23]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[9]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[51]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[7]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'get_data': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[8]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[0]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[10]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[3]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[6]': Starting task run...
[2020-10-12 22:12:44] INFO - prefect.TaskRunner | Task 'inc[5]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[1]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[53]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[22]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[51]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[45]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[8]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[4]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[6]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[5]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[12]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[11]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[1]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[22]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[53]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[30]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[2]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[29]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[45]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[40]': Starting task run...
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[4]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:45] INFO - prefect.TaskRunner | Task 'inc[20]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[12]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[35]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[11]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[41]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[30]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[2]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[29]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[47]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[21]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[40]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[46]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[20]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[31]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[35]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[50]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[37]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[41]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[27]': Starting task run...
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[47]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[46]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:46] INFO - prefect.TaskRunner | Task 'inc[21]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[54]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[38]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[26]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[50]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[31]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[37]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[27]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[44]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[70]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[28]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[42]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[54]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[38]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[26]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[48]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[49]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[67]': Starting task run...
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[44]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[28]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[70]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[42]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:47] INFO - prefect.TaskRunner | Task 'inc[90]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[34]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[63]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[25]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[48]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[49]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[36]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[67]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[24]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[69]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[90]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[34]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[63]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[25]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[103]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[52]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[130]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[39]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[36]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[102]': Starting task run...
[2020-10-12 22:12:48] INFO - prefect.TaskRunner | Task 'inc[24]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[69]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[43]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[62]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[103]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[52]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[130]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[39]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[80]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[124]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[66]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[77]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[102]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[97]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[43]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[62]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[64]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[65]': Starting task run...
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[80]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[66]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:49] INFO - prefect.TaskRunner | Task 'inc[124]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[77]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[57]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[128]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[83]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[97]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[94]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[100]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[64]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[88]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[65]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[84]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[57]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[92]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[128]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[83]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[94]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[100]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[88]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[76]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[109]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[84]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[87]': Starting task run...
[2020-10-12 22:12:50] INFO - prefect.TaskRunner | Task 'inc[79]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[82]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[72]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[92]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[74]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[76]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[109]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[56]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[79]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[87]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[82]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[136]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[72]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[89]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[58]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[86]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[74]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[98]': Starting task run...
[2020-10-12 22:12:51] INFO - prefect.TaskRunner | Task 'inc[68]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[56]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[112]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[136]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[106]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[89]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[58]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[86]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[98]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[68]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[81]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[117]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[101]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[99]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[85]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[112]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[129]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[106]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[118]': Starting task run...
[2020-10-12 22:12:52] INFO - prefect.TaskRunner | Task 'inc[81]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[61]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[101]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[117]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[85]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[99]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[114]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[113]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[129]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[93]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[118]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[91]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[120]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[135]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[61]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[132]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[113]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[114]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[105]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[121]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[93]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[91]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[73]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[120]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[75]': Starting task run...
[2020-10-12 22:12:53] INFO - prefect.TaskRunner | Task 'inc[135]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[116]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[115]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[132]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[107]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[105]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[121]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[125]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[127]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[73]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[123]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[75]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[116]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[115]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[59]': Starting task run...
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[107]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:54] INFO - prefect.TaskRunner | Task 'inc[138]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[160]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[125]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[127]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[123]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[126]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[134]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[131]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[111]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[59]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[60]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[138]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[165]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[160]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[147]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[126]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[148]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[111]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[134]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[131]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[60]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[104]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[119]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[152]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[78]': Starting task run...
[2020-10-12 22:12:55] INFO - prefect.TaskRunner | Task 'inc[165]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[153]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[147]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[154]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[148]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[163]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[104]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[152]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[119]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[78]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[122]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[153]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[141]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[110]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[133]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[154]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[161]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[163]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[144]': Starting task run...
[2020-10-12 22:12:56] INFO - prefect.TaskRunner | Task 'inc[164]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[122]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[133]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[157]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[141]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[110]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[150]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[161]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[145]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[108]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[164]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[159]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[144]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[142]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[162]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[157]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[169]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[150]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[158]': Starting task run...
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[145]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:57] INFO - prefect.TaskRunner | Task 'inc[108]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[159]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[140]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[149]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[156]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[142]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[162]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[139]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[151]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[169]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[174]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[158]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[143]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[140]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[149]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[156]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[146]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[168]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[139]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[170]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[151]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[155]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[171]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[174]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[177]': Starting task run...
[2020-10-12 22:12:58] INFO - prefect.TaskRunner | Task 'inc[143]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[172]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[146]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[167]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[168]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[170]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[176]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[166]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[171]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[155]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[177]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[178]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[172]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[179]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[173]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[175]': Starting task run...
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[167]': finished task run for task with final state: 'Success'
[2020-10-12 22:12:59] INFO - prefect.TaskRunner | Task 'inc[176]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[166]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[178]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[179]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[173]': finished task run for task with final state: 'Success'
[2020-10-12 22:13:00] INFO - prefect.TaskRunner | Task 'inc[175]': finished task run for task with final state: 'Success'
distributed.worker - WARNING -  Compute Failed
Function:  _maybe_run
args:      ('prefect-ea630327465142ab9810ea3f6966882e', <function run_task at 0x7fdaad74c310>)
kwargs:    {'task': <Task: accum>, 'state': None, 'upstream_states': {<Edge(key=data, mapped=False, flattened=False): inc to accum>: <Mapped: "Ready to proceed with mapping.">}, 'context': {'config': <Box: {'debug': False, 'home_dir': '/Users/jcristharif/.prefect', 'backend': 'cloud', 'server': {'host': 'http://localhost', 'port': 4200, 'host_port': 4200, 'endpoint': 'http://localhost:4200', 'database': {'host': 'localhost', 'port': 5432, 'host_port': 5432, 'name': 'prefect_server', 'username': 'prefect', 'password': 'test-password', 'connection_url': 'postgresql://prefect:test-password@localhost:5432/prefect_server', 'volume_path': '/Users/jcristharif/.prefect/pg_data'}, 'graphql': {'host': '0.0.0.0', 'port': 4201, 'host_port': 4201, 'debug': False, 'path': '/graphql/'}, 'hasura': {'host': 'localhost', 'port': 3000, 'host_port': 3000, 'admin_secret': '', 'claims_namespace': 'hasura-claims', 'graphql_url': 'http://localhost:3000/v1alpha1/graphql', 'ws_url': 'ws://localhost:3000/v1alpha1/graphql',
Exception: AttributeError("'NoneType' object has no attribute 'result'")

distributed.scheduler - INFO - Remove client variable-prefect-ea630327465142ab9810ea3f6966882e
distributed.scheduler - INFO - Remove client Client-05baa1fe-0cd8-11eb-9fd6-acde48001122
distributed.scheduler - INFO - Remove client Client-05baa1fe-0cd8-11eb-9fd6-acde48001122
distributed.scheduler - INFO - Close client connection: Client-05baa1fe-0cd8-11eb-9fd6-acde48001122

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions