Skip to content

Vine/WQ: Serial Task Latency #3432

@dthain

Description

@dthain

@benclifford reports:

The parsl+wq and parsl+taskvine test suite takes 493s to run, vs 8s for 
the same tests run in parsl+htex.

Right now that's the dominant time in our CI - lets say 15 minutes out of 
a 22 minute CI runtime.

I think that's coming from sequential task handling somewhere in the 
WQExecutor/WQ stack taking around 1 second: where task B is not submitted 
until the parsl DFK has become aware of task A completing. (task A being 
one test, task B being the next test, or in some cases task B being a task 
inside a test that depends on task A completing).

Is there a way to make this faster? In principle, parsl+wq coprocesses 
should be down at the 8s mark too.
The bit of CI that does this test is this:

pytest parsl/tests/ -k "not cleannet and not issue363" --config parsl/tests/configs/workqueue_ex.py

(that's the same test suite parameterised by different test 
configurations, so --config parsl/tests/configs/htex_local.py will get you 
the same tests but through the high throughput executor)

Metadata

Metadata

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions