-
Notifications
You must be signed in to change notification settings - Fork 125
Vine/WQ: Serial Task Latency #3432
Copy link
Copy link
Closed
Labels
ParslTaskVineWork QueuebugFor modifications that fix a flaw in the code.For modifications that fix a flaw in the code.
Description
@benclifford reports:
The parsl+wq and parsl+taskvine test suite takes 493s to run, vs 8s for
the same tests run in parsl+htex.
Right now that's the dominant time in our CI - lets say 15 minutes out of
a 22 minute CI runtime.
I think that's coming from sequential task handling somewhere in the
WQExecutor/WQ stack taking around 1 second: where task B is not submitted
until the parsl DFK has become aware of task A completing. (task A being
one test, task B being the next test, or in some cases task B being a task
inside a test that depends on task A completing).
Is there a way to make this faster? In principle, parsl+wq coprocesses
should be down at the 8s mark too.
The bit of CI that does this test is this:
pytest parsl/tests/ -k "not cleannet and not issue363" --config parsl/tests/configs/workqueue_ex.py
(that's the same test suite parameterised by different test
configurations, so --config parsl/tests/configs/htex_local.py will get you
the same tests but through the high throughput executor)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
ParslTaskVineWork QueuebugFor modifications that fix a flaw in the code.For modifications that fix a flaw in the code.