TaskVine Futures: High latency on chained tasks. #3892
Description
This is related to PR #3876, but different enough that I figured I'd open it as a separate issue to track.
I was using the vine.FuturesExecutor and getting pretty poor task latency (> 1s) with no-op tasks (using a compute-zen3 node at CHI@TACC). To figure out if it was my vine.FuturesExecutor executor, I also tested with the Parsl TaskVineExecutor and got similarly bad performance.
I wrote a script to test round-trip task time, and it seems the vine.FuturesExecutor and TaskVineExecutor are much slower than just creating PythonTask directly. Is there a reason for this performance difference or have I configured something wrong? I also see quite some variance in the results for the "executor" and "parsl" configurations (as low as 0.25 and as high as 10). The "basic" configuration is a pretty consistent 25 tasks/s.
$ python taskvine.py --tasks 100 basic
Basic -- 25.752 tasks/s
$ python taskvine.py --tasks 100 executor
Executor -- 4.396 tasks/s
$ python taskvine.py --tasks 100 parsl
Parsl -- 1.394 tasks/simport argparse
import time
import ndcctools.taskvine as vine
from parsl.concurrent import ParslPoolExecutor
from parsl.config import Config
from parsl.executors.taskvine import (
TaskVineExecutor,
TaskVineFactoryConfig,
TaskVineManagerConfig,
)
def noop() -> None:
pass
def basic(tasks: int) -> None:
manager = vine.Manager(9123)
factory = vine.Factory(manager=manager)
factory.min_workers = 1
factory.start()
start = time.perf_counter()
for i in range(tasks):
task = vine.PythonTask(noop)
manager.submit(task)
while not manager.empty():
result = manager.wait(5)
if result:
if isinstance(result.output, vine.PythonTaskNoResult):
print(f'Task {result.id} failed.')
total = time.perf_counter() - start
print(f'Basic -- {tasks/total:.3f} tasks/s')
def executor(tasks: int) -> None:
with vine.FuturesExecutor(
manager_name='taskvine-manager',
opts={'min_workers': 1},
) as executor:
start = time.perf_counter()
for i in range(tasks):
future = executor.submit(noop)
future.result()
total = time.perf_counter() - start
print(f'Executor -- {tasks/total:.3f} tasks/s')
def parsl(tasks: int) -> None:
config = Config(
executors=[
TaskVineExecutor(
label="parsl-vine-example",
manager_config=TaskVineManagerConfig(),
factory_config=TaskVineFactoryConfig(min_workers=1),
)
],
run_dir='parsl-runinfo',
)
with ParslPoolExecutor(config) as executor:
start = time.perf_counter()
for i in range(tasks):
future = executor.submit(noop)
future.result()
total = time.perf_counter() - start
print(f'Parsl -- {tasks/total:.3f} tasks/s')
def main():
parser = argparse.ArgumentParser()
parser.add_argument('method', choices=['basic', 'executor', 'parsl'])
parser.add_argument('--tasks', default=10, type=int)
args = parser.parse_args()
if args.method == 'basic':
basic(args.tasks)
elif args.method == 'executor':
executor(args.tasks)
elif args.method == 'parsl':
parsl(args.tasks)
if __name__ == '__main__':
main()@benclifford pointed me at Issue #3432 which seems like a similar performance problem. It's unclear from the comments if the issue was only resolved in WorkQueue or also TaskVine.