fix(cli/task): run recursive workspace tasks in parallel#34512
Conversation
`deno task -r <name>` (and `deno task --filter <pattern> <name>`) iterated workspace members sequentially, so a long-running task in one member blocked every other member from starting. The fix flattens all matched packages' resolved tasks into one topology that runs through the existing parallel task runner, and extends the colored prefix label to `[pkg#task]` when concurrent tasks span multiple workspace members so identically-named siblings stay distinguishable.
fibibot
left a comment
fibibot
left a comment
There was a problem hiding this comment.
Flattening all matched packages into one shared topology and running it through run_tasks_in_parallel is the right shape: IDs come from the shared sorted.len() so they stay globally unique across packages, and sort_visit's dedup-by-(name, folder_url) now collapses a shared root-task dependency into a single node across members instead of re-running it per package.
One concern on the new test. recursive_parallel deadlocks whenever available_parallelism() (or DENO_JOBS) is < 3. The three dev tasks each block until all three marker files exist, so they must sit in the queue simultaneously — but run_tasks_in_parallel only admits self.concurrency tasks at a time (while queue.len() < self.concurrency). On a 2-core runner the third dev never starts, the first two wait forever, and the test hangs until the 15s timeout fails it. Pin it so it does not depend on host core count: add "envs": { "DENO_JOBS": "3" } to __test__.jsonc.
Holding approval until CI is green — debug builds and linux specs are still running.
The three `dev` tasks block on a filesystem barrier waiting for all three marker files to exist, so they must run concurrently. On runners where `available_parallelism()` returns less than 3, only two of the tasks are admitted into the queue and the test deadlocks until the 15s timeout. Pin `DENO_JOBS=3` so the test is independent of host core count.
lunadogbot
left a comment
lunadogbot
left a comment
There was a problem hiding this comment.
The DENO_JOBS=3 pin in __test__.jsonc (commit 6b39371) addresses my earlier concern about the recursive_parallel test deadlocking on available_parallelism() < 3 runners — the three filesystem-barrier dev tasks now reliably get admitted into the queue regardless of host core count.
Architecture is clean:
sort_tasks_toposwitches from building a per-packageVecto appending into a shared&mut Vec, sosort_visit's dedup-by-(name, folder_url)collapses a shared root-task dependency into a single node across members rather than re-running per package.run_all_tasksreplaces the sequential per-package for-loop with one call to the existingrun_tasks_in_parallel. NotFound semantics correctly shift to "error only when the combined sorted Vec is empty", which is the right shape for recursive/filtered mode.[pkg#task]label expansion is gated on whether concurrent tasks actually span more than one folder URL, so single-package recursive runs keep the old[task]form. Root-package tasks also render as bare[task]becausetask_label_namereturnsNonefor the workspace root — visible independencies_shadowed_root_name.out.
User-visible behavior change worth flagging in release notes (not a blocker): recursive task output is now non-deterministic in ordering — reflected in all the filter/*.out files and recursive_order/task.out switching to UNORDERED_START/END. Anyone string-matching deno task -r output in CI may need to adjust.
CI is fully green.
LGTM.
) `deno task -r <name>` (and `deno task --filter <pattern> <name>`) walked workspace members one at a time, so a long-running task in the first matched member blocked every other member from starting. This was especially painful for the common dev-server case described in denoland#27586, where running `dev` recursively across an `apps/server` + `apps/web` workspace would only ever boot one of them. The fix flattens all matched packages' resolved tasks into a single topology and runs them through the existing parallel task runner, so cross-package siblings execute concurrently while intra-package task dependencies still order correctly. When concurrent tasks span more than one workspace member, the colored output prefix becomes `[pkg#task]` so identically-named siblings stay distinguishable; if a member has no `name` field the folder's directory name is used, except for the workspace root which falls back to the bare task name. Fixes denoland#27586
3 participants
deno task -r <name>(anddeno task --filter <pattern> <name>)walked workspace members one at a time, so a long-running task in the
first matched member blocked every other member from starting. This was
especially painful for the common dev-server case described in #27586,
where running
devrecursively across anapps/server+apps/webworkspace would only ever boot one of them.
The fix flattens all matched packages' resolved tasks into a single
topology and runs them through the existing parallel task runner, so
cross-package siblings execute concurrently while intra-package task
dependencies still order correctly. When concurrent tasks span more
than one workspace member, the colored output prefix becomes
[pkg#task]so identically-named siblings stay distinguishable; if amember has no
namefield the folder's directory name is used, exceptfor the workspace root which falls back to the bare task name.
Fixes #27586