I've been experimenting with parallelizing powershell unit tests with some success. That use case is mostly CPU-bound and involves large numbers of invocations and importing large script modules. From what I can tell there are a number of non-trivial challenges to overcome to make Invoke-Parallel or parallel ForEach-Object intuitive, robust, and performant for general use.

Below is a summary of the main challenges I have noted during my experimentation.

(BTW, thanks @RamblingCookieMonster and @proxb for your blogs and repos on parallelization. Your work saved me a lot of time.)

Module Importing and Contention

Each concurrent powershell needs its own runspace. Each runspace must import its own modules. This means all the modules used by each runspace must be imported for each runspace. For script modules, importing modules for each scriptblock invocation can easily result in worse performance than a corresponding single-runspace implementation. This is simply because importing script modules takes a significant amount of processor time.

There seems to be contention that prevents runspaces from being opened using multiple cores in parallel (see also #7153 and #7035). I have not yet found a way to, in a single process, open multiple runspaces that import the same script module in a manner that performs better than single-threaded. I suspect this is possible with some changes to PowerShell, though, because importing the same script module in parallel in multiple instances of pwsh.exe seems to parallelize nicely.

Currently there are a couple of open issues related to reliably opening runspaces with imported modules (see #7377 and #7034). I have found some tentative workarounds, but they aren't exactly supported techniques and it's hard to predict how robust such a workaround is in the first place.

Note also that ResetRunspaceState() is limited to variables, so runspaces currently can't simply be re-used in a manner that is guaranteed to be side-effect free.

This all seems to lead to the need for some sort of runspace provider that is more sophisticated than the current RunspacePool. It seems to me like such a runspace provider should share at least compiled script modules between runspaces -- the current RunspacePool doesn't seem to perform in a manner consistent with such sharing. Parallelizing the CPU-bound portions of module importing where possible would also be an improvement.

Output Behavior

Invoking runspaces in parallel means that you can have any combination of progressing, succeeding, and faulting powershells as a result of a single invocation of Invoke-Parallel or ForEach-Object. What, exactly, Invoke-Parallel and ForEach-Object should do with those results is a messy business. Should it throw AggregateException? If so, immediately? Or after all the runspaces are complete? Should the PSDataStreams of all the runspaces be output by Invoke-Parallel? If so, should the outputs be interleaved? Should outputs from the PSDataStreams of one runspace be kept together? From my experiments with some of these behaviors there are pros and cons to the different possible answers to these questions. For each of these questions there doesn't seem to be one good answer that works best for all uses.

Functions Mentioned in Scriptblock

As @powercode pointed out, the functions mentioned in the scriptblock of parallel ForEach-Object might not be available because the scriptblock is being invoked in a difference runspace. This is further complicated in the case where a large script module is auto-loaded when one of its exported functions is mentioned. This will usually cause all the runspaces to try to load that same module at the same time, and because of the contention mentioned above, takes approximately the time to import the module once times the number of runspaces. So for doing a large amount of things in parallel you can easily be waiting minutes just to load modules before any real work begins to occur.

Add a parallel % (foreach-object) #3008

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Add a parallel `%` (foreach-object) #3008