Is there a way to use RunspacePools such that module loading uses more than one processor? #7035
Description
I'm experimenting with running PowerShell unit tests in parallel. It seems like one open runspace per thread is needed to achieve speedup. I'm seeing good speedup after the runspaces are open (see #6965(comment). I haven't, however, found a reliable way to parallelize module importing that occurs when opening the runspaces. (Runspaces.OpenAsync() looks promising, but it seems to suffer from #7034.)
Calling PowerShell.BeginInvoke() on a several of PowerShell instances sharing a runspace pool seems to block while module loading completes. The result is that despite that there might be many processors available and one runspace for each, only one processor is utilized to perform module importing for all of the runspaces.
Steps to reproduce
$processorCount = [System.Environment]::ProcessorCount
Write-Host "Processor Count: $processorCount"
$moduleContent = {
function fibonacci {
param([int]$n)
[bigint]$a=0
[bigint]$b=1
foreach ($x in 0..$n)
{
$a,$b = $b,($a+$b)
}
$b
}
0..0 | % { fibonacci 100000 }
}
$modulePath = "$([System.IO.Path]::GetTempPath())slowLoading.psm1"
$moduleContent | Set-Content $modulePath
$t_import = Measure-Command {
Import-Module $modulePath -Force
}
$initialSessionState = [initialsessionstate]::CreateDefault()
$initialSessionState.ImportPSModule($modulePath)
$rsp = [runspacefactory]::CreateRunspacePool($initialSessionState)
$rsp.SetMaxRunspaces($processorCount) | Out-Null
$rsp.Open()
$ps = 1..$processorCount | % { [powershell]::Create().AddScript({'done'}) }
$ps | % { $_.RunspacePool = $rsp }
$t_begin = Measure-Command {
$invocation = $ps.BeginInvoke()
}
$t_wait = Measure-Command {
while ( $invocation.IsCompleted -contains $false )
{
sleep 0.1
}
}
[pscustomobject]@{
'name' = 'Import-Module slowLoading.psm1'
'time(ms)' = [int]$t_import.TotalMilliseconds
}
[pscustomobject]@{
'name' = 'BeginInvoke()'
'time(ms)' = [int]$t_begin.TotalMilliseconds
}
[pscustomobject]@{
'name' = 'Wait'
'time(ms)' = [int]$t_wait.TotalMilliseconds
}
Expected behavior
Processor Count: 16
name time(ms)
---- --------
Import-Module slowLoading.psm1 5122
BeginInvoke() 100 (approximately)
Wait 6000 (approximately)
Actual behavior
Processor Count: 16
name time(ms)
---- --------
Import-Module slowLoading.psm1 5122
BeginInvoke() 19452
Wait 13
Here is the CPU utilization graph for the above test run:
Environment data
> $PSVersionTable
Name Value
---- -----
PSVersion 6.1.0-preview.691
PSEdition Core
GitCommitId v6.1.0-preview.691
OS Microsoft Windows 6.3.9600
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0