@bergmeister Could you please clarify - running some Test-ModuleManifest-s in parallel raises exceptions? For the same module or different module?

@iSazonov For the same module manifest if I understand the workings of PSSA correctly (not just because it was easier to repro but I think this is also what happens in sporadic production failures because PSSA runs all rules in parallel foreach file and there are 2 rules that call it to get the PSModuleInfo and check if the module manifest is in a certain expected, 'healthy' state). Yes, in theory, PSSA could try to have a (thread safe) cache similar to the CommandInfo cache but to me it seems a general issue that thread safety has been missed out in this area of PowerShell or maybe experienced reviewers can say if the problem should be fixed at a higher architectural level.

I'm not sure this is the right fix.

First, I wonder why a ModuleInfo is shared across threads. If the instance comes from the same runspace, then I think this is a PSSA problem - much of PowerShell avoids locks as it assumes you'll only use something from the origin runspace.

If the ModuleInfo is meant to be shared, then I'd expect locking somewhere else. With this change, I wouldn't be surprised to see duplicate items added.

I'm not sure this is the right fix.

Yes, I agree too. At first look the exceptions come from enumeration (ex., exportedCmdlets property) in time when second runspace (thread) changes the same property - what is the second action? module autoloading? If so second thread must wait until first one finish to load the module and a right fix would be to isolate the module until finish the module loading.
Athother fix. I remember the suggestion that this cmdlet should not cause a module (module manifest) to be loaded/executed.

@lzybkr @iSazonov
For context, here is how PSSA calls Test-ModuleManifest in the Singleton Helper class by creating an instance of PowerShell each time (that gets disposed afterwards) but with a re-used runspace pool (previous versions did not re-use the runspace pool and suffered from the same issue as well though).
https://github.com/PowerShell/PSScriptAnalyzer/blob/0bef1c8b8ddf5b570dd0b6c90f0d0257b94eb374/Engine/Helper.cs#L306-L315

using (var ps = System.Management.Automation.PowerShell.Create())
{
    ps.RunspacePool = _runSpacePool;
    ps.AddCommand("Test-ModuleManifest")
        .AddParameter("Path", filePath)
        .AddParameter("WarningAction", ActionPreference.SilentlyContinue);
    try
    {
        psObj = ps.Invoke();
    }
    { ... }
}

The above code gets called from from different threads (each rule has its own thread) that are started as follows here:
https://github.com/PowerShell/PSScriptAnalyzer/blob/a68b13956ab1136af91066d1ce61113574447945/Engine/ScriptAnalyzer.cs#L2075

allowedRules.Select(scriptRule => Task.Factory.StartNew(() => .... );

Do me this looks like legit usage of the PowerShell API, i.e. the PSModuleInfo is not really shared across PSSA's threads if I understand correctly. Would you otherwise expect any user of the PowerShell API to have a (heavy) lock around PowerShell.Invoke() calls? PSSA is by nature very multi-threaded and at least for Get-Command, concurrency has not been an issue.

@bergmeister - the PowerShell api usage looks fine, so I'm assuming a PowerShell data structure is being shared across threads, in which case this comment applies:

If the ModuleInfo is meant to be shared, then I'd expect locking somewhere else. With this change, I wouldn't be surprised to see duplicate items added.

@lzybkr OK, so should the locking then be somewhere in the Test-ModuleManifest cmdlet or somewhere deep inside ModuleCmdletBase or PowerShell.cs? I assume the latter. Can you please give some pointers where to look, what to debug, etc.
The stack trace is below and I uploaded a minidump with heap here from the latest master commit bdc8548700118cea6c31420c2f7478785e6d84ac of PowerShell for one of the most common exceptions (CmdletInvocationException with inner exception from System.Collections.ListDictionaryInternal: 'Collection was modified; enumeration operation may not execute.')

   at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input) in C:\Users\chris\PowerShell\src\System.Management.Automation\engine\hostifaces\pipelinebase.cs:line 416
   at System.Management.Automation.Runspaces.Pipeline.Invoke() in C:\Users\chris\PowerShell\src\System.Management.Automation\engine\hostifaces\Pipeline.cs:line 531
   at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke) in C:\Users\chris\PowerShell\src\System.Management.Automation\engine\hostifaces\PowerShell.cs:line 5619
   at System.Management.Automation.PowerShell.Worker.CreateRunspaceIfNeededAndDoWork(Runspace rsToUse, Boolean isSync) in C:\Users\chris\PowerShell\src\System.Management.Automation\engine\hostifaces\PowerShell.cs:line 5466
   at System.Management.Automation.PowerShell.CoreInvokeHelper[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings) in C:\Users\chris\PowerShell\src\System.Management.Automation\engine\hostifaces\PowerShell.cs:line 4588
   at System.Management.Automation.PowerShell.CoreInvoke[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings) in C:\Users\chris\PowerShell\src\System.Management.Automation\engine\hostifaces\PowerShell.cs:line 4780
   at System.Management.Automation.PowerShell.CoreInvoke[TOutput](IEnumerable input, PSDataCollection`1 output, PSInvocationSettings settings) in C:\Users\chris\PowerShell\src\System.Management.Automation\engine\hostifaces\PowerShell.cs:line 4530
   at System.Management.Automation.PowerShell.Invoke(IEnumerable input, PSInvocationSettings settings) in C:\Users\chris\PowerShell\src\System.Management.Automation\engine\hostifaces\PowerShell.cs:line 2425
   at System.Management.Automation.PowerShell.Invoke() in C:\Users\chris\PowerShell\src\System.Management.Automation\engine\hostifaces\PowerShell.cs:line 2296
   at Microsoft.Windows.PowerShell.ScriptAnalyzer.BuiltinRules.AvoidUsingDeprecatedManifestFields.AnalyzeScript(Ast ast, String fileName)+MoveNext()

I wonder if the call to return Output.NonBlockingRead(Int32.MaxValue); in pipelinebase.cs (line 416) would need to be blocking?

@bergmeister - I'm honestly not too sure, but ModuleCmdletBase is probably the closest to where I'd look at. Definitely not PowerShell.cs and maybe not Test-ModuleManifest. Maybe we should be cloning module info instances somewhere?

I'd really have to dig to provide a more concrete direction.

@bergmeister If it is time critical you could make a workaround in PSSA for the rule. (I guess that the fix can take more time in PowerShell engine).

@KirkMunro If you are digging in Import-Module you could keep in mind the race condition issue too.

@iSazonov We have merged a PR yesterday in PSSA to workaround this by having a lock around calls to Test-ModuleManifest but since this might be a general issue in the PS code base it might still be wortwhile trying to fix it. I think I will open an issue with more details and easier REPRO steps in the next steps. Do you want me to close this PR then once I've logged the issue?

@bergmeister I agree to close the PR and open new issue.

Opened issue #9889 and will therefore close this PR for now