Mapping file descriptors of pwsh process to Powershell streams

Summary

I need a way to specify (using command line arguments to pwsh) a mapping between Powershell streams and process file descriptors . Right now I can't see a way to prevent warnings, progress information, and debug messages from being munged together on stdout alongside data that needs to be in a particular format (e.g. JSON).

Steps to reproduce

Put the following code into a file test:

#!/usr/bin/env pwsh
Write-Warning "test"
Write-Output "[{ foo: 1 }]"

In some shell of your choosing, run the equivalent of:

chmod +x ./test
./test 1>output 2>error

Expected behavior

The file output should contain:

[{ foo: 1 }]

Actual behavior

The file output contains:

WARNING: test
[{ foo: 1 }]

Environment data

> $PSVersionTable
Name                           Value
----                           -----
PSVersion                      6.1.0
PSEdition                      Core
GitCommitId                    6.1.0
OS                             Linux 4.18.9-arch1-1-ARCH #1 SMP PREEMPT Wed Sep 19 21:19:17 UTC 2018
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Alternately, the streams should just be serialized to the file descriptor corresponding to the stream numbers documented here, and anyone who wants to see progress, warning etc. output on stdout can just manually redirect all of those file descriptors into 1.

PowerShell can redirect streams https://blogs.technet.microsoft.com/heyscriptingguy/2014/03/30/understanding-streams-redirection-and-write-host-in-powershell/

Yeah, you'll need to redirect the streams from within a PS session. There's not a complete 1:1 mapping of all PS streams to other console streams, so you'll have to perform this redirection when calling from a PS shell or from within the script itself to be sure PowerShell is actually handling the redirections.

Otherwise, I believe most of these will get squashed into the output / stderr stream(s) by default.

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32) I don't always have access to the script though (nor to the scripts that it transitively calls). From my perspective I'm just trying to evaluate some black box binary which emits data to a variety of output streams, and I only care about retrieving one of them, which is trivial with other shells and script interpreters.

If some functions somewhere do Write-Warning or Write-Debug or whatever, I want to be able to retrieve the output sans that content.

PowerShell has more streams than STDOUT and STDERR, but these are only understood from within PowerShell. By default, I believe it is expected for the extended streams to be sent to STDOUT from pwsh.exe as it can't send it anywhere else. Trying to understand the use case here. If you don't want the Warning stream, you could just add $WarningPreference = "SilentlyContinue" into the script. If you want to redirect those streams to something else, you can do it within the script as well.

Are you asking for something like:

pwsh -warning warning.txt

Steve Lee (@SteveL-MSFT)

Steve Lee (@SteveL-MSFT) Processes also have more file descriptors than just 1 and 2, so maybe the output for each Powershell stream >2 could go to the corresponding file descriptors (at least when you're not running pwsh in a tty).

The use case is to have a script that uses various modules (e.g. PowerCLI) and emits a fixed format. It's going to live alongside a variety of other scripts (most of which are not implemented in Powershell) that emit an identical format.

Right now the script pollutes stdout with warning/progress/debug information emitted by the various modules it uses. If this was instead a bash script, stuff that is emitted on fd 3 by a sourced third party script or exe would not contaminate the output of fd 1 (i.e. stdout).

I can certainly try and suppress everything besides the Powershell output stream at a global level, but I'd prefer to still have this information available if needed. More importantly, I'd like to avoid having to modify the script at all.

Steve Lee (@SteveL-MSFT)

Steve Lee (@SteveL-MSFT)

Are you asking for something like:
pwsh -warning warning.txt

Yes, something like that would be very useful, provided it can be specified exhaustively for all Powershell streams.

Steve Lee (@SteveL-MSFT)

Arguably, what PowerShell should always have done is to map all streams other than the success stream to stderr (this is basically what native shells / utilities are forced to do: anything that's not data must go to stderr) - throwing everything into stdout was never a good idea.

It is another unfortunate example of PowerShell treating the outside world as an afterthought - which was less of a problem on Windows, but Unix users will expect conformant behavior.

It would be a breaking change, Steve Lee (@SteveL-MSFT), but perhaps it's not too late for Core?

+1 for the option to capture streams individually.

P.S.:

While everything currently goes to stdout in the absence of redirections or in the presence of only stdout redirection (>, 1>), at least with a 2> (stderr) redirection it is currently possible to capture the error stream separately (e.g., pwsh ... 2>errs.txt) - but only the error stream: streams 3 - 6 invariably go to stdout.

Here are variations of a simple test command that exercises all streams:

# From Bash:
$ pwsh -nop -command "write-output 1; write-error 2; write-warning 3; write-verbose -vb 4; \$DebugPreference = 'Continue'; Write-Debug 5; Write-Information -InformationAction continue 6; Write-Host host"


# From cmd.exe
C:> pwsh -nop -command "write-output 1; write-error 2; write-warning 3; write-verbose -vb 4; $DebugPreference = 'Continue'; Write-Debug 5; Write-Information -InformationAction continue 6; Write-Host host"

# From cmd.exe, Windows PowerShell
C:> powershell -nop -command "write-output 1; write-error 2; write-warning 3; write-verbose -vb 4; $DebugPreference = 'Continue'; Write-Debug 5; Write-Information -InformationAction continue 6; Write-Host host"

Michael Klement (@mklement0)

Michael Klement (@mklement0) changing from STDOUT to STDERR is probably too big a breaking change to take at this point, adding a switch to pwsh to enable this is doable if people will actually use it.

Steve Lee (@SteveL-MSFT)

Steve Lee (@SteveL-MSFT), I just remembered the discussion about providing a separate pwsh-np binary for starting PowerShell without loading $PROFILE, for use in shebang lines (see #992).

Perhaps we should consider fixing all the broken things there, to provide a CLI that fits better into the Unix world and has saner defaults overall (perhaps with a different name than pwsh-np).

Aside from not loading $PROFILE, this could include:

Sending PS streams 2 - 6 to stderr by default, as discussed here.
Fixing the CLI issues listed in Align PowerShell's CLI / startup behavior with POSIX-like shells such as Bash - command-line arguments #3743

Michael Klement (@mklement0)

Michael Klement (@mklement0) that may be a good solution as it wouldn't be a breaking change, but then pwsh-np may not be a good name to capture the differences to pwsh

I just came across this thread as I wanted to capture warning/debug/verbose streams separately too. Turns out you can run pwsh with `pwsh -o XML -c "Write-Verbose 123" which pipes XML based messages to stdout and stderr with all of the information (and more) that you can use to capture all output. I have almost managed to make it work properly, my current problem is incorrect order of incoming data e.g. from below script:

$ErrorActionPreference = "Continue"
$DebugPreference = "Continue"
$VerbosePreference = "Continue"

$PSVersionTable | Out-Host

Write-Debug "Hello DEBUG"
Write-Verbose "Hello VERBOSE"
Write-Output "Hello OUTPUT"
Write-Host "Hello HOST"
Write-Information "Hello INFORMATION"
Write-Warning "Hello WARNING"
Write-Error "Hello ERROR"
throw "Hello THROW"

I see Hello DEBUG and Hello VERBOSE before $PSVersionTable output.

I am using node.js to process the output:

const child_process = require('child_process');
const fs = require('fs');
const path = require('path');
const util = require('util');
const Saxophone = require('saxophone');

const cp = child_process.spawn(
  'pwsh',
  ['-NoProfile', '-NoLogo', '-NonInteractive', '-OutputFormat', 'XML', '-File', path.join(__dirname, 'script.ps1')],
  );

new CliXMLParser(cp.stdout, 'stdout');
new CliXMLParser(cp.stderr, 'stderr');
  
function CliXMLParser(stream, name) {
  this.stream = stream;
  this.name = name;
  this.currentTag = null;
  this.debugOut = fs.createWriteStream(util.format('%s.xml', name), 'utf-8')
  const self = this;

  this.stream.on('data', (chunk) => {
    const text = chunk.toString('utf-8');
    console.log('chunk received', text.slice(0, 100));
    if (text.indexOf('#< CLIXML') === 0) {
  
      self.parser = new Saxophone();
      
      self.parser.on('tagopen', tag => {
        self.currentTag = tag;
      })
      
      self.parser.on('tagclosed', () => {
        self.currentTag = null;
      })
      
      self.parser.on('text', (text) => {
        if (!self.currentTag) {
          return;
        }
  
        const attributes = Saxophone.parseAttrs(self.currentTag.attrs);
        
        if (text.contents === '_x000A_') {
          return; // process.stdout.write("\n");
        }

        if(self.currentTag.name === 'ToString') {
          console.log(text.contents)
        }

        if(attributes.S && attributes.S !== "Output") {
          process.stdout.write(util.format('##%s ', attributes.S.toLowerCase()))
        }
        
        if(self.currentTag.name === 'S') {
          console.log(text.contents)
        }
      })
    
      self.parser.on('finish', () => {
        console.log('Parsing finished.');
      });
      
      self.parser.on('error', (err) => {
        console.error(err);
      });
    } else {
      self.debugOut.write(chunk);

      if (self.parser){
        self.parser.write(chunk);
      }
    }
  })

  this.stream.on('error', err => {
    console.error(err);
  })

  this.stream.on('end', () => {
    console.log("Closing debug out")
    self.debugOut.close();
  })
}



cp.on('exit', (code) => {
  console.log("Pwsh exited with code %d", code);
})


console.log("Started pwsh process: %s", cp.pid);

You may have hit a bug, which I've just reported in #12489.

Also, it seems that host output is captured as success-stream output (S="Output") rather than information-stream output (S="information") - but that may
be by (historical) design and is probably retained for backward compatibility (the information stream, to which Write-Host (but not Out-Host) now writes, wasn't introduced until v5).

Navigation Menu

Repository navigation

Mapping file descriptors of pwsh process to Powershell streams #7989

Summary

Steps to reproduce

Expected behavior

Actual behavior

Environment data

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Mapping file descriptors of pwsh process to Powershell streams #7989

Description

Summary

Steps to reproduce

Expected behavior

Actual behavior

Environment data

Activity

masaeedu commented on Oct 10, 2018

iSazonov commented on Oct 11, 2018

vexx32 commented on Oct 11, 2018

masaeedu commented on Oct 11, 2018

SteveL-MSFT commented on Oct 11, 2018

masaeedu commented on Oct 11, 2018

masaeedu commented on Oct 11, 2018

mklement0 commented on Oct 18, 2018

mklement0 commented on Oct 18, 2018

SteveL-MSFT commented on Oct 18, 2018

mklement0 commented on Oct 18, 2018

SteveL-MSFT commented on Oct 18, 2018

sponte commented on Apr 25, 2020

mklement0 commented on Apr 25, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions