Skip to content

Exec failing with internal error when piping large output #818

@ealmansi

Description

@ealmansi

Node version (or tell us if you're using electron or some other framework):

8

ShellJS version (the most recent version/Github branch you see the bug on):

0.8.0

Operating system:

Ubuntu 16.04.3

Description of the bug:

Piping medium to large amounts of output between two exec's causes the second one to fail with 'Error: exec: internal error'.

Example ShellJS command to reproduce the error:

var shell = require('shelljs')
shell.config.fatal = true
shell.exec('cat small.txt', { silent: true }).exec('wc -l', { silent: true }).to('few-lines.txt')
shell.exec('cat big.txt', { silent: true }).exec('wc -l', { silent: true }).to('many-lines.txt')

Here, small.txt is a small file (under 1kb), while big.txt is a 2MB file.
The first command executes perfectly well, while the second one fails with exec: internall error.
Without fatal = true, the second exec on the second command simply returns null and the command fails due to to not being a property of null. (Likely related: #724)

Note

I did some digging and it seems the problem is here:

https://github.com/shelljs/shelljs/blob/master/src/exec.js#L68

That call to execFileSync receives execArgs, which contains the serialized output from the first exec. So it's essentially calling a process with an argument which is over 2MB, which is breaking a limit on my OS. That call to execFileSync ends with an Error: spawnSync /usr/bin/node E2BIG being thrown by node.

Metadata

Metadata

Assignees

No one assigned

    Labels

    fixBug/defect, or a fix for such a problem

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions