Skip to content

fix: propagate errors when using pipelines (#2560)#2624

Merged
ddelgrosso1 merged 5 commits intogoogleapis:mainfrom
wvanderdeijl:pipelines
Aug 11, 2025
Merged

fix: propagate errors when using pipelines (#2560)#2624
ddelgrosso1 merged 5 commits intogoogleapis:mainfrom
wvanderdeijl:pipelines

Conversation

@wvanderdeijl
Copy link
Contributor

@wvanderdeijl wvanderdeijl commented Aug 6, 2025

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

This Pull Request add two tests relating to issues #2367 and #2560.

The first one is about the application hanging when a pipeline is used that writes to a File stream and that pipeline errors. This was fixed in PR #2437 but without any unit tests. That change introduced a regression that is logged as #2560.
Since both issues are related I tried to add unit tests for both scenarios.

If you undo the changes form PR #2437 the "should close upstream when pipeline fails" test fails as we reintroduce the bug. But the "should error pipeline if source stream emits error before any data" test succeeds as this problem did not exist before v7.10.0.

If you run the tests with the latest version (so with #2437 changes in place) the "should close upstream when pipeline fails" test now succeeds, but the "should error pipeline if source stream emits error before any data" fails with an "Uncaught Error: Error before first chunk".

Here is what Gemini thinks of this:

The Cause of the "Uncaught Error"

The test fails with an "Uncaught Error" because an internal stream within your createWriteStream method emits an 'error' event that has no listener attached to it. In Node.js, an unhandled 'error' event on an EventEmitter (which streams are) will crash the process.

Here's a step-by-step breakdown of what's happening in your test:

  1. Your test creates a writable stream by calling file.createWriteStream().
  2. It then uses stream.pipeline() to pipe a source stream that is designed to fail immediately (before producing any data) into this writable stream.
  3. The pipeline utility correctly detects the error from the source stream and, as part of its cleanup, calls writable.destroy(error).
  4. This causes the writable stream to emit an 'error' event. Your test's pipeline callback correctly catches this, and the test should pass.
  5. Here's the problem: Inside createWriteStream, there's a listener on the writable stream's 'error' event: writeStream.once('error', e => { emitStream.destroy(e); });. This is intended to propagate the error to an internal emitStream.
  6. However, the error handling for emitStream is set up inside a writeStream.on('writing', ...) listener. This 'writing' event is only emitted when data starts to flow.
  7. In your test's scenario, the source stream fails before any data is sent. The 'writing' event never fires, so the error handling for emitStream (which would be the inner pipeline) is never attached.
  8. When emitStream.destroy(error) is called, emitStream emits an 'error' event. With no listeners attached, this becomes an uncaught exception, and Mocha reports it as a test failure.

The code changes were also suggested by Gemini.

Fixes #2560

@wvanderdeijl wvanderdeijl requested a review from a team August 6, 2025 10:46
@wvanderdeijl wvanderdeijl requested a review from a team as a code owner August 6, 2025 10:46
@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: storage Issues related to the googleapis/nodejs-storage API. labels Aug 6, 2025
@wvanderdeijl
Copy link
Contributor Author

I don't understand why the new test fails in CI for node 14. I only have a apple sillicon mac and nodejs 14 is not available for that architecture, so I can't test locally.

@ddelgrosso1
Copy link
Contributor

cc: @thiyaguk09

@ddelgrosso1 ddelgrosso1 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 11, 2025
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Aug 11, 2025
@ddelgrosso1 ddelgrosso1 merged commit a43b490 into googleapis:main Aug 11, 2025
14 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the googleapis/nodejs-storage API. size: m Pull request size is medium.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Uncatchable error when upload stream is used in failed pipeline.

4 participants