Skip to content

Comments

src: avoid deferred gc/cleanup for Buffer.from#38337

Closed
jasnell wants to merge 2 commits intonodejs:masterfrom
jasnell:avoid-deferred-cleanup-for-buffer-fro
Closed

src: avoid deferred gc/cleanup for Buffer.from#38337
jasnell wants to merge 2 commits intonodejs:masterfrom
jasnell:avoid-deferred-cleanup-for-buffer-fro

Conversation

@jasnell
Copy link
Member

@jasnell jasnell commented Apr 21, 2021

/cc @addaleax @Daninet

Previously, the code path would allocated a tracked ArrayBuffer that defers cleanup and deallocation of the underlying data with a native SetImmediate. Avoid the unnecessary deferral by just allocating a BackingStore directly and writing into it.

This should be considered an alternative to #38336

Fixes: #38300
Refs: #38336

@jasnell
src: avoid deferred gc/cleanup for Buffer.from
6805449 
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: nodejs#38300
Refs: nodejs#38336
@nodejs-github-bot nodejs-github-bot added buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run. labels Apr 21, 2021
@nodejs-github-bot

This comment has been minimized.

Sign in to view
@jasnell jasnell mentioned this pull request Apr 21, 2021
Closed
addaleax
addaleax approved these changes Apr 22, 2021
View reviewed changes
@jasnell @addaleax
[Squash] nit
f805a1e 
Co-authored-by: Anna Henningsen <[email protected]>
@jasnell jasnell mentioned this pull request Apr 22, 2021
Closed
XadillaX
XadillaX approved these changes Apr 22, 2021
View reviewed changes
Copy link
Contributor

@XadillaX XadillaX left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After testing:

const str = 'x'.repeat(10000);

console.log('start', process.memoryUsage());

console.time('1');
  for(let i = 0; i < 1e6; i++) {
    Buffer.from(str); // memory leak happens here
    if (i % 1e5 === 0) {
      // await wait(100);
      gc();
      console.log('step', i, 'rss', process.memoryUsage().rss);
    }
  }
console.timeEnd('1');

  gc();
  console.log('end', process.memoryUsage());

The new PR #38337 runs about 3.9s in my computer. And 39f42d1a2d (before using BackingStore) runs about 3.7s. I think this performance loss is acceptable.

mcollina
mcollina approved these changes Apr 22, 2021
View reviewed changes
Copy link
Member

@mcollina mcollina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@nodejs-github-bot

This comment has been minimized.

Sign in to view
@nodejs-github-bot

This comment has been minimized.

Sign in to view
@jasnell jasnell added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Apr 22, 2021
@nodejs-github-bot

This comment has been minimized.

Sign in to view
@nodejs-github-bot
Copy link
Collaborator

nodejs-github-bot commented Apr 23, 2021

CI: https://ci.nodejs.org/job/node-test-pull-request/37635/

@jasnell
Copy link
Member Author

jasnell commented Apr 23, 2021

@nodejs/build ... is there a known issue with the node-test-commit-arm-fanned cluster? It is failing consistently on multiple PRs

@richardlau
Copy link
Member

richardlau commented Apr 23, 2021

@nodejs/build ... is there a known issue with the node-test-commit-arm-fanned cluster? It is failing consistently on multiple PRs

I cleaned up leftover node process on three of the pi 2's this morning. AFAIK things should be okay now.

@nodejs-github-bot
Copy link
Collaborator

nodejs-github-bot commented Apr 23, 2021

CI: https://ci.nodejs.org/job/node-test-pull-request/37643/

@jasnell
Copy link
Member Author

jasnell commented Apr 23, 2021

@richardlau ... no joy it seems. See https://ci.nodejs.org/job/node-test-binary-arm-12+/10459/RUN_SUBSET=1,label=pi2-docker/console for #38331, for instance.

targos pushed a commit that referenced this pull request Apr 24, 2021
@jasnell @targos
src: avoid deferred gc/cleanup for Buffer.from
720fdf2 
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: #38300
Refs: #38336

PR-URL: #38337
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Khaidi Chu <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
@targos
Copy link
Member

targos commented Apr 24, 2021

Landed in 720fdf2

@targos targos closed this Apr 24, 2021
targos pushed a commit that referenced this pull request Apr 29, 2021
@jasnell @targos
src: avoid deferred gc/cleanup for Buffer.from
3649ec5 
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: #38300
Refs: #38336

PR-URL: #38337
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Khaidi Chu <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
@targos targos mentioned this pull request May 3, 2021
Merged
targos pushed a commit that referenced this pull request May 30, 2021
@jasnell @targos
src: avoid deferred gc/cleanup for Buffer.from
c0aa3f0 
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: #38300
Refs: #38336

PR-URL: #38337
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Khaidi Chu <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
targos pushed a commit that referenced this pull request Jun 5, 2021
@jasnell @targos
src: avoid deferred gc/cleanup for Buffer.from
ebf79c6 
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: #38300
Refs: #38336

PR-URL: #38337
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Khaidi Chu <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
@targos targos mentioned this pull request Jun 6, 2021
Merged
targos pushed a commit that referenced this pull request Jun 11, 2021
@jasnell @targos
src: avoid deferred gc/cleanup for Buffer.from
e279b02 
Previously, the code path would allocated a tracked ArrayBuffer
that defers cleanup and deallocation of the underlying data with
a SetImmediate. Avoid the unnecessary deferral by just allocating
a `BackingStore` directly and writing into it.

Fixes: #38300
Refs: #38336

PR-URL: #38337
Reviewed-By: Anna Henningsen <[email protected]>
Reviewed-By: Khaidi Chu <[email protected]>
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Joyee Cheung <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mcollina mcollina mcollina approved these changes

@addaleax addaleax addaleax approved these changes

@joyeecheung joyeecheung joyeecheung approved these changes

+1 more reviewer

@XadillaX XadillaX XadillaX approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

author ready PRs that have at least one approval, no pending requests for changes, and a CI started. buffer Issues and PRs related to the buffer subsystem. c++ Issues and PRs that require attention from people who are familiar with C++. needs-ci PRs that need a full CI run.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Buffer.from(str) memory leak

8 participants

@jasnell @nodejs-github-bot @richardlau @targos @mcollina @addaleax @XadillaX @joyeecheung