Skip to content

Manually compact GC before fork #2093

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
schneems merged 1 commit into from
Dec 18, 2019
Merged

Manually compact GC before fork #2093

schneems merged 1 commit into from
Dec 18, 2019

Conversation

@schneems
Copy link
Contributor

@schneems schneems commented Dec 18, 2019 •
edited
Loading

Rub 2.7.0 introduced GC.compact which allows manual compaction of Ruby slots. A good time to do this is before forking so that memory fragmentation can be reduced.

One issue with memory fragmentation when forking is that while a page in memory might have only one free slot, as that slot is written to after fork, the entire page is written so the benefit of CoW optimizations are greatly reduced. Manually compacting GC reduces the number of pages with empty slots.

This PR manually compacts memory right before Puma forks and after other before_fork hooks are called.

Description

Please describe your pull request. Thank you for contributing! You're the best.

Your checklist for this pull request

  • I have reviewed the guidelines for contributing to this repository.
  • I have added an entry to History.md if this PR fixes a bug or adds a feature. If it doesn't need an entry to HISTORY.md, I have added [changelog skip] the pull request title.
  • I have added appropriate tests if this PR fixes a bug or adds a feature.
  • My pull request is 100 lines added/removed or less so that it can be easily reviewed.
  • If this PR doesn't need tests (docs change), I added [ci skip] to the title of the PR.
  • If this closes any issues, I have added "Closes #issue" to the PR description or my commit messages.
  • I have updated the documentation accordingly.
  • All new and existing tests passed, including Rubocop.

@schneems schneems force-pushed the schneems/compact-gc branch from a62f551 to 2b4eb74 Compare December 18, 2019 16:28
@MSP-Greg
Copy link
Member

MSP-Greg commented Dec 18, 2019

JFYI, using ruby master in actions:

https://github.com/ruby/psych/blob/master/.github/workflows/ubuntu-rvm.yml

@liaden
Copy link

liaden commented Dec 18, 2019 •
edited
Loading

Would it be worthwhile to add a GC.start to increase the resulting compactness? Edit: Nevermind, it already invokes GC https://github.com/ruby/ruby/blob/master/gc.c#L8438

At work, we often will use eye to monitor child processes for memory bloat and have eye kill them. It would be beneficial to have this only done on the first fork instead of every time we fork.

mauro-oto
mauro-oto reviewed Dec 18, 2019
View reviewed changes
@schneems schneems force-pushed the schneems/compact-gc branch 2 times, most recently from 9c46fea to 020e4d8 Compare December 18, 2019 17:23
@schneems
Copy link
Contributor Author

schneems commented Dec 18, 2019

I'm getting failures on GitHub actions for 2.7.0 and 2.3.x and 2.3.8 with just

Run actions/setup-ruby@v1
  with:
    ruby-version: 2.7.0-preview3
##[error]Version 2.7.0-preview3 not found
##[error]Node run failed with exit code 1

Any idea how to debug or what needs to be changed @MSP-Greg ?

@MSP-Greg
Copy link
Member

MSP-Greg commented Dec 18, 2019

@schneems

I'm getting failures on GitHub actions for 2.7.0 and 2.3.x and 2.3.8

At present, standard GitHub actions just has Ruby 2.4.x, 2.5.x, & 2.6.x installed. Anything else needs code using a container, installing and using rvm, or a special action step.

@dentarg & PR #2087 adds Ruby 2.3 in a container. The Ruby/Psych code I linked to adds ruby-head using rvm. I can add 2.3 & ruby-head to Windows, but that won't run this code as fork is not available...

@schneems
Manually compact GC before fork
abd3f35 
Rub 2.7.0 introduced `GC.compact` which allows manual compaction of Ruby slots. A good time to do this is before forking so that memory fragmentation can be reduced.

- https://www.ruby-lang.org/en/news/2019/12/17/ruby-2-7-0-rc1-released/
- https://www.youtube.com/watch?v=1F3gXYhQsAY

One issue with memory fragmentation when forking is that while a page in memory might have only one free slot, as that slot is written to after fork, the entire page is written so the benefit of CoW optimizations are greatly reduced. Manually compacting GC reduces the number of pages with empty slots.

This PR manually compacts memory right before Puma forks and after other `before_fork` hooks are called.
@schneems schneems force-pushed the schneems/compact-gc branch from 020e4d8 to abd3f35 Compare December 18, 2019 18:11
@MSP-Greg
Copy link
Member

MSP-Greg commented Dec 18, 2019

All the Travis 2.7.0 tests passed, and all the Actions tests that are not broken passed.

@schneems
Copy link
Contributor Author

schneems commented Dec 18, 2019

All the Travis 2.7.0 tests passed, and all the Actions tests that are not broken passed.

Seems good!

@schneems schneems merged commit b646fc5 into master Dec 18, 2019
@nateberkopec nateberkopec mentioned this pull request May 12, 2020
Closed
@dentarg dentarg deleted the schneems/compact-gc branch February 10, 2021 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@mauro-oto mauro-oto mauro-oto left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@schneems @MSP-Greg @liaden @mauro-oto