MMTk binding for Ruby

This repository hosts the binding code for MMTk Ruby.

In order for this binding to work, changes have been made to the Ruby core language to support generic third party heaps. Eventually, the aim is to upstream these changes. Until then, the modifications can be found under our fork here, on the branch third-party-heap.

This repository is based on previous work of Angus Atkinson, and the original repository can be found here, and the original Ruby fork can be found here.

Building/installation instructions

Checkout repositories

You need to clone both the Ruby fork and the MMTk Ruby binding. The location does not matter.

git clone https://github.com/mmtk/ruby.git
git clone https://github.com/mmtk/mmtk-ruby.git

Build the MMTk binding, first.

pushd mmtk-ruby/mmtk
cargo build --release
popd

This will give you a libmmtk_ruby.so in the target/release directory.

By default, mmtk-ruby uses the mmtk crate from the master branch of its official repository. If you want to hack the MMTk core itself, you can edit mmtk-ruby/mmtk/Cargo.toml to point to your local repository.

Then build our forked Ruby repository.

Configure.

cd ruby
./autogen.sh
./configure --with-mmtk-ruby=../mmtk-ruby --prefix=$PWD/build

The miniruby executable should be able to execute simple Ruby programs. You can try the following command:

# Run with vanilla Ruby GC
./miniruby -e 'puts "Hello world!"'

# Run with MMTk GC
./miniruby --mmtk --mmtk-plan=MarkSweep -e 'puts "Hello world!"'

You can continue to build the full Ruby and install it with

make install -j

Debug build

You can build with debug options enabled for easy debugging.

Remove the --release option to build mmtk-ruby for debug. Note that the Cargo build system is smart enough to let the result of debug build and release build to coexist in the target/debug and target/release directories.

pushd mmtk-ruby/mmtk
cargo build
popd

By default, ./configure searches for libmmtk_ruby.so in mmtk-ruby/mmtk/target/debug, and the linker will subsequently link miniruby and ruby to that .so. Add --with-mmtk-ruby-debug so it will search for libmmtk_ruby.so in mmtk-ruby/mmtk/target/debug, instead.

Set the compiler option -DRUBY_DEBUG=1 to enable most assertions in Ruby.

Add the -O0 optimization flag so the debug can see the values of most local variables. But if it is too slow, try -O1 instead.

The --disable-install-doc will disable the generation of documentations. It can make the build process much faster.

The following is an example of configuring for debugging.

./configure --with-mmtk-ruby=../mmtk-ruby --with-mmtk-ruby-debug cppflags='-DRUBY_DEBUG=1' optflags='-O0' --prefix=$PWD/build --disable-install-doc

Also note that you can use the release build of mmtk-ruby with a debug build of ruby (with optflags='-O0' but without --with-mmtk-ruby-debug) and vice versa (with --with-mmtk-ruby-debug and optflags='-O1' or higher).

Adjusting heap size

By default, MMTk dynamically adjust the heap size (i.e. when to trigger the next GC) between 1MiB and 80% of the physical memory, depending on the heap utility after the previous GC. It is convenient for production settings. However, when doing experiments, you may want to set the heap size to a fixed value so the GC behaviour becomes more deterministic. You can do it in two ways:

1. Using the --mmtk-max-heap command line option.
2. Using the THIRD_PARTY_HEAP_LIMIT environment variable.

Both of them accept IEC suffixes "KiB", "MiB", "GiB" and "TiB". Therefore, "16777216" and "16MiB" are equivalent.

Test

To test Ruby, it is recommended that you add the ADDITIONAL_EXCLUDES option to exclude tests which make assumptions based on Ruby's current GC implementation, or are extremely memory intensive.

make test-all ADDITIONAL_EXCLUDES="--excludes-dir=./test/excludes/_third_party_heap"

Current status

Known working:

• ./miniruby ./basictest/test.rb
• make test
• Regular Ruby programs (note: this hasn't been tested on any extensive real-world programs, only <100 line dummy programs)
• Basic Rails 5 app utilising a sqlite3 database:

  gem install rails -v 5.2.0
  rails new hello
  cd hello
  rails generate scaffold User name:string email:string
  rails db:migrate
  rails server
  
  # If you run into installation issues along the way, you may need to...
  gem install sqlite3
  gem install puma
  # ...and try again

Known issues:

• make test-all fails. Many tests are GC implementation-dependent, so exclusion files have been created to ignore most of these. There are still >50 errors, but not all have been triaged or fixed yet.
• make test-rubyspec is currently failing; need to find a way to exclude GC-specific specifications.
• GC implementation-specific modules (e.g. ObjectSpace, GC, WeakRef) and anything that relies on them (e.g. Coverage) are not supported. For now, there are no plans to implement these as many of the APIs are irrelevant (e.g. GC.stat); however some may be fixed in the future (e.g. ObjectSpace.each_object)
• MJIT is not supported.

TODO

• Add a runtime flag to enable MMTk/ruby, using environment variables. See #1
• Rebase my changes onto the variable-sized objects heap (currently being developed by Shopify here)
• Use separate mutators for every thread for cache locality benefits (and correctness in case the GVL is ever removed)
• Implement allocation fast paths
• Fix tests

Licensing

This work is dual-licensed under the MIT and Apache licenses, to be compatible with the MMTk-Core project. See the license notices in the root of this repository for further details.