jruby jar-dependencies cannot be updated out-of-band from jruby

I want to make feature improvements to jar-dependencies, but when I install the updated gem into my system, bundler will refuse to load it because the default gem has already been installed. I've determined that the problem is that the rubygems/defaults/jruby.rb requires a file from jar-dependencies extremely early in the boot process of any jruby process in order to hook into the rubygems install process. I plan on resolving this and sending PRs for both jruby and jar-dependencies, but want a tracking ticket first since it spans both projects.

Environment Information

jruby 9.3.1.0 (2.6.8) 2021-10-13 2e01e7199d OpenJDK 64-Bit Server VM 11.0.15+10-Ubuntu-0ubuntu0.22.04.1 on 11.0.15+10-Ubuntu-0ubuntu0.22.04.1 [linux-x86_64]
Linux opennuc 5.15.0-37-generic #39-Ubuntu SMP Wed Jun 1 19:16:45 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Expected Behavior

Gemfile:

gem "jar-dependencies", "~> 1.0" # installed manually locally; no actual code changes required other than the change in version number

Running bundle console should boot without error.

Actual Behavior

cody@opennuc:~/test$ bundle console
[DEPRECATED] bundle console will be replaced by `bin/console` generated by `bundle gem <name>`
[DEPRECATED] This Gemfile does not include an explicit global source. Not using an explicit global source may result in a different lockfile being generated depending on the gems you have installed locally before bundler is run. Instead, define a global source in your Gemfile like this: source "https://rubygems.org".
Resolving dependencies...
Gem::LoadError: You have already activated jar-dependencies 0.4.1, but your Gemfile requires jar-dependencies 1.0.0. Since jar-dependencies is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports jar-dependencies as a default gem.
  check_for_activated_spec! at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/runtime.rb:309
                      setup at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/runtime.rb:25
                       each at org/jruby/RubyArray.java:1865
                       each at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/spec_set.rb:140
                        map at org/jruby/RubyEnumerable.java:799
                      setup at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/runtime.rb:24
                      setup at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler.rb:162
                    require at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler.rb:187
                        run at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/cli/console.rb:15
                    console at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/cli.rb:513
                        run at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/vendor/thor/lib/thor/command.rb:27
             invoke_command at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/vendor/thor/lib/thor/invocation.rb:127
                   dispatch at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/vendor/thor/lib/thor.rb:392
                   dispatch at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/cli.rb:31
                      start at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/vendor/thor/lib/thor/base.rb:485
                      start at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/cli.rb:25
                     <main> at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/exe/bundle:48
       with_friendly_errors at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/lib/bundler/friendly_errors.rb:120
                     <main> at /home/cody/.gem/jruby/2.6.8/gems/bundler-2.3.17/exe/bundle:36
                       load at org/jruby/RubyKernel.java:1052
                     <main> at /home/cody/.gem/jruby/2.6.8/bin/bundle:25

Jun 30 '22 15:06 ccutrer

Good find! Yes, we should defer loading jruby-jars until RubyGems/Bundler can be activated.

Jul 02 '22 08:07 headius

I merged #7264 but it seems to have broken some things. We'll need to revert it or fix those issues before we can release 9.3.7.0.

Aug 04 '22 20:08 headius

I don't know if this is the right place to report this, but I received an issue report yesterday in the glimmer-dsl-swt project (JRuby Desktop Development GUI Framework), which depends on jar-dependencies, that it suddenly started displaying this error upon scaffolding brand new projects (which I confirmed in my system):

Gem::LoadError: You have already activated jar-dependencies 0.4.1, but your Gemfile requires jar-dependencies 0.4.2. Since jar-dependencies is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports jar-dependencies as a default gem.
  check_for_activated_spec! at /Users/jordancech/.rvm/gems/jruby-9.3.6.0@dads_converter_ruby/gems/bundler-2.3.19/lib/bundler/runtime.rb:308
                      setup at /Users/jordancech/.rvm/gems/jruby-9.3.6.0@dads_converter_ruby/gems/bundler-2.3.19/lib/bundler/runtime.rb:25
                       each at org/jruby/RubyArray.java:1865
                       each at /Users/jordancech/.rvm/gems/jruby-9.3.6.0@dads_converter_ruby/gems/bundler-2.3.19/lib/bundler/spec_set.rb:148
                        map at org/jruby/RubyEnumerable.java:799
                      setup at /Users/jordancech/.rvm/gems/jruby-9.3.6.0@dads_converter_ruby/gems/bundler-2.3.19/lib/bundler/runtime.rb:24
                      setup at /Users/jordancech/.rvm/gems/jruby-9.3.6.0@dads_converter_ruby/gems/bundler-2.3.19/lib/bundler.rb:163
                     <main> at /Users/jordancech/.rvm/gems/jruby-9.3.6.0@dads_converter_ruby/gems/bundler-2.3.19/lib/bundler/setup.rb:10
                 with_level at /Users/jordancech/.rvm/gems/jruby-9.3.6.0@dads_converter_ruby/gems/bundler-2.3.19/lib/bundler/ui/shell.rb:136
                    silence at /Users/jordancech/.rvm/gems/jruby-9.3.6.0@dads_converter_ruby/gems/bundler-2.3.19/lib/bundler/ui/shell.rb:88
                     <main> at /Users/jordancech/.rvm/gems/jruby-9.3.6.0@dads_converter_ruby/gems/bundler-2.3.19/lib/bundler/setup.rb:10
                    require at org/jruby/RubyKernel.java:1017
                    require at /Users/jordancech/.rvm/rubies/jruby-9.3.6.0/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:85

The scaffolded app Gemfile looked like this:

# frozen_string_literal: true

source 'https://rubygems.org'

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

gem 'glimmer-dsl-swt', '~> 4.24.1.0'

group :development do
  gem 'rspec', '~> 3.5.0'
  gem 'juwelier', '2.4.9'
  gem 'warbler', '2.0.5'
  gem 'simplecov', '>= 0'
end

In Gemfile.lock, psych 4.0.4 is the dependency that requires jar-dependencies (it is open about accepting the greatest version available of jar-dependencies unless psych is updated to avoid the open version range):

    psych (4.0.4-java)
      jar-dependencies (>= 0.1.7)

The latest psych (>= 4.0.0) is required by the latest rdoc 6.4.0:

    rdoc (6.4.0)
      psych (>= 4.0.0)

The latest rdoc (>= 0) is required by juwelier 2.4.9.

When I tried to enter the scaffolded project and run bundle update jar-dependencies, I got this error towards the end:

Bundler attempted to update jar-dependencies but its version stayed the same

There are other weird issues that were encountered, but they all were caused by the jar-dependencies gem update.

I was able to resolve all these issues by reverting/freezing the jar-dependencies gem version dependency in the new glimmer-dsl-swt v4.24.1.1 patch release as 0.4.1

In any case, I can help out by testing a future fix when you make a new release (e.g. jruby 9.3.7.0).

Aug 05 '22 17:08 AndyObtiva

Ugh, you didn't have jar-dependencies in the Gemfile at all, so yanking 0.4.2 and publishing as 1.0.0 wouldn't even help :(. That means for this transition it's going to be required that users either pin to 0.4.1, or update jruby :(.

Aug 05 '22 17:08 ccutrer

Ugh, you didn't have jar-dependencies in the Gemfile at all, so yanking 0.4.2 and publishing as 1.0.0 wouldn't even help :(. That means for this transition it's going to be required that users either pin to 0.4.1, or update jruby :(.

Actually, I just updated my previous comment to include details about the Gemfile and Gemfile.lock of the scaffolded application.

Apparently, what brings in jar-dependencies is psych 4.0.4 because it has a dependency on jar-dependencies (>= 0.1.7. If psych was updated to a newer version that has a cap on the max version of jar-dependencies (e.g. '>= 0.1.7', '<= 0.4.1'), then that might perhaps resolve the issue for users without forcing them to pin jar-dependencies manually to 0.4.1 or update jruby.

Aug 05 '22 18:08 AndyObtiva

We've been forced to revert this effort and take another look.

The problems were manifold:

Existing JRuby installations cannot load the new jar-dependencies exactly because of this bug; 0.4.1 gets activated too early and errors result.
At least one library, Psych, has jar-dependencies in its deps (to retrieve and load SnakeYAML) but its only version specified is > 0.1.7 which means all new versions are eligible and anyone updating Psych will run into the version mismatch above.
At the same time, portions of JRuby's own build are run using older versions of JRuby, which of course have this same bug and are incompatible with the 0.4.2 that gets installed during the build.

There's a path forward but we don't know all the twists yet.

It seems to me the primary issue is that 0.4.2 is not compatible with (and breaks) JRuby versions 9.3.6 and earlier, and 0.4.1 is not compatible with (and breaks) any JRuby version that depends on the 0.4.2 gem hooks. So we end up needing a hard break in compatibility that causes problems across the board.

Aug 09 '22 15:08 headius