Skip to content

Only remove connection for an existing pool if the config is different - #45450

Merged
eileencodes merged 1 commit into
rails:mainfrom
eileencodes:only-remove-connection-if-config-is-different
Jun 29, 2022
Merged

Only remove connection for an existing pool if the config is different#45450
eileencodes merged 1 commit into
rails:mainfrom
eileencodes:only-remove-connection-if-config-is-different

Conversation

@eileencodes

@eileencodes eileencodes commented Jun 23, 2022

Copy link
Copy Markdown
Member

Previously Rails would always remove the connection if it found a
matching class in the pool manager. Therefore if
ActiveRecord::Base.establish_connection was called with the same
config, each time it was called it would be clobbered, even though the
config hasn't changed and the existing connection is prefectly fine. As
far as I can tell from conversations and reading the history this
functionality was added for ActiveRecord tests to be able to clobber the
connection and use a new config, then re-establish the old connection.
Essentially outside Rake tasks and AR tests, this functionality doesn't
have a ton of value.

On top of not adding a ton of value, this has resulted in a few bugs. In
Rails 6.0 I made it so that if you established a connection on
ApplicationRecord Rails would treat that connection the same as
ActiveRecord::Base. The reason for this is that the Railtie
establishes a connection on boot to the first database, but then if
you're using multiple databases you're calling connects_to in your
ApplicationRecord or primary abstract class which essentially doubles
your connections to the same database. To avoid opening 2 connections to
the same database, Rails treats them the same.

However, because we have this code that removes existing connections,
when an application boots, ApplicationRecord will clobber the
connection that the Railtie established even though the connection
configs are the same.

This removal of the connection caused bugs in migrations that load up a
model connected to ApplicationRecord (ex Post.first) and then calls
execute("SELECT 1") (obviously a simplified example). When execute
runs the connection is different from the one opened to run the
migration and essentially it is lost when the remove_connection code
is called.

To fix this I've updated the code to only remove the connection if the
database config is different. Ultimately I'd like to remove this code
altogether but to do that we first need to stop using
Base.establish_connection in the rake tasks and tests. This will fix
the major bugs until I come up with a solution for the areas that
currently need to call establish_connection on Base.

The added benefit of this change is that if your app is calling
establish_connection multiple times with the same config, it is now
3x faster than the previous implementation because we can return the
found pool instead of setting it up again. To benchmark this I
duplicated the establish_connection method to use the new behavior
with a new name.

Benchmark script:

require "active_record"
require "logger"
require "benchmark/ips"

config_hash = { "development" => { "primary" => { "adapter" => "mysql2", "username" => "rails", "database" => "activerecord_unittest"}}}
ActiveRecord::Base.configurations = config_hash

db_config = ActiveRecord::Base.configurations.configs_for(env_name: "development", name: "primary")

ActiveRecord::Base.connected_to(role: :writing, prevent_writes: true) do
  Benchmark.ips do |x|
    x.report "establish_connection with remove" do
      ActiveRecord::Base.establish_connection(db_config)
    end

    x.report "establish_connection without remove" do
      ActiveRecord::Base.establish_connection_no_remove(db_config)
    end

    x.compare!
  end
end

Benchmark results:

Warming up --------------------------------------
establish_connection with remove
                            4.677k i/100ms
establish_connection without remove
                        19.501k i/100ms
Calculating -------------------------------------
establish_connection with remove
                            41.252k (±11.3%) i/s -    205.788k in   5.075525s
establish_connection without remove
                        179.205k (± 6.9%) i/s -    897.046k in   5.029742s

Comparison:
establish_connection without remove:   179205.1 i/s
establish_connection with remove:    41252.3 i/s - 4.34x  (± 0.00) slower

Other changes:

  1. sqlite3 now disconnects and reconnects the connection when purge is
    called. This is necessary now that a new connection isn't created
    everyt time establish_connection is called. Without this change to
    purge the new database is left in an inaccessible state causing a
    readonly error from the sqlite3 client. This wasn't happening in mysql
    or postgres because they were already reconnecting the db connection.
  2. I added remove_connection to tests that use ApplicationRecord.
    This is required because ApplicationRecord or any class that is a
    primary_abstract_class will be treated the same as
    ActiveRecord::Base. This is fine in applications because they are
    shared connections, but in the AR test environment, we don't want those
    connnections to stick around (we want AR::Base back).
  3. In the async tests I removed 2 calls to establish_connection. These
    were causing sqlite3 tests to leak the state of async_executor because
    it's stored on the connection. I'm not sure why these were calling
    establish_connection but it's not necessary and was leaking state when
    now that we are no longer removing the connection.

Fixes: #41855
Fixes: #41876
Fixes: #42873
Fixes: #43004

cc/ @matthewd as we talked about this today.

@eileencodes
eileencodes force-pushed the only-remove-connection-if-config-is-different branch 3 times, most recently from b012ee0 to 2bec974 Compare June 23, 2022 20:16
@eileencodes

Copy link
Copy Markdown
Member Author

This has a change of behavior in that if you used to depend on establish_connection to remove an existing connection that's identical to the one you're establishing, it no longer is removed. This would avoid a database closed error but I can't imagine why you'd want this functionality. If applications really want to remove connections before establishing, then I think they should do that explicitly.

@matthewd matthewd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to write out the words: we could deprecate this behaviour... but I don't feel like that's necessary.

In general I don't see this as an API-observable behaviour change: whether a subsequently-checked-out connection comes from a fresh pool or one that already existed is immaterial to the caller. I think you'd need to be doing something notably odd to detect the difference; and correspondingly there isn't a good way for a caller to respond to a deprecation... adding their own if to avoid calling establish_connection if the new and existing configs match would be ugly, ultimately pointless, and force them to interact with much lower-level APIs than they were before.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this retain the ActiveRecord::Base / :primary protection mentioned in the above-removed comment? (Assuming we need to)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was for when we were deprecating primary as a name for ActiveRecord::Base and since that's been deprecated and removed I think we no longer need this. I wrote the original comment and went back to the original PR to verify that.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unlikely to matter either way in practice (mostly because the whole point of being okay with changing this is that we don't think real people are doing widespread same-config re-establishment in the first place), but I feel like we should only do this if the call had an effect.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea I was on the fence about this. My first implementation didn't instrument but then I wasn't sure if someone still wanted to know that establish_connection was called. Removing it makes an existing pool 4x faster than the prior behavior.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm leaving this in place for now. I made a refactor that skipped and included other changes when we have a pool, but that refactor resulted in a mysql2 test run with a bunch of flakes I couldn't reproduce locally. I'm going to do that refactor in a followup PR so I can more easily isolate what's going on there from the other fixes here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our own code inside fixtures consume this instrumentation, but this instrumentation is private (that is why it starts with !) so people should not be using.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh good to know, I want to change the keys in a followup PR anyway so I'll hopefully get rid of it then.

@eileencodes
eileencodes force-pushed the only-remove-connection-if-config-is-different branch 4 times, most recently from b65acc2 to 78fbb48 Compare June 29, 2022 14:06

@eileencodes eileencodes left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've solved all the failures in the sqlite3 tests. I've left comments on all the changes and why I made the changes I did.

CC'ing a few before merge - let me know if you think this change is problematic. The TL;DR is that if we have an existing pool with the same config, role and shard, AR no longer removes the connection and creates a new one, it simply returns the existing connection. If apps want to clobber an existing connection with a new config that still works the same as before. I've only changed behavior for calling establish_connection when the pool already exists. This change fixes some bugs in migrations when the connection goes away (because connects_to was called) and the migration expects it to be there (via execute).

cc/ @byroot @matthewd @jhawthorn @tenderlove

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm leaving this in place for now. I made a refactor that skipped and included other changes when we have a pool, but that refactor resulted in a mysql2 test run with a bunch of flakes I couldn't reproduce locally. I'm going to do that refactor in a followup PR so I can more easily isolate what's going on there from the other fixes here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action Mailbox and other related libraries are relying on loading the schema before test runs via db:test:prepare which calls db:schema:load which calls db:purge. With the changes in connection management we no longer are re-establishing the connection so anything that called purge would fail with a sqlite3 error. I noticed that this behavior was not reproducible in mysql2 and found that the purge method there reconnects to the new database. I implemented that same behavior in purge here. Previously this was working "fine" because we were always removing the connection even if it was the same. Now that we're not removing the connection we want to reset it so we don't have a bad connection.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rails apps treat ApplicationRecord OR the class named primary_abstract_class (example below) the same as ActiveRecord::Base in order to avoid opening duplicate connections to the same database. In a real app we want these connections to be shared and if ApplicationRecord.establish_connection is called it shouldn't replace or duplicate ActiveRecord::Base.

However, in the AR tests we don't actually want ApplicationRecord shared with ActiveRecord::Base beyond these tests so we need to remove the connection when the test is complete. Previously establish_connection was handling this for us, but now we need to do it explicitly. In an app these would actually be shared, but in our tests we don't want to share this global state.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed these because I found that the sqlite3 async tests were failing with some seeds and my changes. After debugging I found that the connection that was opened had asycn_query_executor set to nil rather than the global executor default. I don't know why these tests were re-establishing a connection, there would be one in the pool to pick up. Removing this fixed the tests that incorrectly had the wrong query executor. I believe this works because it will open a new connection for both tests (there wont be one in the pool). Previously calling establish_connection was ensuring there was an existing pool (with async executor set to nil) but we don't want to use that one so it's better not to establish a connection at all.

@eileencodes
eileencodes force-pushed the only-remove-connection-if-config-is-different branch from 78fbb48 to 5e8a8cf Compare June 29, 2022 14:35
@rails-bot rails-bot Bot added the railties label Jun 29, 2022
Previously Rails would always remove the connection if it found a
matching class in the pool manager. Therefore if
`ActiveRecord::Base.establish_connection` was called with the same
config, each time it was called it would be clobbered, even though the
config hasn't changed and the existing connection is prefectly fine. As
far as I can tell from conversations and reading the history this
functionality was added for ActiveRecord tests to be able to clobber the
connection and use a new config, then re-establish the old connection.
Essentially outside Rake tasks and AR tests, this functionality doesn't
have a ton of value.

On top of not adding a ton of value, this has resulted in a few bugs. In
Rails 6.0 I made it so that if you established a connection on
`ApplicationRecord` Rails would treat that connection the same as
`ActiveRecord::Base.` The reason for this is that the Railtie
establishes a connection on boot to the first database, but then if
you're using multiple databases you're calling `connects_to` in your
`ApplicationRecord` or primary abstract class which essentially doubles
your connections to the same database. To avoid opening 2 connections to
the same database, Rails treats them the same.

However, because we have this code that removes existing connections,
when an application boots, `ApplicationRecord` will clobber the
connection that the Railtie established even though the connection
configs are the same.

This removal of the connection caused bugs in migrations that load up a
model connected to `ApplicationRecord` (ex `Post.first`) and then calls
`execute("SELECT 1")` (obviously a simplified example). When `execute`
runs the connection is different from the one opened to run the
migration and essentially it is lost when the `remove_connection` code
is called.

To fix this I've updated the code to only remove the connection if the
database config is different. Ultimately I'd like to remove this code
altogether but to do that we first need to stop using
`Base.establish_connection` in the rake tasks and tests. This will fix
the major bugs until I come up with a solution for the areas that
currently need to call `establish_connection` on Base.

The added benefit of this change is that if your app is calling
`establish_connection` multiple times with the same config, it is now
3x faster than the previous implementation because we can return the
found pool instead of setting it up again. To benchmark this I
duplicated the `establish_connection` method to use the new behavior
with a new name.

Benchmark script:

```ruby
require "active_record"
require "logger"
require "benchmark/ips"

config_hash = { "development" => { "primary" => { "adapter" => "mysql2", "username" => "rails", "database" => "activerecord_unittest"}}}
ActiveRecord::Base.configurations = config_hash

db_config = ActiveRecord::Base.configurations.configs_for(env_name: "development", name: "primary")

p "Same model same config"
ActiveRecord::Base.connected_to(role: :writing, prevent_writes: true) do
  Benchmark.ips do |x|
    x.report "establish_connection with remove" do
      ActiveRecord::Base.establish_connection(db_config)
    end

    x.report "establish_connection without remove" do
      ActiveRecord::Base.establish_connection_no_remove(db_config)
    end

    x.compare!
  end
end
```

Benchmark results:

```
Warming up --------------------------------------
establish_connection with remove
                         4.677k i/100ms
establish_connection without remove
                        19.501k i/100ms
Calculating -------------------------------------
establish_connection with remove
                         41.252k (±11.3%) i/s -    205.788k in   5.075525s
establish_connection without remove
                        179.205k (± 6.9%) i/s -    897.046k in   5.029742s

Comparison:
establish_connection without remove:   179205.1 i/s
establish_connection with remove:    41252.3 i/s - 4.34x  (± 0.00) slower
```

Other changes:

1) sqlite3 now disconnects and reconnects the connection when `purge` is
called. This is necessary now that a new connection isn't created
everyt time `establish_connection` is called. Without this change to
purge the new database is left in an inaccessible state causing a
readonly error from the sqlite3 client. This wasn't happening in mysql
or postgres because they were already reconnecting the db connection.
2) I added `remove_connection` to tests that use `ApplicationRecord`.
This is required because `ApplicationRecord` or any class that is a
`primary_abstract_class` will be treated the same as
`ActiveRecord::Base`. This is fine in applications because they are
shared connections, but in the AR test environment, we don't want those
connnections to stick around (we want AR::Base back).
3) In the async tests I removed 2 calls to `establish_connection`. These
were causing sqlite3 tests to leak the state of async_executor because
it's stored on the connection. I'm not sure why these were calling
`establish_connection` but it's not necessary and was leaking state when
now that we are no longer removing the connection.

Fixes: rails#41855
Fixes: rails#41876
Fixes: rails#42873
Fixes: rails#43004
@eileencodes
eileencodes force-pushed the only-remove-connection-if-config-is-different branch from 5e8a8cf to adb64db Compare June 29, 2022 15:25
@eileencodes

eileencodes commented Jun 29, 2022

Copy link
Copy Markdown
Member Author

I'm going to merge this mainly because I need to make more changes and don't want to deal with conflicts. If anyone has an objection please let me know and I will revert.

I won't be backporting this due to the behavior change. Calling a model and then execute is not that common in migrations. There is a valid workaround in that apps can call ActiveRecord::Base.connection.execute instead of execute directly.

@eileencodes
eileencodes merged commit 462545c into rails:main Jun 29, 2022
@eileencodes
eileencodes deleted the only-remove-connection-if-config-is-different branch June 29, 2022 17:39
eileencodes added a commit to eileencodes/rails that referenced this pull request Jun 30, 2022
In the prior code we were getting the pool manager a bunch of times -
once in the retrieve_connection_pool, once in remove_connection_pool,
and once in the else conditional after setting it.

This change ensures we're only getting the pool manager once. If there
is no pool manager for the given key then we create a new one in the new
`set_pool_manager` method and use that.

The refactor also has a change in that we no longer instrument if a new
connection was not established. This instrumentation is private though
(denoted by the !) so it's safe to make this change.

Followup to rails#45450
eileencodes added a commit to eileencodes/rails that referenced this pull request Jul 19, 2022
In rails#45450 we stopped removing connections when establish_connection was
called and an identical connection existed in the pool. Unfortunately
this broke granular connection swapping for the primary_abstract_class
because the class stored in the pool was incorrect.

What was happening was connection for the writing role were getting
stored with the class ActiveRecord::Base and connections for the reading
role were getting ApplicationRecord (or whatever the app set as their
`primary_abstract_class`. If we made sure that the reading connections
also got ActiveRecord::Base that would break granular swapping for both
writing and reading because Active Record doesn't know whether you
wanted global swapping or granular swapping for prevent writes. This bug
only manifests on prevent writes because it's the only value we actually
lookup on the connection pool (because at the point we need it we don't
have access to self).

To fix this I've decided to update the class on the existing pool if it
doesn't match the owner_name passed. This is kind of gross I admit, but
it's safe because the way we find connections is on the
`connection_name`. The class is only stored so that the connection can
find it's `current_preventing_writes` value. This also required me to
move the ivar to an instance method for connection_class on the pool
because we don't want to cache the value and want to make sure we're
getting it from the pool config directly.
eileencodes added a commit to eileencodes/rails that referenced this pull request Oct 12, 2022
In rails#38235 I moved advisory locks to their own named connections, then
in rails#39758 the advisory lock was left on Base.connection but then moved
it it's own connection handler. I believe with rails#45450 that this change
was made obsolete and can be returned to the prior behavior without
having to open an additional connection. The tests added pass and I also
tested this in my local demo to ensure that this is working correctly.

When I originally changed the behavior here Matthew noted that this
could be surprising for some setups that expect only one connection for
a running migration. I thought there was an issue related to this but I
can't find it.
eileencodes added a commit to eileencodes/rails that referenced this pull request Jul 5, 2023
Fixes 2 bugs in parallel testing.

The first bug was related to changes made in rails#45450 which meant that we were
no longer replacing the connection in parallel testing because the
config object is equal (we simply merge a new db name but the object id
of the config stays the same). This bug only manifested in mysql and
sqlite3 interestingly. It would fail on the internal metadata tables
because they were missing in the schema version check.

To fix this I introduced a `clobber: true` kwarg onto the connection
handler that allows us to bypass the functionality that won't make a new
connection if the config is the same. This is an easy way to fall back
to the old behavior from before this change. I only added `clobber`
to the `reconstruct_from_schema` call because we need to actually
replace the connection for these. It's not safe to add everywhere since
we don't always want to replace the connection.

After implementing this fix I was still seeing failures in the mysql
demo app I made due to the fact that `purge` was not re-establishing the
connection to a config that had a database defined. Neither sqlite3 or
postgresql were missing this.

I added a test for mysql2 so we don't have regressions in the future. I
think this was missed because sqlite3 only demonstrates the bug if it
was never successful on that worker and postgresql was fine.

Fixes rails#48547
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

3 participants