Only remove connection for an existing pool if the config is different - #45450
Conversation
b012ee0 to
2bec974
Compare
|
This has a change of behavior in that if you used to depend on |
matthewd
left a comment
There was a problem hiding this comment.
Just to write out the words: we could deprecate this behaviour... but I don't feel like that's necessary.
In general I don't see this as an API-observable behaviour change: whether a subsequently-checked-out connection comes from a fresh pool or one that already existed is immaterial to the caller. I think you'd need to be doing something notably odd to detect the difference; and correspondingly there isn't a good way for a caller to respond to a deprecation... adding their own if to avoid calling establish_connection if the new and existing configs match would be ugly, ultimately pointless, and force them to interact with much lower-level APIs than they were before.
There was a problem hiding this comment.
Does this retain the ActiveRecord::Base / :primary protection mentioned in the above-removed comment? (Assuming we need to)
There was a problem hiding this comment.
That was for when we were deprecating primary as a name for ActiveRecord::Base and since that's been deprecated and removed I think we no longer need this. I wrote the original comment and went back to the original PR to verify that.
There was a problem hiding this comment.
It's unlikely to matter either way in practice (mostly because the whole point of being okay with changing this is that we don't think real people are doing widespread same-config re-establishment in the first place), but I feel like we should only do this if the call had an effect.
There was a problem hiding this comment.
Yea I was on the fence about this. My first implementation didn't instrument but then I wasn't sure if someone still wanted to know that establish_connection was called. Removing it makes an existing pool 4x faster than the prior behavior.
There was a problem hiding this comment.
I'm leaving this in place for now. I made a refactor that skipped and included other changes when we have a pool, but that refactor resulted in a mysql2 test run with a bunch of flakes I couldn't reproduce locally. I'm going to do that refactor in a followup PR so I can more easily isolate what's going on there from the other fixes here.
There was a problem hiding this comment.
Our own code inside fixtures consume this instrumentation, but this instrumentation is private (that is why it starts with !) so people should not be using.
There was a problem hiding this comment.
Oh good to know, I want to change the keys in a followup PR anyway so I'll hopefully get rid of it then.
b65acc2 to
78fbb48
Compare
eileencodes
left a comment
There was a problem hiding this comment.
I've solved all the failures in the sqlite3 tests. I've left comments on all the changes and why I made the changes I did.
CC'ing a few before merge - let me know if you think this change is problematic. The TL;DR is that if we have an existing pool with the same config, role and shard, AR no longer removes the connection and creates a new one, it simply returns the existing connection. If apps want to clobber an existing connection with a new config that still works the same as before. I've only changed behavior for calling establish_connection when the pool already exists. This change fixes some bugs in migrations when the connection goes away (because connects_to was called) and the migration expects it to be there (via execute).
There was a problem hiding this comment.
I'm leaving this in place for now. I made a refactor that skipped and included other changes when we have a pool, but that refactor resulted in a mysql2 test run with a bunch of flakes I couldn't reproduce locally. I'm going to do that refactor in a followup PR so I can more easily isolate what's going on there from the other fixes here.
There was a problem hiding this comment.
Action Mailbox and other related libraries are relying on loading the schema before test runs via db:test:prepare which calls db:schema:load which calls db:purge. With the changes in connection management we no longer are re-establishing the connection so anything that called purge would fail with a sqlite3 error. I noticed that this behavior was not reproducible in mysql2 and found that the purge method there reconnects to the new database. I implemented that same behavior in purge here. Previously this was working "fine" because we were always removing the connection even if it was the same. Now that we're not removing the connection we want to reset it so we don't have a bad connection.
There was a problem hiding this comment.
Rails apps treat ApplicationRecord OR the class named primary_abstract_class (example below) the same as ActiveRecord::Base in order to avoid opening duplicate connections to the same database. In a real app we want these connections to be shared and if ApplicationRecord.establish_connection is called it shouldn't replace or duplicate ActiveRecord::Base.
However, in the AR tests we don't actually want ApplicationRecord shared with ActiveRecord::Base beyond these tests so we need to remove the connection when the test is complete. Previously establish_connection was handling this for us, but now we need to do it explicitly. In an app these would actually be shared, but in our tests we don't want to share this global state.
There was a problem hiding this comment.
I removed these because I found that the sqlite3 async tests were failing with some seeds and my changes. After debugging I found that the connection that was opened had asycn_query_executor set to nil rather than the global executor default. I don't know why these tests were re-establishing a connection, there would be one in the pool to pick up. Removing this fixed the tests that incorrectly had the wrong query executor. I believe this works because it will open a new connection for both tests (there wont be one in the pool). Previously calling establish_connection was ensuring there was an existing pool (with async executor set to nil) but we don't want to use that one so it's better not to establish a connection at all.
78fbb48 to
5e8a8cf
Compare
Previously Rails would always remove the connection if it found a
matching class in the pool manager. Therefore if
`ActiveRecord::Base.establish_connection` was called with the same
config, each time it was called it would be clobbered, even though the
config hasn't changed and the existing connection is prefectly fine. As
far as I can tell from conversations and reading the history this
functionality was added for ActiveRecord tests to be able to clobber the
connection and use a new config, then re-establish the old connection.
Essentially outside Rake tasks and AR tests, this functionality doesn't
have a ton of value.
On top of not adding a ton of value, this has resulted in a few bugs. In
Rails 6.0 I made it so that if you established a connection on
`ApplicationRecord` Rails would treat that connection the same as
`ActiveRecord::Base.` The reason for this is that the Railtie
establishes a connection on boot to the first database, but then if
you're using multiple databases you're calling `connects_to` in your
`ApplicationRecord` or primary abstract class which essentially doubles
your connections to the same database. To avoid opening 2 connections to
the same database, Rails treats them the same.
However, because we have this code that removes existing connections,
when an application boots, `ApplicationRecord` will clobber the
connection that the Railtie established even though the connection
configs are the same.
This removal of the connection caused bugs in migrations that load up a
model connected to `ApplicationRecord` (ex `Post.first`) and then calls
`execute("SELECT 1")` (obviously a simplified example). When `execute`
runs the connection is different from the one opened to run the
migration and essentially it is lost when the `remove_connection` code
is called.
To fix this I've updated the code to only remove the connection if the
database config is different. Ultimately I'd like to remove this code
altogether but to do that we first need to stop using
`Base.establish_connection` in the rake tasks and tests. This will fix
the major bugs until I come up with a solution for the areas that
currently need to call `establish_connection` on Base.
The added benefit of this change is that if your app is calling
`establish_connection` multiple times with the same config, it is now
3x faster than the previous implementation because we can return the
found pool instead of setting it up again. To benchmark this I
duplicated the `establish_connection` method to use the new behavior
with a new name.
Benchmark script:
```ruby
require "active_record"
require "logger"
require "benchmark/ips"
config_hash = { "development" => { "primary" => { "adapter" => "mysql2", "username" => "rails", "database" => "activerecord_unittest"}}}
ActiveRecord::Base.configurations = config_hash
db_config = ActiveRecord::Base.configurations.configs_for(env_name: "development", name: "primary")
p "Same model same config"
ActiveRecord::Base.connected_to(role: :writing, prevent_writes: true) do
Benchmark.ips do |x|
x.report "establish_connection with remove" do
ActiveRecord::Base.establish_connection(db_config)
end
x.report "establish_connection without remove" do
ActiveRecord::Base.establish_connection_no_remove(db_config)
end
x.compare!
end
end
```
Benchmark results:
```
Warming up --------------------------------------
establish_connection with remove
4.677k i/100ms
establish_connection without remove
19.501k i/100ms
Calculating -------------------------------------
establish_connection with remove
41.252k (±11.3%) i/s - 205.788k in 5.075525s
establish_connection without remove
179.205k (± 6.9%) i/s - 897.046k in 5.029742s
Comparison:
establish_connection without remove: 179205.1 i/s
establish_connection with remove: 41252.3 i/s - 4.34x (± 0.00) slower
```
Other changes:
1) sqlite3 now disconnects and reconnects the connection when `purge` is
called. This is necessary now that a new connection isn't created
everyt time `establish_connection` is called. Without this change to
purge the new database is left in an inaccessible state causing a
readonly error from the sqlite3 client. This wasn't happening in mysql
or postgres because they were already reconnecting the db connection.
2) I added `remove_connection` to tests that use `ApplicationRecord`.
This is required because `ApplicationRecord` or any class that is a
`primary_abstract_class` will be treated the same as
`ActiveRecord::Base`. This is fine in applications because they are
shared connections, but in the AR test environment, we don't want those
connnections to stick around (we want AR::Base back).
3) In the async tests I removed 2 calls to `establish_connection`. These
were causing sqlite3 tests to leak the state of async_executor because
it's stored on the connection. I'm not sure why these were calling
`establish_connection` but it's not necessary and was leaking state when
now that we are no longer removing the connection.
Fixes: rails#41855
Fixes: rails#41876
Fixes: rails#42873
Fixes: rails#43004
5e8a8cf to
adb64db
Compare
|
I'm going to merge this mainly because I need to make more changes and don't want to deal with conflicts. If anyone has an objection please let me know and I will revert. I won't be backporting this due to the behavior change. Calling a model and then execute is not that common in migrations. There is a valid workaround in that apps can call |
In the prior code we were getting the pool manager a bunch of times - once in the retrieve_connection_pool, once in remove_connection_pool, and once in the else conditional after setting it. This change ensures we're only getting the pool manager once. If there is no pool manager for the given key then we create a new one in the new `set_pool_manager` method and use that. The refactor also has a change in that we no longer instrument if a new connection was not established. This instrumentation is private though (denoted by the !) so it's safe to make this change. Followup to rails#45450
In rails#45450 we stopped removing connections when establish_connection was called and an identical connection existed in the pool. Unfortunately this broke granular connection swapping for the primary_abstract_class because the class stored in the pool was incorrect. What was happening was connection for the writing role were getting stored with the class ActiveRecord::Base and connections for the reading role were getting ApplicationRecord (or whatever the app set as their `primary_abstract_class`. If we made sure that the reading connections also got ActiveRecord::Base that would break granular swapping for both writing and reading because Active Record doesn't know whether you wanted global swapping or granular swapping for prevent writes. This bug only manifests on prevent writes because it's the only value we actually lookup on the connection pool (because at the point we need it we don't have access to self). To fix this I've decided to update the class on the existing pool if it doesn't match the owner_name passed. This is kind of gross I admit, but it's safe because the way we find connections is on the `connection_name`. The class is only stored so that the connection can find it's `current_preventing_writes` value. This also required me to move the ivar to an instance method for connection_class on the pool because we don't want to cache the value and want to make sure we're getting it from the pool config directly.
In rails#38235 I moved advisory locks to their own named connections, then in rails#39758 the advisory lock was left on Base.connection but then moved it it's own connection handler. I believe with rails#45450 that this change was made obsolete and can be returned to the prior behavior without having to open an additional connection. The tests added pass and I also tested this in my local demo to ensure that this is working correctly. When I originally changed the behavior here Matthew noted that this could be surprising for some setups that expect only one connection for a running migration. I thought there was an issue related to this but I can't find it.
Fixes 2 bugs in parallel testing. The first bug was related to changes made in rails#45450 which meant that we were no longer replacing the connection in parallel testing because the config object is equal (we simply merge a new db name but the object id of the config stays the same). This bug only manifested in mysql and sqlite3 interestingly. It would fail on the internal metadata tables because they were missing in the schema version check. To fix this I introduced a `clobber: true` kwarg onto the connection handler that allows us to bypass the functionality that won't make a new connection if the config is the same. This is an easy way to fall back to the old behavior from before this change. I only added `clobber` to the `reconstruct_from_schema` call because we need to actually replace the connection for these. It's not safe to add everywhere since we don't always want to replace the connection. After implementing this fix I was still seeing failures in the mysql demo app I made due to the fact that `purge` was not re-establishing the connection to a config that had a database defined. Neither sqlite3 or postgresql were missing this. I added a test for mysql2 so we don't have regressions in the future. I think this was missed because sqlite3 only demonstrates the bug if it was never successful on that worker and postgresql was fine. Fixes rails#48547
Previously Rails would always remove the connection if it found a
matching class in the pool manager. Therefore if
ActiveRecord::Base.establish_connectionwas called with the sameconfig, each time it was called it would be clobbered, even though the
config hasn't changed and the existing connection is prefectly fine. As
far as I can tell from conversations and reading the history this
functionality was added for ActiveRecord tests to be able to clobber the
connection and use a new config, then re-establish the old connection.
Essentially outside Rake tasks and AR tests, this functionality doesn't
have a ton of value.
On top of not adding a ton of value, this has resulted in a few bugs. In
Rails 6.0 I made it so that if you established a connection on
ApplicationRecordRails would treat that connection the same asActiveRecord::Base.The reason for this is that the Railtieestablishes a connection on boot to the first database, but then if
you're using multiple databases you're calling
connects_toin yourApplicationRecordor primary abstract class which essentially doublesyour connections to the same database. To avoid opening 2 connections to
the same database, Rails treats them the same.
However, because we have this code that removes existing connections,
when an application boots,
ApplicationRecordwill clobber theconnection that the Railtie established even though the connection
configs are the same.
This removal of the connection caused bugs in migrations that load up a
model connected to
ApplicationRecord(exPost.first) and then callsexecute("SELECT 1")(obviously a simplified example). Whenexecuteruns the connection is different from the one opened to run the
migration and essentially it is lost when the
remove_connectioncodeis called.
To fix this I've updated the code to only remove the connection if the
database config is different. Ultimately I'd like to remove this code
altogether but to do that we first need to stop using
Base.establish_connectionin the rake tasks and tests. This will fixthe major bugs until I come up with a solution for the areas that
currently need to call
establish_connectionon Base.The added benefit of this change is that if your app is calling
establish_connectionmultiple times with the same config, it is now3x faster than the previous implementation because we can return the
found pool instead of setting it up again. To benchmark this I
duplicated the
establish_connectionmethod to use the new behaviorwith a new name.
Benchmark script:
Benchmark results:
Other changes:
purgeiscalled. This is necessary now that a new connection isn't created
everyt time
establish_connectionis called. Without this change topurge the new database is left in an inaccessible state causing a
readonly error from the sqlite3 client. This wasn't happening in mysql
or postgres because they were already reconnecting the db connection.
remove_connectionto tests that useApplicationRecord.This is required because
ApplicationRecordor any class that is aprimary_abstract_classwill be treated the same asActiveRecord::Base. This is fine in applications because they areshared connections, but in the AR test environment, we don't want those
connnections to stick around (we want AR::Base back).
establish_connection. Thesewere causing sqlite3 tests to leak the state of async_executor because
it's stored on the connection. I'm not sure why these were calling
establish_connectionbut it's not necessary and was leaking state whennow that we are no longer removing the connection.
Fixes: #41855
Fixes: #41876
Fixes: #42873
Fixes: #43004
cc/ @matthewd as we talked about this today.