Use execute_batch2 rather than execute_batch to fix performance regression for fixture loading
#35844
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…egression for fixture loading d8d6bd5 makes fixture loading to bulk statements by using `execute_batch` for sqlite3 adapter. But `execute_batch` is slower and it caused the performance regression for fixture loading. In sqlite3 1.4.0, it have new batch method `execute_batch2`. I've confirmed `execute_batch2` is extremely faster than `execute_batch`. So I think it is worth to upgrade sqlite3 to 1.4.0 to use that method. Before: ``` % ARCONN=sqlite3 bundle exec ruby -w -Itest test/cases/associations/eager_test.rb -n test_eager_loading_too_may_ids Using sqlite3 Run options: -n test_eager_loading_too_may_ids --seed 35790 # Running: . Finished in 202.437406s, 0.0049 runs/s, 0.0049 assertions/s. 1 runs, 1 assertions, 0 failures, 0 errors, 0 skips ARCONN=sqlite3 bundle exec ruby -w -Itest -n test_eager_loading_too_may_ids 142.57s user 60.83s system 98% cpu 3:27.08 total ``` After: ``` % ARCONN=sqlite3 bundle exec ruby -w -Itest test/cases/associations/eager_test.rb -n test_eager_loading_too_may_ids Using sqlite3 Run options: -n test_eager_loading_too_may_ids --seed 16649 # Running: . Finished in 8.471032s, 0.1180 runs/s, 0.1180 assertions/s. 1 runs, 1 assertions, 0 failures, 0 errors, 0 skips ARCONN=sqlite3 bundle exec ruby -w -Itest -n test_eager_loading_too_may_ids 10.71s user 1.36s system 95% cpu 12.672 total ```
eileencodes
approved these changes
Apr 3, 2019
Member
eileencodes
left a comment
eileencodes
left a comment
There was a problem hiding this comment.
Thanks for the quick fix ❤️
rafaelfranca
approved these changes
Apr 3, 2019
kaspth
approved these changes
Apr 3, 2019
Contributor
kaspth
left a comment
kaspth
left a comment
There was a problem hiding this comment.
Looks like the build passed too!
bogdanvlviv
reviewed
Apr 6, 2019
| require "active_record/connection_adapters/sqlite3/schema_statements" | ||
|
|
||
| gem "sqlite3", "~> 1.3", ">= 1.3.6" | ||
| gem "sqlite3", "~> 1.4" |
Contributor
There was a problem hiding this comment.
We might need to add changelog entry to note that Active Record is compatible with sqlite3 ~> 1.4 since Rails 6.0.
There was a problem hiding this comment.
Agreed. This has just caught us out in cucumber-rails as we were using a 1.3 version of sqllite across the board. We'll put a conditional in for rails 6.
mracos
added a commit
to heartcombo/devise
that referenced
this pull request
May 4, 2019
Also bumped sqlite from 1.3.6 to 1.4 because besides conflicting with the version that the sqlite adapter was trying to load [0], it is supported officially since rails 6 [1]. Related: [0] rails/rails#35153 [1] rails/rails#35844
mracos
added a commit
to heartcombo/devise
that referenced
this pull request
May 4, 2019
Also bumped sqlite from 1.3.6 to 1.4 because besides conflicting with the version that the sqlite adapter was trying to load [0], it is supported officially since rails 6 [1]. Related: [0] rails/rails#35153 [1] rails/rails#35844
y-yagi
added a commit
to y-yagi/enumerize
that referenced
this pull request
May 25, 2019
Also, moved sqlite3 gem to individual gemfiles because Rails 6 requires sqlite3 upper 1.4.0. Ref: rails/rails#35844
jtapia
pushed a commit
to jtapia/devise
that referenced
this pull request
Jun 6, 2019
Also bumped sqlite from 1.3.6 to 1.4 because besides conflicting with the version that the sqlite adapter was trying to load [0], it is supported officially since rails 6 [1]. Related: [0] rails/rails#35153 [1] rails/rails#35844
mygreatstory
pushed a commit
to mygreatstory/rubenumerize
that referenced
this pull request
Jan 9, 2021
Also, moved sqlite3 gem to individual gemfiles because Rails 6 requires sqlite3 upper 1.4.0. Ref: rails/rails#35844
alimi
pushed a commit
to 1debit/rails-properties
that referenced
this pull request
May 29, 2024
* Starting in Rails 6, ActiveRecord's Sqlite adapter restricted the
sqlite3 gem version to ~> 1.4. This is still true in Rails 7.1. See
rails/rails#35844.
* Use the same restriction here to ensure a compatible sqlite3 version
is installed for the dev and test.
* The version restriction has been relaxed on Rails edge but it hasn't
landed in a released Rails version yet. See
rails/rails#51592.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
d8d6bd5 makes fixture loading to bulk statements by using
execute_batchfor sqlite3 adapter. Butexecute_batchis slower andit caused the performance regression for fixture loading.
In sqlite3 1.4.0, it have new batch method
execute_batch2. I'veconfirmed
execute_batch2is extremely faster thanexecute_batch.So I think it is worth to upgrade sqlite3 to 1.4.0 to use that method.
Before:
After:
What do you think?