RUM-3796: Optimise BatchFileOrchestator performance#1968
Merged
0xnm merged 6 commits intoApr 8, 2024
Conversation
xgouchet
approved these changes
Apr 5, 2024
xgouchet
left a comment
xgouchet
left a comment
Contributor
There was a problem hiding this comment.
Really nice job there !
| get() = File("${this.path}_metadata") | ||
| private fun listSortedBatchFiles(): List<File> { | ||
| // note: since it is using File#compareTo, lexicographical sorting will be used, meaning "10" comes before "9". | ||
| // but for our needs it is fine, because the moment when Unix timestamp adds one more digit will be in 2286. |
Contributor
There was a problem hiding this comment.
Few, I guess it'll be for others to deal with this then…
Member
Author
There was a problem hiding this comment.
Unless SDK is installed on time machine device
0xnm
force-pushed
the
nogorodnikov/rum-3796/optimise-batch-file-orchestrator
branch
from
April 5, 2024 12:49
916d402 to
949f2c9
Compare
0xnm
force-pushed
the
nogorodnikov/rum-3796/optimise-batch-file-orchestrator
branch
from
April 5, 2024 12:50
949f2c9 to
63471a2
Compare
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## develop #1968 +/- ##
===========================================
- Coverage 83.40% 83.38% -0.01%
===========================================
Files 489 489
Lines 17923 17902 -21
Branches 2669 2669
===========================================
- Hits 14947 14927 -20
- Misses 2237 2241 +4
+ Partials 739 734 -5
|
mariusc83
approved these changes
Apr 8, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR optimise
BatchFileOrchestatorperformance by doing the following changes:File.isFilecheck - this is expensive syscall and it should be only files in the particular folder anyway (it is onlyBatchFileOrchestratorwho works with that folder) and we check file names anyway. AlsoFile.isFiledoc says:Any non-directory file created by a Java application is guaranteed to be a normal file.Long.File.isFilecheck is removed as benchmarks show below)Benchmarks
The testing approach was the following: generate 500 files of size ranging from 1Kb to 32Kb, evenly distributed in the last 24h by their names as timestamps. Then do 100 calls to
getWritableFilepassingforceNewFile = trueat every even call, andforceNewFile = falseat every odd call.Tests:
getWritableFile)Table below shows the average duration of each
getWritableFilecall in milliseconds.File.isFileremovedNB: it shouldn't be considered as a proper benchmarking methodology (JMH is not compatible with AGP, there is some variability, etc.), but can give an idea of the impact for the improvements.
Last 2 changes (cache removal and change of the cleanup frequency) are not listed, because they directly impact tests.
All changes are in separate commits.
Review checklist (to be filled by reviewers)