Skip to content

MOD-7745: Add Benchmarks For Document And Field Expiration#5030

Merged
kei-nan merged 12 commits intomasterfrom
master_jk_add_expiration_benchmarks
Sep 30, 2024
Merged

MOD-7745: Add Benchmarks For Document And Field Expiration#5030
kei-nan merged 12 commits intomasterfrom
master_jk_add_expiration_benchmarks

Conversation

@kei-nan
Copy link
Collaborator

@kei-nan kei-nan commented Sep 16, 2024

Add benchmarks for document and field expiration

  1. Currently there aren't any benchmark tests for document and field expiration
  2. This PR adds benchmarks for document and field expiration

Which issues this PR fixes

  1. MOD-7745

Main objects this PR modified
1.Benchmarks

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

@codecov
Copy link

codecov bot commented Sep 16, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.30%. Comparing base (45de454) to head (b7bf7ef).
Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #5030   +/-   ##
=======================================
  Coverage   86.30%   86.30%           
=======================================
  Files         190      190           
  Lines       34601    34601           
=======================================
+ Hits        29861    29863    +2     
+ Misses       4740     4738    -2     
Flag Coverage Δ
86.30% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@raz-mon raz-mon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

@kei-nan kei-nan requested a review from raz-mon September 17, 2024 13:49
@kei-nan kei-nan force-pushed the master_jk_add_expiration_benchmarks branch from 8f9b63a to c348ad0 Compare September 24, 2024 08:38
@fcostaoliveira
Copy link
Contributor

fcostaoliveira commented Sep 24, 2024

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 30 stable tests between versions.
  • Detected a total of 8 improvements above the improvement water line.
  • Detected a total of 6 regressions bellow the regression water line 5.0.

You can check a comparison in detail via the grafana link

Comparison between master and master_jk_add_expiration_benchmarks.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison master_jk_add_expiration_benchmarks (median obs. +- std.dev) % change (higher-better) Note
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby 72 69 -3.2% potential REGRESSION
ftsb-10K-enwiki_abstract-hashes-term-prefix 6934 7591 9.5% IMPROVEMENT
ftsb-10K-enwiki_abstract-hashes-term-suffix 2192 2157 -1.6% No Change
ftsb-10K-enwiki_abstract-hashes-term-suffix-withsuffixtrie 72078 64510 -10.5% REGRESSION
ftsb-10K-enwiki_abstract-hashes-term-wildcard 12041 12387 2.9% No Change
ftsb-10K-enwiki_pages-hashes-fulltext-mixed_simple-1word-query_write_1_to_read_20.yml 1207 1291 7.0% IMPROVEMENT
ftsb-10K-enwiki_pages-hashes-load 42862 42476 -0.9% No Change
ftsb-10K-multivalue-numeric-json 774 813 5.1% IMPROVEMENT
ftsb-10K-singlevalue-numeric-json 333 341 2.5% No Change
ftsb-1K-enwiki_abstract-hashes-term-contains 1812 1772 -2.2% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query N/A N/A 0.0%
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable 35 44 25.0% IMPROVEMENT
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query@100_ops_sec 100 100 0.0% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query 2654 2915 9.8% IMPROVEMENT
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable 1007 952 -5.5% REGRESSION
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query@100_ops_sec 100 100 -0.0% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query 1045 899 -14.0% REGRESSION
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query@100_ops_sec 100 100 0.1% No Change
ftsb-1M-enwiki_abstract-hashes-load 21458 21827 1.7% No Change
ftsb-1M-nyc_taxis-ftadd-load 17896 17559 -1.9% No Change
ftsb-1M-nyc_taxis-hashes-load 19890 19088 -4.0% potential REGRESSION
search-aggregate-post-filter-simple.yml 89332 95156 6.5% IMPROVEMENT
search-filtering-tag-numeric 178 200 12.6% IMPROVEMENT
search-filtering-tag-numeric-filter-pipeline 18730 19649 4.9% potential IMPROVEMENT
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie 40668 40670 0.0% No Change
search-ftsb-10K-enwiki_abstract-hashes-term-withsuffix-trie 40720 38767 -4.8% potential REGRESSION
search-ftsb-1700K-docs-union-iterators-q3 N/A N/A 0.0%
search-ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable@50_ops_sec 50 50 0.9% No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable@100_ops_sec 100 100 -0.0% No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable 147 142 -3.9% potential REGRESSION
search-ftsb-370K-docs-union-iterators-q4 6.50 6.6 1.4% No Change
search-ftsb-5500K-docs-union-iterators-q2 0.92 0.9 -2.2% No Change
search-geo 184 181 -2.1% No Change
search-high-cardinality-negation-term-baseline N/A N/A 0.0%
search-high-cardinality-negation-term-comparison_union_all_other_terms 7.10 6.8 -4.2% potential REGRESSION
search-numeric 3405 3061 -10.1% REGRESSION
search-numeric-optimize 9266 9153 -1.2% No Change
search-numeric-sortby 2608 3202 22.8% IMPROVEMENT
search-numeric-sortby-desc 5105 2750 -46.1% REGRESSION
search-numeric-sortby-desc-optimize 48 34 -28.1% REGRESSION
search-numeric-sortby-optimize 19 19 -0.1% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter 480 485 1.2% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-numeric-filter 104 106 1.4% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter 58897 58074 -1.4% No Change

@kei-nan kei-nan requested review from nafraf and oshadmi and removed request for oshadmi and raz-mon September 24, 2024 09:17
@kei-nan kei-nan requested a review from nafraf September 26, 2024 16:00
@kei-nan kei-nan enabled auto-merge September 26, 2024 16:00
@kei-nan kei-nan requested a review from oshadmi September 29, 2024 06:09
@kei-nan kei-nan requested a review from oshadmi September 29, 2024 07:10
oshadmi
oshadmi previously approved these changes Sep 29, 2024
@kei-nan kei-nan force-pushed the master_jk_add_expiration_benchmarks branch from be57882 to b7bf7ef Compare September 29, 2024 19:19
@kei-nan kei-nan added this pull request to the merge queue Sep 30, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 30, 2024
@kei-nan kei-nan added this pull request to the merge queue Sep 30, 2024
Merged via the queue into master with commit 3aeb659 Sep 30, 2024
@kei-nan kei-nan deleted the master_jk_add_expiration_benchmarks branch September 30, 2024 15:29
github-actions bot pushed a commit that referenced this pull request Sep 30, 2024
* * initial commit

* * rename internal names

* * rename numeric field benchmark

* * use second resolution to avoid actually expiring stuff and ruining our dataset

* * typo

* * add key limitations to benchmarks

* * add two tests that will actually expire fields and documents
* will probably be less deterministic but we want to see how it actually behaves

* * change to a dataset where the __key__ can be effective

* * fix benchmark name

* * fix benchmark failures

* * remove requests limitation

* * updated lazy expiration benchmarks description

(cherry picked from commit 3aeb659)
@github-actions
Copy link

Successfully created backport PR for 2.10:

github-actions bot pushed a commit that referenced this pull request Sep 30, 2024
* * initial commit

* * rename internal names

* * rename numeric field benchmark

* * use second resolution to avoid actually expiring stuff and ruining our dataset

* * typo

* * add key limitations to benchmarks

* * add two tests that will actually expire fields and documents
* will probably be less deterministic but we want to see how it actually behaves

* * change to a dataset where the __key__ can be effective

* * fix benchmark name

* * fix benchmark failures

* * remove requests limitation

* * updated lazy expiration benchmarks description

(cherry picked from commit 3aeb659)
@github-actions
Copy link

Successfully created backport PR for 8.0:

github-merge-queue bot pushed a commit that referenced this pull request Sep 30, 2024
)

MOD-7745: Add Benchmarks For Document And Field Expiration (#5030)

* * initial commit

* * rename internal names

* * rename numeric field benchmark

* * use second resolution to avoid actually expiring stuff and ruining our dataset

* * typo

* * add key limitations to benchmarks

* * add two tests that will actually expire fields and documents
* will probably be less deterministic but we want to see how it actually behaves

* * change to a dataset where the __key__ can be effective

* * fix benchmark name

* * fix benchmark failures

* * remove requests limitation

* * updated lazy expiration benchmarks description

(cherry picked from commit 3aeb659)

Co-authored-by: kei-nan <[email protected]>
github-merge-queue bot pushed a commit that referenced this pull request Oct 7, 2024
MOD-7745: Add Benchmarks For Document And Field Expiration (#5030)

* * initial commit

* * rename internal names

* * rename numeric field benchmark

* * use second resolution to avoid actually expiring stuff and ruining our dataset

* * typo

* * add key limitations to benchmarks

* * add two tests that will actually expire fields and documents
* will probably be less deterministic but we want to see how it actually behaves

* * change to a dataset where the __key__ can be effective

* * fix benchmark name

* * fix benchmark failures

* * remove requests limitation

* * updated lazy expiration benchmarks description

(cherry picked from commit 3aeb659)

Co-authored-by: kei-nan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants