Skip to content

[8.0] MOD-7745: Add Benchmarks For Document And Field Expiration#5064

Merged
kei-nan merged 2 commits into8.0from
backport-5030-to-8.0
Oct 7, 2024
Merged

[8.0] MOD-7745: Add Benchmarks For Document And Field Expiration#5064
kei-nan merged 2 commits into8.0from
backport-5030-to-8.0

Conversation

@github-actions
Copy link

@github-actions github-actions bot commented Sep 30, 2024

Description

Backport of #5030 to 8.0.

@kei-nan @github-actions
MOD-7745: Add Benchmarks For Document And Field Expiration (#5030)
ca26a6c 
* * initial commit

* * rename internal names

* * rename numeric field benchmark

* * use second resolution to avoid actually expiring stuff and ruining our dataset

* * typo

* * add key limitations to benchmarks

* * add two tests that will actually expire fields and documents
* will probably be less deterministic but we want to see how it actually behaves

* * change to a dataset where the __key__ can be effective

* * fix benchmark name

* * fix benchmark failures

* * remove requests limitation

* * updated lazy expiration benchmarks description

(cherry picked from commit 3aeb659)
@github-actions github-actions bot mentioned this pull request Sep 30, 2024
Merged
2 tasks
@GuyAv46 GuyAv46 marked this pull request as draft September 30, 2024 15:29
@GuyAv46 GuyAv46 marked this pull request as ready for review September 30, 2024 15:29
@GuyAv46 GuyAv46 requested a review from kei-nan September 30, 2024 15:29
@codecov
Copy link

codecov bot commented Sep 30, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.25%. Comparing base (0e40c71) to head (fdc3691).
Report is 5 commits behind head on 8.0.

Additional details and impacted files 
@@           Coverage Diff           @@
##              8.0    #5064   +/-   ##
=======================================
  Coverage   86.24%   86.25%           
=======================================
  Files         189      189           
  Lines       34313    34313           
=======================================
+ Hits        29594    29595    +1     
+ Misses       4719     4718    -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kei-nan kei-nan self-assigned this Oct 1, 2024
@kei-nan kei-nan enabled auto-merge October 7, 2024 13:30
@fcostaoliveira
Copy link
Contributor

fcostaoliveira commented Oct 7, 2024 •
edited
Loading

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 24 stable tests between versions.
  • Detected a total of 6 highly unstable benchmarks.
  • Detected a total of 7 improvements above the improvement water line.
  • Detected a total of 4 regressions bellow the regression water line 5.0.

You can check a comparison in detail via the grafana link

Comparison between master and backport-5030-to-8.0.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison backport-5030-to-8.0 (median obs. +- std.dev) % change (higher-better) Note
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby 70 +- 1.8% (5 datapoints) 68.00 -2.2% No Change
ftsb-10K-enwiki_abstract-hashes-term-prefix 7064 +- 5.6% (5 datapoints) 7440.00 5.3% waterline=5.6%. potential IMPROVEMENT
ftsb-10K-enwiki_abstract-hashes-term-suffix 2155 +- 1.6% (5 datapoints) 1964.00 -8.9% REGRESSION
ftsb-10K-enwiki_abstract-hashes-term-suffix-withsuffixtrie 67838 +- 4.2% (5 datapoints) 71605.00 5.6% IMPROVEMENT
ftsb-10K-enwiki_abstract-hashes-term-wildcard 12041 +- 4.3% (5 datapoints) 12390.00 2.9% No Change
ftsb-10K-enwiki_pages-hashes-load 45555 +- 2.9% (5 datapoints) 43741.00 -4.0% potential REGRESSION
ftsb-10K-multivalue-numeric-json 796 +- 3.9% (5 datapoints) 841.00 5.7% IMPROVEMENT
ftsb-10K-singlevalue-numeric-json 333 +- 1.4% (5 datapoints) 350.00 5.2% IMPROVEMENT
ftsb-1K-enwiki_abstract-hashes-term-contains 1812 +- 1.3% (5 datapoints) 1658.00 -8.5% REGRESSION
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable 39 +- 9.5% (5 datapoints) 34.00 -12.6% waterline=9.5%. REGRESSION
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query@100_ops_sec 100 +- 0.0% (5 datapoints) 100.00 0.0% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query 2728 +- 7.5% (5 datapoints) 2428.00 -11.0% waterline=7.5%. REGRESSION
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable 1007 +- 9.5% (5 datapoints) 986.00 -2.1% waterline=9.5%. No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query@100_ops_sec 100 +- 0.0% (5 datapoints) 100.00 0.0% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query 1045 +- 10.5% UNSTABLE (5 datapoints) 1046.00 0.1% UNSTABLE (very high variance)
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query@100_ops_sec 100 +- 0.0% (5 datapoints) 100.00 0.0% No Change
ftsb-1M-enwiki_abstract-hashes-load 21806 +- 3.0% (5 datapoints) 22481.00 3.1% potential IMPROVEMENT
ftsb-1M-nyc_taxis-ftadd-load 18081 +- 2.7% (4 datapoints) 19606.00 8.4% IMPROVEMENT
ftsb-1M-nyc_taxis-hashes-load 19605 +- 2.5% (5 datapoints) 21903.00 11.7% IMPROVEMENT
search-aggregate-post-filter-simple.yml 93527 +- 4.5% (5 datapoints) 91798.00 -1.8% No Change
search-filtering-tag-numeric 178 +- 11.8% UNSTABLE (5 datapoints) 202.00 13.3% UNSTABLE (very high variance)
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie 40000 +- 3.2% (5 datapoints) 39747.00 -0.6% No Change
search-ftsb-10K-enwiki_abstract-hashes-term-withsuffix-trie 40720 +- 2.1% (5 datapoints) 41735.00 2.5% No Change
search-ftsb-1700K-docs-union-iterators-q3 6.1 +- 2.1% (4 datapoints) 6.30 3.4% potential IMPROVEMENT
search-ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable@50_ops_sec 50 +- 7.7% (5 datapoints) 50.00 0.3% waterline=7.7%. No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable@100_ops_sec 100 +- 0.0% (5 datapoints) 100.00 -0.0% No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable 147 +- 6.7% (5 datapoints) 160.00 8.5% waterline=6.7%. IMPROVEMENT
search-ftsb-370K-docs-union-iterators-q4 6.5 +- 1.7% (5 datapoints) 6.70 1.5% No Change
search-ftsb-5500K-docs-union-iterators-q2 0.91 +- 0.5% (4 datapoints) 0.92 1.1% No Change
search-geo 179 +- 3.4% (5 datapoints) 193.00 7.9% IMPROVEMENT
search-high-cardinality-negation-term-baseline 20 +- 1.4% (4 datapoints) 21.00 4.2% potential IMPROVEMENT
search-high-cardinality-negation-term-comparison_union_all_other_terms 7.2 +- 4.2% (4 datapoints) 7.30 0.3% No Change
search-numeric 3075 +- 11.8% UNSTABLE (5 datapoints) 2738.00 -11.0% UNSTABLE (very high variance)
search-numeric-optimize 9210 +- 0.9% (5 datapoints) 9127.00 -0.9% No Change
search-numeric-sortby 3068 +- 35.8% UNSTABLE (5 datapoints) 4401.00 43.4% UNSTABLE (very high variance)
search-numeric-sortby-desc 2680 +- 52.4% UNSTABLE (5 datapoints) 2669.00 -0.4% UNSTABLE (very high variance)
search-numeric-sortby-desc-optimize 48 +- 21.5% UNSTABLE (5 datapoints) 45.00 -5.8% UNSTABLE (very high variance)
search-numeric-sortby-optimize 20 +- 3.2% (5 datapoints) 19.00 -2.8% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter 480 +- 3.4% (5 datapoints) 491.00 2.3% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-numeric-filter 103 +- 7.0% (5 datapoints) 102.00 -0.4% waterline=7.0%. No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter 58418 +- 1.0% (5 datapoints) 55602.00 -4.8% potential REGRESSION

@kei-nan kei-nan added this pull request to the merge queue Oct 7, 2024
Merged via the queue into 8.0 with commit 8643499 Oct 7, 2024
@kei-nan kei-nan deleted the backport-5030-to-8.0 branch October 7, 2024 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kei-nan kei-nan kei-nan approved these changes

Assignees

@kei-nan kei-nan

Labels

action:run-benchmark

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@fcostaoliveira @kei-nan