Skip to content

[8.0] [MOD-7480] Single build path + benchmark fix#5036

Merged
alonre24 merged 7 commits into8.0from
single_build_path_cp
Sep 26, 2024
Merged

[8.0] [MOD-7480] Single build path + benchmark fix#5036
alonre24 merged 7 commits into8.0from
single_build_path_cp

Conversation

@alonre24
Copy link
Collaborator

Describe the changes in the pull request

CP #4935 and #5026

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

kei-nan and others added 5 commits September 10, 2024 21:56
* * initial commit

* * Use RediSearch branch mirroring fork

* * Code Review - Round #1
* * initial commit

* * code review
* 7.99.0

* redis unstable tests
* Fix benchmark to use new build path

* Take Raz's fix for benchmarks

* fix default module path

* make benchmark fail

* fix path

* fail external flow

* remove fail fast and continue on error

* addressing CR

* Splitting the standalone work among 6 groups. Only profile if explictly enabled. Only run >4 primaries if extended if enabled.

* Specify name of benchmark step on GH flow

* Using perf-cto-base-image-ubuntu20.04-redis-7.4.0-tuned-latency on spot instances as well.

* Fixed RedisJSON module path on benchmark setup. Enforcing ReJSON module to be present

* Changing from 6 to 9 oss-standalone runners to achieve a time for full feedback < 30min

* MODULE_FILE not working on Prepare ReJSON Module: using default location

* Using redisbench-admin >= 0.11.3 due to multi-module issue

* Using redisbench-admin >= 0.11.3 due to multi-module issue

* Added missing --allowed-setups info for oss-standalone config

* redisbench-admin >= 0.11.4

* Remove vecsim-ann-benchmarks_LOAD_1c_redisearch-hnsw_glove-200-angular_M-4.yml

* Using redisbench-admin >= 0.11.5 due to ssh timeout issues on heavy benchmarks or heavy prunning at the end

* Increasing concurrency of standalone benchmarks to reduce time to feedback

* redisbench-admin >= 0.11.6. removed unstable test vecsim-ann-benchmarks_QUERY_1c_redisearch-hnsw_glove-200-angular_M-4

* redisbench-admin >= 0.11.7. running cluster tests faster and also ensuring we use a redisbench-admin with keepalive on ssh tunnels

* Removed search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie from cluster tests due to keyspace error check

* Removed search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable from cluster tests due to client error. created RediSearch/ftsb#97

* Not running *.rdb based benchmarks on cluster due to orphaned keys (redis/redis#9506)

---------

Co-authored-by: fcostaoliveira <[email protected]>
@fcostaoliveira
Copy link
Contributor

fcostaoliveira commented Sep 23, 2024

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 30 stable tests between versions.
  • Detected a total of 10 improvements above the improvement water line.
  • Detected a total of 4 regressions bellow the regression water line 5.0.

You can check a comparison in detail via the grafana link

Comparison between master and single_build_path_cp.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison single_build_path_cp (median obs. +- std.dev) % change (higher-better) Note
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby 72 67 -6.3% REGRESSION
ftsb-10K-enwiki_abstract-hashes-term-prefix 6934 7410 6.9% IMPROVEMENT
ftsb-10K-enwiki_abstract-hashes-term-suffix 2192 2102 -4.1% potential REGRESSION
ftsb-10K-enwiki_abstract-hashes-term-suffix-withsuffixtrie 72078 66027 -8.4% REGRESSION
ftsb-10K-enwiki_abstract-hashes-term-wildcard 12041 12400 3.0% No Change
ftsb-10K-enwiki_pages-hashes-fulltext-mixed_simple-1word-query_write_1_to_read_20.yml 1207 1230 1.9% No Change
ftsb-10K-enwiki_pages-hashes-load 42862 52120 21.6% IMPROVEMENT
ftsb-10K-multivalue-numeric-json 774 795 2.7% No Change
ftsb-10K-singlevalue-numeric-json 333 326 -2.2% No Change
ftsb-1K-enwiki_abstract-hashes-term-contains 1812 1854 2.4% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query N/A N/A 0.0%
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable 35 40 12.8% IMPROVEMENT
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query@100_ops_sec 100 100 0.0% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query 2654 2524 -4.9% potential REGRESSION
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable 1007 1062 5.4% IMPROVEMENT
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query@100_ops_sec 100 100 -0.0% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query 1045 1024 -2.0% No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query@100_ops_sec 100 100 0.0% No Change
ftsb-1M-enwiki_abstract-hashes-load 21458 22523 5.0% potential IMPROVEMENT
ftsb-1M-nyc_taxis-ftadd-load 17896 20352 13.7% IMPROVEMENT
ftsb-1M-nyc_taxis-hashes-load 19890 21695 9.1% IMPROVEMENT
search-aggregate-post-filter-simple.yml 89332 100601 12.6% IMPROVEMENT
search-filtering-tag-numeric 178 211 18.8% IMPROVEMENT
search-filtering-tag-numeric-filter-pipeline 18730 19455 3.9% potential IMPROVEMENT
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie 40668 40782 0.3% No Change
search-ftsb-10K-enwiki_abstract-hashes-term-withsuffix-trie 40720 39483 -3.0% potential REGRESSION
search-ftsb-1700K-docs-union-iterators-q3 N/A N/A 0.0%
search-ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable@50_ops_sec 50 50 0.6% No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable@100_ops_sec 100 100 -0.0% No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable 147 148 0.2% No Change
search-ftsb-370K-docs-union-iterators-q4 6.50 6.80 4.6% potential IMPROVEMENT
search-ftsb-5500K-docs-union-iterators-q2 0.92 0.93 1.1% No Change
search-geo 184 178 -3.7% potential REGRESSION
search-high-cardinality-negation-term-baseline N/A N/A 0.0%
search-high-cardinality-negation-term-comparison_union_all_other_terms 7.10 6.90 -2.8% No Change
search-numeric 3405 3166 -7.0% REGRESSION
search-numeric-optimize 9266 9335 0.7% No Change
search-numeric-sortby 2608 2748 5.4% IMPROVEMENT
search-numeric-sortby-desc 5105 3070 -39.9% REGRESSION
search-numeric-sortby-desc-optimize 48 47 -1.3% No Change
search-numeric-sortby-optimize 19 21 7.3% IMPROVEMENT
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter 480 479 -0.2% No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-numeric-filter 104 108 3.7% potential IMPROVEMENT
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter 58897 56850 -3.5% potential REGRESSION

@codecov
Copy link

codecov bot commented Sep 23, 2024

Codecov Report

Attention: Patch coverage is 90.33898% with 114 lines in your changes missing coverage. Please review.

Project coverage is 86.24%. Comparing base (7d1c19d) to head (a3d19b3).
Report is 2 commits behind head on 8.0.

Files with missing lines Patch % Lines
src/module.c 90.33% 114 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##              8.0    #5036      +/-   ##
==========================================
+ Coverage   86.12%   86.24%   +0.12%     
==========================================
  Files         192      189       -3     
  Lines       34416    34305     -111     
==========================================
- Hits        29641    29587      -54     
+ Misses       4775     4718      -57     
Flag Coverage Δ
86.24% <90.33%> (+0.12%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@GuyAv46 GuyAv46 changed the title [MOD-7480] Single build path + benchmark fix [0.8] [0.8] [MOD-7480] Single build path + benchmark fix Sep 23, 2024
@alonre24 alonre24 changed the title [0.8] [MOD-7480] Single build path + benchmark fix [8.0] [MOD-7480] Single build path + benchmark fix Sep 23, 2024
@alonre24 alonre24 enabled auto-merge September 23, 2024 13:26
@alonre24 alonre24 added this pull request to the merge queue Sep 23, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 23, 2024
@alonre24 alonre24 force-pushed the single_build_path_cp branch from 4b5ed2e to 938f9c2 Compare September 25, 2024 09:17
@alonre24 alonre24 enabled auto-merge September 25, 2024 09:20
@alonre24 alonre24 added this pull request to the merge queue Sep 25, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 25, 2024
@alonre24 alonre24 added this pull request to the merge queue Sep 26, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 26, 2024
@alonre24 alonre24 added this pull request to the merge queue Sep 26, 2024
Merged via the queue into 8.0 with commit 16abfe1 Sep 26, 2024
@alonre24 alonre24 deleted the single_build_path_cp branch September 26, 2024 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants