add gcc11:bullseye to platforms [MOD-9529]#6198
Merged
BenGoldberger merged 8 commits intomasterfrom Jun 3, 2025
Merged
Conversation
BenGoldberger
added
x:platform
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #6198 +/- ##
==========================================
- Coverage 88.31% 88.24% -0.07%
==========================================
Files 227 227
Lines 39953 39953
Branches 2592 2592
==========================================
- Hits 35285 35258 -27
- Misses 4653 4680 +27
Partials 15 15
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
nafraf
reviewed
May 26, 2025
|
|
||
| # Ubuntu and Debian distributions need git pre-checkout dependencies | ||
| debian_based_os = ['ubuntu:noble', 'ubuntu:jammy', 'ubuntu:focal', 'debian:bullseye', 'gcc:12-bookworm'] | ||
| debian_based_os = ['ubuntu:noble', 'ubuntu:jammy', 'ubuntu:focal', 'debian:bullseye', 'gcc:12-bookworm', 'gcc:11-bullseye'] |
Collaborator
There was a problem hiding this comment.
Hi, shouldn't we remove debian:bullseye as we did with debian:bookworm when it was replaced by gcc:12-bookworm ?
Collaborator
There was a problem hiding this comment.
It's a good question, since gcc11 is not natively available in bullseye, we still want to test against the common use case. But maybe it is not needed. WDYT? @BenGoldberger @nafraf
Collaborator
Author
There was a problem hiding this comment.
as long as we deploy search on debian11 I think it should be tested.
The gcc11 enabled optimizations are going to be the default?
Collaborator
There was a problem hiding this comment.
probably not. Let's have both on nightly, and only the native (gcc10) debian on the merge queue then as did on vecsim
Collaborator
There was a problem hiding this comment.
why not the other way around? (gcc11 on merge queue)
It will be useful to identify failures (both in vecsim in search) on svs when a pr is merged rather than when it will fail the nightly flow
Collaborator
Author
meiravgri
reviewed
May 29, 2025
Collaborator
meiravgri
left a comment
meiravgri
left a comment
There was a problem hiding this comment.
looks good
if there's a mod number related to the pr please add to title so it will be linked to the ticket :)
Does verify_build_deps.sh require any adjustments?
BenGoldberger
changed the title
Collaborator
Author
I dont think so, it still runs on debian environment. |
meiravgri
approved these changes
May 29, 2025
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
nafraf
pushed a commit
that referenced
this pull request
Jun 4, 2025
* add gcc11:bullseye to platforms * add naming * remove deps * add nightly platforms option * set merge queue to false * fix flow * fix flow
LukeMathWalker
pushed a commit
that referenced
this pull request
Jun 4, 2025
* add gcc11:bullseye to platforms * add naming * remove deps * add nightly platforms option * set merge queue to false * fix flow * fix flow
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jun 16, 2025
* Normalize multi-byte char terms * Fix rm_strdupcase_utf8 * Remove debug log * Add Russian alphabet and diacritic tests * Don't use DefaultNormalize_utf8() for chinese * print available locales * Check locale before using utf8 normalization * Fix DefaultNormalize() * Skip multibyte tests if 'en_US.UTF-8' locale is not available * Test multibyte stopwords * Revert changes in install_script.sh * run sanitizer using ubuntu:latest * Test_cn: fix language_field * Remove unused strtolower function from misc.c and misc.h * Fixes from code review * Enhance multibyte character tests * Validate queries using multi-byte stopwords * Add MULTIBYTE_CHARS config param and install locales * Use setlocale instead of querylocale for better compatibility * revert changes in event-pull_request.yml and install French locale for debian/ubuntu * Convert to lowercase using nunicode library * nunicode_tolower() returns zero terminated string * Refactor nunicode_tolower() to improve readability * Refactor nunicode_tolower() to use destination buffer and improve memory management * Remove null termination in nunicode_tolower() * Improve documentation * Support multi-byte chars for tags * Support multi-byte chars for synonyms * unicode_tolower(): avoid duplicating encoded input * Increase SSO_MAX_LENGTH * Update test to use FT.DEBUG DUMP_TAGIDX instead of FT.TAGVALS * Keep previous test_cn:testSynonym and test_cn:testMixedHighlight * Update CMake configuration for consistent multi-byte char sorting * Test JSON index * Fix JSON test and remove unneeded tolower() * Fix query_EvalSingleTagNode() * Refactor string normalization to use case folding instead of lowercase conversion * Refactor tests to use run_command_on_all_shards for setting DEFAULT_DIALECT * BWC: Add function to convert string to single codepoint folded runes * Rename rm_strdupcase() to rm_normalize() * Add stemming test * Refactor string normalization to use lowercase transformation instead of folding. Modify FT.SUGGET flow to be BWC * Refactor filtering functions to use a transformation callback for rune processing. Remove dead code. * Update testDFAFilter to use strToSingleCodepointFoldedRunes for rune processing * Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation * Revert "Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation" This reverts commit ebcf994. * Add case sensitivity option to tag string processing functions * Rename tag_strtofold to tag_strtolower for clarity and update related references to reflect lowercase transformation * Simplify length checks in tag_strtolower and rm_normalize functions * Add documentation after review * WIP: Enhance unicode_tolower function to handle memory allocation for transformed strings * WIP: Fixing unicode_tolower usage * Fix tag_strtolower() to support memory realloc * skip testToLowerSize() on cluster * Fix rm_normalize and add more tests * Fix memory leak * Fix memory leak 2 * MOD-9609: Pass Private Data Context Back To UnblockClient API (#6237) * Apparently private data had to be sent back when unblocking the client in order for the free function to get called * code review comment * solve memory leak * MOD-9774: Stop Compiling Redis-With Coverage (#6231) * stop compiling redis-server with coverage * bring back coverage as requirement * Add unit test for unicode_tolower * Fix memory management in UnicodeToLowerTest and update expected results in multibyte character tests * Add tests for unsupported Unicode characters in tolower conversion. * MOD-9768: Add Rust abstractions for Buffer* (#6175) * Add Rust abstractions for Buffer* * Add tests * Make Buffer fields pub For consistency with BufferReader and BufferWriter. Also there is a very good chance of us needing to use them directly in the near future. * Expand the docs a bit more * Add and improve upon safety comments * fmt fix * Add forgotten license headers * Revert "Make Buffer fields pub" This reverts commit 1243590. * Address review comments * Address review comments * Better document safety of `Buffer` fields * Apply suggestions from code review Co-authored-by: Luca Palmieri <[email protected]> * Remove an unneeded import * Use reallocation in mock `Buffer_Grow` * Satisfy MIRI about some pointer access * Drop unneeded dep * Remove unnecessary #includes * Split FFI into a separate crate * Add buffer.h API to ffi crate * Buffer API now based on generated bindings for the C Buffer API * CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env Just acting on the advice from a panic: ``` To improve backtraces for build dependencies, set the CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation. ``` * Drop linking of internal libs in ffi crate * Apply suggestions from code review Co-authored-by: Luca Palmieri <[email protected]> * Revert "CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env" This reverts commit 9bd1d87. * Drop a redundant import * Add a few debug_asserts * Bring back an accidentally deleted assertion in a test * Fix a .gitignore file * Make a safety assumption explicit * Dont build `buffer` crate as static lib * Don't build bindings for oxidized triemap code * Make use of BufferWritter::buffer_mut * Fix build on Alpine --------- Co-authored-by: Luca Palmieri <[email protected]> * refactor EmptyIterator (#6240) * refactor EmptyIterator * fix useless imports and doc * update empty_iterator.c to add newline * add gcc11:bullseye to platforms [MOD-9529] (#6198) * add gcc11:bullseye to platforms * add naming * remove deps * add nightly platforms option * set merge queue to false * fix flow * fix flow * chore: Improve granularity of build invalidation logic for FFI bindings (#6244) * Add `tests/deps/RedisJSON` to `.gitignore` (#6249) * Skip tests of unsupported unicode 9.0.0 chars * Fix mem leak 3 * Test stopwords which require memory reallocation * unicode_tolower update out_len always * Refactor unicode_tolower function to receive a single *inout_len argument * Update documentation for tag_strtolower() * Fixes after review * tokenizeTagString: Rename origLen as len * Test prefix/infix/suffix search with supported unicode chars * Update test description * Fix testToLowerConversionExactMatch * testTextToLowerConversionSimilarMatch: remove unused doc_id in queries * fix crash in test * Fix test failure on focal and bullseye: avoid nested f-strings with curly brackets * Fix test failure on mac: invalid escape sequence * Use SSO in runesToStr and strToLowerRunes * Add utf8_len arg to strToLowerRunes() * Align and move strToLowerRunes comment to header --------- Co-authored-by: kei-nan <[email protected]> Co-authored-by: Zeeshan Ali Khan <[email protected]> Co-authored-by: Luca Palmieri <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Co-authored-by: BenGoldberger <[email protected]> Co-authored-by: oshadmi <[email protected]>
redisearch-backport-pull-request bot
pushed a commit
that referenced
this pull request
Jun 16, 2025
* Normalize multi-byte char terms * Fix rm_strdupcase_utf8 * Remove debug log * Add Russian alphabet and diacritic tests * Don't use DefaultNormalize_utf8() for chinese * print available locales * Check locale before using utf8 normalization * Fix DefaultNormalize() * Skip multibyte tests if 'en_US.UTF-8' locale is not available * Test multibyte stopwords * Revert changes in install_script.sh * run sanitizer using ubuntu:latest * Test_cn: fix language_field * Remove unused strtolower function from misc.c and misc.h * Fixes from code review * Enhance multibyte character tests * Validate queries using multi-byte stopwords * Add MULTIBYTE_CHARS config param and install locales * Use setlocale instead of querylocale for better compatibility * revert changes in event-pull_request.yml and install French locale for debian/ubuntu * Convert to lowercase using nunicode library * nunicode_tolower() returns zero terminated string * Refactor nunicode_tolower() to improve readability * Refactor nunicode_tolower() to use destination buffer and improve memory management * Remove null termination in nunicode_tolower() * Improve documentation * Support multi-byte chars for tags * Support multi-byte chars for synonyms * unicode_tolower(): avoid duplicating encoded input * Increase SSO_MAX_LENGTH * Update test to use FT.DEBUG DUMP_TAGIDX instead of FT.TAGVALS * Keep previous test_cn:testSynonym and test_cn:testMixedHighlight * Update CMake configuration for consistent multi-byte char sorting * Test JSON index * Fix JSON test and remove unneeded tolower() * Fix query_EvalSingleTagNode() * Refactor string normalization to use case folding instead of lowercase conversion * Refactor tests to use run_command_on_all_shards for setting DEFAULT_DIALECT * BWC: Add function to convert string to single codepoint folded runes * Rename rm_strdupcase() to rm_normalize() * Add stemming test * Refactor string normalization to use lowercase transformation instead of folding. Modify FT.SUGGET flow to be BWC * Refactor filtering functions to use a transformation callback for rune processing. Remove dead code. * Update testDFAFilter to use strToSingleCodepointFoldedRunes for rune processing * Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation * Revert "Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation" This reverts commit ebcf994. * Add case sensitivity option to tag string processing functions * Rename tag_strtofold to tag_strtolower for clarity and update related references to reflect lowercase transformation * Simplify length checks in tag_strtolower and rm_normalize functions * Add documentation after review * WIP: Enhance unicode_tolower function to handle memory allocation for transformed strings * WIP: Fixing unicode_tolower usage * Fix tag_strtolower() to support memory realloc * skip testToLowerSize() on cluster * Fix rm_normalize and add more tests * Fix memory leak * Fix memory leak 2 * MOD-9609: Pass Private Data Context Back To UnblockClient API (#6237) * Apparently private data had to be sent back when unblocking the client in order for the free function to get called * code review comment * solve memory leak * MOD-9774: Stop Compiling Redis-With Coverage (#6231) * stop compiling redis-server with coverage * bring back coverage as requirement * Add unit test for unicode_tolower * Fix memory management in UnicodeToLowerTest and update expected results in multibyte character tests * Add tests for unsupported Unicode characters in tolower conversion. * MOD-9768: Add Rust abstractions for Buffer* (#6175) * Add Rust abstractions for Buffer* * Add tests * Make Buffer fields pub For consistency with BufferReader and BufferWriter. Also there is a very good chance of us needing to use them directly in the near future. * Expand the docs a bit more * Add and improve upon safety comments * fmt fix * Add forgotten license headers * Revert "Make Buffer fields pub" This reverts commit 1243590. * Address review comments * Address review comments * Better document safety of `Buffer` fields * Apply suggestions from code review Co-authored-by: Luca Palmieri <[email protected]> * Remove an unneeded import * Use reallocation in mock `Buffer_Grow` * Satisfy MIRI about some pointer access * Drop unneeded dep * Remove unnecessary #includes * Split FFI into a separate crate * Add buffer.h API to ffi crate * Buffer API now based on generated bindings for the C Buffer API * CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env Just acting on the advice from a panic: ``` To improve backtraces for build dependencies, set the CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation. ``` * Drop linking of internal libs in ffi crate * Apply suggestions from code review Co-authored-by: Luca Palmieri <[email protected]> * Revert "CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env" This reverts commit 9bd1d87. * Drop a redundant import * Add a few debug_asserts * Bring back an accidentally deleted assertion in a test * Fix a .gitignore file * Make a safety assumption explicit * Dont build `buffer` crate as static lib * Don't build bindings for oxidized triemap code * Make use of BufferWritter::buffer_mut * Fix build on Alpine --------- Co-authored-by: Luca Palmieri <[email protected]> * refactor EmptyIterator (#6240) * refactor EmptyIterator * fix useless imports and doc * update empty_iterator.c to add newline * add gcc11:bullseye to platforms [MOD-9529] (#6198) * add gcc11:bullseye to platforms * add naming * remove deps * add nightly platforms option * set merge queue to false * fix flow * fix flow * chore: Improve granularity of build invalidation logic for FFI bindings (#6244) * Add `tests/deps/RedisJSON` to `.gitignore` (#6249) * Skip tests of unsupported unicode 9.0.0 chars * Fix mem leak 3 * Test stopwords which require memory reallocation * unicode_tolower update out_len always * Refactor unicode_tolower function to receive a single *inout_len argument * Update documentation for tag_strtolower() * Fixes after review * tokenizeTagString: Rename origLen as len * Test prefix/infix/suffix search with supported unicode chars * Update test description * Fix testToLowerConversionExactMatch * testTextToLowerConversionSimilarMatch: remove unused doc_id in queries * fix crash in test * Fix test failure on focal and bullseye: avoid nested f-strings with curly brackets * Fix test failure on mac: invalid escape sequence * Use SSO in runesToStr and strToLowerRunes * Add utf8_len arg to strToLowerRunes() * Align and move strToLowerRunes comment to header --------- Co-authored-by: kei-nan <[email protected]> Co-authored-by: Zeeshan Ali Khan <[email protected]> Co-authored-by: Luca Palmieri <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Co-authored-by: BenGoldberger <[email protected]> Co-authored-by: oshadmi <[email protected]> (cherry picked from commit dfe511e)
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jun 17, 2025
MOD-8799: Support special utf8 (#5637) * Normalize multi-byte char terms * Fix rm_strdupcase_utf8 * Remove debug log * Add Russian alphabet and diacritic tests * Don't use DefaultNormalize_utf8() for chinese * print available locales * Check locale before using utf8 normalization * Fix DefaultNormalize() * Skip multibyte tests if 'en_US.UTF-8' locale is not available * Test multibyte stopwords * Revert changes in install_script.sh * run sanitizer using ubuntu:latest * Test_cn: fix language_field * Remove unused strtolower function from misc.c and misc.h * Fixes from code review * Enhance multibyte character tests * Validate queries using multi-byte stopwords * Add MULTIBYTE_CHARS config param and install locales * Use setlocale instead of querylocale for better compatibility * revert changes in event-pull_request.yml and install French locale for debian/ubuntu * Convert to lowercase using nunicode library * nunicode_tolower() returns zero terminated string * Refactor nunicode_tolower() to improve readability * Refactor nunicode_tolower() to use destination buffer and improve memory management * Remove null termination in nunicode_tolower() * Improve documentation * Support multi-byte chars for tags * Support multi-byte chars for synonyms * unicode_tolower(): avoid duplicating encoded input * Increase SSO_MAX_LENGTH * Update test to use FT.DEBUG DUMP_TAGIDX instead of FT.TAGVALS * Keep previous test_cn:testSynonym and test_cn:testMixedHighlight * Update CMake configuration for consistent multi-byte char sorting * Test JSON index * Fix JSON test and remove unneeded tolower() * Fix query_EvalSingleTagNode() * Refactor string normalization to use case folding instead of lowercase conversion * Refactor tests to use run_command_on_all_shards for setting DEFAULT_DIALECT * BWC: Add function to convert string to single codepoint folded runes * Rename rm_strdupcase() to rm_normalize() * Add stemming test * Refactor string normalization to use lowercase transformation instead of folding. Modify FT.SUGGET flow to be BWC * Refactor filtering functions to use a transformation callback for rune processing. Remove dead code. * Update testDFAFilter to use strToSingleCodepointFoldedRunes for rune processing * Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation * Revert "Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation" This reverts commit ebcf994. * Add case sensitivity option to tag string processing functions * Rename tag_strtofold to tag_strtolower for clarity and update related references to reflect lowercase transformation * Simplify length checks in tag_strtolower and rm_normalize functions * Add documentation after review * WIP: Enhance unicode_tolower function to handle memory allocation for transformed strings * WIP: Fixing unicode_tolower usage * Fix tag_strtolower() to support memory realloc * skip testToLowerSize() on cluster * Fix rm_normalize and add more tests * Fix memory leak * Fix memory leak 2 * MOD-9609: Pass Private Data Context Back To UnblockClient API (#6237) * Apparently private data had to be sent back when unblocking the client in order for the free function to get called * code review comment * solve memory leak * MOD-9774: Stop Compiling Redis-With Coverage (#6231) * stop compiling redis-server with coverage * bring back coverage as requirement * Add unit test for unicode_tolower * Fix memory management in UnicodeToLowerTest and update expected results in multibyte character tests * Add tests for unsupported Unicode characters in tolower conversion. * MOD-9768: Add Rust abstractions for Buffer* (#6175) * Add Rust abstractions for Buffer* * Add tests * Make Buffer fields pub For consistency with BufferReader and BufferWriter. Also there is a very good chance of us needing to use them directly in the near future. * Expand the docs a bit more * Add and improve upon safety comments * fmt fix * Add forgotten license headers * Revert "Make Buffer fields pub" This reverts commit 1243590. * Address review comments * Address review comments * Better document safety of `Buffer` fields * Apply suggestions from code review * Remove an unneeded import * Use reallocation in mock `Buffer_Grow` * Satisfy MIRI about some pointer access * Drop unneeded dep * Remove unnecessary #includes * Split FFI into a separate crate * Add buffer.h API to ffi crate * Buffer API now based on generated bindings for the C Buffer API * CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env Just acting on the advice from a panic: ``` To improve backtraces for build dependencies, set the CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation. ``` * Drop linking of internal libs in ffi crate * Apply suggestions from code review * Revert "CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env" This reverts commit 9bd1d87. * Drop a redundant import * Add a few debug_asserts * Bring back an accidentally deleted assertion in a test * Fix a .gitignore file * Make a safety assumption explicit * Dont build `buffer` crate as static lib * Don't build bindings for oxidized triemap code * Make use of BufferWritter::buffer_mut * Fix build on Alpine --------- * refactor EmptyIterator (#6240) * refactor EmptyIterator * fix useless imports and doc * update empty_iterator.c to add newline * add gcc11:bullseye to platforms [MOD-9529] (#6198) * add gcc11:bullseye to platforms * add naming * remove deps * add nightly platforms option * set merge queue to false * fix flow * fix flow * chore: Improve granularity of build invalidation logic for FFI bindings (#6244) * Add `tests/deps/RedisJSON` to `.gitignore` (#6249) * Skip tests of unsupported unicode 9.0.0 chars * Fix mem leak 3 * Test stopwords which require memory reallocation * unicode_tolower update out_len always * Refactor unicode_tolower function to receive a single *inout_len argument * Update documentation for tag_strtolower() * Fixes after review * tokenizeTagString: Rename origLen as len * Test prefix/infix/suffix search with supported unicode chars * Update test description * Fix testToLowerConversionExactMatch * testTextToLowerConversionSimilarMatch: remove unused doc_id in queries * fix crash in test * Fix test failure on focal and bullseye: avoid nested f-strings with curly brackets * Fix test failure on mac: invalid escape sequence * Use SSO in runesToStr and strToLowerRunes * Add utf8_len arg to strToLowerRunes() * Align and move strToLowerRunes comment to header --------- (cherry picked from commit dfe511e) Co-authored-by: nafraf <[email protected]> Co-authored-by: kei-nan <[email protected]> Co-authored-by: Zeeshan Ali Khan <[email protected]> Co-authored-by: Luca Palmieri <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Co-authored-by: BenGoldberger <[email protected]> Co-authored-by: oshadmi <[email protected]>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pr adds ci tests for compiling with gcc > 10 on debian bullseye(11)