chore: Improve granularity of build invalidation logic for FFI bindings by LukeMathWalker · Pull Request #6244 · RediSearch/RediSearch

LukeMathWalker · 2025-06-03T09:44:00Z

Describe the changes in the pull request

Scope build invalidation to *.c and *.h files.
We also restructure the build script to simplify future additions.

Mark if applicable

This PR introduces API changes
This PR introduces serialization changes

JoanFM · 2025-06-03T09:48:05Z

src/redisearch_rs/ffi/build.rs

+        bindings = bindings.clang_arg(format!("-I{}", include.display()));
+        // Re-run the build script if any of the C files in the included
+        // directory changes
+        rerun_if_c_changes(&include);


why do we call here implicitly and for headers we use the println?

I see the comment now

rerun_if_c_changes expects a directory as input, while headers are plain files.

codecov · 2025-06-03T10:13:28Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.83%. Comparing base (74451de) to head (d4b3cfc).
Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6244      +/-   ##
==========================================
- Coverage   88.86%   88.83%   -0.03%     
==========================================
  Files         236      236              
  Lines       40204    40204              
  Branches     3167     3167              
==========================================
- Hits        35728    35716      -12     
- Misses       4441     4453      +12     
  Partials       35       35

Flag	Coverage Δ
flow	`82.80% <ø> (-0.15%)`	⬇️
unit	`45.75% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…gs (#6244)

* Normalize multi-byte char terms * Fix rm_strdupcase_utf8 * Remove debug log * Add Russian alphabet and diacritic tests * Don't use DefaultNormalize_utf8() for chinese * print available locales * Check locale before using utf8 normalization * Fix DefaultNormalize() * Skip multibyte tests if 'en_US.UTF-8' locale is not available * Test multibyte stopwords * Revert changes in install_script.sh * run sanitizer using ubuntu:latest * Test_cn: fix language_field * Remove unused strtolower function from misc.c and misc.h * Fixes from code review * Enhance multibyte character tests * Validate queries using multi-byte stopwords * Add MULTIBYTE_CHARS config param and install locales * Use setlocale instead of querylocale for better compatibility * revert changes in event-pull_request.yml and install French locale for debian/ubuntu * Convert to lowercase using nunicode library * nunicode_tolower() returns zero terminated string * Refactor nunicode_tolower() to improve readability * Refactor nunicode_tolower() to use destination buffer and improve memory management * Remove null termination in nunicode_tolower() * Improve documentation * Support multi-byte chars for tags * Support multi-byte chars for synonyms * unicode_tolower(): avoid duplicating encoded input * Increase SSO_MAX_LENGTH * Update test to use FT.DEBUG DUMP_TAGIDX instead of FT.TAGVALS * Keep previous test_cn:testSynonym and test_cn:testMixedHighlight * Update CMake configuration for consistent multi-byte char sorting * Test JSON index * Fix JSON test and remove unneeded tolower() * Fix query_EvalSingleTagNode() * Refactor string normalization to use case folding instead of lowercase conversion * Refactor tests to use run_command_on_all_shards for setting DEFAULT_DIALECT * BWC: Add function to convert string to single codepoint folded runes * Rename rm_strdupcase() to rm_normalize() * Add stemming test * Refactor string normalization to use lowercase transformation instead of folding. Modify FT.SUGGET flow to be BWC * Refactor filtering functions to use a transformation callback for rune processing. Remove dead code. * Update testDFAFilter to use strToSingleCodepointFoldedRunes for rune processing * Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation * Revert "Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation" This reverts commit ebcf994. * Add case sensitivity option to tag string processing functions * Rename tag_strtofold to tag_strtolower for clarity and update related references to reflect lowercase transformation * Simplify length checks in tag_strtolower and rm_normalize functions * Add documentation after review * WIP: Enhance unicode_tolower function to handle memory allocation for transformed strings * WIP: Fixing unicode_tolower usage * Fix tag_strtolower() to support memory realloc * skip testToLowerSize() on cluster * Fix rm_normalize and add more tests * Fix memory leak * Fix memory leak 2 * MOD-9609: Pass Private Data Context Back To UnblockClient API (#6237) * Apparently private data had to be sent back when unblocking the client in order for the free function to get called * code review comment * solve memory leak * MOD-9774: Stop Compiling Redis-With Coverage (#6231) * stop compiling redis-server with coverage * bring back coverage as requirement * Add unit test for unicode_tolower * Fix memory management in UnicodeToLowerTest and update expected results in multibyte character tests * Add tests for unsupported Unicode characters in tolower conversion. * MOD-9768: Add Rust abstractions for Buffer* (#6175) * Add Rust abstractions for Buffer* * Add tests * Make Buffer fields pub For consistency with BufferReader and BufferWriter. Also there is a very good chance of us needing to use them directly in the near future. * Expand the docs a bit more * Add and improve upon safety comments * fmt fix * Add forgotten license headers * Revert "Make Buffer fields pub" This reverts commit 1243590. * Address review comments * Address review comments * Better document safety of `Buffer` fields * Apply suggestions from code review Co-authored-by: Luca Palmieri <[email protected]> * Remove an unneeded import * Use reallocation in mock `Buffer_Grow` * Satisfy MIRI about some pointer access * Drop unneeded dep * Remove unnecessary #includes * Split FFI into a separate crate * Add buffer.h API to ffi crate * Buffer API now based on generated bindings for the C Buffer API * CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env Just acting on the advice from a panic: ``` To improve backtraces for build dependencies, set the CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation. ``` * Drop linking of internal libs in ffi crate * Apply suggestions from code review Co-authored-by: Luca Palmieri <[email protected]> * Revert "CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env" This reverts commit 9bd1d87. * Drop a redundant import * Add a few debug_asserts * Bring back an accidentally deleted assertion in a test * Fix a .gitignore file * Make a safety assumption explicit * Dont build `buffer` crate as static lib * Don't build bindings for oxidized triemap code * Make use of BufferWritter::buffer_mut * Fix build on Alpine --------- Co-authored-by: Luca Palmieri <[email protected]> * refactor EmptyIterator (#6240) * refactor EmptyIterator * fix useless imports and doc * update empty_iterator.c to add newline * add gcc11:bullseye to platforms [MOD-9529] (#6198) * add gcc11:bullseye to platforms * add naming * remove deps * add nightly platforms option * set merge queue to false * fix flow * fix flow * chore: Improve granularity of build invalidation logic for FFI bindings (#6244) * Add `tests/deps/RedisJSON` to `.gitignore` (#6249) * Skip tests of unsupported unicode 9.0.0 chars * Fix mem leak 3 * Test stopwords which require memory reallocation * unicode_tolower update out_len always * Refactor unicode_tolower function to receive a single *inout_len argument * Update documentation for tag_strtolower() * Fixes after review * tokenizeTagString: Rename origLen as len * Test prefix/infix/suffix search with supported unicode chars * Update test description * Fix testToLowerConversionExactMatch * testTextToLowerConversionSimilarMatch: remove unused doc_id in queries * fix crash in test * Fix test failure on focal and bullseye: avoid nested f-strings with curly brackets * Fix test failure on mac: invalid escape sequence * Use SSO in runesToStr and strToLowerRunes * Add utf8_len arg to strToLowerRunes() * Align and move strToLowerRunes comment to header --------- Co-authored-by: kei-nan <[email protected]> Co-authored-by: Zeeshan Ali Khan <[email protected]> Co-authored-by: Luca Palmieri <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Co-authored-by: BenGoldberger <[email protected]> Co-authored-by: oshadmi <[email protected]>

* Normalize multi-byte char terms * Fix rm_strdupcase_utf8 * Remove debug log * Add Russian alphabet and diacritic tests * Don't use DefaultNormalize_utf8() for chinese * print available locales * Check locale before using utf8 normalization * Fix DefaultNormalize() * Skip multibyte tests if 'en_US.UTF-8' locale is not available * Test multibyte stopwords * Revert changes in install_script.sh * run sanitizer using ubuntu:latest * Test_cn: fix language_field * Remove unused strtolower function from misc.c and misc.h * Fixes from code review * Enhance multibyte character tests * Validate queries using multi-byte stopwords * Add MULTIBYTE_CHARS config param and install locales * Use setlocale instead of querylocale for better compatibility * revert changes in event-pull_request.yml and install French locale for debian/ubuntu * Convert to lowercase using nunicode library * nunicode_tolower() returns zero terminated string * Refactor nunicode_tolower() to improve readability * Refactor nunicode_tolower() to use destination buffer and improve memory management * Remove null termination in nunicode_tolower() * Improve documentation * Support multi-byte chars for tags * Support multi-byte chars for synonyms * unicode_tolower(): avoid duplicating encoded input * Increase SSO_MAX_LENGTH * Update test to use FT.DEBUG DUMP_TAGIDX instead of FT.TAGVALS * Keep previous test_cn:testSynonym and test_cn:testMixedHighlight * Update CMake configuration for consistent multi-byte char sorting * Test JSON index * Fix JSON test and remove unneeded tolower() * Fix query_EvalSingleTagNode() * Refactor string normalization to use case folding instead of lowercase conversion * Refactor tests to use run_command_on_all_shards for setting DEFAULT_DIALECT * BWC: Add function to convert string to single codepoint folded runes * Rename rm_strdupcase() to rm_normalize() * Add stemming test * Refactor string normalization to use lowercase transformation instead of folding. Modify FT.SUGGET flow to be BWC * Refactor filtering functions to use a transformation callback for rune processing. Remove dead code. * Update testDFAFilter to use strToSingleCodepointFoldedRunes for rune processing * Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation * Revert "Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation" This reverts commit ebcf994. * Add case sensitivity option to tag string processing functions * Rename tag_strtofold to tag_strtolower for clarity and update related references to reflect lowercase transformation * Simplify length checks in tag_strtolower and rm_normalize functions * Add documentation after review * WIP: Enhance unicode_tolower function to handle memory allocation for transformed strings * WIP: Fixing unicode_tolower usage * Fix tag_strtolower() to support memory realloc * skip testToLowerSize() on cluster * Fix rm_normalize and add more tests * Fix memory leak * Fix memory leak 2 * MOD-9609: Pass Private Data Context Back To UnblockClient API (#6237) * Apparently private data had to be sent back when unblocking the client in order for the free function to get called * code review comment * solve memory leak * MOD-9774: Stop Compiling Redis-With Coverage (#6231) * stop compiling redis-server with coverage * bring back coverage as requirement * Add unit test for unicode_tolower * Fix memory management in UnicodeToLowerTest and update expected results in multibyte character tests * Add tests for unsupported Unicode characters in tolower conversion. * MOD-9768: Add Rust abstractions for Buffer* (#6175) * Add Rust abstractions for Buffer* * Add tests * Make Buffer fields pub For consistency with BufferReader and BufferWriter. Also there is a very good chance of us needing to use them directly in the near future. * Expand the docs a bit more * Add and improve upon safety comments * fmt fix * Add forgotten license headers * Revert "Make Buffer fields pub" This reverts commit 1243590. * Address review comments * Address review comments * Better document safety of `Buffer` fields * Apply suggestions from code review Co-authored-by: Luca Palmieri <[email protected]> * Remove an unneeded import * Use reallocation in mock `Buffer_Grow` * Satisfy MIRI about some pointer access * Drop unneeded dep * Remove unnecessary #includes * Split FFI into a separate crate * Add buffer.h API to ffi crate * Buffer API now based on generated bindings for the C Buffer API * CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env Just acting on the advice from a panic: ``` To improve backtraces for build dependencies, set the CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation. ``` * Drop linking of internal libs in ffi crate * Apply suggestions from code review Co-authored-by: Luca Palmieri <[email protected]> * Revert "CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env" This reverts commit 9bd1d87. * Drop a redundant import * Add a few debug_asserts * Bring back an accidentally deleted assertion in a test * Fix a .gitignore file * Make a safety assumption explicit * Dont build `buffer` crate as static lib * Don't build bindings for oxidized triemap code * Make use of BufferWritter::buffer_mut * Fix build on Alpine --------- Co-authored-by: Luca Palmieri <[email protected]> * refactor EmptyIterator (#6240) * refactor EmptyIterator * fix useless imports and doc * update empty_iterator.c to add newline * add gcc11:bullseye to platforms [MOD-9529] (#6198) * add gcc11:bullseye to platforms * add naming * remove deps * add nightly platforms option * set merge queue to false * fix flow * fix flow * chore: Improve granularity of build invalidation logic for FFI bindings (#6244) * Add `tests/deps/RedisJSON` to `.gitignore` (#6249) * Skip tests of unsupported unicode 9.0.0 chars * Fix mem leak 3 * Test stopwords which require memory reallocation * unicode_tolower update out_len always * Refactor unicode_tolower function to receive a single *inout_len argument * Update documentation for tag_strtolower() * Fixes after review * tokenizeTagString: Rename origLen as len * Test prefix/infix/suffix search with supported unicode chars * Update test description * Fix testToLowerConversionExactMatch * testTextToLowerConversionSimilarMatch: remove unused doc_id in queries * fix crash in test * Fix test failure on focal and bullseye: avoid nested f-strings with curly brackets * Fix test failure on mac: invalid escape sequence * Use SSO in runesToStr and strToLowerRunes * Add utf8_len arg to strToLowerRunes() * Align and move strToLowerRunes comment to header --------- Co-authored-by: kei-nan <[email protected]> Co-authored-by: Zeeshan Ali Khan <[email protected]> Co-authored-by: Luca Palmieri <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Co-authored-by: BenGoldberger <[email protected]> Co-authored-by: oshadmi <[email protected]> (cherry picked from commit dfe511e)

MOD-8799: Support special utf8 (#5637) * Normalize multi-byte char terms * Fix rm_strdupcase_utf8 * Remove debug log * Add Russian alphabet and diacritic tests * Don't use DefaultNormalize_utf8() for chinese * print available locales * Check locale before using utf8 normalization * Fix DefaultNormalize() * Skip multibyte tests if 'en_US.UTF-8' locale is not available * Test multibyte stopwords * Revert changes in install_script.sh * run sanitizer using ubuntu:latest * Test_cn: fix language_field * Remove unused strtolower function from misc.c and misc.h * Fixes from code review * Enhance multibyte character tests * Validate queries using multi-byte stopwords * Add MULTIBYTE_CHARS config param and install locales * Use setlocale instead of querylocale for better compatibility * revert changes in event-pull_request.yml and install French locale for debian/ubuntu * Convert to lowercase using nunicode library * nunicode_tolower() returns zero terminated string * Refactor nunicode_tolower() to improve readability * Refactor nunicode_tolower() to use destination buffer and improve memory management * Remove null termination in nunicode_tolower() * Improve documentation * Support multi-byte chars for tags * Support multi-byte chars for synonyms * unicode_tolower(): avoid duplicating encoded input * Increase SSO_MAX_LENGTH * Update test to use FT.DEBUG DUMP_TAGIDX instead of FT.TAGVALS * Keep previous test_cn:testSynonym and test_cn:testMixedHighlight * Update CMake configuration for consistent multi-byte char sorting * Test JSON index * Fix JSON test and remove unneeded tolower() * Fix query_EvalSingleTagNode() * Refactor string normalization to use case folding instead of lowercase conversion * Refactor tests to use run_command_on_all_shards for setting DEFAULT_DIALECT * BWC: Add function to convert string to single codepoint folded runes * Rename rm_strdupcase() to rm_normalize() * Add stemming test * Refactor string normalization to use lowercase transformation instead of folding. Modify FT.SUGGET flow to be BWC * Refactor filtering functions to use a transformation callback for rune processing. Remove dead code. * Update testDFAFilter to use strToSingleCodepointFoldedRunes for rune processing * Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation * Revert "Refactor string normalization to use case folding (single codepoint) instead of lowercase transformation" This reverts commit ebcf994. * Add case sensitivity option to tag string processing functions * Rename tag_strtofold to tag_strtolower for clarity and update related references to reflect lowercase transformation * Simplify length checks in tag_strtolower and rm_normalize functions * Add documentation after review * WIP: Enhance unicode_tolower function to handle memory allocation for transformed strings * WIP: Fixing unicode_tolower usage * Fix tag_strtolower() to support memory realloc * skip testToLowerSize() on cluster * Fix rm_normalize and add more tests * Fix memory leak * Fix memory leak 2 * MOD-9609: Pass Private Data Context Back To UnblockClient API (#6237) * Apparently private data had to be sent back when unblocking the client in order for the free function to get called * code review comment * solve memory leak * MOD-9774: Stop Compiling Redis-With Coverage (#6231) * stop compiling redis-server with coverage * bring back coverage as requirement * Add unit test for unicode_tolower * Fix memory management in UnicodeToLowerTest and update expected results in multibyte character tests * Add tests for unsupported Unicode characters in tolower conversion. * MOD-9768: Add Rust abstractions for Buffer* (#6175) * Add Rust abstractions for Buffer* * Add tests * Make Buffer fields pub For consistency with BufferReader and BufferWriter. Also there is a very good chance of us needing to use them directly in the near future. * Expand the docs a bit more * Add and improve upon safety comments * fmt fix * Add forgotten license headers * Revert "Make Buffer fields pub" This reverts commit 1243590. * Address review comments * Address review comments * Better document safety of `Buffer` fields * Apply suggestions from code review * Remove an unneeded import * Use reallocation in mock `Buffer_Grow` * Satisfy MIRI about some pointer access * Drop unneeded dep * Remove unnecessary #includes * Split FFI into a separate crate * Add buffer.h API to ffi crate * Buffer API now based on generated bindings for the C Buffer API * CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env Just acting on the advice from a panic: ``` To improve backtraces for build dependencies, set the CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation. ``` * Drop linking of internal libs in ffi crate * Apply suggestions from code review * Revert "CI: Set `CARGO_PROFILE_OPTIMISED_TEST_BUILD_OVERRIDE_DEBUG` env" This reverts commit 9bd1d87. * Drop a redundant import * Add a few debug_asserts * Bring back an accidentally deleted assertion in a test * Fix a .gitignore file * Make a safety assumption explicit * Dont build `buffer` crate as static lib * Don't build bindings for oxidized triemap code * Make use of BufferWritter::buffer_mut * Fix build on Alpine --------- * refactor EmptyIterator (#6240) * refactor EmptyIterator * fix useless imports and doc * update empty_iterator.c to add newline * add gcc11:bullseye to platforms [MOD-9529] (#6198) * add gcc11:bullseye to platforms * add naming * remove deps * add nightly platforms option * set merge queue to false * fix flow * fix flow * chore: Improve granularity of build invalidation logic for FFI bindings (#6244) * Add `tests/deps/RedisJSON` to `.gitignore` (#6249) * Skip tests of unsupported unicode 9.0.0 chars * Fix mem leak 3 * Test stopwords which require memory reallocation * unicode_tolower update out_len always * Refactor unicode_tolower function to receive a single *inout_len argument * Update documentation for tag_strtolower() * Fixes after review * tokenizeTagString: Rename origLen as len * Test prefix/infix/suffix search with supported unicode chars * Update test description * Fix testToLowerConversionExactMatch * testTextToLowerConversionSimilarMatch: remove unused doc_id in queries * fix crash in test * Fix test failure on focal and bullseye: avoid nested f-strings with curly brackets * Fix test failure on mac: invalid escape sequence * Use SSO in runesToStr and strToLowerRunes * Add utf8_len arg to strToLowerRunes() * Align and move strToLowerRunes comment to header --------- (cherry picked from commit dfe511e) Co-authored-by: nafraf <[email protected]> Co-authored-by: kei-nan <[email protected]> Co-authored-by: Zeeshan Ali Khan <[email protected]> Co-authored-by: Luca Palmieri <[email protected]> Co-authored-by: Joan Fontanals <[email protected]> Co-authored-by: BenGoldberger <[email protected]> Co-authored-by: oshadmi <[email protected]>

chore: Improve granularity of build invalidation logic for FFI bindings

d4b3cfc

LukeMathWalker requested a review from JoanFM June 3, 2025 09:44

github-actions bot added the size:S label Jun 3, 2025

LukeMathWalker mentioned this pull request Jun 3, 2025

[MOD-9806] Index encoder & decoder interfaces #6227

Merged

2 tasks

JoanFM reviewed Jun 3, 2025

View reviewed changes

JoanFM approved these changes Jun 3, 2025

View reviewed changes

LukeMathWalker enabled auto-merge June 3, 2025 09:51

LukeMathWalker added this pull request to the merge queue Jun 3, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 3, 2025

LukeMathWalker added this pull request to the merge queue Jun 3, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 3, 2025

LukeMathWalker added this pull request to the merge queue Jun 3, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jun 3, 2025

JoanFM added this pull request to the merge queue Jun 3, 2025

Merged via the queue into master with commit 0157f75 Jun 3, 2025
14 checks passed

JoanFM deleted the improve-ffi branch June 3, 2025 17:58

nafraf pushed a commit that referenced this pull request Jun 4, 2025

chore: Improve granularity of build invalidation logic for FFI bindin…

aad29c0

…gs (#6244)

LukeMathWalker added a commit that referenced this pull request Jun 4, 2025

chore: Improve granularity of build invalidation logic for FFI bindin…

5f2ec0e

…gs (#6244)

LukeMathWalker restored the improve-ffi branch February 6, 2026 09:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Improve granularity of build invalidation logic for FFI bindings#6244

chore: Improve granularity of build invalidation logic for FFI bindings#6244
JoanFM merged 1 commit intomasterfrom
improve-ffi

LukeMathWalker commented Jun 3, 2025

Uh oh!

JoanFM Jun 3, 2025

Uh oh!

JoanFM Jun 3, 2025

Uh oh!

LukeMathWalker Jun 3, 2025

Uh oh!

codecov bot commented Jun 3, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

LukeMathWalker commented Jun 3, 2025

Describe the changes in the pull request

Mark if applicable

Uh oh!

JoanFM Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

JoanFM Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

LukeMathWalker Jun 3, 2025

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Jun 3, 2025 •

edited

Loading