MOD-8097, MOD-8114 Fix memory counting#5204
Conversation
replaces all OpenNumericIndex calls.
rename createIndex to createPopulateTermsInvIndex add function to tests/index_utils generalize FGCTest suit add inheriting suite for tags some cosmetics in cpp_range ExtTest.testDynamicLoading: log when assert fails (the test halts upon assertion and the log will be nerver pronted) in IndexTest and QueryTest use IndexSpec_RemoveFromGlobals instead of StrongRef_Release to ensure the spec is properly cleaned otherwise some structs like the global prefix trie nodes, potentially accessed by following tests, might hold a copy of the spec refrence that was already released and cuase a crash.
update in gc and in addition add cpp_range test and cpp_forkgc test
decrease gc->stats.totalCollected by bytesAdded to not count bytes added by us remove reseting currNode->range->invertedIndexSize in applyNumIdx called upon each node recieved from the child process instead, we will count the memory released when trimming the empty leaves call IndexSpec_TotalMemUsage from RediSearch_MemUsage instead of code duplication add IndexSpec_collect_numeric_overhead that counts the memory of NumericRangeTree in a given spec call this function to add it to IndexSpec_TotalMemUsage update LLApiTest testInfo* accordingly
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5204 +/- ##
==========================================
- Coverage 86.59% 86.59% -0.01%
==========================================
Files 193 193
Lines 34769 34767 -2
==========================================
- Hits 30109 30107 -2
Misses 4660 4660 ☔ View full report in Codecov by Sentry. 🚨 Try these New Features:
|
… if it was an empty tree
it will be counted when the range is deleted
don't intialize the numeric inverted index in the child if it's empty
fix IndexSpec_collect_numeric_overhead and add test fix sending NULL to FGC_sendFixed
tests/pytests/common.py
Outdated
| env.assertEqual(db_res, res, message=message) | ||
|
|
||
| # fields should be in capital letters | ||
| def getInvertedIndexInitialSize_MB(fields: list[str]) -> float: |
There was a problem hiding this comment.
No need to fix it here as well?
|
Backport failed for Please cherry-pick the changes locally and resolve any conflicts. git fetch origin 2.8
git worktree add -d .worktree/backport-5204-to-2.8 origin/2.8
cd .worktree/backport-5204-to-2.8
git switch --create backport-5204-to-2.8
git cherry-pick -x 357d356db01fac23d522344bfb0855cb8a38f4d2 |
|
Backport failed for Please cherry-pick the changes locally and resolve any conflicts. git fetch origin 2.6
git worktree add -d .worktree/backport-5204-to-2.6 origin/2.6
cd .worktree/backport-5204-to-2.6
git switch --create backport-5204-to-2.6
git cherry-pick -x 357d356db01fac23d522344bfb0855cb8a38f4d2 |
|
Backport failed for Please cherry-pick the changes locally and resolve any conflicts. git fetch origin 2.10
git worktree add -d .worktree/backport-5204-to-2.10 origin/2.10
cd .worktree/backport-5204-to-2.10
git switch --create backport-5204-to-2.10
git cherry-pick -x 357d356db01fac23d522344bfb0855cb8a38f4d2 |
|
Backport failed for Please cherry-pick the changes locally and resolve any conflicts. git fetch origin 8.0
git worktree add -d .worktree/backport-5204-to-8.0 origin/8.0
cd .worktree/backport-5204-to-8.0
git switch --create backport-5204-to-8.0
git cherry-pick -x 357d356db01fac23d522344bfb0855cb8a38f4d2 |
* expose openNumericKeysDict, requires a write/read argument. replaces all OpenNumericIndex calls. * fork gc -initialiize index pointer only once rename createIndex to createPopulateTermsInvIndex add function to tests/index_utils generalize FGCTest suit add inheriting suite for tags some cosmetics in cpp_range ExtTest.testDynamicLoading: log when assert fails (the test halts upon assertion and the log will be nerver pronted) in IndexTest and QueryTest use IndexSpec_RemoveFromGlobals instead of StrongRef_Release to ensure the spec is properly cleaned otherwise some structs like the global prefix trie nodes, potentially accessed by following tests, might hold a copy of the spec refrence that was already released and cuase a crash. * add inverted index total size to the numeric tree root update in gc and in addition add cpp_range test and cpp_forkgc test * FIX: decrease gc->stats.totalCollected by bytesAdded to not count bytes added by us remove reseting currNode->range->invertedIndexSize in applyNumIdx called upon each node recieved from the child process instead, we will count the memory released when trimming the empty leaves call IndexSpec_TotalMemUsage from RediSearch_MemUsage instead of code duplication add IndexSpec_collect_numeric_overhead that counts the memory of NumericRangeTree in a given spec call this function to add it to IndexSpec_TotalMemUsage update LLApiTest testInfo* accordingly * fixes * fix new cpp forkgc test * fix unsafe read * fix status check * numeric tree debug commands - if tree was not intizlied yet, reply as if it was an empty tree * fix reset range->invertedIndexSize in applyNumIdx when index is empty. it will be counted when the range is deleted * add initial inverted index size if numeric to test don't intialize the numeric inverted index in the child if it's empty * fix gc warning * little fix * changes rt->invertedIndexSize to rt->invertedIndexesSize fix IndexSpec_collect_numeric_overhead and add test fix sending NULL to FGC_sendFixed * CalculateNumericInvertedIndexMemory fix identation and break when failing * rename openNumericKeysDict variable write to initialize and change type to bool * change OPEN_INDEX_READ to DONT_CREATE_INDEX and OPEN_INDEX_WRITE to CREATE_INDEX add coments to tests * use Likst instead of list to support older python versopns * remove type from function args * fix --------- Co-authored-by: GuyAv46 <[email protected]> (cherry picked from commit 357d356)
* expose openNumericKeysDict, requires a write/read argument. replaces all OpenNumericIndex calls. * fork gc -initialiize index pointer only once rename createIndex to createPopulateTermsInvIndex add function to tests/index_utils generalize FGCTest suit add inheriting suite for tags some cosmetics in cpp_range ExtTest.testDynamicLoading: log when assert fails (the test halts upon assertion and the log will be nerver pronted) in IndexTest and QueryTest use IndexSpec_RemoveFromGlobals instead of StrongRef_Release to ensure the spec is properly cleaned otherwise some structs like the global prefix trie nodes, potentially accessed by following tests, might hold a copy of the spec refrence that was already released and cuase a crash. * add inverted index total size to the numeric tree root update in gc and in addition add cpp_range test and cpp_forkgc test * FIX: decrease gc->stats.totalCollected by bytesAdded to not count bytes added by us remove reseting currNode->range->invertedIndexSize in applyNumIdx called upon each node recieved from the child process instead, we will count the memory released when trimming the empty leaves call IndexSpec_TotalMemUsage from RediSearch_MemUsage instead of code duplication add IndexSpec_collect_numeric_overhead that counts the memory of NumericRangeTree in a given spec call this function to add it to IndexSpec_TotalMemUsage update LLApiTest testInfo* accordingly * fixes * fix new cpp forkgc test * fix unsafe read * fix status check * numeric tree debug commands - if tree was not intizlied yet, reply as if it was an empty tree * fix reset range->invertedIndexSize in applyNumIdx when index is empty. it will be counted when the range is deleted * add initial inverted index size if numeric to test don't intialize the numeric inverted index in the child if it's empty * fix gc warning * little fix * changes rt->invertedIndexSize to rt->invertedIndexesSize fix IndexSpec_collect_numeric_overhead and add test fix sending NULL to FGC_sendFixed * CalculateNumericInvertedIndexMemory fix identation and break when failing * rename openNumericKeysDict variable write to initialize and change type to bool * change OPEN_INDEX_READ to DONT_CREATE_INDEX and OPEN_INDEX_WRITE to CREATE_INDEX add coments to tests * use Likst instead of list to support older python versopns * remove type from function args * fix --------- Co-authored-by: GuyAv46 <[email protected]> (cherry picked from commit 357d356)
* expose openNumericKeysDict, requires a write/read argument. replaces all OpenNumericIndex calls. * fork gc -initialiize index pointer only once rename createIndex to createPopulateTermsInvIndex add function to tests/index_utils generalize FGCTest suit add inheriting suite for tags some cosmetics in cpp_range ExtTest.testDynamicLoading: log when assert fails (the test halts upon assertion and the log will be nerver pronted) in IndexTest and QueryTest use IndexSpec_RemoveFromGlobals instead of StrongRef_Release to ensure the spec is properly cleaned otherwise some structs like the global prefix trie nodes, potentially accessed by following tests, might hold a copy of the spec refrence that was already released and cuase a crash. * add inverted index total size to the numeric tree root update in gc and in addition add cpp_range test and cpp_forkgc test * FIX: decrease gc->stats.totalCollected by bytesAdded to not count bytes added by us remove reseting currNode->range->invertedIndexSize in applyNumIdx called upon each node recieved from the child process instead, we will count the memory released when trimming the empty leaves call IndexSpec_TotalMemUsage from RediSearch_MemUsage instead of code duplication add IndexSpec_collect_numeric_overhead that counts the memory of NumericRangeTree in a given spec call this function to add it to IndexSpec_TotalMemUsage update LLApiTest testInfo* accordingly * fixes * fix new cpp forkgc test * fix unsafe read * fix status check * numeric tree debug commands - if tree was not intizlied yet, reply as if it was an empty tree * fix reset range->invertedIndexSize in applyNumIdx when index is empty. it will be counted when the range is deleted * add initial inverted index size if numeric to test don't intialize the numeric inverted index in the child if it's empty * fix gc warning * little fix * changes rt->invertedIndexSize to rt->invertedIndexesSize fix IndexSpec_collect_numeric_overhead and add test fix sending NULL to FGC_sendFixed * CalculateNumericInvertedIndexMemory fix identation and break when failing * rename openNumericKeysDict variable write to initialize and change type to bool * change OPEN_INDEX_READ to DONT_CREATE_INDEX and OPEN_INDEX_WRITE to CREATE_INDEX add coments to tests * use Likst instead of list to support older python versopns * remove type from function args * fix --------- Co-authored-by: GuyAv46 <[email protected]> (cherry picked from commit 357d356)
* MOD-8097, MOD-8114 Fix memory counting (#5204) * expose openNumericKeysDict, requires a write/read argument. replaces all OpenNumericIndex calls. * fork gc -initialiize index pointer only once rename createIndex to createPopulateTermsInvIndex add function to tests/index_utils generalize FGCTest suit add inheriting suite for tags some cosmetics in cpp_range ExtTest.testDynamicLoading: log when assert fails (the test halts upon assertion and the log will be nerver pronted) in IndexTest and QueryTest use IndexSpec_RemoveFromGlobals instead of StrongRef_Release to ensure the spec is properly cleaned otherwise some structs like the global prefix trie nodes, potentially accessed by following tests, might hold a copy of the spec refrence that was already released and cuase a crash. * add inverted index total size to the numeric tree root update in gc and in addition add cpp_range test and cpp_forkgc test * FIX: decrease gc->stats.totalCollected by bytesAdded to not count bytes added by us remove reseting currNode->range->invertedIndexSize in applyNumIdx called upon each node recieved from the child process instead, we will count the memory released when trimming the empty leaves call IndexSpec_TotalMemUsage from RediSearch_MemUsage instead of code duplication add IndexSpec_collect_numeric_overhead that counts the memory of NumericRangeTree in a given spec call this function to add it to IndexSpec_TotalMemUsage update LLApiTest testInfo* accordingly * fixes * fix new cpp forkgc test * fix unsafe read * fix status check * numeric tree debug commands - if tree was not intizlied yet, reply as if it was an empty tree * fix reset range->invertedIndexSize in applyNumIdx when index is empty. it will be counted when the range is deleted * add initial inverted index size if numeric to test don't intialize the numeric inverted index in the child if it's empty * fix gc warning * little fix * changes rt->invertedIndexSize to rt->invertedIndexesSize fix IndexSpec_collect_numeric_overhead and add test fix sending NULL to FGC_sendFixed * CalculateNumericInvertedIndexMemory fix identation and break when failing * rename openNumericKeysDict variable write to initialize and change type to bool * change OPEN_INDEX_READ to DONT_CREATE_INDEX and OPEN_INDEX_WRITE to CREATE_INDEX add coments to tests * use Likst instead of list to support older python versopns * remove type from function args * fix --------- Co-authored-by: GuyAv46 <[email protected]> (cherry picked from commit 357d356) * !! align FGC_parentHandleMissingDocs with master !! update gc stats after removing the invecrted index
MOD-8097, MOD-8114 Fix memory counting (#5204) * expose openNumericKeysDict, requires a write/read argument. replaces all OpenNumericIndex calls. * fork gc -initialiize index pointer only once rename createIndex to createPopulateTermsInvIndex add function to tests/index_utils generalize FGCTest suit add inheriting suite for tags some cosmetics in cpp_range ExtTest.testDynamicLoading: log when assert fails (the test halts upon assertion and the log will be nerver pronted) in IndexTest and QueryTest use IndexSpec_RemoveFromGlobals instead of StrongRef_Release to ensure the spec is properly cleaned otherwise some structs like the global prefix trie nodes, potentially accessed by following tests, might hold a copy of the spec refrence that was already released and cuase a crash. * add inverted index total size to the numeric tree root update in gc and in addition add cpp_range test and cpp_forkgc test * FIX: decrease gc->stats.totalCollected by bytesAdded to not count bytes added by us remove reseting currNode->range->invertedIndexSize in applyNumIdx called upon each node recieved from the child process instead, we will count the memory released when trimming the empty leaves call IndexSpec_TotalMemUsage from RediSearch_MemUsage instead of code duplication add IndexSpec_collect_numeric_overhead that counts the memory of NumericRangeTree in a given spec call this function to add it to IndexSpec_TotalMemUsage update LLApiTest testInfo* accordingly * fixes * fix new cpp forkgc test * fix unsafe read * fix status check * numeric tree debug commands - if tree was not intizlied yet, reply as if it was an empty tree * fix reset range->invertedIndexSize in applyNumIdx when index is empty. it will be counted when the range is deleted * add initial inverted index size if numeric to test don't intialize the numeric inverted index in the child if it's empty * fix gc warning * little fix * changes rt->invertedIndexSize to rt->invertedIndexesSize fix IndexSpec_collect_numeric_overhead and add test fix sending NULL to FGC_sendFixed * CalculateNumericInvertedIndexMemory fix identation and break when failing * rename openNumericKeysDict variable write to initialize and change type to bool * change OPEN_INDEX_READ to DONT_CREATE_INDEX and OPEN_INDEX_WRITE to CREATE_INDEX add coments to tests * use Likst instead of list to support older python versopns * remove type from function args * fix --------- Co-authored-by: GuyAv46 <[email protected]> (cherry picked from commit 357d356)
MOD-8097, MOD-8114 Fix memory counting (#5204) * expose openNumericKeysDict, requires a write/read argument. replaces all OpenNumericIndex calls. * fork gc -initialiize index pointer only once rename createIndex to createPopulateTermsInvIndex add function to tests/index_utils generalize FGCTest suit add inheriting suite for tags some cosmetics in cpp_range ExtTest.testDynamicLoading: log when assert fails (the test halts upon assertion and the log will be nerver pronted) in IndexTest and QueryTest use IndexSpec_RemoveFromGlobals instead of StrongRef_Release to ensure the spec is properly cleaned otherwise some structs like the global prefix trie nodes, potentially accessed by following tests, might hold a copy of the spec refrence that was already released and cuase a crash. * add inverted index total size to the numeric tree root update in gc and in addition add cpp_range test and cpp_forkgc test * FIX: decrease gc->stats.totalCollected by bytesAdded to not count bytes added by us remove reseting currNode->range->invertedIndexSize in applyNumIdx called upon each node recieved from the child process instead, we will count the memory released when trimming the empty leaves call IndexSpec_TotalMemUsage from RediSearch_MemUsage instead of code duplication add IndexSpec_collect_numeric_overhead that counts the memory of NumericRangeTree in a given spec call this function to add it to IndexSpec_TotalMemUsage update LLApiTest testInfo* accordingly * fixes * fix new cpp forkgc test * fix unsafe read * fix status check * numeric tree debug commands - if tree was not intizlied yet, reply as if it was an empty tree * fix reset range->invertedIndexSize in applyNumIdx when index is empty. it will be counted when the range is deleted * add initial inverted index size if numeric to test don't intialize the numeric inverted index in the child if it's empty * fix gc warning * little fix * changes rt->invertedIndexSize to rt->invertedIndexesSize fix IndexSpec_collect_numeric_overhead and add test fix sending NULL to FGC_sendFixed * CalculateNumericInvertedIndexMemory fix identation and break when failing * rename openNumericKeysDict variable write to initialize and change type to bool * change OPEN_INDEX_READ to DONT_CREATE_INDEX and OPEN_INDEX_WRITE to CREATE_INDEX add coments to tests * use Likst instead of list to support older python versopns * remove type from function args * fix --------- Co-authored-by: GuyAv46 <[email protected]> (cherry picked from commit 357d356)
|
Won't backport to 2.6 as #4218, introducing improvements of inverted indexes memory tracking, essential for this pr, wasn't merged into 2.6 In addition, to avoid potential breaking changes, opening the numeric index for read won't be merged into 2.6 as well, since the mechanism of opening the numeric index is a bit different. |
This reverts commit a97e925. Revert "[2.10] MOD-8601: Fix error message for LOAD (#5531) (#5550)" This reverts commit 5748233. Revert "[2.10] MOD-8561: Fix Inverted Index SeekTo Edge Case (#5536)" This reverts commit f9923b7. Revert "[2.10] Fix legacy geofilter leak - [MOD-8568] (#5516)" This reverts commit 8b75285. Revert "[2.10] MOD-7872: Add Types To Hide User Input (#5485)" This reverts commit d6acfed. Revert "[2.10] MOD-7864: Obfuscation API (#5487)" This reverts commit 259d417. Revert "MOD-7570 Backport 5422 to 2.10 (#5472)" This reverts commit e699dbc. Revert "[2.10] [MOD-8462] rename total_active_writes -> total_active_write_threads (#5446)" This reverts commit a90cfc3. Revert "[2.10] MOD-8058: Avoid error with _NUM_SSTRING (#5424)" This reverts commit 885cffb. Revert "[2.10] MOD-7634: Fix NOSTEM (#5392)" This reverts commit 4df508d. Revert "[2.10] Fixes for inverted indexes encoding - [MOD-8248] (#5388)" This reverts commit bc003c1. Revert "[2.10] Improve "Raw doc id" encoding - [MOD-8255] (#5372)" This reverts commit 9789e30. Revert "[2.10] Improve SkipToBlock logic - [MOD-8255] (#5370)" This reverts commit 56b16ea. Revert "[2.10] Improve counting active writes - [MOD-8151] (#5362)" This reverts commit a3d447d. Revert "[2.10] [MOD-8039] Reorganize stats files (#5356)" This reverts commit a1b8b29. Revert "[2.10] Improve text indexing performance - [MOD-8249] (#5352)" This reverts commit c5c9245. Revert "[2.10] [MOD-8115] Free spec resources in the main thread (#5327)" This reverts commit 42bb520. Revert "[2.10] Sorting Vector - [MOD-6783] (#5314)" This reverts commit 166a00e. Revert "[2.10] Fix long PREFIX/SUFFIX/INFIX/WILDCARD queries - [MOD-7882] (#5300)" This reverts commit 5c4ed74. Revert "[2.10] Remove assertion in optimizer (#5292)" This reverts commit ba55efb. Revert "[2.10] Fix missing expansions in text queries - [MOD-8142] (#5284)" This reverts commit 10f32d3. Revert "[2.10] Fix Tag OnReopen Callback - [MOD-8011] (#5282)" This reverts commit 2c029ee. Revert "[2.10] Fix MRIterator ownership mechanism - [MOD-8108] (#5266)" This reverts commit 457536e. Revert "[2.10] [MOD-8125] Create inverted index in write operations only (#5257)" This reverts commit 555557c. Revert "[2.10] [MOD-8152] rename and move new metrics (#5255)" This reverts commit a1f460e. Revert "[2.10] MOD-8129: Fix indexed_percent info stat (#5247)" This reverts commit d55afb5. Revert "[2.10] MOD-8097, MOD-8114 Fix memory counting (#5204) (#5241)" This reverts commit 52e58af. Revert "[MOD-7949] Avoid lazy expire upon scan keys in background [2.10] (#5294)" This reverts commit 6dfb503. Revert "[2.10] Cleanup for the trimming tree logic (#5226)" This reverts commit 7dfe6a2. Revert "[2.10] Add global marked deleted to stats [MOD-6053] (#5208)" This reverts commit f396732. Revert "Fix and add skips in pytest (#5210)" This reverts commit a8b505c. Revert "[2.10] [MOD-8035] Add field index error for JSON field indexing failures (#5202)" This reverts commit ed1919e. Revert "[2.10] Improve usage of RedisModuleCtx (#5181)" This reverts commit 8e686c7. Revert "[2.10] Add Active Indices Metrics - [MOD-7952] (#5179)" This reverts commit b72fc38. Revert "[2.10] MOD-8009: Allow Users To Configure Cursor Index Limitation Through Global Config (#5166)" This reverts commit 92e1dd8. Revert "[2.10] Send `SCORER` to the shards with `FT.AGGREGATE` on cluster mode - [MOD-8060] (#5163)" This reverts commit 4982ea2. Revert "[2.10] [MOD-8034] expose indexing failures to INFO modules (#5158)" This reverts commit d03f8de. Revert "[2.10] Add a counter of total query execution time - [MOD-7961] (#5153)" This reverts commit d45365d. Revert "[2.10] Add metrics of min and max memory consumption of indexes - [MOD-7584] (#5146)" This reverts commit 094318f. Revert "[2.10] Remove an unnecessary assertion - [MOD-8008] (#5142)" This reverts commit 3156f08. Revert "[2.10] Skip test_multithread:test_async_updates_sanity with sanitizer (#5125)" This reverts commit 5339033. Revert "Count Queries - [MOD-6492] (#5072)" This reverts commit 1006065. Revert "[2.10] Improve Error Message on VecSim Syntax Error - [MOD-7887] (#5109)" This reverts commit 9621fdc. Revert "[2.10] Fix flaky test (#5086)" This reverts commit 5aff118. Revert "[2.10] Fix flaky test_async_updates_sanity - avoid timeout in force invoke GC [MOD-7867] (#5081)" This reverts commit 2b85289. Revert "[2.10] MOD-7745: Add Benchmarks For Document And Field Expiration (#5063)" This reverts commit f99b0cf. Revert "[2.10] MOD-7863: Add HPEXPIRETIME Mock Support (#5028)" This reverts commit 79dfd0f.
This PR fixes bugs in statistics updates during gc.
wrong update of
range->invertedIndexSizewhen returns empty from the fork processIn the previous implementation, when the inverted index of a numeric range returned with no docs from the fork process,
range->invertedIndexSizestat was set to 0.This is wrong since an empty range consumes memory of inverted index metadata struct size + block struct size + initial block capacity size.
In this PR
range->invertedIndexSizeis decreased bynbytesCollectedand increased bynbytesAdded.In addition, once the gc parent finishes iterating all the nodes received from the fork, it starts cleaning empty nodes.
NumericRangeTree_TrimEmptyLeavesreturns the amount of memory released due to empty nodes removal.This is where we update the stats of a completely removed range (inverted index).
fix overflow in spec->stats.invertedSize during
FGC_updateStatsbefore the fix:
Since all the variables are of type size_t, and
bytesAddedmight be smaller thanbytesCollected, sctx->spec->stats.invertedSize might get overflowed.the fix is to split this into two lines.
Fix bytes collected by the gc statistics
Currently, the number of bytes collected by the gc inclused
bytesAdded, but it doesn’t includebytesAddedduring the cleaning process.This might result in counting bytes added by the parent in
bytesCollected, when releasing an entire inverted index.After the fix, we will include also
bytesAddedin the gc statsAdditional changes in this PR
invertedIndexSizeto the numeric tree structIndexSpec_TotalMemUsageAdditional bug fixes