Skip to content

fix hincrby* update volatile key tracking#2974

Merged
ranshid merged 1 commit intovalkey-io:unstablefrom
ranshid:fix-hincrby-update-volatile-tracking
Dec 29, 2025
Merged

fix hincrby* update volatile key tracking#2974
ranshid merged 1 commit intovalkey-io:unstablefrom
ranshid:fix-hincrby-update-volatile-tracking

Conversation

@ranshid
Copy link
Member

@ranshid ranshid commented Dec 24, 2025

Following Hash-Field-Expiration feature, a hash object can hold volatile fields.
volatile fields which are already expired are deleted and reclaimed ONLY by the active-expiration background job.
This means that hash object can contain items which have not yet expired.
In case mutations are requesting to set a value on these "already-expired" fields, they will be overwritten with the new value.
In such cases, though, it is requiered to update the global per-db tracking map by removing the key if it has no more volatile fields.
This was implemented in all mutation cases of the hash commands but the INCRBY and INCRBYFLOAT.
This can lead to a dangling object which has no volatile items, which might lead to assertion during the active-expiration job:

example reproduction:

DEBUG SET-ACTIVE-EXPIRE 0
hset myhash f1 10
hexpire myhash 1 FIELDS 1 f1
sleep(10)
hincrby myhash f1 1
DEBUG SET-ACTIVE-EXPIRE 1

NOTE: we actually had tests for this scenario, only the test did not include explicit assertion in case the item is still tracked after the mutation.

@ranshid ranshid added the bug Something isn't working label Dec 24, 2025
@ranshid ranshid moved this to To be backported in Valkey 9.0 Dec 24, 2025
@ranshid ranshid moved this to In Progress in Valkey 9.1 Dec 24, 2025
@codecov
Copy link

codecov bot commented Dec 24, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.76%. Comparing base (992b886) to head (2b6fc27).
⚠️ Report is 17 commits behind head on unstable.

Additional details and impacted files
@@             Coverage Diff              @@
##           unstable    #2974      +/-   ##
============================================
+ Coverage     73.73%   73.76%   +0.03%     
============================================
  Files           125      125              
  Lines         68911    69357     +446     
============================================
+ Hits          50813    51164     +351     
- Misses        18098    18193      +95     
Files with missing lines Coverage Δ
src/t_hash.c 94.61% <100.00%> (-0.03%) ⬇️

... and 27 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ranshid ranshid mentioned this pull request Dec 24, 2025
@ranshid ranshid merged commit 7cef1e8 into valkey-io:unstable Dec 29, 2025
70 of 71 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in Valkey 9.1 Dec 29, 2025
ranshid pushed a commit that referenced this pull request Jan 1, 2026
## Add Fuzzing Capability to Valkey

### Overview
This PR adds a fuzzing capability to Valkey, allowing developers and
users to stress test their Valkey deployments with randomly generated
commands. The fuzzer is integrated with the existing valkey-benchmark
tool, making it easy to use without requiring additional dependencies.

### Key Features
• **Command Generator**: Automatically generates Valkey commands by
retrieving command information directly from the server
• **Two Fuzzing Modes**:
- normal: Generates only valid commands, doesn't modify server
configurations
- aggressive: Includes malformed commands and allows CONFIG SET
operations

• **Multi-threaded Testing**: Each client runs in a dedicated thread to
maximize interaction between clients and enable testing of complicated
scenarios

• **Integration with valkey-benchmark**: Uses the existing CLI interface

### Implementation Details
• Added new files:
- `fuzzer_command_generator.h/c`: Dynamically generates valkey commands.
- `fuzzer_client.c`: Orchestrate all the client threads, report test
progress, and handle errors.

• Modified existing files:
  - valkey-benchmark.c: Added fuzzing mode options and integration

### Command Generation Approach
The fuzzer dynamically retrieves command information from the server,
allowing it to adapt to different Valkey versions and custom modules.
Since the command information generated from JSON files is sometimes
limited, not all generated commands will be valid, but approximately 95%
valid command generation is achieved.

It is important to generate valid commands to cover as much code path as
possible and not just the invalid command/args path. The fuzzer
prioritizes generating syntactically and semantically correct commands
to ensure thorough testing of the server's core functionality, while
still including a small percentage of invalid commands in `aggressive`
mode to test error handling paths

#### Config modification
For CONFIG SET command, the situation is more complex as the server
currently provides limited information through CONFIG GET *. Some
hardcoded logic is implemented that will need to be modified in the
future. Ideally, the server should provide self-inspection commands to
retrieve config keys-values with their properties (enum values,
modifiability status, etc.).

### Issue Detection
The fuzzer is designed to identify several types of issues:
• Server crashes
• Server memory corruptions / memory leaks(when compiled with ASAN)
• Server unresponsiveness
• Server malformed replies

For unresponsiveness detection, command timeout limits are implemented
to ensure no command blocks for excessive periods. If a server doesn't
respond within 30 seconds, the fuzzer signals that something is wrong.

### Proven Effectiveness
When running against the latest unstable version, the fuzzer has already
identified several issues, demonstrating its effectiveness:
* #2111
* #2112
* #2109
* #2113
* #2108
* #2137
* #2106
* #2347
* #2973
* #2974
### How to Use
Run the fuzzer using the valkey-benchmark tool with the --fuzz flag:

```bash
# Basic usage (10000 commands 1000 commands per client, 10 clients)
./src/valkey-benchmark --fuzz -h 127.0.0.1 -p 6379 -n 10000 -c 10

# With aggressive fuzzing mode
./src/valkey-benchmark --fuzz --fuzz-level aggressive -h 127.0.0.1 -p 6379 -n 10000 -c 10

# With detailed logging
./src/valkey-benchmark --fuzz --fuzz-log-level debug -h 127.0.0.1 -p 6379 -n 10000 -c 10
```

The fuzzer supports existing valkey-benchmark options, including TLS and
cluster mode configuration.

---------

Signed-off-by: Uri Yagelnik <[email protected]>
@ranshid ranshid added the release-notes This issue should get a line item in the release notes label Jan 4, 2026
jdheyburn pushed a commit to jdheyburn/valkey that referenced this pull request Jan 8, 2026
Following Hash-Field-Expiration feature, a hash object can hold volatile
fields.
volatile fields which are already expired are deleted and reclaimed ONLY
by the active-expiration background job.
This means that hash object can contain items which have not yet
expired.
In case mutations are requesting to set a value on these
"already-expired" fields, they will be overwritten with the new value.
In such cases, though, it is requiered to update the global per-db
tracking map by removing the key if it has no more volatile fields.
This was implemented in all mutation cases of the hash commands but the
`INCRBY` and `INCRBYFLOAT`.
This can lead to a dangling object which has no volatile items, which
might lead to assertion during the active-expiration job:

example reproduction:
```
DEBUG SET-ACTIVE-EXPIRE 0
hset myhash f1 10
hexpire myhash 1 FIELDS 1 f1
sleep(10)
hincrby myhash f1 1
DEBUG SET-ACTIVE-EXPIRE 1
```   

NOTE: we actually had tests for this scenario, only the test did not
include explicit assertion in case the item is still tracked after the
mutation.

Signed-off-by: Ran Shidlansik <[email protected]>
jdheyburn pushed a commit to jdheyburn/valkey that referenced this pull request Jan 8, 2026
## Add Fuzzing Capability to Valkey

### Overview
This PR adds a fuzzing capability to Valkey, allowing developers and
users to stress test their Valkey deployments with randomly generated
commands. The fuzzer is integrated with the existing valkey-benchmark
tool, making it easy to use without requiring additional dependencies.

### Key Features
• **Command Generator**: Automatically generates Valkey commands by
retrieving command information directly from the server
• **Two Fuzzing Modes**:
- normal: Generates only valid commands, doesn't modify server
configurations
- aggressive: Includes malformed commands and allows CONFIG SET
operations

• **Multi-threaded Testing**: Each client runs in a dedicated thread to
maximize interaction between clients and enable testing of complicated
scenarios

• **Integration with valkey-benchmark**: Uses the existing CLI interface

### Implementation Details
• Added new files:
- `fuzzer_command_generator.h/c`: Dynamically generates valkey commands.
- `fuzzer_client.c`: Orchestrate all the client threads, report test
progress, and handle errors.

• Modified existing files:
  - valkey-benchmark.c: Added fuzzing mode options and integration

### Command Generation Approach
The fuzzer dynamically retrieves command information from the server,
allowing it to adapt to different Valkey versions and custom modules.
Since the command information generated from JSON files is sometimes
limited, not all generated commands will be valid, but approximately 95%
valid command generation is achieved.

It is important to generate valid commands to cover as much code path as
possible and not just the invalid command/args path. The fuzzer
prioritizes generating syntactically and semantically correct commands
to ensure thorough testing of the server's core functionality, while
still including a small percentage of invalid commands in `aggressive`
mode to test error handling paths

#### Config modification
For CONFIG SET command, the situation is more complex as the server
currently provides limited information through CONFIG GET *. Some
hardcoded logic is implemented that will need to be modified in the
future. Ideally, the server should provide self-inspection commands to
retrieve config keys-values with their properties (enum values,
modifiability status, etc.).

### Issue Detection
The fuzzer is designed to identify several types of issues:
• Server crashes
• Server memory corruptions / memory leaks(when compiled with ASAN)
• Server unresponsiveness
• Server malformed replies

For unresponsiveness detection, command timeout limits are implemented
to ensure no command blocks for excessive periods. If a server doesn't
respond within 30 seconds, the fuzzer signals that something is wrong.

### Proven Effectiveness
When running against the latest unstable version, the fuzzer has already
identified several issues, demonstrating its effectiveness:
* valkey-io#2111
* valkey-io#2112
* valkey-io#2109
* valkey-io#2113
* valkey-io#2108
* valkey-io#2137
* valkey-io#2106
* valkey-io#2347
* valkey-io#2973
* valkey-io#2974
### How to Use
Run the fuzzer using the valkey-benchmark tool with the --fuzz flag:

```bash
# Basic usage (10000 commands 1000 commands per client, 10 clients)
./src/valkey-benchmark --fuzz -h 127.0.0.1 -p 6379 -n 10000 -c 10

# With aggressive fuzzing mode
./src/valkey-benchmark --fuzz --fuzz-level aggressive -h 127.0.0.1 -p 6379 -n 10000 -c 10

# With detailed logging
./src/valkey-benchmark --fuzz --fuzz-log-level debug -h 127.0.0.1 -p 6379 -n 10000 -c 10
```

The fuzzer supports existing valkey-benchmark options, including TLS and
cluster mode configuration.

---------

Signed-off-by: Uri Yagelnik <[email protected]>
ranshid added a commit to ranshid/valkey that referenced this pull request Jan 26, 2026
Following Hash-Field-Expiration feature, a hash object can hold volatile
fields.
volatile fields which are already expired are deleted and reclaimed ONLY
by the active-expiration background job.
This means that hash object can contain items which have not yet
expired.
In case mutations are requesting to set a value on these
"already-expired" fields, they will be overwritten with the new value.
In such cases, though, it is requiered to update the global per-db
tracking map by removing the key if it has no more volatile fields.
This was implemented in all mutation cases of the hash commands but the
`INCRBY` and `INCRBYFLOAT`.
This can lead to a dangling object which has no volatile items, which
might lead to assertion during the active-expiration job:

example reproduction:
```
DEBUG SET-ACTIVE-EXPIRE 0
hset myhash f1 10
hexpire myhash 1 FIELDS 1 f1
sleep(10)
hincrby myhash f1 1
DEBUG SET-ACTIVE-EXPIRE 1
```

NOTE: we actually had tests for this scenario, only the test did not
include explicit assertion in case the item is still tracked after the
mutation.

Signed-off-by: Ran Shidlansik <[email protected]>
@zuiderkwast zuiderkwast moved this from To be backported to 9.0.2 WIP in Valkey 9.0 Jan 28, 2026
ranshid added a commit to ranshid/valkey that referenced this pull request Jan 28, 2026
Following Hash-Field-Expiration feature, a hash object can hold volatile
fields.
volatile fields which are already expired are deleted and reclaimed ONLY
by the active-expiration background job.
This means that hash object can contain items which have not yet
expired.
In case mutations are requesting to set a value on these
"already-expired" fields, they will be overwritten with the new value.
In such cases, though, it is requiered to update the global per-db
tracking map by removing the key if it has no more volatile fields.
This was implemented in all mutation cases of the hash commands but the
`INCRBY` and `INCRBYFLOAT`.
This can lead to a dangling object which has no volatile items, which
might lead to assertion during the active-expiration job:

example reproduction:
```
DEBUG SET-ACTIVE-EXPIRE 0
hset myhash f1 10
hexpire myhash 1 FIELDS 1 f1
sleep(10)
hincrby myhash f1 1
DEBUG SET-ACTIVE-EXPIRE 1
```   

NOTE: we actually had tests for this scenario, only the test did not
include explicit assertion in case the item is still tracked after the
mutation.

Signed-off-by: Ran Shidlansik <[email protected]>
ranshid added a commit that referenced this pull request Jan 29, 2026
Following Hash-Field-Expiration feature, a hash object can hold volatile
fields.
volatile fields which are already expired are deleted and reclaimed ONLY
by the active-expiration background job.
This means that hash object can contain items which have not yet
expired.
In case mutations are requesting to set a value on these
"already-expired" fields, they will be overwritten with the new value.
In such cases, though, it is requiered to update the global per-db
tracking map by removing the key if it has no more volatile fields.
This was implemented in all mutation cases of the hash commands but the
`INCRBY` and `INCRBYFLOAT`.
This can lead to a dangling object which has no volatile items, which
might lead to assertion during the active-expiration job:

example reproduction:
```
DEBUG SET-ACTIVE-EXPIRE 0
hset myhash f1 10
hexpire myhash 1 FIELDS 1 f1
sleep(10)
hincrby myhash f1 1
DEBUG SET-ACTIVE-EXPIRE 1
```   

NOTE: we actually had tests for this scenario, only the test did not
include explicit assertion in case the item is still tracked after the
mutation.

Signed-off-by: Ran Shidlansik <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working release-notes This issue should get a line item in the release notes

Projects

Status: 9.0.2
Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants