Hey David, Thanks for starting this conversation.

Can you tell us a bit more about what is the expectation here? Is the expectation that these need to be part of the main project, i.e. why not a separate project, and are we going to be asked to keep writing more fuzz tests over time?

You are fuzzing some rather uninteresting functions, we probably want to be doing more fuzzing along the core Redis command processing path.

Also, in the nicest way possible, is OSS-Fuzz good? There have been several reports over the years where people give us a dump of "possible bugs" that aren't actually possible and we just waste our time verifying that. I don't want this to end up just being noise that is distracting from real problems we need to be addressing.

In essence there are no expectations. However, naturally if you do not intend on fixing bugs provided then it would be wasted resources to fuzz. However, no one will do anything if you do not fix the bugs and nothing stops you from opting out at any point.

We can keep the fuzzers in other repositories: if you are most comfortable having it outside the main repository then we can store them over in the main oss-fuzz repository: https:/github.com/google/oss-fuzz . If you are happy to try it out like this first then you can provide me with an email and I will fix it all up so in the main OSS-Fuzz repository so you can get the experience without merging anything into the Redis main repo. Naturally, fuzzing is really important for ensuring security of a project so if you like the results of the fuzzing then we can proceed in the future to get it more integrated into the Redis repo and perhaps CI as well.

You will not be asked to do anyting over time - you do not sign up for anything. The only specific caveat is that bugs are made public after a 90 day deadline, i.e. if the fuzzers find a bug and file a bug report, then if the bug is not fixed within 90 days the bug report will be public on the bug tracker.

In regards to whether OSS-Fuzz is good, then I will strongly argue so. Check out the list of projects integrated, which contains more than 400 significant open source projects: https://github.com/google/oss-fuzz/tree/master/projects Fuzzing is a key part of providing security for software and OSS-Fuzz is an excellent way to do so for open source software.

We can create the fuzzers such that the bugs reported within the threat model of redis, which will avoid being random bugs that are less of a priority to other tasks. In this context it would be very helpful if you could guide me to some APIs as I am not an expert in the Redis codebase but know a lot about fuzzing. From the bug reports you also get nice saintizer reports which show you the full stack trace, inputs that trigger it and more introspection, which eases the root-cause analysis process a lot.

not sure i understand where we aim for, but here are a few random thoughts (after looking the the PR code changes):

1. the value of fuzzing a single method of some low level subsystem is not very high IMHO, often the bugs are either in edge cases of the whole system (combination of several subsystems, and the way functions integrate with one another).
2. if i understand this correctly, it looks very coupled with the internals (fuzzing input for internal functions), this means that if we refactor or change this function, the fuzzer breaks, and someone needs to maintain it, this becomes a bigger problem if the fuzzer is hosted in another repo).
3. actually many times we actually have bug validating user input (for instance a missing argument of a redis command causes it to crash), so it would be nice to have some smart fuzzer that sends redis commands. obviously it should not be completely random, the syntax must pass some initial validations to make it to the valuable code (correct syntax, on a key of the right type, etc).
4. as @madolson pointed out, the current approach can indeed expose a lot of false positives in redis, things that look like bugs but actually aren't (due to how the code is used), and looking into these reports can be time consuming.

Thanks for the reply @oranagra

In general I agree with your statements, although fuzzing internal functions can often be a nice way of ensuring no memory corruption exists in them. However, it's not needed to do a ton of fuzzing in these.

In essence, my interest is in integrating fuzzing into the Redis project, and also having it done continuously by way of OSS-Fuzz. If I create the fuzzers such that they produce valid commands and do the maintenance of it, i.e. change the fuzzers when needed, would you be happy to integrate them? In case I can no longer maintain it I can make this clear and also remove the fuzzers if that's what you prefer at that given time.

@DavidKorczynski a fuzzer that generates commands that make sense and have good chance to pass initial checks and reach valuable code is something we would like very much.
generally commands are always backwards compatible so there's no reason this would break. it would be hard for such a fuzzer to validate correct output, but, it can check for crashes, leaks, and double or missing replies.

i.e. if some error handling code for redis responds with two replies, the next command will consume the second reply of the previous one, and if it doesn't reply the client is hung.

you can have a quick look at this work in progress PR #7807 in which i tried to write a simple fuzzer for the purpose of catching crashes and leaks (util.tcl).

@oranagra I was going to do this but simply did not find the time. I think it's probably best closing this one for now and then re-opening if I can come up with such a fuzzer as you suggests.