Hello @hshs1103

*I'm not used to open-source activities. Please tell me if the method of PR is wrong.

Nothing wrong making the PR in my opinion - thanks for sharing this.

@itamarhaber
Thank you for your reply. I'm so glad there's no problem.
I will wait for your review.

Hello @itamarhaber . Can you tell if the review process is going on now?
I check the notification that pull request needs to be reviewed, and i leave a comment.

@hshs1103 @addb-swstarlab can you please outline the differences and advantages here?
sorry, but i rather read the top level description before diving into the code, and don't have an account to view the PDF.

p.s. we can't obviously merge a PR for redis 4. but we can review the diff, provide feedback, and then make a new one for the unstable branch.

@oranagra

Compared to the existing persistence method, AOF-Preamble, there are two differences.

First, LESS method can guarantee data persistence without using a Rewrite buffer for newly requested commands during the operation of the child process.
Second, LESS method manages RDB file created by the child process and AOF file created by the main process separately rather than merged.

The advantages of these differences are as follows.

First, since the LESS method does not use the Rewrite buffer, the memory usage of LESS method is less than that of the AOF-Preamble method. During the AOF-Preamble operation, Redis store the generated log record redundantly in two buffers(AOF buffer & Rewrite buffer), which leads to surge in memory usage. However, in LESS method, the generated log record is stored in only AOF buffer, so memory usage does not increase sharply. For the same memory capacity, LESS method increases the available memory space to store data when compared to the AOF-Preamble.

Second, our method improves throughput over the original method. This is because LESS method does not perform excessive disk I/O. In the AOF-Preamble method, the main process incurs heavy disk I/O owing to the merging of the RDB and AOF files. LESS solves the heavy disk I/O problem by storing the log files separately rather than merging the AOF and RDB files.

As the paper on the LESS module has been published, I will give you another link to the paper for your understanding. Please review it.
http://delab.yonsei.ac.kr/assets/files/publication/international/conference/08679377.pdf

I would appreciate if you could review the diff and provide feedback.

Ohh, IIUC, this idea is was around for a while (we called it multipart-aof, not as fancy as LESS), IIRC it was proposed by @inbareyal several years ago and rejected by Salvatore due to introducing complexity for the operator needing to move several files around.

Just to make sure i understand (didn't fully read it), it boils down to this:

• As soon as redis forks to create the new AOF (RDB snapshot), the main process stops logging to the original AOF file, and starts logging into a new one.
• This new aof file can be considered the second (or last) part of the previous AOF file (once we used till now).
• When the fork completes, the output of the fork is now considered the first part, and the new file created by the main process is the second part.
• This way we avoid keeping the commands arrived since the fork was created in memory, we avoid writing these commands to the disk twice, and most importantly, we avoid the freeze that happens when flushing that last part to the disk from the parent process.

Let me know if i'm right, or did i miss anything.
p.s. unlike Salvatore, i think the gains here outweigh the operator complications.

@oranagra

Thank you for your reply. As a result of checking the comments you left, I think you understood them correctly.

To help you understand more in detail, I attach some pictures of motivation for developing the LESS method, the operation process of LESS method, and the data recovery method if an error occurs during LESS operation.

Motivation

LESS logging operation

LESS recovery mechanism

If you have additional questions, please leave a comment.

Interesting idea, it seems we need two AOF file if I understand correctly, I'd like to call them appendonly.base and appendonly.incr, btw I agree with @oranagra , LESS is not easy to understand.

Actually we have a similar implementation, we call it AOF rotate and create a new file aof.index to manage all AOF, and we expand AOF to make PITR be possible, it's another topic ; )

@oranagra @soloestoy

Hello. May I know what the future progress will be about the pull request(#6584 ) I posted? Is there anything I can do to help?

@hshs1103 as i stated in the past, this PR is not mergable because it was done against redis 4.0 rather than the unstable branch (but also redis changed quite a lot since when so it'll have many conflicts and need to be re-done).

back when you submitted this paper and PR, the idea of multi-part AOF already existed in our future roadmap (at least in RedisLabs), but we weren't ready to go there just yet (there was a different maintainer back then, a different development cadence, and different priorities).

Recently the new redis-core-team decided to pick this up for redis 7.0, and and the Alibaba team stepped up to do the detailed design and implementation (hence #9539).
I did ask them to take a look here to see how you resolved the problem of atomic rename of two files (you can see recent comments in the other PR), let's continue the discussion about that there.

@oranagra Thanks for your reply.

In Less, file name change is not handled atomically for the operation of multiple files. Instead, we designed each file to be renamed sequentially, one by one, so we know when the error occurred.

If there is anything I can help with later, please feel free to comment. :-)