Replies: 3 comments 1 reply
-
|
many of of these ideas sound interesting, but we may need to tackle them one by one. |
Beta Was this translation helpful? Give feedback.
-
|
@ShooterIT I think the ability to annotate is going to be very useful for AOF. I find the timestamps and replication offset annotations most useful. Eviction logging is a valid point but the way I see it, it's a debug logging feature and piggy backing it on top of AOF makes less sense to me. The same applies to auditing where as @oranagra mentioned we also need to handle reads, compliance of requires those files to be append-only and rotated (rather than being re-rewritten), etc. |
Beta Was this translation helpful? Give feedback.
-
|
It seems core team agree this proposal, let me create issue from discussion to follow the development of this feature |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
As we know, in AOF, there are only Redis commands, in some cases, we may want to know context of commands for updating database. In RocksDB, it allows us to annotate each
Putwith some arbitrary metadata, so we can implement interesting features based on that. Of course, Redis is a server instead of a library, not casual for changing data format. But there may be several common requires that we can support, at least, we also can make "annotations" is legal in AOF, AOF tools could skip them if they don't want to parse them.In current idea, the annotations doesn't fully break AOF format, just add some lines that starts with
#, like some languages annotation, after#, we could add useful information which we want. The annotations are only text instead of binary safe, of course, there can be multi lines together to explain context much clearly.The followings are several scenarios I think of.
Restore from a specific point-in-time
Users may ask us to provide one ability: restore data from a specific point-in-time, but there is no time info even during writing AOF (rather than AOF rewrite). One idea is that add some timestamp in AOF, timestamp precision may be minute or second, to avoid making AOF over-inflated. If there are some timestamps, it is easy to restore from a specific point-in-time by replaying AOF to a certain point in timestamp.
Keys eviction
Redis sometimes evicts keys silently, especially before Incremental eviction processing #7653, actually, sometimes we may want to know keys are evicted or deleted, and the reason why redis start to evict keys (we lose context forever after finishing eviction, moreover, since of eviction is blocking, we can't learn useful things by
INFOcommand), so it is very useful to record these important infos if possible.PSYNC-AOF
AFAIK, there is not only one company that implement PSYNC based on AOF, antirez also opened an issue PSYNC3: next improvements to Redis replication #4357 to discuss this topic, he also designed one kind of AOF annotations for implementing PSYNC based on AOF. Maybe we can record some commands' offset, after syncing several commands in AOF, and then force to reset replica's replication offset by the record replication offset in AOF (because the content of AOF is not the same as replication backlog). This feature may provide convenience for implementing PSYNC-AOF.
Dangerous commands
In production environments, for some dangerous command, we usually block them by
rename-commandor ACL rules, but if actually they are executed, we still want to know who did this operation, when it happened, maybe other things, and we may also want to get this information fo audit. Moreover, forslowlogthat update database, we maybe could put the metrics of slow log around the write command, so we could see much more slow logs on disk instead of just only those on limited memory. Of course, we can record these in server log, but we always store these commands in AOF, so it is not costly, but this is worth arguing.If you do secondary development based on Redis, maybe you can add more infos for your purpose, such as active-active geo-distribution, audit log. Welcome to communicate!
Beta Was this translation helpful? Give feedback.
All reactions