Skip to content

Disk buffer fixes#901

Merged
MrAnno merged 4 commits intoaxoflow:mainfrom
bshifter:disk-buffer-fixes
Jan 15, 2026
Merged

Disk buffer fixes#901
MrAnno merged 4 commits intoaxoflow:mainfrom
bshifter:disk-buffer-fixes

Conversation

@bshifter
Copy link
Member

@bshifter bshifter commented Jan 9, 2026

This pr contains a set of diskbuffer changes:

  • detailed debug logs for load/save operations displaying the inner state of non-reliable-diskbuffer
  • balancing the front-cache items loaded between front-cache and front-cache-output by keeping the front-cache-limit
  • fixing possible writes over the front-cache limit
  • possible fix for memory usage metrics reset on reload

writer_sck_builder);


log_writer_set_queue(self->writer, NULL);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this fix. Can we discuss it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a workaround, the real resolution would be, for example, to completely get rid of log_queue_memory_usage_sub() calls in LogQueue::_unregister_shared_counters() and do the decrements locally in each implementation at the right time.

… a diskbuffer file

front_cache_output get filled after front_cache limit reached

Signed-off-by: shifter <[email protected]>
On reload, AFsocket destinations release the previous LogQueue
after creating a new one, which can wipe memory usage metrics recorded
by the new LogQueue.

This fix needs deeper investigation; possible side effects are unclear.This fix requires a strong examine.

Signed-off-by: shifter <[email protected]>
else
{
return self->hdr->length +
self->hdr->backlog_pos.count +
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

backlog is missing from the new calculation, but it's not the end of the world

@MrAnno MrAnno merged commit 9592b0e into axoflow:main Jan 15, 2026
21 of 22 checks passed
Copy link
Member

@bazsi bazsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mid air collision of my comments, submitting anyway.

}

if (post_op_cb && user_data)
post_op_cb(QDISK_SAVE, self->hdr, user_data);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why this if becoming a callback initiated from the depths of qdisk_start/stop

Rather than doing the same printouts from wherever we call qdisk_start/stop.

I don't see we are publishing information or a state that would not be available otherwise.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have the qdisk hdr this way. we also use it's data in debug logs. also the timing if logging is crucial.

}

static gboolean
_iv_list_partition(struct iv_list_head *src, struct iv_list_head *dst, gint index)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not have an iv_list specific name, rather something that explains that it is actually splitting the list in two, starting at a specific index.

dst->prev = src->prev;
src->prev->next = dst;
src->prev = pos->prev;
pos->prev = dst;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not reimplement low level ivykis list functions. (Don't remember exact names but I am fairly sure there are easier to read functions for removing/adding an element)

Also applies to list iteration.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably my bad, but didn't find any existing iv_function to split lists. I'm gonna double check it!

return;
if (self->front_cache_output.len <= self->front_cache.limit)
{
iv_list_splice_tail_init(&self->front_cache_output.items, &self->front_cache.items);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not performance sensitive as much, so maybe just always using the split function would works, simplifying the function.

I am ok with keeping this a separate case.

{
LogQueueDiskNonReliable *self = (LogQueueDiskNonReliable *) user_data;

if (op == QDISK_LOAD)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, the callback does not seem to be required for this one either.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is avoidable in case we figure out a working solution not to use callbacks here. I'm going to ask the other's opinions about it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants