Skip to content

Conversation

@Watson1978
Copy link
Contributor

@Watson1978 Watson1978 commented Mar 4, 2025

Which issue(s) this PR fixes:
Fixes #1657

What this PR does / why we need it:
Seems that a full GC have to run in order to collect the wasted String object in @chunk.
However, full GC is not performed frequently.

This patch purges internal memory in wasted String object immediately.

Here is memory usage retrieved by ObjectSpace.memsize_of_all.

Note

Ruby has various optimization, such as reusing freed heap areas by pooling them.
For this reason, it may not be reflected in the process's memory usage immediately even if it clear the object.

Reproduce

  • Configuration
<source>
  @type tcp
  tag testing
  format json
  port 10130
</source>
<match **>
  @type forward
  require_ack_response true
  <server>
    host 127.0.0.1
    port 10131
  </server>

  ## Tune the buffer parameters
  <buffer tag>
    flush_mode interval
    flush_interval 1s
  </buffer>
</match>
  • test data
require "socket"
require "json"

i = 0

Thread.new do
  s = TCPSocket.open("localhost", 10130)
  data = {"message": "a" * 512 * 1024}

  loop do
    puts "[#{i}]"
    data["number"] = i
    s.puts data.to_json
    i += 1
    sleep 0.01
  end

  s.close
end

sleep 20

Docs Changes:
Not needed.

Release Note:
buf_memory: improve memory performance

@Watson1978 Watson1978 changed the title memory_chunk: clear internal memory use in string memory_chunk: clear internal memory use in buffer string Mar 4, 2025
@daipom daipom added this to the v1.19.0 milestone Mar 4, 2025
@daipom daipom added the memory label Mar 4, 2025
@Watson1978 Watson1978 force-pushed the memory_chunk branch 2 times, most recently from 9a6db75 to 1f97eec Compare March 10, 2025 06:05
@Watson1978 Watson1978 marked this pull request as ready for review March 10, 2025 07:32
@Watson1978 Watson1978 requested a review from daipom March 10, 2025 07:33
Copy link
Contributor

@daipom daipom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this improvement!
Basically looks good to me!

I have commented on some points for the test code.
It could be simpler.
Please check them.

@daipom
Copy link
Contributor

daipom commented Mar 11, 2025

Thanks! Waiting CI...

@daipom daipom merged commit 2e896ec into fluent:master Mar 11, 2025
9 of 10 checks passed
@Watson1978 Watson1978 deleted the memory_chunk branch March 11, 2025 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fluentd retains excessive amounts of memory after handling traffic peaks

2 participants