formatter_csv: fix memory leak #4864

Watson1978 · 2025-03-14T05:12:56Z

Which issue(s) this PR fixes:
Fixes #3627

What this PR does / why we need it:
Seems in_exec plugin creats Thread object each interval and it invoke formatter_csv on the thread.

If it use a Thread object as hash key when caching with formatter_csv, that thread will be permanently referenceable and will not be collected by the GC.

Here is memory usage of Ruby's process.

Reproduce

See #3627

<system>
  process_name fluentd_monitor
</system>

<source>
  @type exec
  tag hardware.memory
  run_interval 0.1
  command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh
  <parse>
    @type json
    types mem_available:float,swap_total:float,swap_free:float
  </parse>
</source>

<match **>
  @type exec
  command /home/watson/prj/sandbox/fluentd/psql_memory.sh
  <format>
    @type csv
    force_quotes false
    delimiter |
    fields time_local,mem_available,swap_total,swap_free
  </format>
  <buffer time>
    @type file
    path /home/watson/prj/sandbox/fluentd/log
    timekey 50s
    timekey_wait 0s
    flush_mode lazy
    total_limit_size 200MB
    retry_type periodic  # exponential backoff is broken https://github.com/fluent/fluentd/issues/3609
    retry_max_times 0  # need to go to secondary as soon as possible https://github.com/fluent/fluentd/issues/3610
  </buffer>
</match>

Docs Changes:

Release Note:
formatter_csv: fix memory leak

Signed-off-by: Shizuo Fujita <[email protected]>

daipom

Thanks! Good catch!

It seems that in #2529, input plugins were assumed to spawn only a fixed number of threads for emitting.
However, in_exec for example, makes threads for emitting without limit.
The cache would then become infinitely large.
We need to fix this cache.

daipom

The thread key would need to be unique for each plugin instance.

The following out_http code has a similar feature.
I want to align the code with this one unless there is a special reason not to.

fluentd/lib/fluent/plugin/out_http.rb

Lines 109 to 111 in 2e896ec

    
           def connection_cache_id_thread_key 
        
             "#{plugin_id}_connection_cache_id" 
        
           end

For example, I want the following getter.

def csv_thread_key
  "#{plugin_id}_csv"
end

Watson1978 · 2025-03-14T09:18:33Z

@daipom Thanks. I fixed the key name at last commit.

Signed-off-by: Shizuo Fujita <[email protected]>

lib/fluent/plugin/formatter_csv.rb

daipom

Sorry, I forgot that this is a owned plugin.
It would not be expected to have an owned plugin with PluginId.

So, could you please fix the following points?

(Owned plugins are unique to the owner's id and usage.)

lib/fluent/plugin/formatter_csv.rb

Signed-off-by: Shizuo Fujita <[email protected]>

Watson1978 · 2025-03-17T03:23:29Z

When it invoke CsvFormatter via Fluent::Compat::TextFormatter, looks like it does not set owner...

Signed-off-by: Shizuo Fujita <[email protected]>

Signed-off-by: Daijiro Fukuda <[email protected]>

daipom · 2025-03-18T07:48:31Z

When it invoke CsvFormatter via Fluent::Compat::TextFormatter, looks like it does not set owner...

Sorry, I missed this point.
I have pushed another idea to fix this issue.

80c02fb

@Watson1978 Could you please review this commit?
If there seem to be some problems, we can revert it.

The aim of this commit is to ensure that the compat layer does not affect this cache feature.
(Because I'm not sure how to make the key unique on the compat layer context.)

Signed-off-by: Shizuo Fujita <[email protected]>

daipom

Thanks for fixing the CI failures my commit caused!

It seems possible that there is no owner. (FluentBinlogReader)
It would be a good solution to disable the cache feature in such a case.

I commented on some minor points.
Please check them.

lib/fluent/compat/formatter.rb

lib/fluent/plugin/formatter_csv.rb

Co-authored-by: Daijiro Fukuda <[email protected]> Signed-off-by: Shizuo Fujita <[email protected]>

Signed-off-by: Shizuo Fujita <[email protected]>

daipom

LGTM! Thanks for your contribution!

@type

**Which issue(s) this PR fixes**: Fixes fluent#3627 **What this PR does / why we need it**: Seems `in_exec` plugin creats Thread object each interval and it invoke formatter_csv on the thread. If it use a Thread object as hash key when caching with formatter_csv, that thread will be permanently referenceable and will not be collected by the GC. Here is memory usage of Ruby's process. ![chart](https://github.com/user-attachments/assets/4e74adda-154b-4c00-9cb4-3b1a0e8745fd) ### Reproduce See fluent#3627 ``` <system> process_name fluentd_monitor </system> <source> @type exec tag hardware.memory run_interval 0.1 command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh <parse> @type json types mem_available:float,swap_total:float,swap_free:float </parse> </source> <match **> @type exec command /home/watson/prj/sandbox/fluentd/psql_memory.sh <format> @type csv force_quotes false delimiter | fields time_local,mem_available,swap_total,swap_free </format> <buffer time> @type file path /home/watson/prj/sandbox/fluentd/log timekey 50s timekey_wait 0s flush_mode lazy total_limit_size 200MB retry_type periodic # exponential backoff is broken fluent#3609 retry_max_times 0 # need to go to secondary as soon as possible fluent#3610 </buffer> </match> ``` **Docs Changes**: **Release Note**: formatter_csv: fix memory leak --------- Signed-off-by: Shizuo Fujita <[email protected]> Signed-off-by: Daijiro Fukuda <[email protected]> Co-authored-by: Daijiro Fukuda <[email protected]> Signed-off-by: Kentaro Hayashi <[email protected]>

@type

**Which issue(s) this PR fixes**: Fixes fluent#3627 **What this PR does / why we need it**: Seems `in_exec` plugin creats Thread object each interval and it invoke formatter_csv on the thread. If it use a Thread object as hash key when caching with formatter_csv, that thread will be permanently referenceable and will not be collected by the GC. Here is memory usage of Ruby's process. ![chart](https://github.com/user-attachments/assets/4e74adda-154b-4c00-9cb4-3b1a0e8745fd) ### Reproduce See fluent#3627 ``` <system> process_name fluentd_monitor </system> <source> @type exec tag hardware.memory run_interval 0.1 command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh <parse> @type json types mem_available:float,swap_total:float,swap_free:float </parse> </source> <match **> @type exec command /home/watson/prj/sandbox/fluentd/psql_memory.sh <format> @type csv force_quotes false delimiter | fields time_local,mem_available,swap_total,swap_free </format> <buffer time> @type file path /home/watson/prj/sandbox/fluentd/log timekey 50s timekey_wait 0s flush_mode lazy total_limit_size 200MB retry_type periodic # exponential backoff is broken fluent#3609 retry_max_times 0 # need to go to secondary as soon as possible fluent#3610 </buffer> </match> ``` **Docs Changes**: **Release Note**: formatter_csv: fix memory leak --------- Signed-off-by: Shizuo Fujita <[email protected]> Signed-off-by: Daijiro Fukuda <[email protected]> Co-authored-by: Daijiro Fukuda <[email protected]> Signed-off-by: Kentaro Hayashi <[email protected]>

@type

**Which issue(s) this PR fixes**: Fixes fluent#3627 **What this PR does / why we need it**: Seems `in_exec` plugin creats Thread object each interval and it invoke formatter_csv on the thread. If it use a Thread object as hash key when caching with formatter_csv, that thread will be permanently referenceable and will not be collected by the GC. Here is memory usage of Ruby's process. ![chart](https://github.com/user-attachments/assets/4e74adda-154b-4c00-9cb4-3b1a0e8745fd) ### Reproduce See fluent#3627 ``` <system> process_name fluentd_monitor </system> <source> @type exec tag hardware.memory run_interval 0.1 command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh <parse> @type json types mem_available:float,swap_total:float,swap_free:float </parse> </source> <match **> @type exec command /home/watson/prj/sandbox/fluentd/psql_memory.sh <format> @type csv force_quotes false delimiter | fields time_local,mem_available,swap_total,swap_free </format> <buffer time> @type file path /home/watson/prj/sandbox/fluentd/log timekey 50s timekey_wait 0s flush_mode lazy total_limit_size 200MB retry_type periodic # exponential backoff is broken fluent#3609 retry_max_times 0 # need to go to secondary as soon as possible fluent#3610 </buffer> </match> ``` **Docs Changes**: **Release Note**: formatter_csv: fix memory leak --------- Signed-off-by: Shizuo Fujita <[email protected]> Signed-off-by: Daijiro Fukuda <[email protected]> Co-authored-by: Daijiro Fukuda <[email protected]> Signed-off-by: Kentaro Hayashi <[email protected]>

@type

**Which issue(s) this PR fixes**: Fixes fluent#3627 **What this PR does / why we need it**: Seems `in_exec` plugin creats Thread object each interval and it invoke formatter_csv on the thread. If it use a Thread object as hash key when caching with formatter_csv, that thread will be permanently referenceable and will not be collected by the GC. Here is memory usage of Ruby's process. ![chart](https://github.com/user-attachments/assets/4e74adda-154b-4c00-9cb4-3b1a0e8745fd) ### Reproduce See fluent#3627 ``` <system> process_name fluentd_monitor </system> <source> @type exec tag hardware.memory run_interval 0.1 command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh <parse> @type json types mem_available:float,swap_total:float,swap_free:float </parse> </source> <match **> @type exec command /home/watson/prj/sandbox/fluentd/psql_memory.sh <format> @type csv force_quotes false delimiter | fields time_local,mem_available,swap_total,swap_free </format> <buffer time> @type file path /home/watson/prj/sandbox/fluentd/log timekey 50s timekey_wait 0s flush_mode lazy total_limit_size 200MB retry_type periodic # exponential backoff is broken fluent#3609 retry_max_times 0 # need to go to secondary as soon as possible fluent#3610 </buffer> </match> ``` **Docs Changes**: **Release Note**: formatter_csv: fix memory leak --------- Signed-off-by: Shizuo Fujita <[email protected]> Signed-off-by: Daijiro Fukuda <[email protected]> Co-authored-by: Daijiro Fukuda <[email protected]> Signed-off-by: Kentaro Hayashi <[email protected]>

@type

**Which issue(s) this PR fixes**: Fixes fluent#3627 **What this PR does / why we need it**: Seems `in_exec` plugin creats Thread object each interval and it invoke formatter_csv on the thread. If it use a Thread object as hash key when caching with formatter_csv, that thread will be permanently referenceable and will not be collected by the GC. Here is memory usage of Ruby's process. ![chart](https://github.com/user-attachments/assets/4e74adda-154b-4c00-9cb4-3b1a0e8745fd) ### Reproduce See fluent#3627 ``` <system> process_name fluentd_monitor </system> <source> @type exec tag hardware.memory run_interval 0.1 command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh <parse> @type json types mem_available:float,swap_total:float,swap_free:float </parse> </source> <match **> @type exec command /home/watson/prj/sandbox/fluentd/psql_memory.sh <format> @type csv force_quotes false delimiter | fields time_local,mem_available,swap_total,swap_free </format> <buffer time> @type file path /home/watson/prj/sandbox/fluentd/log timekey 50s timekey_wait 0s flush_mode lazy total_limit_size 200MB retry_type periodic # exponential backoff is broken fluent#3609 retry_max_times 0 # need to go to secondary as soon as possible fluent#3610 </buffer> </match> ``` **Docs Changes**: **Release Note**: formatter_csv: fix memory leak --------- Signed-off-by: Shizuo Fujita <[email protected]> Signed-off-by: Daijiro Fukuda <[email protected]> Co-authored-by: Daijiro Fukuda <[email protected]> Signed-off-by: Kentaro Hayashi <[email protected]>

@type

**Which issue(s) this PR fixes**: Fixes fluent#3627 **What this PR does / why we need it**: Seems `in_exec` plugin creats Thread object each interval and it invoke formatter_csv on the thread. If it use a Thread object as hash key when caching with formatter_csv, that thread will be permanently referenceable and will not be collected by the GC. Here is memory usage of Ruby's process. ![chart](https://github.com/user-attachments/assets/4e74adda-154b-4c00-9cb4-3b1a0e8745fd) ### Reproduce See fluent#3627 ``` <system> process_name fluentd_monitor </system> <source> @type exec tag hardware.memory run_interval 0.1 command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh <parse> @type json types mem_available:float,swap_total:float,swap_free:float </parse> </source> <match **> @type exec command /home/watson/prj/sandbox/fluentd/psql_memory.sh <format> @type csv force_quotes false delimiter | fields time_local,mem_available,swap_total,swap_free </format> <buffer time> @type file path /home/watson/prj/sandbox/fluentd/log timekey 50s timekey_wait 0s flush_mode lazy total_limit_size 200MB retry_type periodic # exponential backoff is broken fluent#3609 retry_max_times 0 # need to go to secondary as soon as possible fluent#3610 </buffer> </match> ``` **Docs Changes**: **Release Note**: formatter_csv: fix memory leak --------- Signed-off-by: Shizuo Fujita <[email protected]> Signed-off-by: Daijiro Fukuda <[email protected]> Co-authored-by: Daijiro Fukuda <[email protected]> Signed-off-by: Kentaro Hayashi <[email protected]>

@type

**Which issue(s) this PR fixes**: Fixes fluent#3627 **What this PR does / why we need it**: Seems `in_exec` plugin creats Thread object each interval and it invoke formatter_csv on the thread. If it use a Thread object as hash key when caching with formatter_csv, that thread will be permanently referenceable and will not be collected by the GC. Here is memory usage of Ruby's process. ![chart](https://github.com/user-attachments/assets/4e74adda-154b-4c00-9cb4-3b1a0e8745fd) ### Reproduce See fluent#3627 ``` <system> process_name fluentd_monitor </system> <source> @type exec tag hardware.memory run_interval 0.1 command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh <parse> @type json types mem_available:float,swap_total:float,swap_free:float </parse> </source> <match **> @type exec command /home/watson/prj/sandbox/fluentd/psql_memory.sh <format> @type csv force_quotes false delimiter | fields time_local,mem_available,swap_total,swap_free </format> <buffer time> @type file path /home/watson/prj/sandbox/fluentd/log timekey 50s timekey_wait 0s flush_mode lazy total_limit_size 200MB retry_type periodic # exponential backoff is broken fluent#3609 retry_max_times 0 # need to go to secondary as soon as possible fluent#3610 </buffer> </match> ``` **Docs Changes**: **Release Note**: formatter_csv: fix memory leak --------- Signed-off-by: Shizuo Fujita <[email protected]> Signed-off-by: Daijiro Fukuda <[email protected]> Co-authored-by: Daijiro Fukuda <[email protected]> Signed-off-by: Kentaro Hayashi <[email protected]>

@type

**Which issue(s) this PR fixes**: Backport #4864 Fixes #3627 **What this PR does / why we need it**: Seems `in_exec` plugin creats Thread object each interval and it invoke formatter_csv on the thread. If it use a Thread object as hash key when caching with formatter_csv, that thread will be permanently referenceable and will not be collected by the GC. Here is memory usage of Ruby's process. ![chart](https://github.com/user-attachments/assets/4e74adda-154b-4c00-9cb4-3b1a0e8745fd) ### Reproduce See #3627 ``` <system> process_name fluentd_monitor </system> <source> @type exec tag hardware.memory run_interval 0.1 command /home/watson/prj/sandbox/fluentd/get_memory_metrics.sh <parse> @type json types mem_available:float,swap_total:float,swap_free:float </parse> </source> <match **> @type exec command /home/watson/prj/sandbox/fluentd/psql_memory.sh <format> @type csv force_quotes false delimiter | fields time_local,mem_available,swap_total,swap_free </format> <buffer time> @type file path /home/watson/prj/sandbox/fluentd/log timekey 50s timekey_wait 0s flush_mode lazy total_limit_size 200MB retry_type periodic # exponential backoff is broken #3609 retry_max_times 0 # need to go to secondary as soon as possible #3610 </buffer> </match> ``` **Docs Changes**: **Release Note**: formatter_csv: fix memory leak Signed-off-by: Shizuo Fujita <[email protected]> Signed-off-by: Daijiro Fukuda <[email protected]> Signed-off-by: Kentaro Hayashi <[email protected]> Co-authored-by: Shizuo Fujita <[email protected]> Co-authored-by: Daijiro Fukuda <[email protected]>

Watson1978 force-pushed the formatter_csv branch from 5111400 to 20f5a97 Compare March 14, 2025 05:18

Watson1978 changed the title ~~formatter_csv: release Thread object~~ formatter_csv: release Thread object when terminated Mar 14, 2025

Watson1978 force-pushed the formatter_csv branch from 20f5a97 to e6dc411 Compare March 14, 2025 06:02

Watson1978 changed the title ~~formatter_csv: release Thread object when terminated~~ formatter_csv: use Thread local variable for cache Mar 14, 2025

Watson1978 changed the title ~~formatter_csv: use Thread local variable for cache~~ formatter_csv: fix memory leak by using Thread local variable for cache Mar 14, 2025

formatter_csv: use Thread local variable for cache

9dd9c1e

Signed-off-by: Shizuo Fujita <[email protected]>

Watson1978 force-pushed the formatter_csv branch from e6dc411 to 9dd9c1e Compare March 14, 2025 06:59

Watson1978 marked this pull request as ready for review March 14, 2025 07:22

Watson1978 requested a review from daipom March 14, 2025 07:22

daipom reviewed Mar 14, 2025

View reviewed changes

daipom requested changes Mar 14, 2025

View reviewed changes

Watson1978 requested a review from daipom March 14, 2025 09:19

formatter_csv: use plugin id as key name for thread local variable

b0b1e88

Signed-off-by: Shizuo Fujita <[email protected]>

Watson1978 force-pushed the formatter_csv branch from 7bad1c1 to b0b1e88 Compare March 14, 2025 09:21

daipom reviewed Mar 14, 2025

View reviewed changes

lib/fluent/plugin/formatter_csv.rb Outdated Show resolved Hide resolved

daipom requested changes Mar 14, 2025

View reviewed changes

lib/fluent/plugin/formatter_csv.rb Outdated Show resolved Hide resolved

lib/fluent/plugin/formatter_csv.rb Outdated Show resolved Hide resolved

formatter_csv: fix thread key

5ec4c74

Signed-off-by: Shizuo Fujita <[email protected]>

formatter_csv: fix thread key

71e2c13

Signed-off-by: Shizuo Fujita <[email protected]>

daipom added this to the v1.19.0 milestone Mar 18, 2025

Watson1978 added the backport to v1.16 We will backport this fix to the LTS branch label Mar 18, 2025

formatter_csv: another idea to fix the compat layer issue

80c02fb

Signed-off-by: Daijiro Fukuda <[email protected]>

formatter_csv: use cache only when owner is provided

0b48703

Signed-off-by: Shizuo Fujita <[email protected]>

daipom reviewed Mar 18, 2025

View reviewed changes

lib/fluent/compat/formatter.rb Outdated Show resolved Hide resolved

lib/fluent/plugin/formatter_csv.rb Outdated Show resolved Hide resolved

daipom self-requested a review March 18, 2025 09:13

formatter: remove unnecessary codes

2591868

Co-authored-by: Daijiro Fukuda <[email protected]> Signed-off-by: Shizuo Fujita <[email protected]>

Watson1978 force-pushed the formatter_csv branch from 2c0611e to 2591868 Compare March 18, 2025 10:26

Watson1978 changed the title ~~formatter_csv: fix memory leak by using Thread local variable for cache~~ formatter_csv: fix memory leak Mar 18, 2025

formatter_csv: fix csv_cacheable? method

eae43ca

Signed-off-by: Shizuo Fujita <[email protected]>

daipom approved these changes Mar 19, 2025

View reviewed changes

daipom merged commit 8a4506b into fluent:master Mar 19, 2025
9 of 10 checks passed

Watson1978 deleted the formatter_csv branch March 19, 2025 04:50

kenhys mentioned this pull request Apr 23, 2025

Backport(v1.16): formatter_csv: fix memory leak (#4864) #4918

Closed

kenhys mentioned this pull request Apr 23, 2025

Backport(v1.16): formatter_csv: fix memory leak (#4864) kenhys/fluentd#11

Closed

kenhys mentioned this pull request Apr 23, 2025

Backport(v1.16): formatter_csv: fix memory leak (#4864) #4920

Merged

kenhys added the backported "backport to LTS" is done label Apr 24, 2025

Watson1978 mentioned this pull request Oct 25, 2025

fluentd memory usage in buffer section #5125

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

formatter_csv: fix memory leak #4864

formatter_csv: fix memory leak #4864

Uh oh!

Watson1978 commented Mar 14, 2025 •

edited

Loading

Uh oh!

daipom left a comment

Uh oh!

daipom left a comment

Uh oh!

Watson1978 commented Mar 14, 2025

Uh oh!

Uh oh!

daipom left a comment

Uh oh!

Uh oh!

Uh oh!

Watson1978 commented Mar 17, 2025

Uh oh!

daipom commented Mar 18, 2025

Uh oh!

daipom left a comment

Uh oh!

Uh oh!

Uh oh!

daipom left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	def connection_cache_id_thread_key
	"#{plugin_id}_connection_cache_id"
	end

formatter_csv: fix memory leak #4864

formatter_csv: fix memory leak #4864

Uh oh!

Conversation

Watson1978 commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reproduce

Uh oh!

daipom left a comment

Choose a reason for hiding this comment

Uh oh!

daipom left a comment

Choose a reason for hiding this comment

Uh oh!

Watson1978 commented Mar 14, 2025

Uh oh!

Uh oh!

daipom left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Watson1978 commented Mar 17, 2025

Uh oh!

daipom commented Mar 18, 2025

Uh oh!

daipom left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

daipom left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Watson1978 commented Mar 14, 2025 •

edited

Loading