Skip to content

Conversation

@Watson1978
Copy link
Contributor

@Watson1978 Watson1978 commented Jun 5, 2025

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
Currently, the “pattern not matched” warning is always generated regardless of the log_level setting. This warning might use a lot of memory when the log is very long.

To reduce memory usage, this PR will generate warnings only when necessary, according to log_level.

Related to #4994

  • config
<system>
  log_level error
</system>

<source>
  @type tail
  tag test
  path "#{File.expand_path '~/tmp/fluentd/access-*.log'}"
  refresh_interval 5s
  read_from_head true
  <parse>
    @type regexp
    expression /^(?<log>a+)+$/
    timeout 0.1s
  </parse>
</source>

<match **>
  @type stdout
</match>

Here is memory usage.

Docs Changes:
Not needed.

Release Note:
Same as the title.

@Watson1978 Watson1978 marked this pull request as ready for review June 5, 2025 02:40
@Watson1978 Watson1978 requested review from daipom and kenhys June 5, 2025 02:58
Copy link
Contributor

@kenhys kenhys left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@daipom daipom changed the title Generate warning messages according to log_level with 'pattern not matched' Limit high-memory logging ('pattern not matched' in parsing) to necessary log_level Jun 5, 2025
@daipom daipom changed the title Limit high-memory logging ('pattern not matched' in parsing) to necessary log_level Limit high-memory logging ('pattern not matched' in parsing) to necessary log level Jun 5, 2025
@daipom daipom added this to the v1.19.0 milestone Jun 5, 2025
@daipom
Copy link
Contributor

daipom commented Jun 5, 2025

Thanks!

Note: Long strings are allocated on the heap, so they are likely not released immediately.
It would be better not to generate unnecessary long strings.
(It probably won't affect normal use cases, though)

@daipom daipom merged commit 8769b57 into fluent:master Jun 5, 2025
15 of 16 checks passed
@Watson1978 Watson1978 deleted the pattern-not-matched branch June 5, 2025 05:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants