parser_json: use JSON as fallback parser instead of Yajl for performance #4813

Watson1978 · 2025-01-31T06:43:02Z

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
Recently, Ruby's json has incredible performance improvements.
It might be faster than oj gem.
So, I think json is a suitable as fallback.

This is similar with #4759

Here is easily benchmark. (I used same log file in #4759)

Before
- It spent 90.50963662 sec to handle 10 GB file
After
- It spent 74.624230077 sec to handle 10 GB file
config

<source>
  @type tail
  path "#{File.expand_path '~/tmp/access*.log'}"
  pos_file "#{File.expand_path '~/tmp/fluentd/access.log.pos'}"
  tag log
  read_from_head true
  <parse>
    @type json
  </parse>
</source>

<match **>
  @type file
  path "#{File.expand_path '~/tmp/fluentd/log'}"
</match>

FYI)

Docs Changes:
fluent/fluentd-docs-gitbook#560

Release Note:

kenhys · 2025-01-31T07:08:00Z

we are happy if we could observe micro benchmark result.

Watson1978 · 2025-01-31T07:36:48Z

Micro benchmark

benchmark code

require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'benchmark-ips'
  gem 'oj'
  gem 'json'
  gem 'yajl-ruby', require: 'yajl'
end

Benchmark.ips do |x|
  json = '{"a":"Alpha","b":true,"c":12345,"d":[true,[false,[-123456789,null],3.9676,["Something else.",false],null]],"e":{"zero":null,"one":1,"two":2,"three":[3],"four":[0,1,2,3,4]},"f":null,"h":{"a":{"b":{"c":{"d":{"e":{"f":{"g":null}}}}}}},"i":[[[[[[[null]]]]]]]}'

  x.report('Oj.load') { Oj.load(json) }
  x.report('JSON.load') { JSON.load(json) }
  x.report('Yajl.load') { Yajl.load(json) }

  x.compare!
end

result

ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [x86_64-linux]
Warming up --------------------------------------
             Oj.load    31.383k i/100ms
           JSON.load    35.513k i/100ms
           Yajl.load    13.839k i/100ms
Calculating -------------------------------------
             Oj.load    319.187k (± 1.5%) i/s    (3.13 μs/i) -      1.601M in   5.015522s
           JSON.load    364.950k (± 0.5%) i/s    (2.74 μs/i) -      1.847M in   5.060214s
           Yajl.load    137.015k (± 2.5%) i/s    (7.30 μs/i) -    691.950k in   5.053679s

Comparison:
           JSON.load:   364950.5 i/s
             Oj.load:   319187.1 i/s - 1.14x  slower
           Yajl.load:   137015.2 i/s - 2.66x  slower

kenhys · 2025-01-31T09:36:22Z

By the way, which json gem version did you tested with?
Do we need to use specific version of json gem?

Watson1978 · 2025-01-31T09:46:15Z

By the way, which json gem version did you tested with? Do we need to use specific version of json gem?

I tested with json v2.9.1 (latest version), it is default version in Ruby 3.4.1
https://github.com/ruby/ruby/blob/master/doc/NEWS/NEWS-3.4.0.md

If we will use Ruby 3.4 or later in the next fluent-package LTS, hmmm, I think we don't need to specify the version...

test/plugin/test_parser_json.rb

kenhys · 2025-02-03T03:14:48Z

NOTE: will not be backported to v1.16

lib/fluent/plugin/parser_json.rb

Signed-off-by: Shizuo Fujita <[email protected]>

daipom

LGTM. Thanks!!
Waiting CI.

Watson1978 · 2025-02-03T08:28:59Z

I created separated PR in #4817

**Which issue(s) this PR fixes**: Fixes # **What this PR does / why we need it**: Ref. #4813 (comment) Fix wrong usage of `JSON.load` method. We should use `JSON.parse` instead. JSON.load performs unnecessary deserialisation. ``` irb(main):001> JSON.load('{ "json_class": "String", "raw": [72, 101, 108, 108, 111] }') => "Hello" irb(main):002> JSON.parse('{ "json_class": "String", "raw": [72, 101, 108, 108, 111] }') => {"json_class" => "String", "raw" => [72, 101, 108, 108, 111]} ``` **Docs Changes**: **Release Note**: Signed-off-by: Shizuo Fujita <[email protected]>

@type

…rmance (#4835) **Which issue(s) this PR fixes**: Fixes # **What this PR does / why we need it**: Recently, Ruby's json has incredible performance improvements. It might be faster than oj gem. So, I think json is a suitable as fallback. This is similar with #4813 * Before * It spent 70.708872503 sec to handle 10 GB file * After * It spent 54.428017716 sec to handle 10 GB file * config ``` <source> @type tail path "#{File.expand_path('~/tmp/fluentd/access*.log')}" pos_file "#{File.expand_path('~/tmp/fluentd/access.log.pos')}" tag log read_from_head true <parse> @type none </parse> </source> <match **> @type file path "#{File.expand_path('~/tmp/fluentd/output/log')}" # use formatter_json format json </match> ``` **Docs Changes**: **Release Note**: --------- Signed-off-by: Shizuo Fujita <[email protected]>

orgads · 2025-09-10T21:08:44Z

Following this, can yajl-ruby be a development_dependency, so that it is not installed by default?

Watson1978 · 2025-09-11T01:10:36Z

The json gem cannot handle stream-based JSON using IO objects, so we still need yajl.

byroot · 2025-09-11T07:55:21Z

The json gem cannot handle stream-based JSON using IO objects

Could you clarify what you mean by streaming here? Is it SAX style parsing ? Or JSONL ? Other ?

I tried searching for YAJL use in that repo, but saw nothing like that.

orgads · 2025-09-11T07:58:27Z

@byroot See in_forward.rb for example. The input is read from a socket in chunks.

Watson1978 · 2025-09-11T08:00:05Z

Could you clarify what you mean by streaming here? Is it SAX style parsing ? Or JSONL ? Other ?

Ah, sorry. It is just IO stream.

byroot · 2025-09-11T08:28:17Z

Ah, sorry. It is just IO stream.

This seem to be the code in question:

fluentd/lib/fluent/plugin/in_forward.rb

Lines 250 to 257 in d63d193

    
           if first == '{' || first == '[' # json 
        
             parser = Yajl::Parser.new 
        
             parser.on_parse_complete = ->(obj){ 
        
               block.call(obj, bytes, serializer) 
        
               bytes = 0 
        
             } 
        
             serializer = :to_json.to_proc 
        
             feeder = ->(d){ parser << d }

But just looking at the code, I'm not 100% sure I get what the format it. Seems to be JSONL ? e.g.

{"foo": 1}
{"foo": 2}

Is that correct?

Watson1978 · 2025-09-11T08:55:56Z

For example, we parse a JSON from an IO stream in the following.

fluentd/lib/fluent/plugin/parser_json.rb

Lines 96 to 102 in d63d193

    
           def parse_io(io, &block) 
        
             y = Yajl::Parser.new 
        
             y.on_parse_complete = ->(record){ 
        
               block.call(parse_time(record), record) 
        
             } 
        
             y.parse(io, @stream_buffer_size) 
        
           end

byroot · 2025-09-11T09:00:44Z

@Watson1978 yes, I saw that, but just the code doesn't really help me. What I'd like to see is the content of that io. Is there a test case perhaps?

Watson1978 · 2025-09-11T09:19:07Z

@byroot We have simple test case for that in

fluentd/test/plugin/test_parser_json.rb

Lines 146 to 168 in d63d193

    
           def test_yajl_parse_io_with_buffer_smaller_than_input 
        
             parser = Fluent::Test::Driver::Parser.new(Fluent::Plugin::JSONParser) 
        
             parser.configure( 
        
                              'keep_time_key' => 'true', 
        
                              'json_parser' => 'yajl', 
        
                              'stream_buffer_size' => 1, 
        
                              ) 
        
             text = "100" 
        
             waiting(5) do 
        
               rd, wr = IO.pipe 
        
               wr.write "{\"time\":\"#{text}\"}" 
        
               parser.instance.parse_io(rd) do |time, record| 
        
                 assert_equal text.to_i, time.sec 
        
                 assert_equal text, record['time'] 
        
                 # Once a record has been received the 'write' end of the pipe must be 
        
                 # closed, otherwise the test will block waiting for more input. 
        
                 wr.close 
        
               end 
        
             end 
        
           end

require 'yajl'

y = Yajl::Parser.new
y.on_parse_complete = ->(record){
  puts "Finished parsing: #{record}"
}

rio, wio = IO.pipe
wio.write '{"message": "hello"}'
wio.close

y.parse(rio)
rio.close

byroot · 2025-09-11T12:18:51Z

Hum, right, that that's a single JSON document. What I'm trying to understand is whether it's expected that the pipe may contain multiple documents one after the other. That's what I think I understand, but I'll try to play a bit more with YAJL, to see what it offer.

Watson1978 mentioned this pull request Jan 31, 2025

parser_json: use JSON as fallback parser instead of Yajl fluent/fluentd-docs-gitbook#560

Merged

Watson1978 requested a review from daipom January 31, 2025 06:49

Watson1978 mentioned this pull request Jan 31, 2025

File watchers might not be handled properly causing gradual increase in CPU/Memory usage #4381

Open

Watson1978 force-pushed the parser_json branch from 2078284 to 9b3887f Compare January 31, 2025 08:10

byroot reviewed Jan 31, 2025

View reviewed changes

test/plugin/test_parser_json.rb Show resolved Hide resolved

Watson1978 force-pushed the parser_json branch 2 times, most recently from 9968af9 to bfc027e Compare February 3, 2025 00:57

daipom reviewed Feb 3, 2025

View reviewed changes

lib/fluent/plugin/parser_json.rb Outdated Show resolved Hide resolved

parser_json: use JSON as fallback parser instead of Yajl for performance

aa67069

Signed-off-by: Shizuo Fujita <[email protected]>

Watson1978 force-pushed the parser_json branch from bfc027e to aa67069 Compare February 3, 2025 07:06

daipom approved these changes Feb 3, 2025

View reviewed changes

daipom added this to the v1.19.0 milestone Feb 3, 2025

daipom merged commit 1d5b0de into fluent:master Feb 3, 2025
10 checks passed

Watson1978 deleted the parser_json branch February 3, 2025 08:25

Watson1978 mentioned this pull request Feb 3, 2025

parser_json: use JSON.parse instead of .load #4817

Merged

Watson1978 mentioned this pull request Feb 18, 2025

formatter_json: use JSON as fallback parser instead of Yajl for performance #4835

Merged

parser_json: use JSON as fallback parser instead of Yajl for performance #4813

parser_json: use JSON as fallback parser instead of Yajl for performance #4813

Uh oh!

Conversation

Watson1978 commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kenhys commented Jan 31, 2025

Uh oh!

Watson1978 commented Jan 31, 2025

Micro benchmark

benchmark code

result

Uh oh!

kenhys commented Jan 31, 2025

Uh oh!

Watson1978 commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kenhys commented Feb 3, 2025

Uh oh!

Uh oh!

daipom left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Watson1978 commented Feb 3, 2025

Uh oh!

orgads commented Sep 10, 2025

Uh oh!

Watson1978 commented Sep 11, 2025

Uh oh!

byroot commented Sep 11, 2025

Uh oh!

orgads commented Sep 11, 2025

Uh oh!

Watson1978 commented Sep 11, 2025

Uh oh!

byroot commented Sep 11, 2025

Uh oh!

Watson1978 commented Sep 11, 2025

Uh oh!

byroot commented Sep 11, 2025

Uh oh!

Watson1978 commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

byroot commented Sep 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Watson1978 commented Jan 31, 2025 •

edited

Loading

Watson1978 commented Jan 31, 2025 •

edited

Loading

Watson1978 commented Sep 11, 2025 •

edited

Loading