mirror of
https://github.com/ruby/ruby.git
synced 2025-08-15 13:39:04 +02:00
Revamp the benchmark suite
There is a large number of outstanding performance PRs that I want to merge, but we need a decent benchmark to judge if they are effective. I went to borrow rapidjson's benchmark suite, which is a good start. I only kept the comparison with Oj and RapidJSON, because YAJL is slower on most benchmarks, so little point comparing to it. Encoding: ``` == Encoding small nested array (121 bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 88.225k i/100ms oj 209.862k i/100ms rapidjson 128.978k i/100ms Calculating ------------------------------------- json 914.611k (± 0.4%) i/s (1.09 μs/i) - 4.588M in 5.016099s oj 2.163M (± 0.2%) i/s (462.39 ns/i) - 10.913M in 5.045964s rapidjson 1.392M (± 1.3%) i/s (718.55 ns/i) - 6.965M in 5.005438s Comparison: json: 914610.6 i/s oj: 2162693.5 i/s - 2.36x faster rapidjson: 1391682.6 i/s - 1.52x faster == Encoding small hash (65 bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 142.093k i/100ms oj 651.412k i/100ms rapidjson 237.706k i/100ms Calculating ------------------------------------- json 1.478M (± 0.7%) i/s (676.78 ns/i) - 7.389M in 5.000866s oj 7.150M (± 0.7%) i/s (139.85 ns/i) - 35.828M in 5.010756s rapidjson 2.250M (± 1.6%) i/s (444.46 ns/i) - 11.410M in 5.072451s Comparison: json: 1477595.1 i/s oj: 7150472.0 i/s - 4.84x faster rapidjson: 2249926.7 i/s - 1.52x faster == Encoding twitter.json (466906 bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 101.000 i/100ms oj 223.000 i/100ms rapidjson 105.000 i/100ms Calculating ------------------------------------- json 1.017k (± 0.7%) i/s (982.83 μs/i) - 5.151k in 5.062786s oj 2.244k (± 0.7%) i/s (445.72 μs/i) - 11.373k in 5.069428s rapidjson 1.069k (± 4.6%) i/s (935.20 μs/i) - 5.355k in 5.016652s Comparison: json: 1017.5 i/s oj: 2243.6 i/s - 2.21x faster rapidjson: 1069.3 i/s - same-ish: difference falls within error == Encoding citm_catalog.json (500299 bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 77.000 i/100ms oj 129.000 i/100ms rapidjson 96.000 i/100ms Calculating ------------------------------------- json 767.217 (± 2.5%) i/s (1.30 ms/i) - 3.850k in 5.021957s oj 1.291k (± 1.5%) i/s (774.45 μs/i) - 6.579k in 5.096439s rapidjson 959.527 (± 1.1%) i/s (1.04 ms/i) - 4.800k in 5.003052s Comparison: json: 767.2 i/s oj: 1291.2 i/s - 1.68x faster rapidjson: 959.5 i/s - 1.25x faster == Encoding canada.json (2090234 bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 1.000 i/100ms oj 3.000 i/100ms rapidjson 1.000 i/100ms Calculating ------------------------------------- json 19.748 (± 0.0%) i/s (50.64 ms/i) - 99.000 in 5.013336s oj 31.016 (± 0.0%) i/s (32.24 ms/i) - 156.000 in 5.029732s rapidjson 19.419 (± 0.0%) i/s (51.50 ms/i) - 98.000 in 5.050382s Comparison: json: 19.7 i/s oj: 31.0 i/s - 1.57x faster rapidjson: 19.4 i/s - 1.02x slower == Encoding many #to_json calls (2661 bytes) oj does not match expected output. Skipping rapidjson unsupported (Invalid object key type: Object) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 2.129k i/100ms Calculating ------------------------------------- json 21.599k (± 0.6%) i/s (46.30 μs/i) - 108.579k in 5.027198s ``` Parsing: ``` == Parsing small nested array (121 bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 47.497k i/100ms oj 54.115k i/100ms oj strict 53.854k i/100ms Oj::Parser 150.904k i/100ms rapidjson 80.775k i/100ms Calculating ------------------------------------- json 481.096k (± 1.1%) i/s (2.08 μs/i) - 2.422M in 5.035657s oj 554.878k (± 0.6%) i/s (1.80 μs/i) - 2.814M in 5.071521s oj strict 547.888k (± 0.7%) i/s (1.83 μs/i) - 2.747M in 5.013212s Oj::Parser 1.545M (± 0.4%) i/s (647.16 ns/i) - 7.847M in 5.078302s rapidjson 822.422k (± 0.6%) i/s (1.22 μs/i) - 4.120M in 5.009178s Comparison: json: 481096.4 i/s Oj::Parser: 1545223.5 i/s - 3.21x faster rapidjson: 822422.4 i/s - 1.71x faster oj: 554877.7 i/s - 1.15x faster oj strict: 547887.7 i/s - 1.14x faster == Parsing small hash (65 bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 154.479k i/100ms oj 220.283k i/100ms oj strict 249.928k i/100ms Oj::Parser 445.062k i/100ms rapidjson 289.615k i/100ms Calculating ------------------------------------- json 1.581M (± 3.0%) i/s (632.55 ns/i) - 8.033M in 5.086476s oj 2.202M (± 3.5%) i/s (454.08 ns/i) - 11.014M in 5.008146s oj strict 2.498M (± 3.5%) i/s (400.25 ns/i) - 12.496M in 5.008245s Oj::Parser 4.640M (± 0.4%) i/s (215.50 ns/i) - 23.588M in 5.083443s rapidjson 3.111M (± 0.3%) i/s (321.44 ns/i) - 15.639M in 5.027097s Comparison: json: 1580898.5 i/s Oj::Parser: 4640298.1 i/s - 2.94x faster rapidjson: 3111005.2 i/s - 1.97x faster oj strict: 2498421.4 i/s - 1.58x faster oj: 2202276.6 i/s - 1.39x faster == Parsing test from oj (256 bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 37.580k i/100ms oj 41.899k i/100ms oj strict 50.731k i/100ms Oj::Parser 74.589k i/100ms rapidjson 50.954k i/100ms Calculating ------------------------------------- json 382.150k (± 1.0%) i/s (2.62 μs/i) - 1.917M in 5.015737s oj 420.282k (± 0.2%) i/s (2.38 μs/i) - 2.137M in 5.084338s oj strict 511.758k (± 0.5%) i/s (1.95 μs/i) - 2.587M in 5.055821s Oj::Parser 759.087k (± 0.3%) i/s (1.32 μs/i) - 3.804M in 5.011388s rapidjson 518.273k (± 1.8%) i/s (1.93 μs/i) - 2.599M in 5.015867s Comparison: json: 382149.6 i/s Oj::Parser: 759087.1 i/s - 1.99x faster rapidjson: 518272.8 i/s - 1.36x faster oj strict: 511758.4 i/s - 1.34x faster oj: 420282.5 i/s - 1.10x faster == Parsing twitter.json (567916 bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 52.000 i/100ms oj 63.000 i/100ms oj strict 74.000 i/100ms Oj::Parser 79.000 i/100ms rapidjson 56.000 i/100ms Calculating ------------------------------------- json 522.896 (± 0.4%) i/s (1.91 ms/i) - 2.652k in 5.071809s oj 624.849 (± 0.6%) i/s (1.60 ms/i) - 3.150k in 5.041398s oj strict 737.779 (± 0.4%) i/s (1.36 ms/i) - 3.700k in 5.015117s Oj::Parser 789.254 (± 0.3%) i/s (1.27 ms/i) - 3.950k in 5.004764s rapidjson 565.663 (± 0.4%) i/s (1.77 ms/i) - 2.856k in 5.049015s Comparison: json: 522.9 i/s Oj::Parser: 789.3 i/s - 1.51x faster oj strict: 737.8 i/s - 1.41x faster oj: 624.8 i/s - 1.19x faster rapidjson: 565.7 i/s - 1.08x faster == Parsing citm_catalog.json (1727030 bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 27.000 i/100ms oj 31.000 i/100ms oj strict 36.000 i/100ms Oj::Parser 42.000 i/100ms rapidjson 38.000 i/100ms Calculating ------------------------------------- json 305.248 (± 0.3%) i/s (3.28 ms/i) - 1.539k in 5.041813s oj 320.265 (± 3.4%) i/s (3.12 ms/i) - 1.612k in 5.039715s oj strict 373.701 (± 1.6%) i/s (2.68 ms/i) - 1.872k in 5.010633s Oj::Parser 457.792 (± 0.4%) i/s (2.18 ms/i) - 2.310k in 5.046049s rapidjson 350.933 (± 8.8%) i/s (2.85 ms/i) - 1.748k in 5.052491s Comparison: json: 305.2 i/s Oj::Parser: 457.8 i/s - 1.50x faster oj strict: 373.7 i/s - 1.22x faster rapidjson: 350.9 i/s - 1.15x faster oj: 320.3 i/s - 1.05x faster == Parsing canada.json (2251051bytes) ruby 3.3.4 (2024-07-09 revisionbe1089c8ec) +YJIT [arm64-darwin23] Warming up -------------------------------------- json 2.000 i/100ms oj 2.000 i/100ms oj strict 2.000 i/100ms Oj::Parser 2.000 i/100ms rapidjson 28.000 i/100ms Calculating ------------------------------------- json 29.216 (± 6.8%) i/s (34.23 ms/i) - 146.000 in 5.053753s oj 24.899 (± 0.0%) i/s (40.16 ms/i) - 126.000 in 5.061915s oj strict 24.828 (± 4.0%) i/s (40.28 ms/i) - 124.000 in 5.003067s Oj::Parser 30.867 (± 3.2%) i/s (32.40 ms/i) - 156.000 in 5.057104s rapidjson 285.761 (± 1.0%) i/s (3.50 ms/i) - 1.456k in 5.095715s Comparison: json: 29.2 i/s rapidjson: 285.8 i/s - 9.78x faster Oj::Parser: 30.9 i/s - same-ish: difference falls within error oj: 24.9 i/s - 1.17x slower oj strict: 24.8 i/s - 1.18x slower ```
This commit is contained in:
Jean Boussier
git
parent
7cdbf93a89
commit
6e2619c968
5 changed files with 66052 additions and 0 deletions
9
benchmark/data/canada.json
Normal file
9
benchmark/data/canada.json
Normal file
File diff suppressed because one or more lines are too long
50469
benchmark/data/citm_catalog.json
Normal file
50469
benchmark/data/citm_catalog.json
Normal file
File diff suppressed because it is too large
Load diff
15482
benchmark/data/twitter.json
Normal file
15482
benchmark/data/twitter.json
Normal file
File diff suppressed because it is too large
Load diff
53
benchmark/encoder.rb
Normal file
53
benchmark/encoder.rb
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
require "benchmark/ips"
|
||||
require "json"
|
||||
require "oj"
|
||||
require "rapidjson"
|
||||
|
||||
if ENV["ONLY"]
|
||||
RUN = ENV["ONLY"].split(/[,: ]/).map{|x| [x.to_sym, true] }.to_h
|
||||
RUN.default = false
|
||||
elsif ENV["EXCEPT"]
|
||||
RUN = ENV["EXCEPT"].split(/[,: ]/).map{|x| [x.to_sym, false] }.to_h
|
||||
RUN.default = true
|
||||
else
|
||||
RUN = Hash.new(true)
|
||||
end
|
||||
|
||||
def implementations(ruby_obj)
|
||||
{
|
||||
json: ["json", proc { JSON.dump(ruby_obj) }],
|
||||
oj: ["oj", proc { Oj.dump(ruby_obj) }],
|
||||
rapidjson: ["rapidjson", proc { RapidJSON.dump(ruby_obj) }],
|
||||
}
|
||||
end
|
||||
|
||||
def benchmark_encoding(benchmark_name, ruby_obj, check_expected: true)
|
||||
json_output = JSON.dump(ruby_obj)
|
||||
puts "== Encoding #{benchmark_name} (#{json_output.bytesize} bytes)"
|
||||
|
||||
Benchmark.ips do |x|
|
||||
expected = ::JSON.dump(ruby_obj) if check_expected
|
||||
implementations(ruby_obj).select { |name| RUN[name] }.values.each do |name, block|
|
||||
begin
|
||||
result = block.call
|
||||
if check_expected && expected != result
|
||||
puts "#{name} does not match expected output. Skipping"
|
||||
next
|
||||
end
|
||||
rescue => error
|
||||
puts "#{name} unsupported (#{error})"
|
||||
next
|
||||
end
|
||||
x.report(name, &block)
|
||||
end
|
||||
x.compare!(order: :baseline)
|
||||
end
|
||||
puts
|
||||
end
|
||||
|
||||
benchmark_encoding "small nested array", [[1,2,3,4,5]]*10
|
||||
benchmark_encoding "small hash", { "username" => "jhawthorn", "id" => 123, "event" => "wrote json serializer" }
|
||||
benchmark_encoding "twitter.json", JSON.load_file("#{__dir__}/data/twitter.json")
|
||||
benchmark_encoding "citm_catalog.json", JSON.load_file("#{__dir__}/data/citm_catalog.json")
|
||||
benchmark_encoding "canada.json", JSON.load_file("#{__dir__}/data/canada.json"), check_expected: false
|
||||
benchmark_encoding "many #to_json calls", [{Object.new => Object.new, 12 => 54.3, Integer => Float, Time.now => Date.today}] * 20
|
||||
39
benchmark/parser.rb
Normal file
39
benchmark/parser.rb
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
require "benchmark/ips"
|
||||
require "json"
|
||||
require "oj"
|
||||
require "rapidjson"
|
||||
|
||||
if ENV["ONLY"]
|
||||
RUN = ENV["ONLY"].split(/[,: ]/).map{|x| [x.to_sym, true] }.to_h
|
||||
RUN.default = false
|
||||
elsif ENV["EXCEPT"]
|
||||
RUN = ENV["EXCEPT"].split(/[,: ]/).map{|x| [x.to_sym, false] }.to_h
|
||||
RUN.default = true
|
||||
else
|
||||
RUN = Hash.new(true)
|
||||
end
|
||||
|
||||
def benchmark_parsing(name, json_output)
|
||||
puts "== Parsing #{name} (#{json_output.size} bytes)"
|
||||
|
||||
Benchmark.ips do |x|
|
||||
x.report("json") { JSON.parse(json_output) } if RUN[:json]
|
||||
x.report("oj") { Oj.load(json_output) } if RUN[:oj]
|
||||
x.report("oj strict") { Oj.strict_load(json_output) } if RUN[:oj]
|
||||
x.report("Oj::Parser") { Oj::Parser.usual.parse(json_output) } if RUN[:oj]
|
||||
x.report("rapidjson") { RapidJSON.parse(json_output) } if RUN[:rapidjson]
|
||||
x.compare!(order: :baseline)
|
||||
end
|
||||
puts
|
||||
end
|
||||
|
||||
benchmark_parsing "small nested array", JSON.dump([[1,2,3,4,5]]*10)
|
||||
benchmark_parsing "small hash", JSON.dump({ "username" => "jhawthorn", "id" => 123, "event" => "wrote json serializer" })
|
||||
|
||||
benchmark_parsing "test from oj", <<JSON
|
||||
{"a":"Alpha","b":true,"c":12345,"d":[true,[false,[-123456789,null],3.9676,["Something else.",false],null]],"e":{"zero":null,"one":1,"two":2,"three":[3],"four":[0,1,2,3,4]},"f":null,"h":{"a":{"b":{"c":{"d":{"e":{"f":{"g":null}}}}}}},"i":[[[[[[[null]]]]]]]}
|
||||
JSON
|
||||
|
||||
benchmark_parsing "twitter.json", File.read("#{__dir__}/data/twitter.json")
|
||||
benchmark_parsing "citm_catalog.json", File.read("#{__dir__}/data/citm_catalog.json")
|
||||
benchmark_parsing "canada.json", File.read("#{__dir__}/data/canada.json")
|
||||
Loading…
Add table
Add a link
Reference in a new issue