archive/ruby
1
0
Fork 0
mirror of https://github.com/ruby/ruby.git synced 2025-08-24 05:25:34 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
88260 commits 30 branches 1174 tags 511 MiB
Commit graph

31 commits

Author SHA1 Message Date
Jean Boussier
165cc6cf40 [ruby/json] json_string_unescape: assume the string doesn't need escaping 
If that assumption holds true, then we don't need to copy the
string into a buffer to unescape it. For small string is just saves
copying, but for large ones it also saves a malloc/free combo.

Before:

```
== Parsing twitter.json (567916 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
                json    52.000 i/100ms
                  oj    61.000 i/100ms
           oj strict    70.000 i/100ms
          Oj::Parser    71.000 i/100ms
           rapidjson    55.000 i/100ms
Calculating -------------------------------------
                json    510.111 (± 2.9%) i/s    (1.96 ms/i) -      2.548k in   5.000029s
                  oj    610.232 (± 3.1%) i/s    (1.64 ms/i) -      3.050k in   5.003725s
           oj strict    713.231 (± 3.2%) i/s    (1.40 ms/i) -      3.570k in   5.010902s
          Oj::Parser    762.598 (± 3.0%) i/s    (1.31 ms/i) -      3.834k in   5.033130s
           rapidjson    553.029 (± 7.4%) i/s    (1.81 ms/i) -      2.750k in   5.022630s

Comparison:
                json:      510.1 i/s
          Oj::Parser:      762.6 i/s - 1.49x  faster
           oj strict:      713.2 i/s - 1.40x  faster
                  oj:      610.2 i/s - 1.20x  faster
           rapidjson:      553.0 i/s - same-ish: difference falls within error

== Parsing citm_catalog.json (1727030 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
                json    28.000 i/100ms
                  oj    33.000 i/100ms
           oj strict    37.000 i/100ms
          Oj::Parser    43.000 i/100ms
           rapidjson    38.000 i/100ms
Calculating -------------------------------------
                json    303.853 (± 3.6%) i/s    (3.29 ms/i) -      1.540k in   5.076079s
                  oj    348.009 (± 2.0%) i/s    (2.87 ms/i) -      1.749k in   5.027738s
           oj strict    396.679 (± 3.3%) i/s    (2.52 ms/i) -      1.998k in   5.042271s
          Oj::Parser    406.699 (± 2.2%) i/s    (2.46 ms/i) -      2.064k in   5.077587s
           rapidjson    393.463 (± 3.3%) i/s    (2.54 ms/i) -      1.976k in   5.028501s

Comparison:
                json:      303.9 i/s
          Oj::Parser:      406.7 i/s - 1.34x  faster
           oj strict:      396.7 i/s - 1.31x  faster
           rapidjson:      393.5 i/s - 1.29x  faster
                  oj:      348.0 i/s - 1.15x  faster
```

After:

```
== Parsing twitter.json (567916 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
                json    56.000 i/100ms
                  oj    62.000 i/100ms
           oj strict    72.000 i/100ms
          Oj::Parser    77.000 i/100ms
           rapidjson    55.000 i/100ms
Calculating -------------------------------------
                json    568.025 (± 2.1%) i/s    (1.76 ms/i) -      2.856k in   5.030272s
                  oj    630.936 (± 1.4%) i/s    (1.58 ms/i) -      3.162k in   5.012630s
           oj strict    705.784 (±11.2%) i/s    (1.42 ms/i) -      3.456k in   5.006706s
          Oj::Parser    783.989 (± 1.7%) i/s    (1.28 ms/i) -      3.927k in   5.010343s
           rapidjson    557.630 (± 2.0%) i/s    (1.79 ms/i) -      2.805k in   5.032388s

Comparison:
                json:      568.0 i/s
          Oj::Parser:      784.0 i/s - 1.38x  faster
           oj strict:      705.8 i/s - 1.24x  faster
                  oj:      630.9 i/s - 1.11x  faster
           rapidjson:      557.6 i/s - same-ish: difference falls within error

== Parsing citm_catalog.json (1727030 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
                json    29.000 i/100ms
                  oj    33.000 i/100ms
           oj strict    38.000 i/100ms
          Oj::Parser    43.000 i/100ms
           rapidjson    37.000 i/100ms
Calculating -------------------------------------
                json    319.271 (± 3.1%) i/s    (3.13 ms/i) -      1.595k in   5.001128s
                  oj    347.946 (± 1.7%) i/s    (2.87 ms/i) -      1.749k in   5.028395s
           oj strict    396.914 (± 3.0%) i/s    (2.52 ms/i) -      2.014k in   5.079645s
          Oj::Parser    409.311 (± 2.7%) i/s    (2.44 ms/i) -      2.064k in   5.046626s
           rapidjson    394.752 (± 1.5%) i/s    (2.53 ms/i) -      1.998k in   5.062776s

Comparison:
                json:      319.3 i/s
          Oj::Parser:      409.3 i/s - 1.28x  faster
           oj strict:      396.9 i/s - 1.24x  faster
           rapidjson:      394.8 i/s - 1.24x  faster
                  oj:      347.9 i/s - 1.09x  faster
```

7e0f66546a
 2024-11-01 13:04:24 +09:00
Jean Boussier
f2b8829df0 Deprecate unsafe default options of JSON.load 
[Feature #19528]

Ref: https://bugs.ruby-lang.org/issues/19528

`load` is understood as the default method for serializer kind of libraries, and
the default options of `JSON.load` has caused many security vulnerabilities over the
years.

The plan is to do like YAML/Psych, deprecate these default options and direct
users toward using `JSON.unsafe_load` so at least it's obvious it should be
used against untrusted data.
 2024-11-01 13:04:24 +09:00
Jean Boussier
5d176436ce [ruby/json] Allocate the FBuffer struct on the stack 
Ref: https://github.com/ruby/json/issues/655

The actual buffer is still on the heap, but this saves a pair
of malloc/free.

This helps a lot on micro-benchmarks

Before:

```
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
                  Oj   531.598k i/100ms
          JSON reuse   417.666k i/100ms
Calculating -------------------------------------
                  Oj      5.735M (± 1.3%) i/s  (174.35 ns/i) -     28.706M in   5.005900s
          JSON reuse      4.604M (± 1.4%) i/s  (217.18 ns/i) -     23.389M in   5.080779s

Comparison:
                  Oj:  5735475.6 i/s
          JSON reuse:  4604380.3 i/s - 1.25x  slower
```

After:

```
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
                  Oj   518.700k i/100ms
          JSON reuse   483.370k i/100ms
Calculating -------------------------------------
                  Oj      5.722M (± 1.8%) i/s  (174.76 ns/i) -     29.047M in   5.077823s
          JSON reuse      5.278M (± 1.5%) i/s  (189.46 ns/i) -     26.585M in   5.038172s

Comparison:
                  Oj:  5722283.8 i/s
          JSON reuse:  5278061.7 i/s - 1.08x  slower
```

Bench:

```ruby
require 'benchmark/ips'
require 'oj'
require 'json'

json_encoder = JSON::State.new(JSON.dump_default_options)
test_data = [1, "string", { a: 1, b: 2 }, [3, 4, 5]]

Oj.default_options = Oj.default_options.merge(mode: :compat)

Benchmark.ips do |x|
  x.config(time: 5, warmup: 2)

  x.report("Oj") do
    Oj.dump(test_data)
  end

  x.report("JSON reuse") do
    json_encoder.generate(test_data)
  end

  x.compare!(order: :baseline)
end
```

72110f7992
 2024-10-30 10:13:48 +09:00
Jean Boussier
a3c21756e9 [ruby/json] Use smaller types for JSON_Parser boolean fields 
7f079b25be
 2024-10-26 18:44:15 +09:00
Jean Boussier
c4d4c6b846 [ruby/json] Speedup Parser initialization 
Extracted from: https://github.com/ruby/json/pull/512

Use `rb_hash_lookup2` to check for hash key existence instead
of going through `rb_funcall`.

43835a0d13

Co-Authored-By: lukeg <luke.gru@gmail.com>
 2024-10-18 11:28:12 +09:00
Jean Boussier
6105bae331 [ruby/json] Get rid of compatibility code for older rubies 
All of these are for rubies older than 2.3.

811297f86a
 2024-10-17 12:22:16 +00:00
Luke T. Shumaker
74d459fd52 [ruby/json] Adjust to the CVTUTF code being gone 
I, Luke T. Shumaker, am the sole author of the added code.

I did not reference CVTUTF when writing it.  I did reference the
Unicode standard (15.0.0), the Wikipedia article on UTF-8, and the
Wikipedia article on UTF-16.  When I saw some tests fail, I did
reference the old deleted code (but a JSON-specific part, inherently
not as based on CVTUTF) to determine that script_safe should also
escape U+2028 and U+2029.

I targeted simplicity and clarity when writing the code--it can likely
be optimized.  In my mind, the obvious next optimization is to have it
combine contiguous non-escaped characters into just one call to
fbuffer_append(), instead of calling fbuffer_append() for each
character.

Regarding the use of the "modern" types `uint32_t`, `uint16_t`, and
`bool`:
 - ruby.h is guaranteed to give us uint32_t and uint16_t.
 - Since Ruby 3.0.0, ruby.h is guaranteed to give us bool... but we
   support down to Ruby 2.3.  But, ruby.h is guaranteed to give us
   HAVE_STDBOOL_H for the C99 stdbool.h; so use that to include
   stdbool.h if we can, and if not then fall back to a copy of the
   same bool definition that Ruby 3.0.5 uses with C89.

c96351f874
 2024-10-08 14:10:05 +09:00
Luke T. Shumaker
6e47968929 [ruby/json] Delete code that is based on CVTUTF 
I did this based on manual inspection, comparing the code to my re-created
history of CVTUTF at https://git.lukeshu.com/2git/cvtutf/ (created by the
scripts at https://git.lukeshu.com/2git/cvtutf-make/)

0819553144
 2024-10-08 14:10:05 +09:00
Jean Boussier
c5ae432ec8
[flori/json] Cleanup useless ifdef 
The json gem now requires Ruby 2.3, so there is no point keeping
compatibility code for older releases that don't have the
TypedData API.

45c86e153f
 2024-06-04 12:23:48 +09:00
Nobuyoshi Nakada
1d170fdc6d
ext/json/parser/parser.h: Add fallback MAYBE_UNUSED 
e2ad91fc20
 2021-05-19 10:16:22 +09:00
Jean Boussier
2de594ca98
[flori/json] Deduplicate strings inside json_string_unescape 
[ci 2]

1982070cb8
 2021-05-17 19:51:51 +09:00
Jean Boussier
1d2b4ccaf2
[flori/json] Refactor json_string_unescape 
f398769332
 2021-05-17 19:51:50 +09:00
Jean Boussier
520e0916af
Implement a freeze: parser option 
If set to true all parsed objects will be
immediately frozen, and strings will be
deduplicated if the Ruby implementation
allows it.
 2020-10-20 21:40:25 +09:00
hsbt
b304078ae6 Merge json-2.1.0 from https://github.com/flori/json 
https://github.com/flori/json/blob/master/CHANGES.md#2017-04-18-210

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59099 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2017-06-16 03:04:46 +00:00
hsbt
1130658738 * ext/json/*, test/json/*: Update json-2.0.1. 
Changes of 2.0.0: f679ebd0c6/CHANGES.md (2015-09-11-200)
  Changes of 2.0.1: f679ebd0c6/CHANGES.md (2016-07-01-201)
  [Feature #12542][ruby-dev:49706][fix GH-1395]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55576 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2016-07-05 11:49:39 +00:00
nobu
cbf902fc33 parser.rl: allocate structs with wrapper 
* ext/json/parser/parser.rl (cJSON_parser_s_allocate): allocate
  structs with making new wrapper objects and get rid of potential
  memory leak.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50660 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2015-05-28 07:17:42 +00:00
hsbt
2e4f0af00f * ext/json/*, test/json/*: Reverted r50231. Because it's not works with 
cross-compile environment.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50267 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2015-04-12 08:36:37 +00:00
hsbt
cfaddc2a32 * ext/json/*, test/json/*, defs/default_gems: Gemify JSON library. 
[fix GH-867][Feature #11057]
* test/ruby/test_extlibs.rb: removed json gem from existence extentions.
* gems/bundled_gems: added json gem into bundled gem.
* lib/rdoc/rubygems_hook.rb: ignored no json environment.
* lib/rubygems/test_case.rb, test/rubygems/*: ditto.
* lib/rdoc/test_case.rb, test/rdoc/*: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50231 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2015-04-11 11:14:36 +00:00
hsbt
f5219fee63 * ext/json: merge upstream from flori/json 
change usage of TypedData. [Feature #10739][ruby-core:67564]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49582 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2015-02-13 05:19:12 +00:00
nobu
5547719573 ext/json: for ancient backward compatibilities 
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49050 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2014-12-28 17:03:36 +00:00
nobu
d29ff24793 json: backward compatibilities 
* ext/json/generator/generator.c (JSON_Generator_State_type): add
  #ifdef for backward compatibility.
* ext/json/parser/parser.rl (JSON_Parser_type): ditto.
* ext/json/generator/generator.h (ZALLOC): add fallback definition.
* ext/json/parser/parser.h (ZALLOC): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49038 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2014-12-27 11:12:58 +00:00
nobu
1e46f02394 json/parser: typed data 
* ext/json/parser/parser.rl (JSON_Parser_type): turn into typed
  data.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47798 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2014-10-04 23:33:11 +00:00
nobu
ba0655ebdd ext: protoize no-arguments functions 
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47793 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2014-10-04 23:29:24 +00:00
naruse
4d8d3184d1 * ext/json: Merge 164a75c8bd2007d32c4d7665d53140d8fc126dcd. 
[ruby-core:41917] [Bug #5846]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@34971 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2012-03-11 13:36:06 +00:00
naruse
b14c060dda * ext/json: Merge json gem 1.5.4+ (2149f4185c598fb97db1). 
[Bug #5173] [ruby-core:38866]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@33122 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2011-08-30 02:23:12 +00:00
naruse
a2e497d5ed * ext/json: Merge json gem 1.5.4+ (f7f78896607b6f6226cd). 
[Bug #4700]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@32493 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2011-07-10 08:01:04 +00:00
nobu
1069e5d665 * ext/json/parser/parser.h (GET_PARSER): raise TypeError. 
* ext/json/parser/parser.rl (cParser_initialize): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30792 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2011-02-05 02:29:18 +00:00
nobu
9cc62abc6b * ext/json/parser/parser.h (GET_PARSER): check if initialized. 
[ruby-core:35079]
* ext/json/parser/parser.rl (cParser_initialize): ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@30791 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2011-02-05 01:30:01 +00:00
naruse
3642494ce5 Recommit of JSON; fix mixed declarations. 
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@27501 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2010-04-26 06:27:27 +00:00
usa
27be3056dc * ext/jason: revert r27493. came again after canceling gcc-ism. 
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@27500 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2010-04-26 04:34:36 +00:00
naruse
54592ad627 * ext/json: Update to JSON 1.4.1. 
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@27493 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
 2010-04-26 00:06:35 +00:00