(https://github.com/ruby/zlib/pull/88)
* Only release the GVL where necessary.
- Several string manipulation methods were invoked while the GVL was
released. This is unsafe.
- The mutex protecting multi-threaded access was not covering buffer state
manipulation, leading to data corruption and out-of-bounds writes.
- Using `rb_str_locktmp` prevents changes to buffer while it's in use.
[Bug #20863]
e445cf3c80
This accounts for a significant number of string allocations when reading rubygems, but we can avoid that in many places by only copying into the outbuf when present
d25ef406c1
extconf.rb
(https://github.com/ruby/zlib/pull/69)
The android NDK (android-ndk-r21e) does not have crc32_z, adler32_z, nor
z_size_t in its zlib.h header file. However, it _does_ have the crc32_z
and adler32_z symbols in the libz.a static library!
mkmf performs two tests for have_func:
* It sees if a program that includes the header and takes the address of
the symbol can compile
* It sees if a program that defines the symbol as `extern void sym_name()`
and calls it can be linked
If either test works, it considers the function present. The
android-ndk-r21e is passing the second test but not the first for
crc32_z/adler32_z. So, we define HAVE_ZLIB_SIZE_T_FUNCS, but then can't
actually compile the extension (since the prototypes aren't in the
header file).
We can keep this working how it was working before by _also_ checking
for `have_type("z_size_t", "zlib.h")`. The have_type check _only_ looks
in the header file for the type; if a program including the header file
and using the type can't compile, the type is considered absent
regardless of what might be in libz.a.
3b9fe962d8
We use the Cloudflare fork of zlib
(https://github.com/cloudflare/zlib), which we find gives improved
performance on AWS Graviton ARM instances. That fork does not define
crc32_z and alder32_z functions.
Until two days ago, Ruby's zlib gem worked fine, because cloudflare zlib
_also_ did not define z_size_t, which meant Ruby did not try and use
these functions.
Since a3ba99596d
however, cloudflare zlib _does_ define z_size_t (but NOT crc32_z or
alder32_z). The zlib gem would try and use these nonexistant
functions and not compile.
This patch fixes it by actually specifically detecting the functions
that the gem wants to call, rather than just the presence of the
z_size_t type.
c96e8b9a57
(https://github.com/ruby/zlib/pull/55)
zstream_discard_input was encoding and character-aware when given input is user-provided, so this discards `len` chars instead of `len` bytes.
Also Zlib.gunzip explains in its rdoc that it is equivalent with the following code, but this doesn't fail for UTF-8 String.
```ruby
string = %w[1f8b0800c28000000003cb48cdc9c9070086a6103605000000].pack("H*").force_encoding('UTF-8')
sio = StringIO.new(string)
p gz.read #=> "hello"
gz&.close
p Zlib.gunzip(string) #=> Zlib::DataError
```
Reported and discovered by eagletmt at https://twitter.com/eagletmt/status/1689692467929694209c5e58bc62a
When Zlib::Inflate#inflate or Zlib::Deflate#deflate is called
recursively inside the block, a crash can occur because of an
use-after-free bug.
50fb8a0338
See also: https://github.com/ruby/csv/issues/117#issuecomment-933289373
How to reproduce with x.csv.gz in the issue comment:
Zlib::GzipReader.open("x.csv.gz") do |rio|
rio.gets(nil, 1024)
while line = rio.gets(nil, 8192)
raise line unless line.valid_encoding?
end
end
Reported by Dimitrij Denissenko. Thanks!!!
b1f182e98f
I'm not sure whether this handles all multithreaded use cases,
but this handles the example that crashes almost immediately
and does 10,000,000 total deflates using 100 separate threads.
To prevent the tests from taking forever, the committed test
for this uses only 10,000 deflates across 10 separate threads,
which still causes a segfault in the previous implementation
almost immediately.
Fixes [Bug #17803]
4b1023b3f2
