[Bug #19896]
fix memory leak in vm_method
This introduces a unified reference_count to clarify who is referencing a method.
This also allows us to treat the refinement method as the def owner since it counts itself as a reference
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
---
gc.c | 4 +-
method.h | 6 +--
rjit_c.rb | 6 +--
test/ruby/test_module.rb | 4 +-
vm_insnhelper.c | 2 +-
vm_method.c | 105 +++++++++++++++++++----------------------------
6 files changed, 54 insertions(+), 73 deletions(-)
[Bug #19778] Add `-I` options for opt-dir to `$INCFLAGS`
These options have been separated from `$CFLAGS` already in the
other places.
---
configure.ac | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Fix premature string collection during append
Previously, the following crashed due to use-after-free
with AArch64 Alpine Linux 3.18.3 (aarch64-linux-musl):
```ruby
str = 'a' * (32*1024*1024)
p({z: str})
```
32 MiB is the default for `GC_MALLOC_LIMIT_MAX`, and the crash
could be dodged by setting `RUBY_GC_MALLOC_LIMIT_MAX` to large values.
Under a debugger, one can see the `str2` of rb_str_buf_append()
getting prematurely collected while str_buf_cat4() allocates capacity.
Add GC guards so the buffer of `str2` lives across the GC run
initiated in str_buf_cat4().
[Bug #19792]
---
string.c | 2 ++
1 file changed, 2 insertions(+)
Fix memory leak in complemented method entries
[Bug #19894]
When a copy of a complemented method entry is created, there are two
issues:
1. IMEMO_FL_USER3 is not copied, so the complemented status is not
copied over.
2. In rb_method_entry_clone we increment both alias_count and
complemented_count. However, when we free the method entry in
rb_method_definition_release, we only decrement one of the two
counters, resulting in the rb_method_definition_t being leaked.
Co-authored-by: Adam Hess <adamhess1991@gmail.com>
---
method.h | 5 +++--
test/ruby/test_module.rb | 29 +++++++++++++++++++++++++++++
vm_method.c | 8 +++++---
3 files changed, 37 insertions(+), 5 deletions(-)
Fix memory leak for incomplete lambdas
[Bug #19836]
The parser does not free the chain of `struct vtable`, which causes
memory leaks.
The following script reproduces this issue:
```
10.times do
100_000.times do
Ripper.parse("-> {")
end
puts `ps -o rss= -p #{$$}`
end
```
---
parse.y | 24 ++++++++++++++----------
test/ripper/test_ripper.rb | 7 +++++++
2 files changed, 21 insertions(+), 10 deletions(-)
Fix memory leak in parser for incomplete tokens
[Bug #19835]
The parser does not free the `tbl` of the `struct vtable` when there are
leftover `lvtbl` in the parser. This causes a memory leak.
The following script reproduces this issue:
```
10.times do
100_000.times do
Ripper.parse("class Foo")
end
puts `ps -o rss= -p #{$$}`
end
```
---
parse.y | 42 ++++++++++++++++++++++++++++--------------
test/ripper/test_ripper.rb | 7 +++++++
2 files changed, 35 insertions(+), 14 deletions(-)
Skip running brew upgrade (#8189)
This has been unstable:
1571398859
and I'm not sure if we need that in the first place, assuming the
OS image itself is maintained by GitHub.
---
.github/actions/setup/macos/action.yml | 1 -
1 file changed, 1 deletion(-)
Fix crash in NoMethodError for dummy frames
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
[Bug #19793]
Dummy frames are created at the top level when requiring another file.
While requiring a file, it will try to convert using encodings. Some of
these encodings will not respond to to_str. If method_missing is
redefined on Object, then it will call method_missing and attempt raise
an error. However, the iseq is invalid as it's a dummy frame so it will
write an invalid iseq to the created NoMethodError.
The following script crashes:
```
GC.stress = true
class Object
public :method_missing
end
File.write("/tmp/empty.rb", "")
require "/tmp/empty.rb"
```
With the following backtrace:
```
frame #0: 0x00000001000fa8b8 miniruby`RVALUE_MARKED(obj=4308637824) at gc.c:1638:12
frame #1: 0x00000001000fb440 miniruby`RVALUE_BLACK_P(obj=4308637824) at gc.c:1763:12
frame #2: 0x00000001000facdc miniruby`gc_writebarrier_incremental(a=4308637824, b=4308332208, objspace=0x000000010180b000) at gc.c:8822:9
frame #3: 0x00000001000faad8 miniruby`rb_gc_writebarrier(a=4308637824, b=4308332208) at gc.c:8864:17
frame #4: 0x000000010016aff0 miniruby`rb_obj_written(a=4308637824, oldv=36, b=4308332208, filename="../iseq.c", line=1279) at gc.h:804:9
frame #5: 0x0000000100162a60 miniruby`rb_obj_write(a=4308637824, slot=0x0000000100d09888, b=4308332208, filename="../iseq.c", line=1279) at gc.h:837:5
frame #6: 0x0000000100165b0c miniruby`iseqw_new(iseq=0x0000000100d09880) at iseq.c:1279:9
frame #7: 0x0000000100165a64 miniruby`rb_iseqw_new(iseq=0x0000000100d09880) at iseq.c:1289:12
frame #8: 0x00000001000d8324 miniruby`name_err_init_attr(exc=4309777920, recv=4304780496, method=827660) at error.c:1830:35
frame #9: 0x00000001000d1b80 miniruby`name_err_init(exc=4309777920, mesg=4308332496, recv=4304780496, method=827660) at error.c:1869:12
frame #10: 0x00000001000d1bd4 miniruby`rb_nomethod_err_new(mesg=4308332496, recv=4304780496, method=827660, args=4308332448, priv=0) at error.c:1957:5
frame #11: 0x000000010039049c miniruby`rb_make_no_method_exception(exc=4304914512, format=4308332496, obj=4304780496, argc=1, argv=0x000000016fdfab00, priv=0) at vm_eval.c:959:16
frame #12: 0x00000001003b3274 miniruby`raise_method_missing(ec=0x0000000100b06f40, argc=1, argv=0x000000016fdfab00, obj=4304780496, last_call_status=MISSING_NOENTRY) at vm_eval.c:999:15
frame #13: 0x00000001003945d4 miniruby`rb_method_missing(argc=1, argv=0x000000016fdfab00, obj=4304780496) at vm_eval.c:944:5
...
frame #23: 0x000000010038f5e4 miniruby`rb_vm_call_kw(ec=0x0000000100b06f40, recv=4304780496, id=2865, argc=1, argv=0x000000016fdfab00, me=0x0000000100cbfcf0, kw_splat=0) at vm_eval.c:326:12
frame #24: 0x00000001003c18e4 miniruby`call_method_entry(ec=0x0000000100b06f40, defined_class=4304927952, obj=4304780496, id=2865, cme=0x0000000100cbfcf0, argc=1, argv=0x000000016fdfab00, kw_splat=0) at vm_method.c:2720:20
frame #25: 0x00000001003c440c miniruby`check_funcall_exec(v=6171896792) at vm_eval.c:589:12
frame #26: 0x00000001000dec00 miniruby`rb_vrescue2(b_proc=(miniruby`check_funcall_exec at vm_eval.c:587), data1=6171896792, r_proc=(miniruby`check_funcall_failed at vm_eval.c:596), data2=6171896792, args="Pȗ") at eval.c:919:18
frame #27: 0x00000001000deab0 miniruby`rb_rescue2(b_proc=(miniruby`check_funcall_exec at vm_eval.c:587), data1=6171896792, r_proc=(miniruby`check_funcall_failed at vm_eval.c:596), data2=6171896792) at eval.c:900:17
frame #28: 0x000000010039008c miniruby`check_funcall_missing(ec=0x0000000100b06f40, klass=4304923536, recv=4304780496, mid=3233, argc=0, argv=0x0000000000000000, respond=-1, def=36, kw_splat=0) at vm_eval.c:666:15
frame #29: 0x000000010038fa60 miniruby`rb_check_funcall_default_kw(recv=4304780496, mid=3233, argc=0, argv=0x0000000000000000, def=36, kw_splat=0) at vm_eval.c:703:21
frame #30: 0x000000010038fb04 miniruby`rb_check_funcall(recv=4304780496, mid=3233, argc=0, argv=0x0000000000000000) at vm_eval.c:685:12
frame #31: 0x00000001001c469c miniruby`convert_type_with_id(val=4304780496, tname="String", method=3233, raise=0, index=-1) at object.c:3061:15
frame #32: 0x00000001001c4a4c miniruby`rb_check_convert_type_with_id(val=4304780496, type=5, tname="String", method=3233) at object.c:3153:9
frame #33: 0x00000001002d59f8 miniruby`rb_check_string_type(str=4304780496) at string.c:2571:11
frame #34: 0x000000010014b7b0 miniruby`io_encoding_set(fptr=0x0000000100d09ca0, v1=4304780496, v2=4, opt=4) at io.c:11655:19
frame #35: 0x0000000100139a58 miniruby`rb_io_set_encoding(argc=1, argv=0x000000016fdfb450, io=4308334032) at io.c:13497:5
frame #36: 0x00000001003c0004 miniruby`ractor_safe_call_cfunc_m1(recv=4308334032, argc=1, argv=0x000000016fdfb450, func=(miniruby`rb_io_set_encoding at io.c:13487)) at vm_insnhelper.c:3271:12
...
frame #43: 0x0000000100390b08 miniruby`rb_funcall(recv=4308334032, mid=16593, n=1) at vm_eval.c:1137:12
frame #44: 0x00000001002a43d8 miniruby`load_file_internal(argp_v=6171899936) at ruby.c:2500:5
...
```
---
error.c | 4 +++-
test/ruby/test_require.rb | 15 +++++++++++++++
2 files changed, 18 insertions(+), 1 deletion(-)
addr2line.c: fix `DW_FORM_ref_addr` parsing for DWARF 2 (#8146)
addr2line.c: fix DW_FORM_ref_addr parsing for DWARF 2
This fixes a crash when retrieving backtrace info with YJIT enabled on
macOS with Rust 1.71.0. Since Rust 1.71.0, the DWARF info generated by
the Rust compiler uses DW_FORM_ref_addr instead of DW_FORM_ref4 for
pointers to other DIEs.
DW_FORM_ref_addr representation in DWARF 2 is different from DWARF 3+,
so we need to handle it separately.
This patch fixes the parsing of DW_FORM_ref_addr for DWARF 2, which is
the default DWARF version Rustc uses on macOS.
See the DWARF 2.0.0 spec, section 7.5.4 Attribute Encodings
https://dwarfstd.org/doc/dwarf-2.0.0.pdfhttps://bugs.ruby-lang.org/issues/19789
---
addr2line.c | 40 ++++++++++++++++++++++++++--------------
1 file changed, 26 insertions(+), 14 deletions(-)
Fix autosplat conditions to handle ruby2_keywords case
Autosplat should not occur if there are two arguments but second
argument is an array containing a ruby2_keywords splat. Only
autosplat if a single argument to be yielded to the block, and there
is no splatted flagged keyword hash passed.
Fixes [Bug #19759]
---
test/ruby/test_proc.rb | 26 ++++++++++++++++++++++++++
vm_args.c | 3 ++-
2 files changed, 28 insertions(+), 1 deletion(-)
Use tools appropriate with CC
To get rid of mysterious errors such as:
```
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm: error: libruby.3.3-static.a(/): The end of the file was unexpectedly encountered
```
and
```
ld: warning: ignoring file ../../libruby.3.3-static.a, building for macOS-x86_64 but attempting to link with file built for macOS-x86_64
Undefined symbols for architecture x86_64:
"_rb_rational_num", referenced from:
```
---
configure.ac | 6 ++++++
1 file changed, 6 insertions(+)
Use an st table for "too complex" objects
st tables will maintain insertion order so we can marshal dump / load
objects with instance variables in the same order they were set on that
particular instance
[ruby-core:112926] [Bug #19535]
Co-Authored-By: Jemma Issroff <jemmaissroff@gmail.com>
---
gc.c | 10 ++++------
include/ruby/st.h | 2 ++
object.c | 2 +-
ractor.c | 43 ++++++++++++++++++++++---------------------
shape.h | 6 +++---
st.c | 6 ++++++
test/ruby/test_shapes.rb | 21 +++++++++++++++++++++
variable.c | 28 ++++++++++++++--------------
vm_insnhelper.c | 2 +-
9 files changed, 74 insertions(+), 46 deletions(-)
Fix `PLATFORM_GET_INC`
On platforms where unaligned word access is not allowed, and if
`sizeof(val)` and `sizeof(type)` differ:
- `val` > `type`, `val` will be a garbage.
- `val` < `type`, outside `val` will be clobbered.
---
regexec.c | 2 +-
regint.h | 9 ++++++++-
2 files changed, 9 insertions(+), 2 deletions(-)
ObjectSpace::WeakMap: clean inverse reference when an entry is
re-assigned
[Bug #19531]
```ruby
wmap[1] = "A"
wmap[1] = "B"
```
In the example above, we need to remove the `"A" => 1` inverse reference
so that when `"A"` is GCed the `1` key isn't deleted.
---
test/ruby/test_weakmap.rb | 17 +++++++++
weakmap.c | 91 ++++++++++++++++++++++++++++++++++++++---------
2 files changed, 91 insertions(+), 17 deletions(-)
Handle unterminated unicode escapes in regexps
This fixes an infinite loop possible after ec3542229b.
For \u{} escapes in regexps, skip validation in the parser, and rely on the regexp
code to handle validation. This is necessary so that invalid unicode escapes in
comments in extended regexps are allowed.
Fixes [Bug #19750]
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
---
parse.y | 97 ++++++++++++++++++++++++++++++++-----------------
test/ruby/test_parse.rb | 16 ++++++++
2 files changed, 79 insertions(+), 34 deletions(-)
YJIT: Fix false object collection when setting ivar
Previously, setinstancevariable could generate code that calls
`rb_ensure_iv_list_size()` without first updating `cfp->sp`. This means
in the event that a GC start from within said routine the top few
objects would not be marked, causing them to be falsly collected.
Call `jit_prepare_routine_call()` first.
[Bug #19601]
---
bootstraptest/test_yjit.rb | 20 ++++++++++++++++++++
yjit/src/codegen.rs | 5 +++++
2 files changed, 25 insertions(+)
YJIT: Remove duplicate `asm.spill_temps()`
`jit_prepare_routine_call()` calls it, and there is another call above on line 2302.
Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
---
yjit/src/codegen.rs | 1 -
1 file changed, 1 deletion(-)
Ensure throw data is not set as cause
[Bug #19593]
rb_ec_setup_exception did not check if errinfo is a throw_data. This can
cause crashes in code since it is assumed that id_cause is an object.
We saw a crash in show_cause due to id_cause of errinfo being a
throw_data. It crashes on rb_obj_is_kind_of since it cannot be called on
T_IMEMO objects.
Unfortunately, we couldn't find a reproduction script, however we
debugged the core dump and rb_ec_setup_exception is the only place where
id_cause is assigned from errinfo without checking if it is a
throw_data.
```
0x0000556c5708e6dd in sigsegv (sig=11, info=0x7f301befa3f0, ctx=0x7f301befa2c0) at signal.c:964
0x00007f301d046420 in <signal handler called> () at /lib/x86_64-linux-gnu/libpthread.so.0
class_search_class_ancestor (c=139844586301760, cl=<optimized out>) at object.c:810
rb_obj_is_kind_of (obj=obj@entry=139839221734880, c=139844586301760) at object.c:861
0x0000556c56f2f00f in show_cause
(errinfo=errinfo@entry=139838840645160, str=str@entry=139839221730520, opt=139839221730480, highlight=0, reverse=reverse@entry=0, backtrace_limit=backtrace_limit@entry=-1, shown_causes=0x7ffe9d1a2d68) at ./include/ruby/internal/special_consts.h:175
```
Co-Authored-By: Jean Boussier <byroot@ruby-lang.org>
---
eval.c | 4 ++++
1 file changed, 4 insertions(+)
hash.c: Fix hash_iter_lev_dec corrupting shape
[Bug #19589]
When decrementing `iter_lev` from `65` to `64` the flags would be
corrupted, causing the shape_id to be invalid.
---
hash.c | 12 +++++++++---
test/ruby/test_hash.rb | 11 +++++++++++
2 files changed, 20 insertions(+), 3 deletions(-)
YJIT: Fix missing argc check in known cfuncs
Previously we were missing a compile-time check that the known cfuncs
receive the correct number of arguments.
We noticied this because in particular when using ARGS_SPLAT, which also
wasn't checked, YJIT would crash on code which was otherwise correct
(didn't raise exceptions in the VM).
This still supports vararg (argc == -1) cfuncs. I added an additional
assertion that when we use the specialized codegen for one of these
known functions that the argc are popped off the stack correctly, which
should help ensure they're implemented correctly (previously the crash
was usually observed on a future `leave` insn).
[Bug #19595]
---
bootstraptest/test_yjit.rb | 32 ++++++++++++++++++++++++++++++++
yjit/src/codegen.rs | 4 +++-
2 files changed, 35 insertions(+), 1 deletion(-)
Check if Bundler is defined
---
tool/lib/leakchecker.rb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Check if `Bundler::EnvironmentPreserver` is defined
Only `Bundler` might be defined. `EnvironmentPreserver` and its
`BUNDLER_PREFIX` would be defined together in the same file.
---
tool/lib/leakchecker.rb | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
Ensure ruby_xfree won't segfault if called after vm_destruct
[Bug #19580]
The real-world scenario motivating this change is libxml2's pthread
code which uses `pthread_key_create` to set up a destructor that is
called at thread exit to free thread-local storage.
There is a small window of time -- after ruby_vm_destruct but before
the process exits -- in which a pthread may exit and the destructor is
called, leading to a segfault.
Please note that this window of time may be relatively large if
`atexit` is being used.
---
gc.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
Fix write barrier order for `klass` to `cme` edge
Previously, the following crashes with
`CFLAGS=-DRGENGC_CHECK_MODE=2 -DRUBY_DEBUG=1 -fno-inline`:
$ ./miniruby -e 'GC.stress = true; Marshal.dump({})'
It crashes with a write barrier (WB) miss assertion on an edge from the
`Hash` class object to a newly allocated negative method entry.
This is due to usages of vm_ccs_create() running the WB too early,
before the method entry is inserted into the cc table, so before the
reference edge is established. The insertion can trigger GC and promote
the class object, so running the WB after the insertion is necessary.
Move the insertion into vm_ccs_create() and run the WB after the
insertion.
Discovered on CI:
http://ci.rvm.jp/results/trunk-asserts@ruby-sp2-docker/4391770
---
vm_eval.c | 3 +--
vm_insnhelper.c | 10 ++++++----
vm_method.c | 3 +--
3 files changed, 8 insertions(+), 8 deletions(-)
Fix crash in Time on 32-bit systems
[Bug #19575]
struct vtm is packed causing it to have a size that is not aligned on
32-bit systems. When allocating it on the stack, it will have unaligned
addresses which means that the fields won't be marked by the GC when
scanning the stack (since the GC only marks aligned addresses). This can
cause crashes when the fields are heap allocated objects like Bignums.
This commit moves the flags in struct time_object into struct vtm for
space efficiency and removes the need for packing.
This is an example of a crash:
ruby(rb_print_backtrace+0xd) [0x56848945] ../src/vm_dump.c:785
ruby(rb_vm_bugreport) ../src/vm_dump.c:1101
ruby(rb_assert_failure+0x7a) [0x56671857] ../src/error.c:878
ruby(vm_search_cc+0x0) [0x56666e47] ../src/vm_method.c:1366
ruby(rb_vm_search_method_slowpath) ../src/vm_insnhelper.c:2090
ruby(callable_method_entry+0x5) [0x568232d3] ../src/vm_method.c:1406
ruby(rb_callable_method_entry) ../src/vm_method.c:1413
ruby(gccct_method_search_slowpath) ../src/vm_eval.c:427
ruby(gccct_method_search+0x20f) [0x568237ef] ../src/vm_eval.c:476
ruby(opt_equality_by_mid_slowpath+0x2c) [0x5682388c] ../src/vm_insnhelper.c:2338
ruby(rb_equal+0x37) [0x566fe577] ../src/object.c:133
ruby(rb_big_eq+0x34) [0x56876ee4] ../src/bignum.c:5554
ruby(rb_int_equal+0x14) [0x566f3ed4] ../src/numeric.c:4640
ruby(rb_int_equal) ../src/numeric.c:4634
ruby(vm_call0_cfunc_with_frame+0x6d) [0x568303c2] ../src/vm_eval.c:148
ruby(vm_call0_cfunc) ../src/vm_eval.c:162
ruby(vm_call0_body) ../src/vm_eval.c:208
ruby(rb_funcallv_scope+0xd1) [0x56833971] ../src/vm_eval.c:85
ruby(RB_TEST+0x0) [0x567e8488] ../src/time.c:78
ruby(eq) ../src/time.c:78
ruby(small_vtm_sub) ../src/time.c:1523
ruby(timelocalw+0x23b) [0x567f3e9b] ../src/time.c:1593
ruby(time_s_alloc+0x0) [0x567f536b] ../src/time.c:3698
ruby(time_new_timew) ../src/time.c:2694
ruby(time_s_mktime) ../src/time.c:3698
---
test/ruby/test_time.rb | 7 ++-----
time.c | 57 ++++++++++++++++++++++++--------------------------
timev.h | 7 +++++--
3 files changed, 34 insertions(+), 37 deletions(-)
Fix memory leak for iclass
[Bug #19550]
If !RCLASS_EXT_EMBEDDED (e.g. 32 bit systems) then the rb_classext_t is
allocated throug malloc so it must be freed.
The issue can be seen in the following script:
```
20.times do
100_000.times do
mod = Module.new
Class.new do
include mod
end
end
# Output the Resident Set Size (memory usage, in KB) of the current Ruby process
puts `ps -o rss= -p #{$$}`
end
```
Before this fix, the max RSS is 280MB, while after this change, it's
30MB.
---
gc.c | 2 +-
test/ruby/test_module.rb | 15 +++++++++++++++
2 files changed, 16 insertions(+), 1 deletion(-)
[Cirrus] Run only when YJIT sources change [skip appveyor]
[ci skip] on other CIs.
Our tasks very often reach a concurrency limit on Cirrus-CI, and get
delayed. Submitting new tasks during the delay seems to make delay
longer and longer. So I think we should restrict the use of Cirrus,
provisionally at least. As YJIT needs ARM build to test code for ARM,
use only for it.
---
.cirrus.yml | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
