YJIT: Fix potential infinite loop when OOM (GH-13186)

Avoid generating an infinite loop in the case where:
1. Block `first` is adjacent to block `second`, and the branch from `first` to
   `second` is a fallthrough, and
2. Block `second` immediately exits to the interpreter, and
3. Block `second` is invalidated and YJIT is OOM

While pondering how to fix this, I think I've stumbled on another related edge case:
1. Block `incoming_one` and `incoming_two` both branch to block `second`. Block
   `incoming_one` has a fallthrough
2. Block `second` immediately exits to the interpreter (so it starts with its exit)
3. When Block `second` is invalidated, the incoming fallthrough branch from
   `incoming_one` might be rewritten first, which overwrites the start of block
   `second` with a jump to a new branch stub.
4. YJIT runs of out memory
5. The incoming branch from `incoming_two` is then rewritten, but because we're
   OOM we can't generate a new stub, so we use `second`'s exit as the branch
   target. However `second`'s exit was already overwritten with a jump to the
   branch stub for `incoming_one`, so `incoming_two` will end up jumping to
   `incoming_one`'s branch stub.

Fixes [Bug #21257]
This commit is contained in:
Rian McGuire 2025-04-28 22:50:29 +10:00 committed by GitHub
parent 37db51b441
commit 80a1a1bb8a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
Notes: git 2025-04-28 12:50:45 +00:00
Merged: https://github.com/ruby/ruby/pull/13186

Merged-By: XrXr
2 changed files with 102 additions and 5 deletions

View file

@ -3667,6 +3667,74 @@ assert_equal 'new', %q{
test
}
# Bug #21257 (infinite jmp)
assert_equal 'ok', %q{
Good = :ok
def first
second
end
def second
::Good
end
# Make `second` side exit on its first instruction
trace = TracePoint.new(:line) { }
trace.enable(target: method(:second))
first
# Recompile now that the constant cache is populated, so we get a fallthrough from `first` to `second`
# (this is need to reproduce with --yjit-call-threshold=1)
RubyVM::YJIT.code_gc if defined?(RubyVM::YJIT)
first
# Trigger a constant cache miss in rb_vm_opt_getconstant_path (in `second`) next time it's called
module InvalidateConstantCache
Good = nil
end
RubyVM::YJIT.simulate_oom! if defined?(RubyVM::YJIT)
first
first
}
assert_equal 'ok', %q{
# Multiple incoming branches into second
Good = :ok
def incoming_one
second
end
def incoming_two
second
end
def second
::Good
end
# Make `second` side exit on its first instruction
trace = TracePoint.new(:line) { }
trace.enable(target: method(:second))
incoming_one
# Recompile now that the constant cache is populated, so we get a fallthrough from `incoming_one` to `second`
# (this is need to reproduce with --yjit-call-threshold=1)
RubyVM::YJIT.code_gc if defined?(RubyVM::YJIT)
incoming_one
incoming_two
# Trigger a constant cache miss in rb_vm_opt_getconstant_path (in `second`) next time it's called
module InvalidateConstantCache
Good = nil
end
incoming_one
}
assert_equal 'ok', %q{
# Try to compile new method while OOM
def foo