Many functions take an outlined code block but do nothing more than
passing it along; only a couple of functions actually make use of it.
So, in most cases the `ocb` parameter is just boilerplate.
Most functions that take `ocb` already also take a `JITState` and this
commit moves `ocb` into `JITState` to remove the visual noise of the
`ocb` parameter.
This commit fixes splat and block handling when calling in to a
forwarding iseq. In the case of a splat we need to avoid expanding the
array to the stack. We need to also ensure the CI write is flushed to
the SP, otherwise it's possible for a block handler to clobber the CI
[ruby-core:118360]
This commit adds `sendforward` and `invokesuperforward` for forwarding
parameters to calls
Co-authored-by: Matt Valentine-House <matt@eightbitraptor.com>
This patch optimizes forwarding callers and callees. It only optimizes methods that only take `...` as their parameter, and then pass `...` to other calls.
Calls it optimizes look like this:
```ruby
def bar(a) = a
def foo(...) = bar(...) # optimized
foo(123)
```
```ruby
def bar(a) = a
def foo(...) = bar(1, 2, ...) # optimized
foo(123)
```
```ruby
def bar(*a) = a
def foo(...)
list = [1, 2]
bar(*list, ...) # optimized
end
foo(123)
```
All variants of the above but using `super` are also optimized, including a bare super like this:
```ruby
def foo(...)
super
end
```
This patch eliminates intermediate allocations made when calling methods that accept `...`.
We can observe allocation elimination like this:
```ruby
def m
x = GC.stat(:total_allocated_objects)
yield
GC.stat(:total_allocated_objects) - x
end
def bar(a) = a
def foo(...) = bar(...)
def test
m { foo(123) }
end
test
p test # allocates 1 object on master, but 0 objects with this patch
```
```ruby
def bar(a, b:) = a + b
def foo(...) = bar(...)
def test
m { foo(1, b: 2) }
end
test
p test # allocates 2 objects on master, but 0 objects with this patch
```
How does it work?
-----------------
This patch works by using a dynamic stack size when passing forwarded parameters to callees.
The caller's info object (known as the "CI") contains the stack size of the
parameters, so we pass the CI object itself as a parameter to the callee.
When forwarding parameters, the forwarding ISeq uses the caller's CI to determine how much stack to copy, then copies the caller's stack before calling the callee.
The CI at the forwarded call site is adjusted using information from the caller's CI.
I think this description is kind of confusing, so let's walk through an example with code.
```ruby
def delegatee(a, b) = a + b
def delegator(...)
delegatee(...) # CI2 (FORWARDING)
end
def caller
delegator(1, 2) # CI1 (argc: 2)
end
```
Before we call the delegator method, the stack looks like this:
```
Executing Line | Code | Stack
---------------+---------------------------------------+--------
1| def delegatee(a, b) = a + b | self
2| | 1
3| def delegator(...) | 2
4| # |
5| delegatee(...) # CI2 (FORWARDING) |
6| end |
7| |
8| def caller |
-> 9| delegator(1, 2) # CI1 (argc: 2) |
10| end |
```
The ISeq for `delegator` is tagged as "forwardable", so when `caller` calls in
to `delegator`, it writes `CI1` on to the stack as a local variable for the
`delegator` method. The `delegator` method has a special local called `...`
that holds the caller's CI object.
Here is the ISeq disasm fo `delegator`:
```
== disasm: #<ISeq:delegator@-e:1 (1,0)-(1,39)>
local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 1] "..."@0
0000 putself ( 1)[LiCa]
0001 getlocal_WC_0 "..."@0
0003 send <calldata!mid:delegatee, argc:0, FCALL|FORWARDING>, nil
0006 leave [Re]
```
The local called `...` will contain the caller's CI: CI1.
Here is the stack when we enter `delegator`:
```
Executing Line | Code | Stack
---------------+---------------------------------------+--------
1| def delegatee(a, b) = a + b | self
2| | 1
3| def delegator(...) | 2
-> 4| # | CI1 (argc: 2)
5| delegatee(...) # CI2 (FORWARDING) | cref_or_me
6| end | specval
7| | type
8| def caller |
9| delegator(1, 2) # CI1 (argc: 2) |
10| end |
```
The CI at `delegatee` on line 5 is tagged as "FORWARDING", so it knows to
memcopy the caller's stack before calling `delegatee`. In this case, it will
memcopy self, 1, and 2 to the stack before calling `delegatee`. It knows how much
memory to copy from the caller because `CI1` contains stack size information
(argc: 2).
Before executing the `send` instruction, we push `...` on the stack. The
`send` instruction pops `...`, and because it is tagged with `FORWARDING`, it
knows to memcopy (using the information in the CI it just popped):
```
== disasm: #<ISeq:delegator@-e:1 (1,0)-(1,39)>
local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
[ 1] "..."@0
0000 putself ( 1)[LiCa]
0001 getlocal_WC_0 "..."@0
0003 send <calldata!mid:delegatee, argc:0, FCALL|FORWARDING>, nil
0006 leave [Re]
```
Instruction 001 puts the caller's CI on the stack. `send` is tagged with
FORWARDING, so it reads the CI and _copies_ the callers stack to this stack:
```
Executing Line | Code | Stack
---------------+---------------------------------------+--------
1| def delegatee(a, b) = a + b | self
2| | 1
3| def delegator(...) | 2
4| # | CI1 (argc: 2)
-> 5| delegatee(...) # CI2 (FORWARDING) | cref_or_me
6| end | specval
7| | type
8| def caller | self
9| delegator(1, 2) # CI1 (argc: 2) | 1
10| end | 2
```
The "FORWARDING" call site combines information from CI1 with CI2 in order
to support passing other values in addition to the `...` value, as well as
perfectly forward splat args, kwargs, etc.
Since we're able to copy the stack from `caller` in to `delegator`'s stack, we
can avoid allocating objects.
I want to do this to eliminate object allocations for delegate methods.
My long term goal is to implement `Class#new` in Ruby and it uses `...`.
I was able to implement `Class#new` in Ruby
[here](https://github.com/ruby/ruby/pull/9289).
If we adopt the technique in this patch, then we can optimize allocating
objects that take keyword parameters for `initialize`.
For example, this code will allocate 2 objects: one for `SomeObject`, and one
for the kwargs:
```ruby
SomeObject.new(foo: 1)
```
If we combine this technique, plus implement `Class#new` in Ruby, then we can
reduce allocations for this common operation.
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Co-Authored-By: Alan Wu <XrXr@users.noreply.github.com>
This mainly aims to make `--yjit-dump-disasm=<relative_path>` more
usable. Previously, it crashed if the program did chdir(2), since it
opened the dump file every time when appending.
Tested with:
./miniruby --yjit-dump-disasm=. --yjit-call-threshold=1 -e 'Dir.chdir("/") {}'
And the `lobsters` benchmark.
Calls to defer_compilation() leave behind a stub and a `struct Block`
that we retain. If the block is empty, it only exits to hold the
`struct Branch` that the stub needs.
This patch transplants the branch out of the empty block into the newly
generated block when the defer_compilation() stub is hit, and deletes
the empty block to save memory.
To assist the transplantation, `Block::outgoing` is now a
`MutableBranchList`, and `Branch::Block` now in a `Cell`. These types
don't incur a size cost.
On the `lobsters` benchmark, `yjit_alloc_size` is roughly 98% of what
it was before the change.
Co-authored-by: Kevin Menard <kevin.menard@shopify.com>
Co-authored-by: Randy Stauner <randy@r4s6.net>
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
* YJIT: add context cache hits stat
This stat should make more sense when it comes to interpreting
the effectiveness of the cache on large deployed apps.
* Implement BitVector data structure for variable-length context encoding
* Rename method to make intent clearer
* Rename write_uint => push_uint to make intent clearer
* Implement debug trait for BitVector
* Fix bug in BitVector::read_uint_at(), enable more tests
* Add one more test for good measure
* Start sketching Context::encode()
* Progress on variable length context encoding
* Add tests. Fix bug.
* Encode stack state
* Add comments. Try to estimate context encoding size.
* More compact encoding for stack size
* Commit before rebase
* Change Context::encode() to take a BitVector as input
* Refactor BitVector::read_uint(), add helper read functions
* Implement Context::decode() function. Add test.
* Fix bug, add tests
* Rename methods
* Add Context::encode() and decode() methods using global data
* Make encode and decode methods use u32 indices
* Refactor YJIT to use variable-length context encoding
* Tag functions as allow unused
* Add a simple caching mechanism and stats for bytes per context etc
* Add comments, fix formatting
* Grow vector of bytes by 1.2x instead of 2x
* Add debug assert to check round-trip encoding-decoding
* Take some rustfmt formatting
* Add decoded_from field to Context to reuse previous encodings
* Remove olde context stats
* Re-add stack_size assert
* Disable decoded_from optimization for now
If you start Ruby with `--yjit-disable`, the `+YJIT` shouldn't be
added until `RubyVM::YJIT.enable` is actually called. Otherwise
it's confusing in crash reports etc.
No plan about migrating to the 2024 edition yet (it's not even
available yet), but this is a simple enough suggestion so we can just
take it.
```
warning: this method call resolves to `<&Box<[T]> as IntoIterator>::into_iter` (due to backwards compatibility), but will resolve to `<Box<[T]> as IntoIterator>::into_iter` in Rust 2024
--> ../yjit/src/core.rs:1003:49
|
1003 | formatter.debug_list().entries(branches.into_iter()).finish()
| ^^^^^^^^^
|
= warning: this changes meaning in Rust 2024
= note: `#[warn(boxed_slice_into_iter)]` on by default
help: use `.iter()` instead of `.into_iter()` to avoid ambiguity
|
1003 | formatter.debug_list().entries(branches.iter()).finish()
| ~~~~
help: or use `IntoIterator::into_iter(..)` instead of `.into_iter()` to explicitly iterate by value
|
1003 | formatter.debug_list().entries(IntoIterator::into_iter(branches)).finish()
| ++++++++++++++++++++++++ ~
```
* YJIT: limit size of call count stats dict
Someone reported that logs were getting bloated because the
ISEQ and C call count dicts were huge, since they include all
of the call sites. I wrote code on the Rust size to limit the
size of the dict to avoid this problem. The size limit is
hardcoded at 20, but I figure this is probably fine?
* Fix bug reported by Kokubun.
They were initially made frozen to avoid false positives for cases such
as:
str = str.dup if str.frozen?
But this may cause bugs and is generally confusing for users.
[Feature #20205]
Co-authored-by: Jean Boussier <byroot@ruby-lang.org>
* YJIT: Fix `Struct` accessors not firing tracing events
Reading and writing to structs should fire `c_call` and `c_return`, but
YJIT wasn't correctly dropping those calls when tracing.
This has been missing since this functionality was added in 3081c83169,
but the added test only fails when ran in isolation with
`--yjit-call-threshold=1`. The test sometimes failed on CI.
* RJIT: YJIT: Fix `Struct` readers not firing tracing events
Same issue as YJIT, but it looks like RJIT doesn't support writing to
structs, so only reading needs changing.
Since we often take the VM lock as the first thing we do when entering
YJIT, and that needs a `src_loc!()`, this removes a allocation from
that. The main trick here is `concat!(file!(), '\0')` to get a C string
statically baked into the binary.
* YJIT: Add specialized codegen function for `TrueClass#===`
TrueClass#=== is currently number 10 in the most frequent C calls list of the lobsters benchmark.
```
require "benchmark/ips"
def wrap
true === true
true === false
true === :x
end
Benchmark.ips do |x|
x.report(:wrap) do
wrap
end
end
```
```
before
Warming up --------------------------------------
wrap 1.791M i/100ms
Calculating -------------------------------------
wrap 17.806M (± 1.0%) i/s - 89.544M in 5.029363s
after
Warming up --------------------------------------
wrap 4.024M i/100ms
Calculating -------------------------------------
wrap 40.149M (± 1.1%) i/s - 201.223M in 5.012527s
```
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Co-authored-by: Takashi Kokubun (k0kubun) <takashikkbn@gmail.com>
Co-authored-by: Kevin Menard <kevin.menard@shopify.com>
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
* Fix the new test for RJIT
---------
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Co-authored-by: Takashi Kokubun (k0kubun) <takashikkbn@gmail.com>
Co-authored-by: Kevin Menard <kevin.menard@shopify.com>
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
Previously, the update was done in the ISEQ callback. That effectively
never updated anything because the callback itself is given an intact
reference, so it could update its content, and `rb_gc_location(iseq)`
never returned a new address. Update the whole table once in the YJIT
root instead.
Types like `Type::CString` really only assert that at one point the object had
its class field equal to `String`. Once a singleton class is created for any
strings, the type makes no assertion about any class field anymore, and becomes
the same as `Type::TString`.
Previously, the `--yjit-verify-ctx` option wasn't allowing objects of these
kind that have have singleton classes to pass verification even though the code
generators handle it just fine.
Found through `ruby/spec`.
* Revert "Revert "YJIT: Optimize local variables when EP == BP" (#10584)"
This reverts commit c878344195.
* YJIT: Take care of GC references in ISEQ invariants
Co-authored-by: Alan Wu <alansi.xingwu@shopify.com>
---------
Co-authored-by: Alan Wu <alansi.xingwu@shopify.com>
Add a specialized codegen function for `Class#superclass`.
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
Co-authored-by: Takashi Kokubun (k0kubun) <takashikkbn@gmail.com>
Co-authored-by: Randy Stauner <randy.stauner@shopify.com>
Co-authored-by: Alan Wu <XrXr@users.noreply.github.com>
* YJIT: Fix shrinking block with assumption too much
Under the very specific circumstances, discovered by a test case in
`ruby/spec`, an `expandarray` block can contain just a branch and carry
a method lookup assumption. Previously, when we regenerated the branch,
we allowed it to shrink to empty, since we put the code at the jump
target immediately after it. That was incorrect and caused a crash while
the block is invalidated, since that left no room to patch in an exit.
When regenerating a branch that makes up a block entirely, and the block
could be invalidated, we need to ensure there is room for invalidation.
When there is code before the branch, they should act as padding, so we
don't need to worry about those cases.
* skip on RJIT
This reverts commit 4cc58ea0b8.
Since the change landed call-threshold=1 CI runs have been timing out.
There has also been `verify-ctx` violations. Revert for now while we debug.
Previously, we got "We are killing the stack canary set by opt_ltlt"
from `$./miniruby --yjit-call-threshold=1 -e 'a = [].freeze; a << 1'`
Found by running ruby-spec with yjit-call-threshold=1.
* YJIT: A64: Add CBZ and CBNZ encoding functions
* YJIT: A64: Use CBZ/CBNZ to check for zero
Instead of emitting `cmp x0, #0` plus `b.z #target`, A64 offers Compare
and Branch on Zero for us to just do `cbz x0, #target`. This commit
utilizes that and the related CBNZ instruction when appropriate.
We check for zero most commonly in interrupt checks:
```diff
# Insn: 0003 leave (stack_size: 1)
# RUBY_VM_CHECK_INTS(ec)
ldur w11, [x20, #0x20]
-tst w11, w11
-b.ne #0x109002164
+cbnz w11, #0x1049021d0
```
* fix copy paste error
Co-authored-by: Randy Stauner <randy@r4s6.net>
---------
Co-authored-by: Randy Stauner <randy@r4s6.net>
Same idea as the x64 equivalent in c2622b5253, removing the register
shuffle coming from the pop two, push one stack motion these VM
instructions perform.
```
# Insn: 0004 opt_or (stack_size: 2)
- orr x11, x1, x9
- mov x1, x11
+ orr x1, x1, x9
```
This is best understood by looking at the change to the output:
```diff
# Insn: 0002 opt_and (stack_size: 2)
- mov rax, rsi
- and rax, rdi
- mov rsi, rax
+ and rsi, rdi
```
It's a bit awkward to match against due to how stack operands are
lowered, but hey, it's nice to save the 2 unnecessary MOVs.
* YJIT: A64: Use ADDS/SUBS/CMP (immediate) when possible
We were loading 1 into a register and then doing ADDS/SUBS previously.
That was particularly bad since those come up in fixnum operations.
```diff
# integer left shift with rhs=1
- mov x11, #1
- subs x11, x1, x11
+ subs x11, x1, #1
lsl x12, x11, #1
asr x13, x12, #1
cmp x13, x11
- b.ne #0x106ab60f8
- mov x11, #1
- adds x12, x12, x11
+ b.ne #0x10903a0f8
+ adds x12, x12, #1
mov x1, x12
```
Note that it's fine to cast between i64 and u64 since the bit pattern is
preserved, and the add/sub themselves don't care about the signedness of
the operands.
CMP is just another mnemonic for SUBS.
* YJIT: A64: Split asm.mul() with immediates properly
There is in fact no MUL on A64 that takes an immediate, so this
instruction was using the wrong split method. No current usages of this
form in YJIT.
---------
Co-authored-by: Maxime Chevalier-Boisvert <maxime.chevalierboisvert@shopify.com>
In `jit_rb_int_lshift()`, we guard against the right hand side changing
since we want to avoid generating variable length shifts. When control
reaches a `putobject` and `opt_ltlt` pair, though, we know that the right
hand side never changes.
This commit detects this situation and substitutes an implementation
that does not guard against the right hand side changing, saving that
work.
Deleted some `putobject` Rust tests since they aren't that valuable and
cause linking issues.
Nice boost to `optcarrot` and `protoboeuf`:
```
---------- ------------------
bench yjit-pre/yjit-post
optcarrot 1.09
protoboeuf 1.12
---------- ------------------
```
This mainly targets things like `T.unsafe()` from Sorbet, which is just an
identity function at runtime and only a hint for the static checker.
Only deal with simple caller and callees (no keywords and splat etc.).
Co-authored-by: Takashi Kokubun (k0kubun) <takashikkbn@gmail.com>
