I think we can safely assume these days that all RubyGems and Bundler
versions that will ever bundle a new gem created in 2023 support
prereleases.
So this non transparent requirement is not necessary.
In my opinion, it should be the gem author to explicitly add this
constraint, not RubyGems.
b165e6d725
If there are MultiTargetNodes within parameters, we need to
iterate over them and compile them individually correctly, once
the locals are all in the correct spaces. We need to add one
getlocal for the hidden variable, and then can recurse into the
MultiTargetNodes themselves
Prior to this commit, we were using `add_ensure_iseq` which compiled
a node as if it was a CRuby node. This commit defines
`pm_add_ensure_iseq` which compiles the Prism node appropriately.
In this commit we're splitting up the call nodes that were in target
positions (that is, for loop indices, rescue error captures, and
multi assign targets).
Previously, we would simply leave the call nodes in place. This had
the benefit of keeping the AST relatively simple, but had the
downside of not being very explicit. If a static analysis tool wanted
to only look at call nodes, it could easily be confused because the
method would have 1 fewer argument than it would actually be called
with.
This also brings some consistency to the AST. All of the nodes in
a target position are now *TargetNode nodes. These should all be
treated the same, and the call nodes can now be treated the same.
Finally, there is benefit to memory. Because being in a target
position ensures we don't have some fields, we can strip down the
number of fields on these nodes.
So this commit introduces two new nodes: CallTargetNode and
IndexTargetNode. For CallTargetNode we get to drop the opening_loc,
closing_loc, arguments, and block. Those can never be present. We
also get to mark their fields as non-null, so they will always be
seen as present.
The IndexTargetNode keeps around most of its fields but gets to
drop both the name (because it will always be []=) and the
message_loc (which was always super confusing because it included
the arguments by virtue of being inside the []).
Overall, this adds complexity to the AST at the expense of memory
savings and explicitness. I believe this tradeoff is worth it in
this case, especially because these are very much not common nodes
in the first place.
3ef71cdb45
The logic within the consequent for the CaseNodes in popped cases
was incorrect as it wouldn't emit consequent instructions for a
popped CaseNode. This commit fixes that.
Arguments that are passed as a hash need special consideration since in certain case they are not treated as keyword arguments. For example, if a call is passing `"a" => 1` as an argument, this will be turned into an implicit hash argument and not a keyword argument.
The existing compiler checks to see if all hash nodes can be treated as keyword arguments. If they can, then it will treat them as keyword arguments. If not, then it will treat them as implicit hash arguments.
This commit implements the same logic inside the Prism compiler.
Our current implementation of rb_postponed_job_register suffers from
some safety issues that can lead to interpreter crashes (see bug #1991).
Essentially, the issue is that jobs can be called with the wrong
arguments.
We made two attempts to fix this whilst keeping the promised semantics,
but:
* The first one involved masking/unmasking when flushing jobs, which
was believed to be too expensive
* The second one involved a lock-free, multi-producer, single-consumer
ringbuffer, which was too complex
The critical insight behind this third solution is that essentially the
only user of these APIs are a) internal, or b) profiling gems.
For a), none of the usages actually require variable data; they will
work just fine with the preregistration interface.
For b), generally profiling gems only call a single callback with a
single piece of data (which is actually usually just zero) for the life
of the program. The ringbuffer is complex because it needs to support
multi-word inserts of job & data (which can't be atomic); but nobody
actually even needs that functionality, really.
So, this comit:
* Introduces a pre-registration API for jobs, with a GVL-requiring
rb_postponed_job_prereigster, which returns a handle which can be
used with an async-signal-safe rb_postponed_job_trigger.
* Deprecates rb_postponed_job_register (and re-implements it on top of
the preregister function for compatability)
* Moves all the internal usages of postponed job register
pre-registration
Previously, block.to_proc was called first, by vm_caller_setup_arg_block.
kw.to_hash was called later inside CALLER_SETUP_ARG or setup_parameters_complex.
This adds a splatkw instruction that is inserted before sends with
ARGS_BLOCKARG and KW_SPLAT and without KW_SPLAT_MUT. This is not needed in the
KW_SPLAT_MUT case, because then you know the value is a hash, and you don't
need to call to_hash on it.
The splatkw instruction checks whether the second to top block is a hash,
and if not, replaces it with the value of calling to_hash on it (using
rb_to_hash_type). As it is always before a send with ARGS_BLOCKARG and
KW_SPLAT, second to top is the keyword splat, and top is the passed block.
There seems to be another manifestation of bug #20021, where some of the
compaction tests are failing on i686 for unrelated PR's because of fake
"live" references to moved objects on the machine stack.
We _could_ solve this by counting how many objects are pinned during
compaction, but doing that involves pushing down the mark & pin bitset
merge into gc_compact_plane and out of gc_compact_page, which I thought
was pretty ugly.
Now that we've solved bug #20022 though, we're able to compact
arbitrarily many objects with GC.verify_compaction_references, so the
number of objects we're moving is now 50,000 instead of 500. Since
that's now much larger than the number of objects likely to be pinned, I
think it's safe enough to just add a fudge-factor to the tests.
Any _other_ change in GC.verify_compaction_references that breaks
compaction is now highly likely to break the assertion by more than 10
objects, since it's operating on so many more in the first place.
[Bug #20021]
It seems that the Ractor sleep GVL event arrives very slightly after the
value becomes available and other threads wake (which makes sense) so we
need a little additional time to ensure we end up in a consisteny state.
This doesn't actually fix the encodings for symbols and regex,
unfortunately. But I wanted to get this change in because it is
the last AST change we're going to make before 3.3 is released.
So, if consumers want, they can start to check these flags to
determine the encoding, even though it will be wrong. Then once we
actually set them correctly, everything should work.
9b35f7e891
So generate_index can be implemented with dependencies, such as the compact index
Took this approach from feedback in https://github.com/rubygems/rubygems/pull/6853
Running `gem generate_index` by default will use an installed rubygems-generate_index, or install and then use the command from the gem
Apply suggestions from code review
fc1cb9bc9e
Co-authored-by: Hiroshi SHIBATA <hsbt@ruby-lang.org>
when the RUBY_FREE_ON_SHUTDOWN environment variable is set, manually free memory at shutdown.
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
BreakNode, ReturnNode and NextNode all compile the ArgumentsNode
directly, but we weren't accounting for multiple arguments. If there
is more than one argument, we need to also emit a newarray
instruction to put the arguments onto the stack
For the following:
```
def f(*a); a end
p f(*a, kw: 3)
```
`setup_parameters_complex` pushes `{kw: 3}` onto `a`. This worked fine
back when `concatarray true` was used and `a` was already a copy. It
does not work correctly with the optimization to switch to
`concatarray false`. This duplicates the array on the callee side
in such a case.
This affects cases when passing a regular splat and a keyword splat
(or literal keywords) in a method call, where the method does not
accept keywords.
This allocation could probably be avoided, but doing so would
make `setup_parameters_complex` more complicated.
This follows the same approach used for attr_reader/attr_writer in
2d98593bf5, skipping the checking for
tracing after the first call using the call cache, and clearing the
call cache when tracing is turned on/off.
Fixes [Bug #18886]
This seems to happen only on VisualStudio:
1941837538
It fails relatively frequently. Nobody seems actively working on it, so
let's skip it until somebody starts working on it.
Since Ruby 2.4, `return` is supported at toplevel. This makes `eval "return"`
also supported at toplevel.
This mostly uses the same tests as direct `return` at toplevel, with a couple
differences:
`END {return if false}` is a SyntaxError, but `END {eval "return" if false}`
is not an error since the eval is never executed. `END {return}` is a
SyntaxError, but `END {eval "return"}` is a LocalJumpError.
The following is a SyntaxError:
```ruby
class X
nil&defined?0--begin e=no_method_error(); return; 0;end
end
```
However, the following is not, because the eval is never executed:
```ruby
class X
nil&defined?0--begin e=no_method_error(); eval "return"; 0;end
end
```
Fixes [Bug #19779]
Since Ruby 3.0, Ruby has passed a keyword splat as a regular
argument in the case of a call to a Ruby method where the
method does not accept keyword arguments, if the method
call does not contain an argument splat:
```ruby
def self.f(obj) obj end
def self.fs(*obj) obj[0] end
h = {a: 1}
f(**h).equal?(h) # Before: true; After: false
fs(**h).equal?(h) # Before: true; After: false
a = []
f(*a, **h).equal?(h) # Before and After: false
fs(*a, **h).equal?(h) # Before and After: false
```
The fact that the behavior differs when passing an empty
argument splat makes it obvious that something is not
working the way it is intended. Ruby 2 always copied
the keyword splat hash, and that is the expected behavior
in Ruby 3.
This bug is because of a missed check in setup_parameters_complex.
If the keyword splat passed is not mutable, then it points to
an existing object and not a new object, and therefore it must
be copied.
Now, there are 3 specs for the broken behavior of directly
using the keyword splatted hash. Fix two specs and add a
new version guard. Do not keep the specs for the broken
behavior for earlier Ruby versions, in case this fix is
backported. For the ruby2_keywords spec, just remove the
related line, since that line is unrelated to what the
spec is testing.
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
The intention of GC.verify_compaction_references is, I believe, to force
every single movable object to be moved, so that it's possible to debug
native extensions which not correctly updating their references to
objects they mark as movable.
To do this, it doubles the number of allocated pages for each size pool,
and sorts the heap pages so that the free ones are swept first; thus,
every object in an old page should be moved into a free slot in one of
the new pages.
This worked fine until movement of objects _between_ size pools during
compaction was implemented. That causes some problems for
verify_compaction_references:
* We were doubling the number of pages in each size pool, but actually
if some objects need to move into a _different_ pool, there's no
guarantee that they'll be enough room in that one.
* It's possible for the sweep & compact cursors to meet in one size pool
before all the objects that want to move into that size pool from
another are processed by the compaction.
You can see these problems by changing some of the movement tests in
test_gc_compact.rb to try and move e.g. 50,000 objects instead of
500; the test is not able to actually move all of the objects in a
single compaction run.
To fix this, we do two things in verify_compaction_references:
* Firstly, we add enough pages to every size pool to make them the same
size. This ensures that their compact cursors will all have space to
move during compaction (even if that means empty pages are
pointlessly compacted)
* Then, we examine every object and determine where it _wants_ to be
compacted into. We use this information to add additional pages to
each size pool to handle all objects which should live there.
With these two changes, we can move arbitrary amounts of objects into
the correct size pool in a single call to verify_compaction_references.
My _motivation_ for performing this work was to try and fix some test
stability issues in test_gc_compact.rb. I now no longer think that we
actually see this particular bug in rubyci.org, but I also think
verify_compaction_references should do what it says on the tin, so it's
worth keeping.
[Bug #20022]
When making an outgoing TCP or UDP connection, set AI_ADDRCONFIG in the
hints we send to getaddrinfo(3) (if supported). This will prompt the
resolver to _NOT_ issue A or AAAA queries if the system does not
actually have an IPv4 or IPv6 address (respectively).
This makes outgoing connections marginally more efficient on
non-dual-stack systems, since we don't have to try connecting to an
address which can't possibly work.
More importantly, however, this works around a race condition present
in some older versions of glibc on aarch64 where it could accidently
send the two outgoing DNS queries with the same DNS txnid, and get
confused when receiving the responses. This manifests as outgoing
connections sometimes taking 5 seconds (the DNS timeout before retry) to
be made.
Fixes#19144
