Commit graph

20692 commits

Author SHA1 Message Date
ydah
feac2b4b77 Implement CASE NODE keyword locations 2024-09-23 09:19:37 +09:00
Jeremy Evans
9f574fa12f Make Complex#{inspect,to_s} work correctly if real part #inspect returns frozen string
Make static f_format function take a non-frozen string to append
to.

This does not result in an additional allocation for #inspect,
but it does result in an additional allocation for #to_s.

Fixes [Bug #20337]
2024-09-21 20:00:23 +02:00
Koichi ITO
75ed086348 [ruby/prism] Fix kDO_LAMBDA token incompatibility for Prism::Translation::Parser::Lexer
## Summary

This PR fixes `kDO_LAMBDA` token incompatibility between Parser gem and `Prism::Translation::Parser` for lambda `do` block.

### Parser gem (Expected)

Returns `kDO_LAMBDA` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 3...5>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]]
```

### `Prism::Translation::Parser` (Actual)

Previously, the parser returned `kDO` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO, ["do", #<Parser::Source::Range example.rb 3...5>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]]
```

After the update, the parser now returns `kDO_LAMBDA` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.4.0dev (2024-09-01T11:00:13Z master eb144ef91e) [x86_64-darwin23]
[[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 3...5>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 6...9>]]]
```

## Additional Information

Unfortunately, this kind of edge case doesn't work as expected; `kDO` is returned instead of `kDO_LAMBDA`.
However, since `kDO` is already being returned in this case, there is no change in behavior.

### Parser gem

Returns `tLAMBDA` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> (foo = -> (bar) {}) do end"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [x86_64-darwin23]
[[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 4...7>]], [:tEQL, ["=", #<Parser::Source::Range example.rb 8...9>]],
[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 10...12>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 13...14>]],
[:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 14...17>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 17...18>]],
[:tLAMBEG, ["{", #<Parser::Source::Range example.rb 19...20>]], [:tRCURLY, ["}", #<Parser::Source::Range example.rb 20...21>]],
[:tRPAREN, [")", #<Parser::Source::Range example.rb 21...22>]], [:kDO_LAMBDA, ["do", #<Parser::Source::Range example.rb 23...25>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 26...29>]]]
```

### `Prism::Translation::Parser`

Returns `kDO` token:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "-> (foo = -> (bar) {}) do end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [x86_64-darwin23]
[[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 0...2>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 3...4>]],
[:tIDENTIFIER, ["foo", #<Parser::Source::Range example.rb 4...7>]], [:tEQL, ["=", #<Parser::Source::Range example.rb 8...9>]],
[:tLAMBDA, ["->", #<Parser::Source::Range example.rb 10...12>]], [:tLPAREN2, ["(", #<Parser::Source::Range example.rb 13...14>]],
[:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 14...17>]], [:tRPAREN, [")", #<Parser::Source::Range example.rb 17...18>]],
[:tLAMBEG, ["{", #<Parser::Source::Range example.rb 19...20>]], [:tRCURLY, ["}", #<Parser::Source::Range example.rb 20...21>]],
[:tRPAREN, [")", #<Parser::Source::Range example.rb 21...22>]], [:kDO, ["do", #<Parser::Source::Range example.rb 23...25>]],
[:kEND, ["end", #<Parser::Source::Range example.rb 26...29>]]]
```

As the intention is not to address such special cases at this point, a comment has been left indicating that this case still returns `kDO`.
In other words, `kDO_LAMBDA` will now be returned except for edge cases after this PR.

2ee480654c
2024-09-20 17:17:21 +00:00
Kevin Newton
f515a1ab4b [ruby/prism] Introduce partial_script option
b28877fa4f
2024-09-20 15:42:12 +00:00
Samuel Giddins
43e3416b70 [rubygems/rubygems] Unconditionally set installed_by_version
It has been supported since RubyGems 2.2.0 via 4525e45a4d

Signed-off-by: Samuel Giddins <segiddins@segiddins.me>

bf39c583e8
2024-09-20 14:26:13 +00:00
tomoya ishida
34e008d075 [ruby/irb] Fix debug command in nomultiline mode
(https://github.com/ruby/irb/pull/1006)

* Fix debug command in nomultiline mode

* context.colorize_code -> context.colorize_input

71f4d6bfb5
2024-09-20 10:13:39 +00:00
Nobuyoshi Nakada
75a8c937e7
[Bug #18651] Pass undefined region in CP51932 as is to CP50220 2024-09-20 16:23:55 +09:00
Jeremy Evans
6118e8a473 Fix method caching bug when including/prepend module A that prepends module B
Fix by always adding the generated iclass to the subclasses list,
otherwise the method cache for the iclass is not cleared when
the method in the module is overwritten.

Fixes [Bug #20716]
2024-09-19 07:34:01 -07:00
Jeremy Evans
268c72377b
Raise a compile error for break/next/redo inside eval in cases where it is optimized away
In cases where break/next/redo are not valid syntax, they should
raise a SyntaxError even if inside a conditional block that is
optimized away.

Fixes [Bug #20597]

Co-authored-by: Kevin Newton <kddnewton@gmail.com>
2024-09-18 16:54:56 -07:00
Jeremy Evans
29f2cb83fb
Fix evaluation order issue in f(**h, &h.delete(key))
Previously, this would delete the key in `h` before keyword
splatting `h`.  This goes against how ruby handles `f(*a, &a.pop)`
and similar expressions.

Fix this by having the compiler check whether the block pass
expression is safe.  If it is not safe, then dup the keyword
splatted hash before evaluating the block pass expression.

For expression: `h=nil; f(**h, &h.delete(:key))`

VM instructions before:

```
0000 putnil                                                           (   1)[Li]
0001 setlocal_WC_0                          h@0
0003 putself
0004 getlocal_WC_0                          h@0
0006 getlocal_WC_0                          h@0
0008 putobject                              :key
0010 opt_send_without_block                 <calldata!mid:delete, argc:1, ARGS_SIMPLE>
0012 splatkw
0013 send                                   <calldata!mid:f, argc:1, ARGS_BLOCKARG|FCALL|KW_SPLAT>, nil
0016 leave
```

VM instructions after:

```
0000 putnil                                                           (   1)[Li]
0001 setlocal_WC_0                          h@0
0003 putself
0004 putspecialobject                       1
0006 newhash                                0
0008 getlocal_WC_0                          h@0
0010 opt_send_without_block                 <calldata!mid:core#hash_merge_kwd, argc:2, ARGS_SIMPLE>
0012 getlocal_WC_0                          h@0
0014 putobject                              :key
0016 opt_send_without_block                 <calldata!mid:delete, argc:1, ARGS_SIMPLE>
0018 send                                   <calldata!mid:f, argc:1, ARGS_BLOCKARG|FCALL|KW_SPLAT|KW_SPLAT_MUT>, nil
0021 leave
```

This is the same as 07d3bf4832, except that
it removes unnecessary hash allocations when using the prism compiler.

Fixes [Bug #20640]
2024-09-18 12:46:07 -07:00
Jeremy Evans
9c12c39ed1 Revert "Fix evaluation order issue in f(**h, &h.delete(key))"
This reverts commit 07d3bf4832.

No failures in the pull request CI, but there are now allocation
test failures.
2024-09-18 11:26:10 -07:00
Jeremy Evans
07d3bf4832
Fix evaluation order issue in f(**h, &h.delete(key))
Previously, this would delete the key in h before keyword
splatting h.  This goes against how ruby handles f(*a, &a.pop)
and similar expressions.

Fix this by having the compiler check whether the block pass
expression is safe.  If it is not safe, then dup the keyword
splatted hash before evaluating the block pass expression.

For expression: `h=nil; f(**h, &h.delete(:key))`

VM instructions before:

```
0000 putnil                                                           (   1)[Li]
0001 setlocal_WC_0                          h@0
0003 putself
0004 getlocal_WC_0                          h@0
0006 getlocal_WC_0                          h@0
0008 putobject                              :key
0010 opt_send_without_block                 <calldata!mid:delete, argc:1, ARGS_SIMPLE>
0012 splatkw
0013 send                                   <calldata!mid:f, argc:1, ARGS_BLOCKARG|FCALL|KW_SPLAT>, nil
0016 leave
```

VM instructions after:

```
0000 putnil                                                           (   1)[Li]
0001 setlocal_WC_0                          h@0
0003 putself
0004 putspecialobject                       1
0006 newhash                                0
0008 getlocal_WC_0                          h@0
0010 opt_send_without_block                 <calldata!mid:core#hash_merge_kwd, argc:2, ARGS_SIMPLE>
0012 getlocal_WC_0                          h@0
0014 putobject                              :key
0016 opt_send_without_block                 <calldata!mid:delete, argc:1, ARGS_SIMPLE>
0018 send                                   <calldata!mid:f, argc:1, ARGS_BLOCKARG|FCALL|KW_SPLAT|KW_SPLAT_MUT>, nil
0021 leave
```

Fixes [Bug #20640]
2024-09-18 11:18:29 -07:00
Kevin Newton
96da3f16a0 [ruby/prism] Allow returns in default parameter values
e98ea15596
2024-09-18 11:52:27 -04:00
Yusuke Endoh
4797b0704a Prevent a warning (retry)
```
test/ruby/test_case.rb:75: warning: 'when' clause on line 75 duplicates 'when' clause on line 75 and is ignored
```
2024-09-18 11:16:22 +09:00
Randy Stauner
7c4b028435
YJIT: Accept key for runtime_stats to return only that stat (#11536) 2024-09-17 20:06:27 -04:00
Peter Zhu
0160dafc4c Replace all GC.disable with EnvUtil.without_gc 2024-09-17 10:34:26 -04:00
Yusuke Endoh
82151a8630 Fix a typo, sorry! 2024-09-17 23:07:44 +09:00
Yusuke Endoh
fd5e583990 Prevent two warnings
```
/home/chkbuild/chkbuild/tmp/build/20240917T123003Z/ruby/test/ruby/test_case.rb:73: warning: 'when' clause on line 73 duplicates 'when' clause on line 73 and is ignored
/home/chkbuild/chkbuild/tmp/build/20240917T123003Z/ruby/test/ruby/test_syntax.rb:333: warning: key :k1 is duplicated and overwritten on line 333
```
2024-09-17 22:52:25 +09:00
Hiroshi SHIBATA
1ecac5d4ed
TestProcess#test_daemon_noclose is working fine with macOS 15.1 beta3 and Xcode 16 RC 2024-09-17 16:16:45 +09:00
Hiroshi SHIBATA
c0116860ba
Removed accidentally commit for snapshot file of prism 2024-09-17 15:37:20 +09:00
NAITOH Jun
d81b0588bb
[ruby/strscan] Accept String as a pattern at non head
(https://github.com/ruby/strscan/pull/106)

It supports non-head match cases such as StringScanner#scan_until.

If we use a String as a pattern, we can improve match performance.
Here is a result of the including benchmark.

## CRuby

It shows String as a pattern is 1.18x faster than Regexp as a pattern.

```
$ benchmark-driver benchmark/check_until.yaml
Warming up --------------------------------------
              regexp     9.403M i/s -      9.548M times in 1.015459s (106.35ns/i)
          regexp_var     9.162M i/s -      9.248M times in 1.009479s (109.15ns/i)
              string     8.966M i/s -      9.274M times in 1.034343s (111.54ns/i)
          string_var    11.051M i/s -     11.190M times in 1.012538s (90.49ns/i)
Calculating -------------------------------------
              regexp    10.319M i/s -     28.209M times in 2.733707s (96.91ns/i)
          regexp_var    10.032M i/s -     27.485M times in 2.739807s (99.68ns/i)
              string     9.681M i/s -     26.897M times in 2.778397s (103.30ns/i)
          string_var    12.162M i/s -     33.154M times in 2.726046s (82.22ns/i)

Comparison:
          string_var:  12161920.6 i/s
              regexp:  10318949.7 i/s - 1.18x  slower
          regexp_var:  10031617.6 i/s - 1.21x  slower
              string:   9680843.7 i/s - 1.26x  slower
```

## JRuby

It shows String as a pattern is 2.11x faster than Regexp as a pattern.

```
$ benchmark-driver benchmark/check_until.yaml
Warming up --------------------------------------
              regexp     7.591M i/s -      7.544M times in 0.993780s (131.74ns/i)
          regexp_var     6.143M i/s -      6.125M times in 0.997038s (162.77ns/i)
              string    14.135M i/s -     14.079M times in 0.996067s (70.75ns/i)
          string_var    14.079M i/s -     14.057M times in 0.998420s (71.03ns/i)
Calculating -------------------------------------
              regexp     9.409M i/s -     22.773M times in 2.420268s (106.28ns/i)
          regexp_var    10.116M i/s -     18.430M times in 1.821820s (98.85ns/i)
              string    21.389M i/s -     42.404M times in 1.982519s (46.75ns/i)
          string_var    20.897M i/s -     42.237M times in 2.021187s (47.85ns/i)

Comparison:
              string:  21389191.1 i/s
          string_var:  20897327.5 i/s - 1.02x  slower
          regexp_var:  10116464.7 i/s - 2.11x  slower
              regexp:   9409222.3 i/s - 2.27x  slower
```

See:
be7815ec02/core/src/main/java/org/jruby/util/StringSupport.java (L1706-L1736)

---------

f9d96c446a

Co-authored-by: Sutou Kouhei <kou@clear-code.com>
2024-09-17 15:12:25 +09:00
Yusuke Endoh
70111fcc45 Prevent a warning due to error recovery of prism 2024-09-17 14:15:44 +09:00
Yusuke Endoh
d90da9408d Prevent warnings for RubyVM::AbstractSyntaxTree.parse in test_ast.rb 2024-09-17 14:10:49 +09:00
Yusuke Endoh
d84b062b63 Prevent warning: assigned but unused variable - message 2024-09-17 13:56:04 +09:00
Yusuke Endoh
76d927f88f Prevent prism warnings in syntax exhaustive tests
[Bug #20736]
2024-09-17 12:44:07 +09:00
KJ Tsanaktsidis
e08d5239b6 Ensure fiber scheduler is woken up when close interrupts read
If one thread is reading and another closes that socket, the close
blocks waiting for the read to abort cleanly. This ensures that Ruby is
totally done with the file descriptor _BEFORE_ we tell the OS to close
and potentially re-use it.

When the read is correctly terminated, the close should be unblocked.
That currently works if closing is happening on a thread, but if it's
happening on a fiber with a fiber scheduler, it does NOT work.

This patch ensures that if the close happened in a fiber scheduled
thread, that the scheduler is notified that the fiber is unblocked.

[Bug #20723]
2024-09-17 10:11:44 +10:00
Kevin Newton
2ea1950510 [ruby/prism] Do not leak explicit encoding
Fixes [Bug #20744]

f1b8b1b2a2
2024-09-16 18:57:54 +00:00
Kevin Newton
d57bc870ac [PRISM] Remove snapshot testing from Prism sync 2024-09-16 08:44:19 -04:00
Yusuke Endoh
532af89e3b Prevent warnings: the block passed to ... may be ignored 2024-09-15 10:06:11 +09:00
Yusuke Endoh
bc13ec735b Prevent warnings: assigned but unused variable 2024-09-15 10:05:57 +09:00
Kevin Newton
ddbd644001 [ruby/prism] Stat file first to check directory
4ed7de537b
2024-09-13 19:30:57 +00:00
Kevin Newton
77521afac1 [PRISM] Do not warn ambiguous ampersand when symbol literal
Fixes [Bug #20735]
2024-09-13 14:09:02 -04:00
Kevin Newton
9afc6a981d [PRISM] Only parse shebang on main script
Fixes [Bug #20730]
2024-09-13 12:51:53 -04:00
Kevin Newton
d42d19059d [PRISM] Allow case/when to be indented with no warning
Fixes [Bug #20731]
2024-09-13 12:51:36 -04:00
Kevin Newton
f0dcbbe9b9 [ruby/prism] Reverse-sync numbered reference range handling
a2f57ef6e3
2024-09-13 13:17:56 +00:00
Nobuyoshi Nakada
24493779b7
[Bug #20725] Should not call compare on nil-endpoint
It means unbounded, always inclusive of other ranges.
2024-09-13 18:23:28 +09:00
Yusuke Endoh
0f3dc2f958 Prevent warnings "the block passed to ... may be ignored" 2024-09-13 16:52:38 +09:00
Yusuke Endoh
6ae05584bd [ruby/net-http] Prevent warnings
```
/home/chkbuild/chkbuild/tmp/build/20240913T003003Z/ruby/test/net/http/utils.rb:32: warning: assigned but unused variable - e
/home/chkbuild/chkbuild/tmp/build/20240913T003003Z/ruby/test/net/http/utils.rb:61: warning: assigned but unused variable - version
/home/chkbuild/chkbuild/tmp/build/20240913T003003Z/ruby/test/net/http/utils.rb:124: warning: method redefined; discarding old query
```

6f818346ce
2024-09-13 02:39:11 +00:00
Yusuke Endoh
5e3f1b6a6d Prevent a warning: assigned but unused variable - exp_events 2024-09-13 11:34:20 +09:00
Yusuke Endoh
d1d0c50c9d Prevent a warning: assigned but unused variable - t0 2024-09-13 11:32:51 +09:00
Kevin Newton
05e02783a5
[PRISM] Ignore test_parse_directory if error is nil 2024-09-12 15:07:04 -04:00
Kevin Newton
38ba15beed [ruby/prism] Check errno for parsing directory
d68ea29d04
2024-09-12 13:43:04 -04:00
Kevin Newton
d5232fd7ae Temporarily exclude some TestRubyOptions test for parse.y
The description has been updated when running with Prism to have
+PRISM, which means that tests asserting against the description
when running with --parser=parse.y will be incorrect if the
subprocess ends up using Prism. We need to fix these tests, but
we are currently disabling them in order to get this over the line.
2024-09-12 13:43:04 -04:00
Kevin Newton
f59b908e47 [PRISM] Omit some TestAST tests when Prism is enabled 2024-09-12 13:43:04 -04:00
Kevin Newton
ea2af5782d Switch the default parser from parse.y to Prism
This commit switches the default parser to Prism. There are a
couple of additional changes related to this that are a part of
this as well to make this happen.

* Switch the default parser in parse.h
* Remove the Prism-specific workflow and add a parse.y-specific
  workflow to CI so that it continues to be tested
* Update a few test exclusions since Prism has the correct
  behavior but parse.y doesn't per
  https://bugs.ruby-lang.org/issues/20504.
* Skips a couple of tests on RBS which are failing because they
  are using RubyVM::AbstractSyntaxTree.of.

Fixes [Feature #20564]
2024-09-12 13:43:04 -04:00
Jeremy Evans
f2919bd11c
Add error checking to readdir, telldir, and closedir calls in dir.c
Raise SystemCallError exception when these functions return an error.

This changes behavior for the following case (found by the tests):

```ruby
dir1 = Dir.new('..')
dir2 = Dir.for_fd(dir1.fileno)
dir1.close
dir2.close
```

The above code is basically broken, as `dir1.close` closed the file
descriptor.  The subsequent `dir2.close` call is undefined behavior.
When run in isolation, it raises Errno::EBADF after the change, but
if another thread opens a file descriptor between the `dir1.close`
and `dir2.close` calls, the `dir2.close` call could close the file
descriptor opened by the other thread.  Raising an exception is much
better in this case as it makes it obvious there is a bug in the code.

For the readdir check, since the GVL has already been released,
reacquire it rb_thread_call_with_gvl if an exception needs to be
raised.

Due to the number of closedir calls, this adds static close_dir_data
and check_closedir functions.  The close_dir_data takes a
struct dir_data * and handles setting the dir entry to NULL regardless
of failure.

Fixes [Bug #20586]

Co-authored-by: Nobuyoshi Nakada <nobu.nakada@gmail.com>
2024-09-12 10:04:10 -07:00
Kevin Newton
15135030e5 [ruby/prism] Do not warn \r in shebang on windows
e8c862ca1f
2024-09-12 15:50:34 +00:00
Luke Gruber
5d358b660d Fix issue with super and forwarding arguments in prism_compile.c
Fixes [Bug #20720]
2024-09-11 16:41:46 -04:00
Kevin Newton
d4d6f1de83 [ruby/prism] UTF-8 characters in file name
487f0ffe78
2024-09-11 19:17:12 +00:00
John Meade
efc77d535b [ruby/psych] Ensure strings with only underscores are not processed as Integer
A string similar to "0x____" should be treated as a string.
Currently it is processed as an Integer.

This alters the regex specified by http://yaml.org/type/int.html
to ensure at least one numerical symbol is present in the string
before converting to Integer.

81479b203e
2024-09-11 17:19:18 +00:00