This commit adds RVALUE_OVERHEAD for storing metadata at the end of the
slot. This commit moves the ractor_belonging_id in debug builds from the
flags to RVALUE_OVERHEAD which frees the 16 bits in the headers for
object shapes.
Raise a `SyntaxError` with the parser error message, in the case
reading from a file instead of the `-e` option or standard input. So
syntax_suggest can get the message from the caught error.
(https://github.com/ruby/irb/pull/449)
* Seamlessly integrate a few debug commands
* Improve the break command support
* Utilize skip_src option if available
* Add step and delete commands
* Write end-to-end tests for each debugger command
* Add documentation
* Add backtrace, info, catch commands
976100c1c2
Implementation for Language Server Protocol (LSP) sometimes needs token information.
For example both `m(1)` and `m(1, )` has same AST structure other than node locations
then it's impossible to check the existence of `,` from AST. However in later case,
it might be better to suggest variables list for the second argument.
Token information is important for such case.
This commit adds these methods.
* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node.
[Feature #19070]
Impacts on memory usage and performance are below:
Memory usage:
```
$ cat test.rb
root = RubyVM::AbstractSyntaxTree.parse_file(File.expand_path('../test/ruby/test_keyword.rb', __FILE__), keep_tokens: true)
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby -v
ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
11408kb
# keep_tokens :false
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
17508kb
# keep_tokens :true
$ /usr/bin/time -f %Mkb /usr/local/bin/ruby test.rb
30960kb
```
Performance:
```
$ cat ../ast_keep_tokens.yml
prelude: |
src = <<~SRC
module M
class C
def m1(a, b)
1 + a + b
end
end
end
SRC
benchmark:
without_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: false)
with_keep_tokens: |
RubyVM::AbstractSyntaxTree.parse(src, keep_tokens: true)
$ make benchmark COMPARE_RUBY="./ruby" ARGS=../ast_keep_tokens.yml
/home/kaneko.y/.rbenv/shims/ruby --disable=gems -rrubygems -I../benchmark/lib ../benchmark/benchmark-driver/exe/benchmark-driver \
--executables="compare-ruby::./ruby -I.ext/common --disable-gem" \
--executables="built-ruby::./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems --disable-gem" \
--output=markdown --output-compare -v ../ast_keep_tokens.yml
compare-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
built-ruby: ruby 3.2.0dev (2022-11-19T09:41:54Z 19070-keep_tokens d3af1b8057) [x86_64-linux]
warming up..
| |compare-ruby|built-ruby|
|:--------------------|-----------:|---------:|
|without_keep_tokens | 21.659k| 21.303k|
| | 1.02x| -|
|with_keep_tokens | 6.220k| 5.691k|
| | 1.09x| -|
```
This doesn't seem to stably work on mswin:
5871633211
For CI stability, it generally seems like a bad idea to run druby tests
on Windows, given that it's pretty much unstable on MinGW as well.
The tests looks for the .so file, which doesn't exist when the extension
is statically linked. In that situation it passes -I to ruby with
nothing after it which ate the -rio/console argument.
We would like to differentiate types of objects via their shape. This
commit adds a special T_OBJECT shape when we allocate an instance of
T_OBJECT. This allows us to avoid testing whether an object is an
instance of a T_OBJECT or not, we can just check the shape.
20221118T033003Z.log.html.gz
```
[21350/22024] TestIRB::ExtendCommandTest#test_show_source_end_finder/home/chkbuild/chkbuild/tmp/build/20221118T033003Z/ruby/test/irb/test_cmd.rb:523: warning: method redefined; discarding old show_source_test_method
/home/chkbuild/chkbuild/tmp/build/20221118T033003Z/ruby/test/irb/test_cmd.rb:523: warning: previous definition of show_source_test_method was here
```
* Windows: Fix warning about undefined if_indextoname()
* Windows: Fix UNIXSocket on MINGW and make .pair more reliable
* Windows: Use nonblock=true for read tests with scheduler
* Windows: Move socket detection from File.socket? to File.stat
Add S_IFSOCK to Windows and interpret reparse points accordingly.
Enable tests that work now.
* Windows: Use wide-char functions to UNIXSocket
This fixes behaviour with non-ASCII characters.
It also fixes deletion of temporary UNIXSocket.pair files.
* Windows: Add UNIXSocket tests for specifics of Windows impl.
* Windows: fix VC build due to missing _snwprintf
Avoid usage of _snwprintf, since it fails linking ruby.dll like so:
linking shared-library x64-vcruntime140-ruby320.dll
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol snwprintf
x64-vcruntime140-ruby320.def : error LNK2001: unresolved external symbol vsnwprintf_l
whereas linking miniruby.exe succeeds.
This patch uses snprintf on the UTF-8 string instead.
Also remove branch GetWindowsDirectoryW, since it doesn't work.
* Windows: Fix dangling symlink test failures
Co-authored-by: Lars Kanis <kanis@comcard.de>
The first `add_pages` call shouldn't hit Code GC yet; otherwise
`compiles` returns false. With the increased code size in runtime_stats
itself, it sometimes hits Code GC too early, at least in arm64 that has
a large code size.
(https://github.com/ruby/irb/pull/442)
* Add test_in_isolation task to run tests in isolation
This simulates how tests are run in Ruby CI and can detect some issues
before they're merged and break Ruby CI later.
* Run test_in_isolation task on CI
* Fix TestRaiseNoBacktraceException's setup
51f23c58b0
The tests failed on windows
5740085169 (step):18:62
```
1) Failure:
TestRegexp#test_s_timeout [D:/a/ruby/ruby/src/test/ruby/test_regexp.rb:1586]:
<0.30000000000000004> expected but was
<0.3>.
2) Failure:
TestRegexp#test_timeout_shorter_than_global [D:/a/ruby/ruby/src/test/ruby/test_regexp.rb:1631]:
<0.30000000000000004> expected but was
<0.3>.
```
(https://github.com/ruby/irb/pull/437)
* Transform ls's --grep/-G option to keyword args
* Make --grep less flexible
* Support -g instead of --grep
* Suppress warnings from symbol aliases
JRuby uses these same files for testing racc. The existing logic will not
find 'racc' in a JRuby project checkout. This change allows it to work by
just assuming 'ruby -S racc' when running tests. This will not change C
Ruby's detection when setting up tests (since earlier checks will find
racc).
b74ce5e018
