Generally I have been good about safely accessing the tokens but failed
to properly guard against no tokens in places
where it could theoretically happen through invalid syntax.
I added a test case for one occurance, other changes are theoretical only.
4a3866af19
Currently Prism returns `42` for code like this:
```ruby
42.tap { it = it; p it } # => 42
```
But parse.y returns `nil`:
```ruby
42.tap { it = it; p it } # => nil
```
In parse.y, it on the right-hand side is parsed as a local variable.
In Prism, it was parsed as the implicit block parameter it, which caused this inconsistent behavior.
This change makes the right-hand side it to be parsed as a local variable, aligning with parse.y's behavior.
Bug ticket: https://bugs.ruby-lang.org/issues/21139cf3bbf9d2c
Previously, endless method definitions in assignment contexts like
`x = def f = p 1` would fail to parse because command calls (method
calls without parentheses) were only accepted when the surrounding
binding power was less than `PM_BINDING_POWER_COMPOSITION`.
This fix specifically checks for assignment context and allows command
calls in those cases while maintaining the existing behavior for other
contexts. This ensures that:
- `x = def f = p 1` parses correctly (previously failed)
- `private def f = puts "Hello"` still produces the expected error
722af59ba3
Strings concatenated with backslash may end up being frozen when they
shouldn't be. This commit fixes the issue. It required a change
upstream in Prism, but also a change to the Prism compiler in CRuby.
https://github.com/ruby/prism/pull/3606
[Bug #21187]
This reverts commit a495e6a44c.
This break extension builds:
```
/Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:321:in 'String#replace': can't modify frozen String: "$(SDKROOT)$(prefix)/include" (FrozenError)
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:321:in 'RbConfig.expand'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:314:in 'block in RbConfig.expand'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:307:in 'String#gsub'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:307:in 'RbConfig.expand'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:314:in 'block in RbConfig.expand'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:307:in 'String#gsub'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:307:in 'RbConfig.expand'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:314:in 'block in RbConfig.expand'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:307:in 'String#gsub'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:307:in 'RbConfig.expand'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:325:in 'block in <module:RbConfig>'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:324:in 'Hash#each_value'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:324:in '<module:RbConfig>'
from /Users/hsbt/Documents/github.com/ruby/ruby/rbconfig.rb:11:in '<top (required)>'
from ./ext/extmk.rb:42:in 'Kernel#require'
from ./ext/extmk.rb:42:in '<main>'
make[1]: *** [ext/configure-ext.mk:70: ext/json/exts.mk] Error 1
```
When inner strings aren't frozen, we need to clear the flags on
interpolated string nodes so that we don't emit wrong instructions.
The compiler is currently incorrectly emitting frozen strings because
the parser is erroneously declaring interpolated strings as "frozen".
We need to fix this behavior in the parser so we can fix the compiler in
CRuby. This patch is a partial fix for [this bug](https://bugs.ruby-lang.org/issues/21187)
eda693f056
Previously, endless method definitions like `x = def f = p 1` would fail
to parse because command calls (method calls without parentheses) were
only accepted when the surrounding binding power was less than
`PM_BINDING_POWER_COMPOSITION` (8). In assignment contexts with binding
power 18, this condition was false, causing parse errors.
This fix ensures command calls are always accepted in endless method
bodies by passing `true` for `accepts_command_call`, making the method
body parse consistently regardless of where the method is defined.
70413ed4dd
When arithmetic expressions like `-1**2` are used in pattern matching contexts,
Ruby crashes with "Unexpected node type in pattern matching expression: PM_CALL_NODE".
This happens because the Prism parser creates `PM_CALL_NODE` for arithmetic operations,
but Ruby's pattern matching compiler doesn't handle call nodes.
This fix adds validation to reject `PM_CALL_NODE` in pattern contexts with a proper
syntax error.
365049a767
* Added `Ractor::Port`
* `Ractor::Port#receive` (support multi-threads)
* `Rcator::Port#close`
* `Ractor::Port#closed?`
* Added some methods
* `Ractor#join`
* `Ractor#value`
* `Ractor#monitor`
* `Ractor#unmonitor`
* Removed some methods
* `Ractor#take`
* `Ractor.yield`
* Change the spec
* `Racotr.select`
You can wait for multiple sequences of messages with `Ractor::Port`.
```ruby
ports = 3.times.map{ Ractor::Port.new }
ports.map.with_index do |port, ri|
Ractor.new port,ri do |port, ri|
3.times{|i| port << "r#{ri}-#{i}"}
end
end
p ports.each{|port| pp 3.times.map{port.receive}}
```
In this example, we use 3 ports, and 3 Ractors send messages to them respectively.
We can receive a series of messages from each port.
You can use `Ractor#value` to get the last value of a Ractor's block:
```ruby
result = Ractor.new do
heavy_task()
end.value
```
You can wait for the termination of a Ractor with `Ractor#join` like this:
```ruby
Ractor.new do
some_task()
end.join
```
`#value` and `#join` are similar to `Thread#value` and `Thread#join`.
To implement `#join`, `Ractor#monitor` (and `Ractor#unmonitor`) is introduced.
This commit changes `Ractor.select()` method.
It now only accepts ports or Ractors, and returns when a port receives a message or a Ractor terminates.
We removes `Ractor.yield` and `Ractor#take` because:
* `Ractor::Port` supports most of similar use cases in a simpler manner.
* Removing them significantly simplifies the code.
We also change the internal thread scheduler code (thread_pthread.c):
* During barrier synchronization, we keep the `ractor_sched` lock to avoid deadlocks.
This lock is released by `rb_ractor_sched_barrier_end()`
which is called at the end of operations that require the barrier.
* fix potential deadlock issues by checking interrupts just before setting UBF.
https://bugs.ruby-lang.org/issues/21262
Because it ends up treating it as a local variable, and `a.x`
is not a valid local variable name.
I'm not big on pattern matching, but conceptually it makes sense to me
to treat anything inside ^() to not be
pattern matching syntax?
80dbd85c45
`StringNode` and `SymbolNode` don't have the same shape
(`content` vs `value`) and that wasn't handled.
I believe the logic for the common case can be reused.
I simply left the special handling for implicit nodes in pattern matching
and fall through otherwise.
NOTE: patterns.txt is not actually tested at the moment,
because it contains syntax that `parser` mistakenly rejects.
But I checked manually that this doesn't introduce other failures.
https://github.com/whitequark/parser/pull/106055adfaa895
It's not my favorite api but for users that currently use the same thing
from `parser`, moving over is more difficult
than it needs to be.
If you plan to support both old and new ruby versions, you definitly need to
branch somewhere on the ruby version
to either choose prism or parser.
But with prism you then need to enumerate all the versions again and choose the correct one.
Also, don't recommend to use `Prism::Translation::Parser` in docs. It's version-less
but actually always just uses Ruby 3.4 which is probably
not what the user intended.
Note: parser also warns when the patch version doesn't match what it expects. But I don't think prism has such a concept,
and anyways it would require releases anytime ruby releases, which I don't think is very desirable
77177f9e92
Instead of requiring the consumer to provide a list of all events which
they wish to handle, we can give them to option of dynamically detecting
them, by scanning the listener's public methods.
This approach is similar to that used by Minitest (scanning for `test_`
methods) and Rails generators (running all public methods in the order
they are defined).
While this is slower than specifying a hard coded list, the penalty is
only during registration. There is no change the the behaviour of
dispatching the events.
781ebed743
In https://github.com/ruby/prism/pull/3494 I added a bit of code
so that using the new builder doesn't break stuff.
This code can be dropped when it is enforced that builder
is _always_ the correct subclass (and makes future issues like that unlikely).
193d4b806d
This can get triggered even if the list of statements only contains
a single statement. This is necessary to properly support compiling
```ruby
defined? (;a)
defined? (a;)
```
as "expression". Previously these were parsed as statements lists
with single statements in them.
b63b5d67a9
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
4edfe9d981
Of course, these won't really be fixtures, but it allows to test against whole codebases
without copying them, doing symlinks or something like that.
For example, I can tell that over the whole RuboCop codebase, there are only 8 files that produce mismatched ast.
Telling what the problem is is a different problem. The ast for real files can and will be huge so I haven't checked yet
(maybe parser bug) but it's nice for discoverability regardless
2184d82ba6
`Integer#chr` performs some validation that we don't want/need. Octal escapes can go above 255, where it will then raise trying to convert.
`append_as_bytes` actually allows to pass a number, so we can just skip that call.
Although, on older rubies of course we still need to handle this in the polyfill.
I don't really like using `pack` but don't know of another way to do so.
For the utf-8 escapes, this is not an issue. Invalid utf-8 in these is simply a syntax error.
161c606b1f
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
4edfe9d981
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
e1c75f304b
