If we hit an EOF token, and the character before the EOF is a newline,
we should make EOF token start at the previous newline. That way any
errors reported will occur on that line.
For example "foo(\n" should report an error on line 1 even though the
EOF technically occurs on line 2.
[Bug #20918]
https://bugs.ruby-lang.org/issues/2091860bc43de8e
In some cases Prism was either not raising an appropriate `void value
expression` error, or raising that error when the syntax is considered
valid.
To fix this Prism needs to check whether we have other clauses on the
`begin` rather than just returning `cast->statements`.
* If the `cast->statements` are null and the `cast->ensure_clause` is
not null, set the code to `cast->ensure_clause`
* else
* If there is a `cast->rescue_clause`
* Check if `cast->statements` are null and `cast->rescue_clause->statements`
are null, and return `NULL`
* Check if there is an `else_clause`, and set the node to
`cast->else_clause`.
* Otherwise return `cast->statements` as the node
* return `cast->statements` as the node
See tests for test cases. Note I took these directly from CRuby so if
desired I can delete them since the test will now pass. This only fixes
one test in the `test_parse` file, taking failures from 14 to 13.
This fixes `TestParse#test_void_value_in_rhs` and is related to
issue #2791.
398152b412
This PR fixes the following incorrect range of `Prism::Location` when `PM_ERR_RETURN_INVALID`.
It may be hard to tell from the text, but this Ruby error highlights `return`:
```console
$ ruby -e 'class Foo return end'
-e:1: Invalid return in class/module body
class Foo return end
-e: compile error (SyntaxError)
```
Previously, the error's `Prism::Location` pointed to `end`:
```console
$ bundle exec ruby -Ilib -rprism -ve 'p Prism.parse("class Foo return end").errors'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[#<Prism::ParseError @type=:return_invalid @message="invalid `return` in a class or module body"
@location=#<Prism::Location @start_offset=17 @length=3 start_line=1> @level=:fatal>]
After this fix, it will indicate `return`.
```console
$ bundle exec ruby -Ilib -rprism -ve 'p Prism.parse("class Foo return end").errors'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[#<Prism::ParseError @type=:return_invalid @message="invalid `return` in a class or module body"
@location=#<Prism::Location @start_offset=10 @length=6 start_line=1> @level=:fatal>]
```
For reference, here are the before and after of `Prism::Translation::Parser`.
Before:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("class Foo return end")'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
(string):1:18: error: invalid `return` in a class or module body
(string):1: class Foo return end
(string):1: ^~~
/Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/diagnostic/engine.rb:72:in `process':
invalid `return` in a class or module body (Parser::SyntaxError)
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:220:in `block in unwrap'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:218:in `each'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:218:in `unwrap'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:49:in `parse'
from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:33:in `parse'
from -e:1:in `<main>'
```
After:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("class Foo return end")'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
(string):1:11: error: invalid `return` in a class or module body
(string):1: class Foo return end
(string):1: ^~~~~~
/Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/diagnostic/engine.rb:72:in `process':
invalid `return` in a class or module body (Parser::SyntaxError)
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:220:in `block in unwrap'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:218:in `each'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:218:in `unwrap'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:49:in `parse'
from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:33:in `parse'
from -e:1:in `<main>'
```
This PR ensures that the originally intended `return` is highlighted as it should be.
1f9af4d2ad
Fixes https://github.com/ruby/prism/pull/2617.
There was an issue with the lexer as follows.
The following are valid regexp options:
```console
$ bundle exec ruby -Ilib -rprism -ve 'p Prism.lex("/x/io").value.map {|token| token[0].type }'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[:REGEXP_BEGIN, :STRING_CONTENT, :REGEXP_END, :EOF]
```
The following are invalid regexp options. Unnecessary the `IDENTIFIER` token is appearing:
```console
$ bundle exec ruby -Ilib -rprism -ve 'p Prism.lex("/x/az").value.map {|token| token[0].type }'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[:REGEXP_BEGIN, :STRING_CONTENT, :REGEXP_END, :IDENTIFIER, :EOF]
```
As a behavior of Ruby, when given `A` to `Z` and `a` to `z`, they act as invalid regexp options. e.g.,
```console
$ ruby -e '/regexp/az'
-e:1: unknown regexp options - az
/regexp/az
-e: compile error (SyntaxError)
```
Thus, it should probably not be construed as `IDENTIFIER` token.
Therefore, `pm_byte_table` has been adapted to accept those invalid regexp option values.
Whether it is a valid regexp option or not is checked by `pm_regular_expression_flags_create`.
For invalid regexp options, `PM_ERR_REGEXP_UNKNOWN_OPTIONS` is added to diagnostics.
d2a6096fcf
