Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
When parent scopes around an eval are forwarding parameters (like
*, **, &, or ...) we need to know that information when we are in
the parser. As such, we need to support passing that information
into the scopes option. In order to do this, unfortunately we need
a bunch of changes.
The scopes option was previously an array of array of strings.
These corresponded to the names of the locals in the parent scopes.
We still support this, but now additionally support passing in a
Prism::Scope instance at each index in the array. This Prism::Scope
class holds both the names of the locals as well as an array of
forwarding parameter names (symbols corresponding to the forwarding
parameters). There is convenience function on the Prism module that
creates a Prism::Scope object using Prism.scope.
In JavaScript, we now additionally support an object much the same
as the Ruby side. In Java, we now have a ParsingOptions.Scope class
that holds that information. In the dump APIs, these objects in all
3 languages will add an additional byte for the forwarding flags in
the middle of the scopes serialization.
All of this is in service of properly parsing the following code:
```ruby
def foo(*) = eval("bar(*)")
```
21abb6b7c4
When there were more actual tokens than expected tokens, the test would still pass.
Now it's possible to receive an assertion like this:
```
expected: []
actual: [:tNL, [nil, #<Parser::Source::Range comment_single.txt 8...9>]]
<[]> expected but was
<[:tNL, [nil, #<Parser::Source::Range comment_single.txt 8...9>]]>
```
e9e9a48e43
This reverts the change made to this test case in commit a0e98d48c9
(Enhance TLS 1.3 support on LibreSSL 3.2/3.3, 2020-12-03).
Part of the test case was skipped on LibreSSL because LibreSSL 3.2.2
introduced a stricter check during creation of the extension. The check
was then relaxed in LibreSSL 3.4.0.
187b176ecd
Ref: https://github.com/ruby/json/pull/718
The existing `Parser` interface is pretty bad, as it forces to
instantiate a new instance for each document.
Instead it's preferable to only take the config and do all the
initialization needed, and then keep the parsing state on the
stack on in ephemeral memory.
This refactor makes the `JSON::Coder` pull request much easier to
implement in a performant way.
c8d5236a92
Co-Authored-By: Étienne Barrié <etienne.barrie@gmail.com>
parsing
(https://github.com/ruby/fiddle/pull/169)
GitHub: fix https://github.com/ruby/fiddle/pull/168
Struct parsing invokes "parse_ctype" on the whole member signature,
which fails if member type is "bool" due to plain string matching for
it. This change updates "bool" type matching to a regexp, so TYPE_BOOL
is correctly parsed for a whole signature like "bool toggle" as well as
just "bool".
---------
71607446d4
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Tests worked around this but the incompatibility is not hard to fix.
This fixes 17 token incompatibilies in tests here that were previously passing
101962526d
In https://github.com/ruby/prism/pull/3393 I made a mistake.
When there is no previous token, it wraps around to -1. Oops
Additionally, if a comment has no newline then the offset should be kept as is
3c266f1de4
Skipping detecting the encoding is almost always right, just for binary it should actually happen.
A symbol containing escapes that are invalid
in utf-8 would fail to parse since symbols must be valid in the script encoding.
Additionally, the parser gem would raise an exception somewhere during string handling
fa0154d9e4
There appear to be a bunch of rules, changing behaviour for
inline comments, multiple comments after another, etc.
This seems to line up with reality pretty closely, token differences for RuboCop tests go from 1129 to 619 which seems pretty impressive
2e1b92670c
These are not line continuations. They either should be taken literally,
or allow the word array to contain the following whitespace (newlines in this case)
Before:
```
0...1: tSTRING_BEG => "'"
1...12: tSTRING_CONTENT => "foobar\\\n"
12...16: tSTRING_CONTENT => "baz\n"
16...17: tSTRING_END => "'"
17...18: tNL => nil
```
After:
```
0...1: tSTRING_BEG => "'"
1...6: tSTRING_CONTENT => "foo\\\n"
6...12: tSTRING_CONTENT => "bar\\\n"
12...16: tSTRING_CONTENT => "baz\n"
16...17: tSTRING_END => "'"
17...18: tNL => nil
```
b6554ad64e
Slightly tweaking the import script becaues of backtrace format changes in Ruby 3.4
Most tests pass in all parsers, with only a handful of failures overall
9b5b785aa4
Much of this logic should be shared between interpolated symbols and regexps.
It's also incorrect when the node contains a literal `\\n` (same as for plain string nodes at the moment)
561914f99b
Turns out, the vast majority of work was already done with handling the same for heredocs
I'm confident this should also apply to actual string nodes (there's even a todo for it) but
no tests change if I apply it there too, so I can't say for sure if the logic would be correct.
The individual test files are a bit too large, maybe something else would break that currently passes.
Leaving it for later to look more closely into that.
6bba1c54e1
Blocks and lambdas inherit anonymous arguments from the method they are a part of.
They themselves don't allow to introduce new anonymous arguments.
While you can write this:
```rb
def foo(*)
bar { |**| }
end
```
referecing the new parameter inside of the block will always be a syntax error.
2cbd27e134
The offset cache contains an entry for each byte so it can't be accessed via the string length.
Adds tests for all variants except for this:
```
"fo
o" "ba
’"
```
For some reason, this still has the wrong offset.
a651126458
(https://github.com/ruby/irb/pull/1062)
Although not documented, `IRB.conf[:SAVE_HISTORY]` used to accept boolean,
which now causes `NoMethodError` when used.
This commit changes the behavior to accept boolean values and
adds tests for the behavior.
8b1a07b2a8
(https://github.com/ruby/irb/pull/1059)
* Gracefully handle incorrect command aliases
Even if the aliased target is a helper method or does not exist, IRB
should not crash.
This commit warns users in such cases and treat the input as normal expression.
* Streamline command parsing and introduce warnings for incorrect command aliases
9fc14eb74b
