Commit graph

80 commits

Author SHA1 Message Date
Nobuyoshi Nakada
991cf2dd4d [ruby/prism] [DOC] Specify markdown mode to RDoc
12af4e144e
2025-05-29 04:45:58 +00:00
viralpraxis
543dd77cc3 [ruby/prism] Fix parsing rescued exception via indexed assignment
Given this code

```ruby
begin
  raise '42'
rescue => A[]
end
```

Prism fails with this backtrace

```
Error: test_unparser/corpus/literal/rescue.txt(Prism::ParserTest): NoMethodError: undefined method `arguments' for nil
prism/lib/prism/translation/parser/compiler.rb:1055:in `visit_index_target_node'
prism/lib/prism/node.rb:9636:in `accept'
prism/lib/prism/compiler.rb:30:in `visit'
prism/lib/prism/translation/parser/compiler.rb:218:in `visit_begin_node'
```

Seems like

```diff
-            visit_all(node.arguments.arguments),
+            visit_all(node.arguments&.arguments || []),
```

fixes the problem.

76d01aeb6c
2025-04-12 17:43:57 +00:00
Earlopain
334c261cc9 [ruby/prism] Fix parser translator when splatting in pattern matching pin
Because it ends up treating it as a local variable, and `a.x`
is not a valid local variable name.

I'm not big on pattern matching, but conceptually it makes sense to me
to treat anything inside ^() to not be
pattern matching syntax?

80dbd85c45
2025-04-02 20:51:54 +00:00
Earlopain
d7e46543b5 [ruby/prism] Fix parser translator when pinning hash with string keys
`StringNode` and `SymbolNode` don't have the same shape
(`content` vs `value`) and that wasn't handled.

I believe the logic for the common case can be reused.
I simply left the special handling for implicit nodes in pattern matching
and fall through otherwise.

NOTE: patterns.txt is not actually tested at the moment,
because it contains syntax that `parser` mistakenly rejects.
But I checked manually that this doesn't introduce other failures.
https://github.com/whitequark/parser/pull/1060

55adfaa895
2025-03-30 17:24:05 +00:00
Kevin Newton
052794bfe1 [ruby/prism] Accept a newline after the defined? keyword
[Bug #21197]

22be955ce9
2025-03-30 13:22:41 -04:00
Kevin Newton
b003d40194 Fix up merge conflicts for prism sync 2025-03-18 13:36:53 -04:00
Earlopain
e3c8464630 [ruby/prism] Only unnest parser mlhs nodes when no rest argument is provided
```
(a,), = []

PARSER====================
s(:masgn,
  s(:mlhs,
    s(:mlhs,
      s(:lvasgn, :a))),
  s(:array))
PRISM====================
s(:masgn,
  s(:mlhs,
    s(:lvasgn, :a)),
  s(:array))
```

8aa1f4690e
2025-03-18 13:36:53 -04:00
Earlopain
94e12ffa39 [ruby/prism] Fix parser translator multiline interpolated symbols
In 2637007929 I added tests but didn't modify them correctly

de021e74de
2025-03-18 13:36:53 -04:00
Earlopain
a8adf5e006 [ruby/prism] Further refine string handling in the parser translator
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation

4edfe9d981
2025-03-18 13:36:53 -04:00
Earlopain
fd7a10cf4a [ruby/prism] Further refine string handling in the parser translator
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation

4edfe9d981
2025-03-18 13:36:53 -04:00
Earlopain
ac728389e2 [ruby/prism] Fix parser translator edge-case when multiline string ends with \n
When the line contains no real newline but contains unescaped ones, then there will be one less entry

4ef093b600
2025-03-18 13:36:53 -04:00
Earlopain
0fcb7fc21d [ruby/prism] Better handle all kinds of multiline strings in the parser translator
This is a followup to #3373, where the implementation
was extracted

2637007929
2025-03-18 13:36:53 -04:00
Earlopain
acf404e20e [ruby/prism] Fix an incompatibility with the parser translator
The offset cache contains an entry for each byte so it can't be accessed via the string length.

Adds tests for all variants except for this:
```
"fo
o" "ba
’"
```

For some reason, this still has the wrong offset.

a651126458
2025-03-18 13:36:53 -04:00
Earlopain
f49a0114e3 [ruby/prism] Fix parser translator rescue location with semicolon body
There are a few other locations that should be included in that check.
I think the end location must always be present but I left it in to be safe (maybe implicit begin somehow?)

545d07ddc3
2025-03-18 13:36:53 -04:00
Earlopain
bc506295a3 [ruby/prism] Further refine string handling in the parser translator
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation

4edfe9d981
2025-03-18 13:36:53 -04:00
Earlopain
4b844f7d9e [ruby/prism] Ensure backwards compatibility with the custom parser builder
Temoprary backwards-compat code so that current users
don't break.

Eventually the Translation::Parser initializer should asser that the correct class is passed in.

66b0162b35
2025-03-13 12:06:58 +00:00
Koichi ITO
6b4453e332 [ruby/prism] Support itblock for Prism::Translation::Parser
## Summary

`itblock` node is added to support the `it` block parameter syntax introduced in Ruby 3.4.

```console
$ ruby -Ilib -rprism -rprism/translation/parser34 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { it }"; \
                                                      p Prism::Translation::Parser34.new.tokenize(buffer)[0]'
s(:itblock,
  s(:send, nil, :proc), :it,
  s(:lvar, :it))
```

This node design is similar to the `numblock` node, which was introduced for the numbered parameter syntax in Ruby 2.7.

```
$ ruby -Ilib -rprism -rprism/translation/parser34 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { _1 }"; \
                                                      p Prism::Translation::Parser34.new.tokenize(buffer)[0]'
s(:numblock,
  s(:send, nil, :proc), 1,
  s(:lvar, :_1))
```

The difference is that while numbered parameters can have multiple parameters, the `it` block parameter syntax allows only a single parameter.

In Ruby 3.3, the conventional node prior to the `it` block parameter syntax is returned.

```console
$ ruby -Ilib -rprism -rprism/translation/parser33 -e 'buffer = Parser::Source::Buffer.new("path"); buffer.source = "proc { it }"; \
                                                      p Prism::Translation::Parser33.new.tokenize(buffer)[0]'
s(:block,
  s(:send, nil, :proc),
  s(:args),
  s(:send, nil, :it))
```

## Development Note

The Parser gem does not yet support the `it` block parameter syntax. This is the first case where Prism's node design precedes that of the Parser gem.
When implementing https://github.com/whitequark/parser/issues/962, this node design will need to be taken into consideration.

c141e1420a
2025-03-10 16:57:46 +00:00
Earlopain
769cccba56 [ruby/prism] Fix parser translator scope issues for implicit hash values
`builder.pair_label` is no good since it makes use of variables that the parser gem encountered.
Since the prism translator doesn't keep proper track of that information, the following code interprets
the implicit value as a local variable, even though it is not in scope:

```rb
def foo
  bar = 123
end

{ bar: }
```

bbeb5b083a
2025-01-20 18:03:13 +00:00
Earlopain
56242ba495 Better handle regexp in the parser translator
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.

There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.

Fixes more than half of the remaining ast differences for rubocop tests
2025-01-14 20:33:11 +00:00
Kevin Newton
14b9098459 [ruby/prism] Frozen strings in the AST
8d9d429155
2025-01-12 18:41:42 +00:00
Earlopain
283037f7e3 [ruby/prism] Better handle all kinds of multiline strings in the parser translator
This is a followup to #3373, where the implementation
was extracted

2637007929
2025-01-11 19:09:05 -05:00
Earlopain
80fe9a1c77 [ruby/prism] Better handle multiline interpolated strings in the parser translator
Much of this logic should be shared between interpolated symbols and regexps.
It's also incorrect when the node contains a literal `\\n` (same as for plain string nodes at the moment)

561914f99b
2025-01-11 19:09:05 -05:00
Earlopain
9f38ee11cb [ruby/prism] Fix parser translator ast for empty regex
In that specific case, no string node is emitted

1166db13dd
2025-01-11 19:09:05 -05:00
Earlopain
a234fd516f [ruby/prism] Fix parser translator ast for regex with line continuation
Turns out, the vast majority of work was already done with handling the same for heredocs

I'm confident this should also apply to actual string nodes (there's even a todo for it) but
no tests change if I apply it there too, so I can't say for sure if the logic would be correct.
The individual test files are a bit too large, maybe something else would break that currently passes.

Leaving it for later to look more closely into that.

6bba1c54e1
2025-01-11 19:09:05 -05:00
Earlopain
d1a70014f9 [ruby/prism] Fix parser translator ast when using anonymous forwarding in blocks/lambda
Blocks and lambdas inherit anonymous arguments from the method they are a part of.
They themselves don't allow to introduce new anonymous arguments.
While you can write this:
```rb
def foo(*)
  bar { |**| }
end
```
referecing the new parameter inside of the block will always be a syntax error.

2cbd27e134
2025-01-11 19:09:05 -05:00
Earlopain
7cbaa3b929 [ruby/prism] Fix an incompatibility with the parser translator
The offset cache contains an entry for each byte so it can't be accessed via the string length.

Adds tests for all variants except for this:
```
"fo
o" "ba
’"
```

For some reason, this still has the wrong offset.

a651126458
2025-01-11 19:09:05 -05:00
Earlopain
c037f5a28c [ruby/prism] Fix parser translator ast for heredoc with written newlines
Heredocs that contain "\\n" don't start a new string node.

61d9d3a15e
2025-01-05 18:12:44 +00:00
Kevin Newton
2ab1b07b84 [ruby/prism] Simplify srange_find in parser compiler
34efacc618
2024-12-16 10:51:22 -05:00
Kevin Newton
cc967a470b [ruby/prism] Add do keyword tracking for While/Until
9686897290
2024-12-16 10:51:22 -05:00
Kevin Newton
417bb8d4fd [PRISM] Field renaming
Rename some fields that do not quite make sense.

* CaseMatchNode#consequent -> CaseMatchNode#else_clause
* CaseNode#consequent -> CaseNode#else_clause
* IfNode#consequent -> IfNode#subsequent
* RescueNode#consequent -> RescueNode#subsequent
* UnlessNode#consequent -> UnlessNode#else_clause
2024-08-28 15:06:53 -04:00
Kevin Newton
41a36b6853 [ruby/prism] Handle chomped bytesize with lines without newlines
1528d3c019
2024-06-07 19:46:27 +00:00
Kevin Newton
79e9dea8de [ruby/prism] Ensure inner heredoc nodes have the correct location
100340bc6b
2024-06-07 19:46:20 +00:00
Kevin Newton
ce0a352e34 [ruby/prism] Use correct newlines for heredoc inner lines
4a9a7a62af

Co-authored-by: Jason Kim <jasonkim@github.com>
Co-authored-by: Adam Hess <HParker@github.com>
2024-06-07 19:46:16 +00:00
Kevin Newton
e440804d7a [ruby/prism] (parser) split up regexp content by lines
85b4a5f804
2024-06-04 20:28:54 +00:00
Kevin Newton
c2d3573c63 [ruby/prism] (parser) handle quoted symbols in hash patterns
461aa5e658
2024-06-04 20:28:54 +00:00
Kevin Newton
4354e45254 [ruby/prism] (parser) fix up srange_find to anchor at the start of the slice
aecce571d8
2024-06-04 20:28:54 +00:00
Kevin Newton
bbf9b5baad [ruby/prism] (parser) fix up nested multi write
12e079c97e
2024-06-04 20:28:53 +00:00
Kevin Newton
fd517a97be [ruby/prism] Revert "Revert "Properly destructure procarg0 in parser translation""
This reverts commit d8ae19d033.

df1eda2811
2024-06-04 15:05:52 +00:00
Kevin Newton
89ef1392f8 [ruby/prism] Revert "Properly destructure procarg0 in parser translation"
This reverts commit 823e931ff2.

d8ae19d033
2024-06-03 21:31:29 +00:00
Kevin Newton
5502890a2f [ruby/prism] Properly destructure procarg0 in parser translation
823e931ff2
2024-06-03 21:12:36 +00:00
Kevin Newton
b5c8fb9a3f [ruby/prism] Fix up heredoc location translation for parser
a4e164e22b
2024-05-31 19:31:53 +00:00
Kevin Newton
63ea77916a [ruby/prism] Match % strings in parser
840185110f
2024-05-31 19:31:53 +00:00
Kevin Newton
1b392ba7c6 [ruby/prism] Use correct opening and closing parenthesis for array pattern in parser
beed43922c
2024-05-31 19:31:53 +00:00
Kevin Newton
47f05dffa1 [ruby/prism] Match match_hash_var when quotes are used
f2a327449a
2024-05-31 19:31:52 +00:00
Kevin Newton
02b27aca50 [ruby/prism] Match parser for match_rest in pattern
785de2c39d
2024-05-31 19:31:52 +00:00
Kevin Newton
e575954887 [ruby/prism] Fix support for 'it' implicit local variable
53bbcfe513
2024-05-22 16:34:04 -04:00
Kevin Newton
89efb94fec [ruby/prism] Reconfigure rationals
This eliminates the subnode on RationalNode and replaces it with two
integer fields, which represent the ratio for the rational. It also
reduces those two integers if they both fit into 32 bits.

Importantly, this PR does not implement bignum reduction. That's something
I'd like to consider for the future, but it's simple enough for now to
leave them unreduced, which makes it more useful than it used to be.

86e06c7068
2024-05-21 14:27:46 -04:00
Kevin Newton
2e44664547 [ruby/prism] operator rename
b5e47f5c42
2024-05-10 11:47:48 -04:00
Kevin Newton
5758e45657 [ruby/prism] Change ConstantPathNode#child to ConstantPathNode#{name,name_loc}
This has been requested for a long time, and I'm finally doing it
now. Unfortunately this is a breaking change for all of the APIs.

I've added in a Ruby method for `#child` that is deprecated so that
existing usage doesn't break, but for everyone else this is going
to be a bit of a pain.

9cbe74464e
2024-05-03 11:11:57 -04:00
Kevin Newton
81433fd0f5 [ruby/prism] srange_find should only look on current line
3604aa15e7
2024-04-23 19:29:20 +00:00