Commit graph

52 commits

Author SHA1 Message Date
Kevin Newton
cc967a470b [ruby/prism] Add do keyword tracking for While/Until
9686897290
2024-12-16 10:51:22 -05:00
Kevin Newton
417bb8d4fd [PRISM] Field renaming
Rename some fields that do not quite make sense.

* CaseMatchNode#consequent -> CaseMatchNode#else_clause
* CaseNode#consequent -> CaseNode#else_clause
* IfNode#consequent -> IfNode#subsequent
* RescueNode#consequent -> RescueNode#subsequent
* UnlessNode#consequent -> UnlessNode#else_clause
2024-08-28 15:06:53 -04:00
Kevin Newton
41a36b6853 [ruby/prism] Handle chomped bytesize with lines without newlines
1528d3c019
2024-06-07 19:46:27 +00:00
Kevin Newton
79e9dea8de [ruby/prism] Ensure inner heredoc nodes have the correct location
100340bc6b
2024-06-07 19:46:20 +00:00
Kevin Newton
ce0a352e34 [ruby/prism] Use correct newlines for heredoc inner lines
4a9a7a62af

Co-authored-by: Jason Kim <jasonkim@github.com>
Co-authored-by: Adam Hess <HParker@github.com>
2024-06-07 19:46:16 +00:00
Kevin Newton
e440804d7a [ruby/prism] (parser) split up regexp content by lines
85b4a5f804
2024-06-04 20:28:54 +00:00
Kevin Newton
c2d3573c63 [ruby/prism] (parser) handle quoted symbols in hash patterns
461aa5e658
2024-06-04 20:28:54 +00:00
Kevin Newton
4354e45254 [ruby/prism] (parser) fix up srange_find to anchor at the start of the slice
aecce571d8
2024-06-04 20:28:54 +00:00
Kevin Newton
bbf9b5baad [ruby/prism] (parser) fix up nested multi write
12e079c97e
2024-06-04 20:28:53 +00:00
Kevin Newton
fd517a97be [ruby/prism] Revert "Revert "Properly destructure procarg0 in parser translation""
This reverts commit d8ae19d033.

df1eda2811
2024-06-04 15:05:52 +00:00
Kevin Newton
89ef1392f8 [ruby/prism] Revert "Properly destructure procarg0 in parser translation"
This reverts commit 823e931ff2.

d8ae19d033
2024-06-03 21:31:29 +00:00
Kevin Newton
5502890a2f [ruby/prism] Properly destructure procarg0 in parser translation
823e931ff2
2024-06-03 21:12:36 +00:00
Kevin Newton
b5c8fb9a3f [ruby/prism] Fix up heredoc location translation for parser
a4e164e22b
2024-05-31 19:31:53 +00:00
Kevin Newton
63ea77916a [ruby/prism] Match % strings in parser
840185110f
2024-05-31 19:31:53 +00:00
Kevin Newton
1b392ba7c6 [ruby/prism] Use correct opening and closing parenthesis for array pattern in parser
beed43922c
2024-05-31 19:31:53 +00:00
Kevin Newton
47f05dffa1 [ruby/prism] Match match_hash_var when quotes are used
f2a327449a
2024-05-31 19:31:52 +00:00
Kevin Newton
02b27aca50 [ruby/prism] Match parser for match_rest in pattern
785de2c39d
2024-05-31 19:31:52 +00:00
Kevin Newton
e575954887 [ruby/prism] Fix support for 'it' implicit local variable
53bbcfe513
2024-05-22 16:34:04 -04:00
Kevin Newton
89efb94fec [ruby/prism] Reconfigure rationals
This eliminates the subnode on RationalNode and replaces it with two
integer fields, which represent the ratio for the rational. It also
reduces those two integers if they both fit into 32 bits.

Importantly, this PR does not implement bignum reduction. That's something
I'd like to consider for the future, but it's simple enough for now to
leave them unreduced, which makes it more useful than it used to be.

86e06c7068
2024-05-21 14:27:46 -04:00
Kevin Newton
2e44664547 [ruby/prism] operator rename
b5e47f5c42
2024-05-10 11:47:48 -04:00
Kevin Newton
5758e45657 [ruby/prism] Change ConstantPathNode#child to ConstantPathNode#{name,name_loc}
This has been requested for a long time, and I'm finally doing it
now. Unfortunately this is a breaking change for all of the APIs.

I've added in a Ruby method for `#child` that is deprecated so that
existing usage doesn't break, but for everyone else this is going
to be a bit of a pain.

9cbe74464e
2024-05-03 11:11:57 -04:00
Kevin Newton
81433fd0f5 [ruby/prism] srange_find should only look on current line
3604aa15e7
2024-04-23 19:29:20 +00:00
Kevin Newton
2e80ceb6ff [ruby/prism] Fix it parameters for parser translation
2f3feb8d51
2024-04-19 16:29:39 +00:00
Kevin Newton
8f908a354e [ruby/prism] "Fix" transpose issue in parser compiler
593d637178
2024-04-18 18:34:42 +00:00
Stan Lo
0924ff2d39 [ruby/prism] Fix parser translation's heredoc whitespace calculation
Given this example:

```rb
<<~HEREDOC
  #{x}
HEREDOC
```

Both the parser gem and Prism's translation layer would generate the following AST:

```
s(:dstr,
  s(:begin,
    s(:int, 1)),
  s(:str, " a\n"))
```

However, the parser gem inserts a empty string node into this node's location, like:

```
<Parser::Source::Map::Heredoc:0x0000000104ce73b8
 @expression=#<Parser::Source::Range (string) 0...10>,
 @heredoc_body=#<Parser::Source::Range (string) 11...20>,
 @heredoc_end=#<Parser::Source::Range (string) 20...27>,
 @node=s(:dstr,
  s(:str, ""),
  s(:begin,
    s(:int, 1)),
  s(:str, " a\n"))>
```

This is required to calculate the correct whitespace for the heredoc body.

We need to adjust the translation layer to account for this.

With this fix, we also won't need to ignore the tilde heredoc fixture anymore.

e7372e3ba5
2024-04-12 13:55:35 +00:00
Kevin Newton
f45c9dbe87 [ruby/prism] Fix up some comments in the parser compiler
e2147cddd8
2024-04-04 15:29:10 -04:00
Koichi ITO
c0b8dee95a [ruby/prism] Fix an AST and token incompatibility for Prism::Translation::Parser
This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for dstring literal:

```ruby
"foo
  #{bar}"
```

## Parser gem (Expected)

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:str, "foo\n"),
  s(:str, "  "),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
[:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
[:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
[:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
[:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
[:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
[:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the AST and tokens returned by the Parser gem were different. In this case, `dstr` node should not be nested:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:dstr,
    s(:str, "foo\n"),
    s(:str, "  ")),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
[:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
[:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
[:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
[:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
[:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
[:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:str, "foo\n"),
  s(:str, "  "),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
[:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
[:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
[:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
[:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
[:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
[:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
```

c1652a9ee7
2024-03-15 12:31:40 +00:00
Kevin Newton
c45ad17fa1 [ruby/prism] Shareable constant nodes
473cfed6d0
2024-03-15 12:31:26 +00:00
Koichi ITO
0f076fa520 [ruby/prism] Fix an AST and token incompatibility for Prism::Translation::Parser
This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for empty xstring literal.

## Parser gem (Expected)

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
[:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the AST and tokens returned by the Parser gem were different:

```console
$ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
[:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
[:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
```

4ac89dcbb5
2024-03-13 16:02:10 +00:00
Koichi ITO
7eea268b70 [ruby/prism] Fix an AST incompatibility for Prism::Translation::Parser
Fixes ruby/prism#2480.

This PR fixes an AST incompatibility between Parser gem and `Prism::Translation::Parser`
for xstring literal with line breaks.

The following case in ruby/prism#2480 has already been addressed in ruby/prism#2576:

```ruby
"foo
bar"
```

cf85e72c55
2024-03-13 11:34:17 +00:00
Koichi ITO
2af6bc26c5 [ruby/prism] Fix an AST and token incompatibility for Prism::Translation::Parser
Fixes https://github.com/ruby/prism/pull/2515.

This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for string literal with line breaks.

c58466e5bf
2024-03-12 10:45:56 +00:00
Koichi ITO
0e4bfd08e5 [ruby/prism] Fix an AST and token incompatibility for Prism::Translation::Parser
Fixes https://github.com/ruby/prism/pull/2506.

This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for symbols quoted with line breaks.

06ab4df8cd
2024-03-07 14:05:20 +00:00
Ufuk Kayserilioglu
de411ef0b6 [ruby/prism] Small changes to make type-checking pass
5b2970e75b
2024-03-06 21:37:54 +00:00
Kevin Newton
4b5fc91617 [ruby/prism] Fix implicit local variables in hashes
05e0c6792c
2024-03-06 21:37:26 +00:00
Kevin Newton
bbaee0b7e4 [ruby/prism] Add stubs for remaining ripper visit methods
4ba9abf664
2024-03-06 16:42:36 +00:00
Kevin Newton
03a73fdc3d [ruby/prism] Add then keyword loc to when nodes
e1e613df16
2024-03-04 16:40:37 +00:00
Kevin Newton
85fe8b6b9c [ruby/prism] Update lib/prism/translation/parser/compiler.rb
dccfd83bc4
2024-03-04 16:26:47 +00:00
Koichi ITO
a03f92923b [ruby/prism] Fix incompatibility AST for regexp match in Prism::Translation::Parser
This PR fixes the following incompatibility AST for regexp match between Parser gem and Prism:

## Parser gem

Returns an `match_with_lvasgn` node:

```console
$ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/foo/ =~ bar")'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
s(:match_with_lvasgn,
  s(:regexp,
    s(:str, "foo"),
    s(:regopt)),
  s(:send, nil, :bar))
```

## Prism (`Prism::Translation::Parser`)

### Before

Returns an `send` node:

```console
$ bundle exec ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
s(:send,
  s(:regexp,
    s(:str, "foo"),
    s(:regopt)), :=~,
  s(:send, nil, :bar))
```

### After

Returns an `match_with_lvasgn` node:

```console
$ bundle exec ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
s(:match_with_lvasgn,
  s(:regexp,
    s(:str, "foo"),
    s(:regopt)),
  s(:send, nil, :bar))
```

## Background

Found due to incompatibility with RuboCop's `Performance/EndWith`, `Performance/StringInclude,
and `Performance/StartWith` cops.

## Note

This is the incompatibility when the receiver is a regular expression literal and `=~` is used.
Based on the node name `:match_with_lvasgn`, it appears that Prism's AST becomes more accurate
in cases like `visit_match_write_node` only.

However, as shown in the background, the current behavior of Parser gem is not like this.
Considering compatibility with the published AST of Parser gem, the AST incompatibility will be addressed.

This lvar-injecting feature appears to have not been supported by Parser gem for a long time:
https://github.com/whitequark/parser/issues/69#issuecomment-19506391

There seems to be no indication that it will be supported.

This PR prioritizes AST compatibility between the Parser gem and Prism.
However, it is unclear whether this is the best approach.

dff4abb170
2024-03-04 16:26:46 +00:00
Koichi ITO
f8dd2342bf [ruby/prism] Fix an incorrect parsing for Prism::Translation::Parser
This PR fixes an incorrect parsing for `Prism::Translation::Parser`
when one-line pattern mathing with Ruby 2.7 runtime.

## Expected

Parsing should be done based on the specified Ruby parsing version,
independent of the Ruby runtime version. When parsing for Ruby 3.3,
it should return `:match_pattern_p` node:

```console
$ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("foo in bar")'
ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-darwin19]
s(:match_pattern_p,
s(:send, nil, :foo),
  s(:match_var, :bar))
```

## Actual

When parsing with Ruby 2.7 runtime, `match_pattern` node is returned,
even though it is expected to parse for Ruby 3.3:

```console
$ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("foo in bar")'
ruby 2.7.8p225 (2023-03-30 revision 1f4d455848) [x86_64-darwin19]
s(:match_pattern,
s(:send, nil, :foo),
  s(:match_var, :bar))
```

The cause was the use of `RUBY_VERSION` for condition logic,
which made it dependent on runtime Ruby version.
`Prism::Translation::Parser` supports parsing for Ruby 3.3+.
Therefore, the condition for parsing Ruby 2.7, which is not supported, is being removed.

## Background

Found due to incompatibility with RuboCop's `Layout/SpaceAroundKeyword` and `Style/TernaryParentheses` cops.

e752e251d2
2024-02-29 16:06:40 +00:00
Kevin Newton
1ce3d9acbf [ruby/prism] Fix xstring heredoc parser translator
4e0f703975
2024-02-26 14:48:17 +00:00
Kevin Newton
99d0f687fc [ruby/prism] Fix parser translator for pinned expression
eeae07193b
2024-02-26 14:35:00 +00:00
Kevin Newton
af3145bb24 [ruby/prism] Handle negated numeric in parser translation
5877a95be4
2024-02-26 14:26:53 +00:00
Kevin Newton
13301587cf [ruby/prism] Fix block_pass for []=
bf79206220
2024-02-16 20:41:52 +00:00
Kevin Newton
14a7277da1 [ruby/prism] Speed up creating Ruby AST
When creating the Ruby AST, we were previously allocating Location
objects for every node and every inner location. Instead, this
commit changes it to pack both the start offset and length into a
single u64 and pass that into the nodes. Then, when the locations
are requested via a reader method, we lazily allocate the Location
objects.

de203dca83

Co-Authored-By: Aaron Patterson <tenderlove@ruby-lang.org>
2024-02-15 20:39:50 +00:00
Kevin Newton
aad3c36bdf [ruby/prism] Support for Ruby 2.7
1a15b70a8e
2024-02-07 16:54:34 +00:00
Kevin Newton
0b5be2f9e9 Sync to latest prism 2024-02-05 11:07:07 -05:00
Kevin Newton
d2f004cf6a [ruby/prism] Fix hash pairs in patterns
b7ab29daa0
2024-02-02 21:16:04 +00:00
Kevin Newton
420a6349ec [ruby/prism] Small fixes for the parser translator
4327051c86
2024-02-02 13:36:23 -05:00
Kevin Newton
731367d0ab [ruby/prism] Fix up CI
224ea85565
2024-01-30 18:45:19 +00:00
Kevin Newton
e050097beb [ruby/prism] Raise diagnostics for parser
102b4a16f5
2024-01-29 16:09:47 +00:00