archive/ruby
1
0
Fork 0
mirror of https://github.com/ruby/ruby.git synced 2025-08-23 13:04:13 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
91898 commits 30 branches 1174 tags 508 MiB
Commit graph

2684 commits

Author SHA1 Message Date
S-H-GAMELINKS
060a71d4e7 Fix Ripper memory allocation size when enabled Universal Parser 
The size of `struct parser_params` is 8 bytes difference in `ripper_s_allocate` and `rb_ruby_parser_allocate` when the universal parser is
enabled.
This causes a situation where `*r->p` is not fully initialized in `ripper_s_allocate` as shown below.

```console
(gdb) p *r->p
$2 = {heap = 0x0, lval = 0x0, yylloc = 0x0, lex = {strterm = 0x0, gets = 0x0, input = 0, string_buffer = {head = 0x0, last = 0x0}, lastlin
e = 0x0,
    nextline = 0x0, pbeg = 0x0, pcur = 0x0, pend = 0x0, ptok = 0x0, gets_ = {ptr = 0, call = 0x0}, state = EXPR_NONE, paren_nest = 0, lpar
_seen = 0,
    debug = 0, has_shebang = 0, token_seen = 0, token_info_enabled = 0, error_p = 0, cr_seen = 0, value = 0, result = 0, parsing_thread = 0, s_value = 0,
    s_lvalue = 0, s_value_stack = 2097}
````

This seems to cause `double free or corruption (!prev)` and SEGV.
So, fixing this by introduce `rb_ripper_parser_params_allocate` and `rb_ruby_parser_config` functions for Ripper, and `struct parser_params` same size is returned.
 2024-03-21 18:10:02 +09:00
yui-knk
8ba4d7d75f Fix unexpected node bug for shareable_constant_value: literal 
[Bug #20339]
[Bug #20341]

`const_decl_path` changes the value of `NODE **dest`, LHS of an assignment,
with `NODE_LIT` created by `const_decl_path`. `shareable_literal_constant` calls
`const_decl_path` via `ensure_shareable_node` multiple times if RHS of an assignment
is array or hash. This means `NODE **dest` argument of `const_decl_path` can be `NODE_LIT`
from the second time then causes `[BUG] unexpected node: NODE_LIT` in
`rb_node_const_decl_val`.
This commit change to not update `NODE **dest` in `const_decl_path` to
fix the issue.
 2024-03-21 11:29:09 +09:00
Nobuyoshi Nakada
0d5b16599a
[Bug #20218] Reject keyword arguments in index 2024-03-17 13:20:23 +09:00
Nobuyoshi Nakada
df5ef28233
[Bug #19918] Reject block passing in index 2024-03-17 13:18:37 +09:00
Peter Zhu
149a449964 Remove unused function in parse.y 
parse.y:2624:1: warning: unused function 'rb_parser_ary_new'
 2024-03-12 11:28:28 -04:00
HASUMI Hitoshi
9a19cfd4cd [Universal Parser] Reduce dependence on RArray in parse.y 
- Introduce `rb_parser_ary_t` structure to partly eliminate RArray from parse.y
  - In this patch, `parser_params->tokens` and `parser_params->ast->node_buffer->tokens` are now `rb_parser_ary_t *`
  - Instead, `ast_node_all_tokens()` internally creates a Ruby Array object from the `rb_parser_ary_t`
  - Also, delete `rb_ast_tokens()` and `rb_ast_set_tokens()` in node.c

- Implement `rb_parser_str_escape()`
  - This is a port of the `rb_str_escape()` function in string.c
  - `rb_parser_str_escape()` does not depend on `VALUE` (RString)
  - Instead, it uses `rb_parser_stirng_t *`
  - This function works when --dump=y option passed

- Because WIP of the universal parser, similar functions like `rb_parser_tokens_free()` exist in both node.c and parse.y. Refactoring them may be needed in some way in the future

- Although we considered redesigning the structure: `ast->node_buffer->tokens` into `ast->tokens`, we leave it as it is because `rb_ast_t` is an imemo. (We will address it in the future)
 2024-03-12 17:17:52 +09:00
Nobuyoshi Nakada
d7730d3a0b
Constify literal_type unless universal parser 2024-02-27 19:40:39 +09:00
ydah
89f0c0ceb5 Use '\n'? instead of opt_nl 2024-02-27 07:50:42 +09:00
eileencodes
50ace992c7 [Bug #20234] Fix segv when parsing begin statement in method definition 
In a method definition, the `begin` may not have an `nd_body`. When that
happens we get a null expr back from `last_expr_node` which causes a
segv for the following examples:

```ruby
def (begin;end).foo; end
def (begin;else;end).foo; end
def (begin;ensure;else;end).foo; end
```

In addition, I've added tests for other cases that weren't causing a
segv but appeared untested.`

Fixes https://bugs.ruby-lang.org/issues/20234
 2024-02-23 10:59:10 -08:00
Peter Zhu
01f9b2ae41 Use rb_str_to_interned_str in parse.y 
This commit changes rb_fstring to rb_str_to_interned_str in parse.y.
rb_fstring is private so it shouldn't be used by ripper.
 2024-02-23 13:33:46 -05:00
yui-knk
3ca6da24e3 [Bug #20295] Fix SEGV when parsing invalid regexp 2024-02-23 11:12:08 +09:00
yui-knk
7d112b04c7 Use rb_encoding * as literal hash of NODE_ENCODING 
This reduces dependency on VALUE.
 2024-02-22 19:08:20 +09:00
ydah
b7f5c80036 Use terms? instead of opt_terms 2024-02-22 18:22:01 +09:00
yui-knk
91cb303531 Remove not used universal parser macros and functions 2024-02-21 13:36:45 +09:00
yui-knk
5fb574ae4e rb_parser_warn_duplicate_keys doesn't need to take care of NODE_LIT anymore 
NODE_LIT is created only for `shareable_constant_value`.
This means hash key node is never NODE_LIT.
 2024-02-21 10:57:32 +09:00
yui-knk
776dbbba72 Remove hack for ripper.y generation 
Before Rearchitect Ripper (89cfc15), parser and ripper used different
semantic value data type for same symbols.
"ext/ripper/tools/preproc.rb" replaced these types when it generated
ripper.y. Starting the line with other than `%token` suppressed
the type replacement.
However, after Rearchitect Ripper, both parser and ripper use
same semantic value data type. Therefore these comments are not needed
anymore.
 2024-02-21 09:16:09 +09:00
yui-knk
e7ab5d891c Introduce NODE_REGX to manage regexp literal 2024-02-21 08:06:48 +09:00
yui-knk
7cb8fd7800 Move ripper_validate_object to ripper_init.c.tmpl 2024-02-20 19:19:31 +09:00
yui-knk
474f41880f Suppress unused function warning for UNIVERSAL_PARSER build 
Suppress the warning:

```
parse.y:2221:1: warning: unused function 'rb_parser_str_hash' [-Wunused-function]
 2221 | rb_parser_str_hash(rb_parser_string_t *str)
      | ^~~~~~~~~~~~~~~~~~
```
 2024-02-20 19:07:40 +09:00
yui-knk
2a4b6ed37c Workaround for Prism::ParseTest#test_filepath for "unparser/corpus/literal/def.txt" 
See the discussion on https://github.com/ruby/ruby/pull/9923
 2024-02-20 17:33:58 +09:00
yui-knk
89cfc15207 [Feature #20257] Rearchitect Ripper 
Introduce another semantic value stack for Ripper so that
Ripper can manage both Node and Ruby Object separately.
This rearchitectutre of Ripper solves these issues.
Therefore adding test cases for them.

* [Bug 10436] https://bugs.ruby-lang.org/issues/10436
* [Bug 18988] https://bugs.ruby-lang.org/issues/18988
* [Bug 20055] https://bugs.ruby-lang.org/issues/20055

Checked the differences of `Ripper.sexp` for files under `/test/ruby`
are only on test_pattern_matching.rb.
The differences comes from the differences between
`new_hash_pattern_tail` functions between parser and Ripper.
Ripper `new_hash_pattern_tail` didn’t call `assignable` then
`kw_rest_arg` wasn’t marked as local variable.
This is also fixed by this commit.

```
--- a/./tmp/before/test_pattern_matching.rb
+++ b/./tmp/after/test_pattern_matching.rb
@@ -3607,7 +3607,7 @@
                  [:in,
                   [:hshptn, nil, [], [:var_field, [:@ident, “a”, [984, 13]]]],
                   [[:binary,
-                    [:vcall, [:@ident, “a”, [985, 10]]],
+                    [:var_ref, [:@ident, “a”, [985, 10]]],
                     :==,
                     [:hash, nil]]],
                   nil]]],
@@ -3662,7 +3662,7 @@
                  [:in,
                   [:hshptn, nil, [], [:var_field, [:@ident, “a”, [993, 13]]]],
                   [[:binary,
-                    [:vcall, [:@ident, “a”, [994, 10]]],
+                    [:var_ref, [:@ident, “a”, [994, 10]]],
                     :==,
                     [:hash,
                      [:assoclist_from_args,
@@ -3813,7 +3813,7 @@
                    [:command,
                     [:@ident, “raise”, [1022, 10]],
                     [:args_add_block,
-                     [[:vcall, [:@ident, “b”, [1022, 16]]]],
+                     [[:var_ref, [:@ident, “b”, [1022, 16]]]],
                      false]]],
                   [:else, [[:var_ref, [:@kw, “true”, [1024, 10]]]]]]]],
                nil,
@@ -3876,7 +3876,7 @@
                      [:@int, “0”, [1033, 15]]],
                     :“&&“,
                     [:binary,
-                     [:vcall, [:@ident, “b”, [1033, 20]]],
+                     [:var_ref, [:@ident, “b”, [1033, 20]]],
                      :==,
                      [:hash, nil]]]],
                   nil]]],
@@ -3946,7 +3946,7 @@
                      [:@int, “0”, [1042, 15]]],
                     :“&&“,
                     [:binary,
-                     [:vcall, [:@ident, “b”, [1042, 20]]],
+                     [:var_ref, [:@ident, “b”, [1042, 20]]],
                      :==,
                      [:hash,
                       [:assoclist_from_args,
@@ -5206,7 +5206,7 @@
                      [[:assoc_new,
                        [:@label, “c:“, [1352, 22]],
                        [:@int, “0”, [1352, 25]]]]]],
-                   [:vcall, [:@ident, “r”, [1352, 29]]]],
+                   [:var_ref, [:@ident, “r”, [1352, 29]]]],
                   false]]],
                [:binary,
                 [:call,
@@ -5299,7 +5299,7 @@
                       [:assoc_new,
                        [:@label, “c:“, [1367, 34]],
                        [:@int, “0”, [1367, 37]]]]]],
-                   [:vcall, [:@ident, “r”, [1367, 41]]]],
+                   [:var_ref, [:@ident, “r”, [1367, 41]]]],
                   false]]],
                [:binary,
                 [:call,
@@ -5931,7 +5931,7 @@
              [:in,
               [:hshptn, nil, [], [:var_field, [:@ident, “r”, [1533, 11]]]],
               [[:binary,
-                [:vcall, [:@ident, “r”, [1534, 8]]],
+                [:var_ref, [:@ident, “r”, [1534, 8]]],
                 :==,
                 [:hash,
                  [:assoclist_from_args,
```
 2024-02-20 17:33:58 +09:00
Nobuyoshi Nakada
b1d70e4264
[Bug #20280] Check by rb_parser_enc_str_coderange 
Co-authored-by: Yuichiro Kaneko <spiketeika@gmail.com>
 2024-02-19 16:33:26 +09:00
Nobuyoshi Nakada
fcc55dc226
[Bug #20280] Raise SyntaxError on invalid encoding symbol 2024-02-19 16:33:26 +09:00
Yusuke Endoh
25d74b9527 Do not include a backtick in error messages and backtraces 
[Feature #16495]
 2024-02-15 18:42:31 +09:00
Nobuyoshi Nakada
c57880e68d
Show the invalid source encoding in messages 2024-02-15 13:39:33 +09:00
Peter Zhu
a71d1ed838 Fix memory leak when parsing invalid hash symbol 
For example:

    10.times do
      100_000.times do
        eval('{"\xC3": 1}')
      rescue EncodingError
      end

      puts `ps -o rss= -p #{$$}`
    end

Before:

    32032
    48464
    66112
    84192
    100592
    117520
    134096
    150656
    167168
    183760

After:

    17120
    17120
    17120
    17120
    18560
    18560
    18560
    18560
    18560
    18560
 2024-02-13 11:05:56 -05:00
yui-knk
038189b61f Use dedicated parser_string hash function 
Define and use `rb_parser_str_hash` for `rb_parser_string_t`
instead of `rb_str_hash` to remove dependency on `rb_str_hash`.
 2024-02-13 10:56:32 +09:00
yui-knk
8a345860d3 Warn duplication of __ENCODING__ on the hash 
```
$ ruby -e 'h = { __ENCODING__ => 1, __ENCODING__ => 2 }'
-e:1: warning: key #<Encoding:UTF-8> is duplicated and overwritten on line 1
```
 2024-02-13 08:40:14 +09:00
yui-knk
7fc89a9262 Use Node for warn_duplicate_keys st_table keys 2024-02-12 17:46:22 +09:00
yui-knk
fdd92c2d61 Fix the variable to be checked 
It should check the result of `rb_parser_search_nonascii`.
 2024-02-10 18:58:42 +09:00
yui-knk
33c1e082d0 Remove ruby object from string nodes 
String nodes holds ruby string object on `VALUE nd_lit`.
This commit changes it to `struct rb_parser_string *string`
to reduce dependency on ruby object.
Sometimes these strings are concatenated with other string
therefore string concatenate functions are needed.
 2024-02-09 14:20:17 +09:00
Peter Zhu
b2392c6be4 Fix memory leak when parsing invalid pattern matching 
If the pattern matching is invalid, then the pvtbl would get leaked. For
example:

    10.times do
      100_000.times do
        eval(<<~RUBY)
          case {a: 1}
          in {"a" => 1}
          end
        RUBY
      rescue SyntaxError
      end

      puts `ps -o rss= -p #{$$}`
    end

Before:

    28096
    44768
    61472
    78512
    94992
    111504
    128096
    144528
    161008
    177472

After:

    14096
    14112
    14112
    14176
    14208
    14240
    14240
    14240
    14240
    14240
 2024-02-07 12:15:33 -05:00
yui-knk
68b57ceb46 Use bool to check ascii only in parse_ident 
No need to use ENC_CODERANGE to record ascii only or not.
 2024-02-03 09:15:41 +09:00
S.H
f3df218f48
Introduced rb_node_const_decl_val function 
Introduce `rb_node_const_decl_val` function to allow `rb_ary_join` and
`rb_ary_reverse` functions to be removed from Universal Parser.
 2024-01-31 13:31:38 +09:00
Nobuyoshi Nakada
23b8337cd1
[Bug #20219] gettable returns NULL on error 2024-01-28 19:15:31 +09:00
Nobuyoshi Nakada
5f733a1ae7
[Bug #20217] rescue block is void only if all children are void 2024-01-28 18:44:09 +09:00
Nobuyoshi Nakada
fed877c791
[Bug #20217] return with ensure is a void value expression 2024-01-28 18:44:09 +09:00
Nobuyoshi Nakada
e018036d89
Rename nd_head in RNode_RESBODY as nd_next 2024-01-28 11:12:22 +09:00
Nobuyoshi Nakada
0f98d284f3
Remove unused nd_resq from RNode_ENSURE 2024-01-28 11:11:13 +09:00
S.H
9b40f42c22
Introduce NODE_ENCODING 
`__ENCODING__ `was managed by `NODE_LIT` with Encoding object. 

Introduce `NODE_ENCODING` for
1. `__ENCODING__` is detectable from AST Node.
2. Reduce dependency Ruby object for parse.y
 2024-01-27 08:11:10 +00:00
yui-knk
68b9a32a62 bvar is not NODE but ID 
Before this commit `ruby -y -e 'tap {|;x, y|}'` failed with SEGV.
This change fixes it.
 2024-01-27 16:50:35 +09:00
Nobuyoshi Nakada
ff55d6b8e1
Use token_seen and simplify comment_at_top 
Instead of scanning before the current comment.
 2024-01-25 15:06:14 +09:00
Jeremy Evans
4f77d8d328 Do not use ruby2_keywords for ... argument forwarding 
This allows ... argument forwarding to benefit from Allocationless
Anonymous Splat Forwarding, allowing the `f` call below to not
allocate an array or a hash.

```ruby
a = [1]
kw = {b: 2}

def c(a, b:)
end

def f(...)
  c(...)
end

f(*a, **kw)
```

This temporarily skips prism locals tests until prism is changed
to use * and ** for ..., instead of using ruby2_keywords.

Ignore failures in rbs bundled gems tests, since they fail due
to this change.
 2024-01-24 18:25:55 -08:00
yui-knk
ee7f63ebba Make lastline and nextline to be rb_parser_string 
This commit changes `struct parser_params` lastline and nextline
from `VALUE` (String object) to `rb_parser_string_t *` so that
dependency on Ruby Object is reduced.
`parser_string_buffer_t string_buffer` is added to `struct parser_params`
to manage `rb_parser_string_t` pointers of each line. All allocated line
strings are freed in `rb_ruby_parser_free`.
 2024-01-23 08:58:16 +09:00
yui-knk
3d19409637 Use index for referring to symbols in args rule instead of named references 
In `args: args ',' arg_splat`, `args` is not unique name.
Currently the associated rule is interpreted as
`$$ = rest_arg_append(p, $$, $3, &@$);`.
The action works as expected because `$$` is initialized with
`$1` before each action is executed.
However it's misleading then change to use index.
 2024-01-22 16:05:43 +09:00
Nobuyoshi Nakada
0610f555ea
Constify rb_global_parser_config 2024-01-14 17:55:11 +09:00
yui-knk
ccd45a1399 Stop using Array to manage dummy end token locations 
Before this commit, Array is used to store token locations
which expect `end` token, e.g. `class` and `module`.
This commit introduces dedicated struct to manage them
so that dependency on Ruby Object is reduced.
 2024-01-13 20:41:22 +09:00
S-H-GAMELINKS
524770d3dc Suppress warnings in parser_set_encode function 2024-01-12 22:46:07 +09:00
yui-knk
b35e21b388 Remove reference counter from rb_parser_config 
It's allocated outside of parser then no need to track
reference count in rb_parser_config.
 2024-01-12 21:17:41 +09:00
yui-knk
52d9e55903 Statically allocate parser config 2024-01-12 21:17:41 +09:00