archive/ruby
1
0
Fork 0
mirror of https://github.com/ruby/ruby.git synced 2025-08-28 15:36:16 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 11
84976 commits 30 branches 1174 tags 526 MiB
Commit graph

2505 commits

Author SHA1 Message Date
HASUMI Hitoshi
9b1e97b211 [Universal parser] DeVALUE of p->debug_lines and ast->body.script_lines 
This patch is part of universal parser work.

## Summary
- Decouple VALUE from members below:
  - `(struct parser_params *)->debug_lines`
  - `(rb_ast_t *)->body.script_lines`
- Instead, they are now `rb_parser_ary_t *`
  - They can also be a `(VALUE)FIXNUM` as before to hold line count
- `ISEQ_BODY(iseq)->variable.script_lines` remains VALUE
  - In order to do this,
  - Add `VALUE script_lines` param to `rb_iseq_new_with_opt()`
  - Introduce `rb_parser_build_script_lines_from()` to convert `rb_parser_ary_t *` into `VALUE`

## Other details
- Extend `rb_parser_ary_t *`. It previously could only store `rb_parser_ast_token *`, now can store script_lines, too
- Change tactics of building the top-level `SCRIPT_LINES__` in `yycompile0()`
  - Before: While parsing, each line of the script is added to `SCRIPT_LINES__[path]`
  - After: After `yyparse(p)`, `SCRIPT_LINES__[path]` will be built from `p->debug_lines`
- Remove the second parameter of `rb_parser_set_script_lines()` to make it simple
- Introduce `script_lines_free()` to be called from `rb_ast_free()` because the GC no longer takes care of the script_lines
- Introduce `rb_parser_string_deep_copy()` in parse.y to maintain script_lines when `rb_ruby_parser_free()` called
  - With regard to this, please see *Future tasks* below

## Future tasks
- Decouple IMEMO from `rb_ast_t *`
  - This lifts the five-members-restriction of Ruby object,
  - So we will be able to move the ownership of the `lex.string_buffer` from parser to AST
  - Then we remove `rb_parser_string_deep_copy()` to make the whole thing simple
 2024-04-15 20:51:54 +09:00
yui-knk
515e52a0b1 Emit warn event for duplicated hash keys on ripper 
Need to use `rb_warn` macro instead of calling `rb_compile_warn`
directly to emit `warn` event on ripper.
 2024-04-15 06:29:25 +09:00
yui-knk
5970386a2e Use rb_parser_string_t * for delayed.token 2024-04-14 09:08:19 +09:00
Nobuyoshi Nakada
e36988450e
[Bug #20423] Disallow anonymous block within argument forwarding 2024-04-12 16:07:49 +09:00
Jean Boussier
1b830740ba compile.c: use rb_enc_interned_str to reduce allocations 
The `rb_fstring(rb_enc_str_new())` pattern is inneficient because:

- It passes a mutable string to `rb_fstring` so if it has to be interned
  it will first be duped.
- It an equivalent interned string already exists, we allocated the string
  for nothing.

With `rb_enc_interned_str` we either directly get the pre-existing string
with 0 allocations, or efficiently directly intern the one we create
without first duping it.
 2024-04-11 09:04:31 +02:00
Nobuyoshi Nakada
207788466e
[Bug #20417] Block local variables do not need to warn about unused 2024-04-11 12:03:55 +09:00
S-H-GAMELINKS
76732b3e7b Remove unused AREF macro 2024-04-11 11:32:09 +09:00
yui-knk
39be11a17a Fix segv when parsing command by ripper 
89cfc15207 made this event dispatch to pass `Qundef`
to user defined callback method by mistake.
This commit fix it to be `nil`.
 2024-04-11 10:28:58 +09:00
yui-knk
70a0dd4162 Remove unused macro 2024-04-07 11:15:09 +09:00
yui-knk
7767db2379 Fix ripper to dispatch warning event for duplicated when clause 
Need to separate `check_literal_when` function for parser and
ripper otherwise warning event is not dispatched because
parser `rb_warning1` is used in ripper.
 2024-04-07 11:15:09 +09:00
yui-knk
02f5e627a4 Remove redundant conversion between int and object 2024-04-06 22:03:26 +09:00
yui-knk
977672731f Fix a variable name 
The first argument of `WARN_SPACE_CHAR` is always `c2` in caller
side, so `c` equals to `c2`.
 2024-04-06 22:03:08 +09:00
yui-knk
9d1d2de2fe Make nd_plen to be int to reduce rb_long2int usage 2024-04-05 14:25:04 +09:00
yui-knk
bd728a625a Remove unused functions from struct rb_parser_config_struct 2024-04-05 10:04:48 +09:00
yui-knk
f057741c5d NODE_LIT is not used anymore 2024-04-04 13:17:26 +09:00
yui-knk
6056773105 Move shareable_constant_value logic from parse.y to compile.c 2024-04-04 08:44:10 +09:00
yui-knk
e816ab0b0c Remove rb_imemo_tmpbuf_t from parser 
No parser semantic value types are `VALUE` then no need to
use imemo for managing semantic value stack anymore.
 2024-04-02 19:37:27 +09:00
yui-knk
799e854897 [Feature #20331] Simplify parser warnings for hash keys duplication and when clause duplication 
This commit simplifies warnings for hash keys duplication and when clause duplication,
based on the discussion of https://bugs.ruby-lang.org/issues/20331.
Warnings are reported only when strings are same to ohters.
 2024-04-02 08:26:58 +09:00
yui-knk
8066e3ea6e Remove VALUE from struct rb_strterm_struct 
In the past, it was imemo. However a075c55 changed it.
Therefore no need to use `VALUE` for the first field.
 2024-04-02 07:26:35 +09:00
Nobuyoshi Nakada
2ab9fb1c2e
[Bug #20398] Terminate token buffer at invalid octal number 2024-03-28 22:28:37 +09:00
Nobuyoshi Nakada
a850cd1a87 [Bug #20392] Block arguments duplication check at super 2024-03-26 17:37:47 +09:00
S-H-GAMELINKS
060a71d4e7 Fix Ripper memory allocation size when enabled Universal Parser 
The size of `struct parser_params` is 8 bytes difference in `ripper_s_allocate` and `rb_ruby_parser_allocate` when the universal parser is
enabled.
This causes a situation where `*r->p` is not fully initialized in `ripper_s_allocate` as shown below.

```console
(gdb) p *r->p
$2 = {heap = 0x0, lval = 0x0, yylloc = 0x0, lex = {strterm = 0x0, gets = 0x0, input = 0, string_buffer = {head = 0x0, last = 0x0}, lastlin
e = 0x0,
    nextline = 0x0, pbeg = 0x0, pcur = 0x0, pend = 0x0, ptok = 0x0, gets_ = {ptr = 0, call = 0x0}, state = EXPR_NONE, paren_nest = 0, lpar
_seen = 0,
    debug = 0, has_shebang = 0, token_seen = 0, token_info_enabled = 0, error_p = 0, cr_seen = 0, value = 0, result = 0, parsing_thread = 0, s_value = 0,
    s_lvalue = 0, s_value_stack = 2097}
````

This seems to cause `double free or corruption (!prev)` and SEGV.
So, fixing this by introduce `rb_ripper_parser_params_allocate` and `rb_ruby_parser_config` functions for Ripper, and `struct parser_params` same size is returned.
 2024-03-21 18:10:02 +09:00
yui-knk
8ba4d7d75f Fix unexpected node bug for shareable_constant_value: literal 
[Bug #20339]
[Bug #20341]

`const_decl_path` changes the value of `NODE **dest`, LHS of an assignment,
with `NODE_LIT` created by `const_decl_path`. `shareable_literal_constant` calls
`const_decl_path` via `ensure_shareable_node` multiple times if RHS of an assignment
is array or hash. This means `NODE **dest` argument of `const_decl_path` can be `NODE_LIT`
from the second time then causes `[BUG] unexpected node: NODE_LIT` in
`rb_node_const_decl_val`.
This commit change to not update `NODE **dest` in `const_decl_path` to
fix the issue.
 2024-03-21 11:29:09 +09:00
Nobuyoshi Nakada
0d5b16599a
[Bug #20218] Reject keyword arguments in index 2024-03-17 13:20:23 +09:00
Nobuyoshi Nakada
df5ef28233
[Bug #19918] Reject block passing in index 2024-03-17 13:18:37 +09:00
Peter Zhu
149a449964 Remove unused function in parse.y 
parse.y:2624:1: warning: unused function 'rb_parser_ary_new'
 2024-03-12 11:28:28 -04:00
HASUMI Hitoshi
9a19cfd4cd [Universal Parser] Reduce dependence on RArray in parse.y 
- Introduce `rb_parser_ary_t` structure to partly eliminate RArray from parse.y
  - In this patch, `parser_params->tokens` and `parser_params->ast->node_buffer->tokens` are now `rb_parser_ary_t *`
  - Instead, `ast_node_all_tokens()` internally creates a Ruby Array object from the `rb_parser_ary_t`
  - Also, delete `rb_ast_tokens()` and `rb_ast_set_tokens()` in node.c

- Implement `rb_parser_str_escape()`
  - This is a port of the `rb_str_escape()` function in string.c
  - `rb_parser_str_escape()` does not depend on `VALUE` (RString)
  - Instead, it uses `rb_parser_stirng_t *`
  - This function works when --dump=y option passed

- Because WIP of the universal parser, similar functions like `rb_parser_tokens_free()` exist in both node.c and parse.y. Refactoring them may be needed in some way in the future

- Although we considered redesigning the structure: `ast->node_buffer->tokens` into `ast->tokens`, we leave it as it is because `rb_ast_t` is an imemo. (We will address it in the future)
 2024-03-12 17:17:52 +09:00
Nobuyoshi Nakada
d7730d3a0b
Constify literal_type unless universal parser 2024-02-27 19:40:39 +09:00
ydah
89f0c0ceb5 Use '\n'? instead of opt_nl 2024-02-27 07:50:42 +09:00
eileencodes
50ace992c7 [Bug #20234] Fix segv when parsing begin statement in method definition 
In a method definition, the `begin` may not have an `nd_body`. When that
happens we get a null expr back from `last_expr_node` which causes a
segv for the following examples:

```ruby
def (begin;end).foo; end
def (begin;else;end).foo; end
def (begin;ensure;else;end).foo; end
```

In addition, I've added tests for other cases that weren't causing a
segv but appeared untested.`

Fixes https://bugs.ruby-lang.org/issues/20234
 2024-02-23 10:59:10 -08:00
Peter Zhu
01f9b2ae41 Use rb_str_to_interned_str in parse.y 
This commit changes rb_fstring to rb_str_to_interned_str in parse.y.
rb_fstring is private so it shouldn't be used by ripper.
 2024-02-23 13:33:46 -05:00
yui-knk
3ca6da24e3 [Bug #20295] Fix SEGV when parsing invalid regexp 2024-02-23 11:12:08 +09:00
yui-knk
7d112b04c7 Use rb_encoding * as literal hash of NODE_ENCODING 
This reduces dependency on VALUE.
 2024-02-22 19:08:20 +09:00
ydah
b7f5c80036 Use terms? instead of opt_terms 2024-02-22 18:22:01 +09:00
yui-knk
91cb303531 Remove not used universal parser macros and functions 2024-02-21 13:36:45 +09:00
yui-knk
5fb574ae4e rb_parser_warn_duplicate_keys doesn't need to take care of NODE_LIT anymore 
NODE_LIT is created only for `shareable_constant_value`.
This means hash key node is never NODE_LIT.
 2024-02-21 10:57:32 +09:00
yui-knk
776dbbba72 Remove hack for ripper.y generation 
Before Rearchitect Ripper (89cfc15), parser and ripper used different
semantic value data type for same symbols.
"ext/ripper/tools/preproc.rb" replaced these types when it generated
ripper.y. Starting the line with other than `%token` suppressed
the type replacement.
However, after Rearchitect Ripper, both parser and ripper use
same semantic value data type. Therefore these comments are not needed
anymore.
 2024-02-21 09:16:09 +09:00
yui-knk
e7ab5d891c Introduce NODE_REGX to manage regexp literal 2024-02-21 08:06:48 +09:00
yui-knk
7cb8fd7800 Move ripper_validate_object to ripper_init.c.tmpl 2024-02-20 19:19:31 +09:00
yui-knk
474f41880f Suppress unused function warning for UNIVERSAL_PARSER build 
Suppress the warning:

```
parse.y:2221:1: warning: unused function 'rb_parser_str_hash' [-Wunused-function]
 2221 | rb_parser_str_hash(rb_parser_string_t *str)
      | ^~~~~~~~~~~~~~~~~~
```
 2024-02-20 19:07:40 +09:00
yui-knk
2a4b6ed37c Workaround for Prism::ParseTest#test_filepath for "unparser/corpus/literal/def.txt" 
See the discussion on https://github.com/ruby/ruby/pull/9923
 2024-02-20 17:33:58 +09:00
yui-knk
89cfc15207 [Feature #20257] Rearchitect Ripper 
Introduce another semantic value stack for Ripper so that
Ripper can manage both Node and Ruby Object separately.
This rearchitectutre of Ripper solves these issues.
Therefore adding test cases for them.

* [Bug 10436] https://bugs.ruby-lang.org/issues/10436
* [Bug 18988] https://bugs.ruby-lang.org/issues/18988
* [Bug 20055] https://bugs.ruby-lang.org/issues/20055

Checked the differences of `Ripper.sexp` for files under `/test/ruby`
are only on test_pattern_matching.rb.
The differences comes from the differences between
`new_hash_pattern_tail` functions between parser and Ripper.
Ripper `new_hash_pattern_tail` didn’t call `assignable` then
`kw_rest_arg` wasn’t marked as local variable.
This is also fixed by this commit.

```
--- a/./tmp/before/test_pattern_matching.rb
+++ b/./tmp/after/test_pattern_matching.rb
@@ -3607,7 +3607,7 @@
                  [:in,
                   [:hshptn, nil, [], [:var_field, [:@ident, “a”, [984, 13]]]],
                   [[:binary,
-                    [:vcall, [:@ident, “a”, [985, 10]]],
+                    [:var_ref, [:@ident, “a”, [985, 10]]],
                     :==,
                     [:hash, nil]]],
                   nil]]],
@@ -3662,7 +3662,7 @@
                  [:in,
                   [:hshptn, nil, [], [:var_field, [:@ident, “a”, [993, 13]]]],
                   [[:binary,
-                    [:vcall, [:@ident, “a”, [994, 10]]],
+                    [:var_ref, [:@ident, “a”, [994, 10]]],
                     :==,
                     [:hash,
                      [:assoclist_from_args,
@@ -3813,7 +3813,7 @@
                    [:command,
                     [:@ident, “raise”, [1022, 10]],
                     [:args_add_block,
-                     [[:vcall, [:@ident, “b”, [1022, 16]]]],
+                     [[:var_ref, [:@ident, “b”, [1022, 16]]]],
                      false]]],
                   [:else, [[:var_ref, [:@kw, “true”, [1024, 10]]]]]]]],
                nil,
@@ -3876,7 +3876,7 @@
                      [:@int, “0”, [1033, 15]]],
                     :“&&“,
                     [:binary,
-                     [:vcall, [:@ident, “b”, [1033, 20]]],
+                     [:var_ref, [:@ident, “b”, [1033, 20]]],
                      :==,
                      [:hash, nil]]]],
                   nil]]],
@@ -3946,7 +3946,7 @@
                      [:@int, “0”, [1042, 15]]],
                     :“&&“,
                     [:binary,
-                     [:vcall, [:@ident, “b”, [1042, 20]]],
+                     [:var_ref, [:@ident, “b”, [1042, 20]]],
                      :==,
                      [:hash,
                       [:assoclist_from_args,
@@ -5206,7 +5206,7 @@
                      [[:assoc_new,
                        [:@label, “c:“, [1352, 22]],
                        [:@int, “0”, [1352, 25]]]]]],
-                   [:vcall, [:@ident, “r”, [1352, 29]]]],
+                   [:var_ref, [:@ident, “r”, [1352, 29]]]],
                   false]]],
                [:binary,
                 [:call,
@@ -5299,7 +5299,7 @@
                       [:assoc_new,
                        [:@label, “c:“, [1367, 34]],
                        [:@int, “0”, [1367, 37]]]]]],
-                   [:vcall, [:@ident, “r”, [1367, 41]]]],
+                   [:var_ref, [:@ident, “r”, [1367, 41]]]],
                   false]]],
                [:binary,
                 [:call,
@@ -5931,7 +5931,7 @@
              [:in,
               [:hshptn, nil, [], [:var_field, [:@ident, “r”, [1533, 11]]]],
               [[:binary,
-                [:vcall, [:@ident, “r”, [1534, 8]]],
+                [:var_ref, [:@ident, “r”, [1534, 8]]],
                 :==,
                 [:hash,
                  [:assoclist_from_args,
```
 2024-02-20 17:33:58 +09:00
Nobuyoshi Nakada
b1d70e4264
[Bug #20280] Check by rb_parser_enc_str_coderange 
Co-authored-by: Yuichiro Kaneko <spiketeika@gmail.com>
 2024-02-19 16:33:26 +09:00
Nobuyoshi Nakada
fcc55dc226
[Bug #20280] Raise SyntaxError on invalid encoding symbol 2024-02-19 16:33:26 +09:00
Yusuke Endoh
25d74b9527 Do not include a backtick in error messages and backtraces 
[Feature #16495]
 2024-02-15 18:42:31 +09:00
Nobuyoshi Nakada
c57880e68d
Show the invalid source encoding in messages 2024-02-15 13:39:33 +09:00
Peter Zhu
a71d1ed838 Fix memory leak when parsing invalid hash symbol 
For example:

    10.times do
      100_000.times do
        eval('{"\xC3": 1}')
      rescue EncodingError
      end

      puts `ps -o rss= -p #{$$}`
    end

Before:

    32032
    48464
    66112
    84192
    100592
    117520
    134096
    150656
    167168
    183760

After:

    17120
    17120
    17120
    17120
    18560
    18560
    18560
    18560
    18560
    18560
 2024-02-13 11:05:56 -05:00
yui-knk
038189b61f Use dedicated parser_string hash function 
Define and use `rb_parser_str_hash` for `rb_parser_string_t`
instead of `rb_str_hash` to remove dependency on `rb_str_hash`.
 2024-02-13 10:56:32 +09:00
yui-knk
8a345860d3 Warn duplication of __ENCODING__ on the hash 
```
$ ruby -e 'h = { __ENCODING__ => 1, __ENCODING__ => 2 }'
-e:1: warning: key #<Encoding:UTF-8> is duplicated and overwritten on line 1
```
 2024-02-13 08:40:14 +09:00
yui-knk
7fc89a9262 Use Node for warn_duplicate_keys st_table keys 2024-02-12 17:46:22 +09:00