272 commits

Author	SHA1	Message	Date
tomoya ishida	a4a6019550	[Bug #21202] Fix wrong token concat while tokenizing nested unterminated heredoc (#13000)	2025-03-29 20:46:43 +09:00
Peter Zhu	dfc9e978fb	[Bug #21004] Fix memory leak with "it" in parse.y ... Parsing `-> do it end` in parse.y leaks memory. We can see this in the Valgrind output: 56 bytes in 1 blocks are definitely lost in loss record 1 of 6 at 0x484E0DC: calloc (vg_replace_malloc.c:1675) by 0x188970: calloc1 (default.c:1472) by 0x188970: rb_gc_impl_calloc (default.c:8208) by 0x188970: ruby_xcalloc_body (gc.c:4598) by 0x18B8BC: ruby_xcalloc (gc.c:4592) by 0x21DCCA70: new_locations_lambda_body (ripper.y:12844) by 0x21DCCA70: ripper_yyparse (ripper.y:5194) by 0x21DDA521: rb_ruby_ripper_parse0 (ripper.y:15798)	2025-01-05 09:06:12 +09:00
Nobuyoshi Nakada	6bbb470dc7	[Bug #20504] Move dynamic regexp concatenation to iseq compiler	2025-01-03 10:25:15 +09:00
Nobuyoshi Nakada	e4ec2128ae	[Bug #20990] Reject escaped multibyte char with control/meta prefix	2024-12-28 18:40:37 +09:00
Nobuyoshi Nakada	0ccc7657f3	Ripper: Fix duplicate regexp errors	2024-12-28 11:35:00 +09:00
Nobuyoshi Nakada	fb18bb183c	[Bug #20989] Ripper: Pass compile_error ... For the universal parser, `rb_parser_reg_fragment_check` function is shared between the parser and ripper. However `parser_params` struct is partially different, and `compile_error` function depends on that part indirectly.	2024-12-28 11:25:57 +09:00
Nobuyoshi Nakada	7b2ae8df90	[Bug #20969] Pass assignable from ripper ... For the universal parser, `rb_reg_named_capture_assign_iter_impl` function is shared between the parser and ripper. However `parser_params` struct is partially different, and `assignable` function depends on that part indirectly.	2024-12-19 23:20:09 +09:00
tompng	6743e6285a	[Bug #20784] Fix incomplete character syntax followed by EOF	2024-10-05 15:59:01 +09:00
Peter Zhu	584559d86a	Fix leak of token_info when Ripper#warn jumps ... For example, the following code leaks: class MyRipper < Ripper def initialize(src, &blk) super(src) @blk = blk end def warn(msg, *args) = @blk.call(msg) end $VERBOSE = true def call_parse = MyRipper.new("if true\n end\n") { |msg| return msg }.parse 10.times do 500_000.times do call_parse end puts `ps -o rss= -p #{$$}` end Before: 37536 53744 70064 86448 102576 119120 135248 151216 167744 183824 After: 19280 19696 19728 20336 20448 21408 21616 21616 21824 21840	2024-08-07 09:14:14 -04:00
Peter Zhu	ced35800d4	Fix leak in warning of duplicate keys when Ripper#warn jumps ... For example, the following code leaks: class MyRipper < Ripper def initialize(src, &blk) super(src) @blk = blk end def warn(msg, *args) = @blk.call(msg) end $VERBOSE = true def call_parse = MyRipper.new("if true\n end\n") { |msg| return msg }.parse 10.times do 500_000.times do call_parse end puts `ps -o rss= -p #{$$}` end Before: 34832 51952 69760 88048 105344 123040 141152 159152 176656 194272 After: 18400 20256 20272 20272 20272 20304 20368 20368 20368 20400	2024-08-06 10:19:50 -04:00
Peter Zhu	6358397490	Fix leak of AST when Ripper#compile_error jumps ... For example, the following script leaks: class MyRipper < Ripper def initialize(src, &blk) super(src) @blk = blk end def compile_error(msg) = @blk.call(msg) end def call_parse = MyRipper.new("/") { |msg| return msg }.parse 10.times do 100_000.times do call_parse end puts `ps -o rss= -p #{$$}` end Before: 93952 169040 244224 318784 394432 468224 544048 618560 693776 768384 After: 19776 19776 20352 20880 20912 21408 21328 21152 21472 20944	2024-07-31 14:47:44 -04:00
Nobuyoshi Nakada	97449338d6	[Bug #20649] Allow nil as 2nd argument of assign_error ... Fallback to the last token element in that case, for the backward compatibilities.	2024-07-24 22:18:36 +09:00
Nobuyoshi Nakada	a1f72a563b	[Bug #20579] ripper: Dispatch spaces at END-OF-INPUT without newline	2024-06-14 17:54:02 +09:00
Nobuyoshi Nakada	2e59cf00cc	[Bug #20578] ripper: Fix dispatching part at invalid escapes	2024-06-14 15:02:15 +09:00
Nobuyoshi Nakada	9e28354705	ripper: Fix excess compile_error at simple backref op_asgn ... Fix up 89cfc15207.	2024-06-07 11:28:38 +09:00
Nobuyoshi Nakada	27321290d9	[Bug #20521] ripper: Clean up strterm	2024-06-06 20:43:56 +09:00
Nobuyoshi Nakada	05553cf22d	[Bug #20517] Make a multibyte character one token at meta escape	2024-06-01 19:33:12 +09:00
Kevin Newton	47f0965269	Update duplicated when clause warning message	2024-05-24 12:36:54 -04:00
Nobuyoshi Nakada	5695c5df95	ripper: Fix opassign when assignment to backref variables	2024-05-12 15:38:22 +09:00
Nobuyoshi Nakada	a1fb6cc978	ripper: Use $& instead of quoting charaters in tests	2024-05-12 15:36:42 +09:00
yui-knk	7e604a0263	Fix SEGV when ripper hits backref_error on command_asgn or arg	2024-05-11 20:47:15 +09:00
Peter Zhu	7ef8bb129f	Fix memory leak in Ripper.sexp ... rb_ast_dispose does not free the rb_ast_t causing it to be leaked. This commit changes it to use rb_ast_free instead. For example: require "ripper" 10.times do 100_000.times do Ripper.sexp("") end puts `ps -o rss= -p #{$$}` end Before: 27648 32512 37376 42240 47232 52224 57344 62208 67072 71936 After: 22784 22784 22784 22784 22912 22912 22912 22912 22912 22912	2024-05-01 11:09:54 -04:00
yui-knk	515e52a0b1	Emit warn event for duplicated hash keys on ripper ... Need to use `rb_warn` macro instead of calling `rb_compile_warn` directly to emit `warn` event on ripper.	2024-04-15 06:29:25 +09:00
yui-knk	39be11a17a	Fix segv when parsing command by ripper ... 89cfc15207 made this event dispatch to pass `Qundef` to user defined callback method by mistake. This commit fix it to be `nil`.	2024-04-11 10:28:58 +09:00
yui-knk	7767db2379	Fix ripper to dispatch warning event for duplicated when clause ... Need to separate `check_literal_when` function for parser and ripper otherwise warning event is not dispatched because parser `rb_warning1` is used in ripper.	2024-04-07 11:15:09 +09:00
yui-knk	89cfc15207	[Feature #20257] Rearchitect Ripper ... Introduce another semantic value stack for Ripper so that Ripper can manage both Node and Ruby Object separately. This rearchitectutre of Ripper solves these issues. Therefore adding test cases for them. * [Bug 10436] https://bugs.ruby-lang.org/issues/10436 * [Bug 18988] https://bugs.ruby-lang.org/issues/18988 * [Bug 20055] https://bugs.ruby-lang.org/issues/20055 Checked the differences of `Ripper.sexp` for files under `/test/ruby` are only on test_pattern_matching.rb. The differences comes from the differences between `new_hash_pattern_tail` functions between parser and Ripper. Ripper `new_hash_pattern_tail` didn’t call `assignable` then `kw_rest_arg` wasn’t marked as local variable. This is also fixed by this commit. ``` --- a/./tmp/before/test_pattern_matching.rb +++ b/./tmp/after/test_pattern_matching.rb @@ -3607,7 +3607,7 @@ [:in, [:hshptn, nil, [], [:var_field, [:@ident, “a”, [984, 13]]]], [[:binary, - [:vcall, [:@ident, “a”, [985, 10]]], + [:var_ref, [:@ident, “a”, [985, 10]]], :==, [:hash, nil]]], nil]]], @@ -3662,7 +3662,7 @@ [:in, [:hshptn, nil, [], [:var_field, [:@ident, “a”, [993, 13]]]], [[:binary, - [:vcall, [:@ident, “a”, [994, 10]]], + [:var_ref, [:@ident, “a”, [994, 10]]], :==, [:hash, [:assoclist_from_args, @@ -3813,7 +3813,7 @@ [:command, [:@ident, “raise”, [1022, 10]], [:args_add_block, - [[:vcall, [:@ident, “b”, [1022, 16]]]], + [[:var_ref, [:@ident, “b”, [1022, 16]]]], false]]], [:else, [[:var_ref, [:@kw, “true”, [1024, 10]]]]]]]], nil, @@ -3876,7 +3876,7 @@ [:@int, “0”, [1033, 15]]], :“&&“, [:binary, - [:vcall, [:@ident, “b”, [1033, 20]]], + [:var_ref, [:@ident, “b”, [1033, 20]]], :==, [:hash, nil]]]], nil]]], @@ -3946,7 +3946,7 @@ [:@int, “0”, [1042, 15]]], :“&&“, [:binary, - [:vcall, [:@ident, “b”, [1042, 20]]], + [:var_ref, [:@ident, “b”, [1042, 20]]], :==, [:hash, [:assoclist_from_args, @@ -5206,7 +5206,7 @@ [[:assoc_new, [:@label, “c:“, [1352, 22]], [:@int, “0”, [1352, 25]]]]]], - [:vcall, [:@ident, “r”, [1352, 29]]]], + [:var_ref, [:@ident, “r”, [1352, 29]]]], false]]], [:binary, [:call, @@ -5299,7 +5299,7 @@ [:assoc_new, [:@label, “c:“, [1367, 34]], [:@int, “0”, [1367, 37]]]]]], - [:vcall, [:@ident, “r”, [1367, 41]]]], + [:var_ref, [:@ident, “r”, [1367, 41]]]], false]]], [:binary, [:call, @@ -5931,7 +5931,7 @@ [:in, [:hshptn, nil, [], [:var_field, [:@ident, “r”, [1533, 11]]]], [[:binary, - [:vcall, [:@ident, “r”, [1534, 8]]], + [:var_ref, [:@ident, “r”, [1534, 8]]], :==, [:hash, [:assoclist_from_args, ```	2024-02-20 17:33:58 +09:00
Yusuke Endoh	25d74b9527	Do not include a backtick in error messages and backtraces ... [Feature #16495]	2024-02-15 18:42:31 +09:00
Nobuyoshi Nakada	e5e1f9813e	[Bug #19838] Flush delayed token nonconsecutive with the next token	2023-12-02 02:41:39 +09:00
Nobuyoshi Nakada	e36b9760fd	Dispatch invalid hex escape content too	2023-12-01 15:04:30 +09:00
Nobuyoshi Nakada	d503e1b95a	[Bug #20030] dispatch invalid escaped character without ignoring it	2023-12-01 15:04:30 +09:00
Hiroshi SHIBATA	7149543f24	Added TestRipper namespace to test_call_colon2	2023-11-02 15:13:40 +09:00
lukeg	1925c6d555	test/ripper: nest helper classes under TestRipper module ... Generic names like Node and NodeList should be namespaced properly.	2023-11-02 15:10:50 +09:00
Nobuyoshi Nakada	40efbc7e40	[Bug #19851] Ripper: Hide internal block argument ID	2023-08-26 02:08:53 +09:00
Peter Zhu	0b8f15575a	Fix memory leak for incomplete lambdas ... [Bug #19836] The parser does not free the chain of `struct vtable`, which causes memory leaks. The following script reproduces this issue: ``` 10.times do 100_000.times do Ripper.parse("-> {") end puts `ps -o rss= -p #{$$}` end ```	2023-08-09 14:06:58 -04:00
Peter Zhu	5bc8fceca8	Fix memory leak in parser for incomplete tokens ... [Bug #19835] The parser does not free the `tbl` of the `struct vtable` when there are leftover `lvtbl` in the parser. This causes a memory leak. The following script reproduces this issue: ``` 10.times do 100_000.times do Ripper.parse("class Foo") end puts `ps -o rss= -p #{$$}` end ```	2023-08-09 14:06:58 -04:00
Peter Zhu	4b6c584023	Remove --disable-gems for assert_separately ... assert_separately adds --disable=gems so we don't need to add --disable-gems when calling assert_separately.	2023-08-03 09:11:08 +09:00
Nobuyoshi Nakada	382678d411	[Bug #19788] Use the result of tCOLON2 event	2023-08-01 19:00:31 +09:00
Nobuyoshi Nakada	5c77402d88	Fix null pointer access in Ripper#initialize ... In `rb_ruby_ripper_parser_allocate`, `r->p` is NULL between creating `self` and `parser_params` assignment. As GC can happen there, the typed-data functions for it need to consider the case.	2023-07-16 15:41:10 +09:00
Peter Zhu	27d3fa2af0	Increase memory leak test timeout ... The test times out on some platforms, so increase the timeout.	2023-06-28 13:47:48 -04:00
Peter Zhu	a500eb9f8c	Fix memory leak in Ripper ... The following script leaks memory in Ripper: ```ruby require "ripper" 20.times do 100_000.times do Ripper.parse("") end puts `ps -o rss= -p #{$$}` end ```	2023-06-28 09:50:51 -04:00
Nobuyoshi Nakada	6be402e172	[Bug #19736] Recover after unterminated interpolation	2023-06-20 20:10:46 +09:00
Nobuyoshi Nakada	ac8a16237c	[Bug #19563] Yield words separators per lines ... So that newlines across a here-doc terminator will be separated tokens. Cf. https://github.com/ruby/irb/pull/558	2023-04-07 23:13:56 +09:00
Kazuki Tsujimoto	4ac8d11724	* in an array pattern should not be parsed as nil in ripper ... After 6c0925ba70, it was impossible to distinguish between the presence or absence of `*`. # Before the commit Ripper.sexp('0 in []')[1][0][2][1] #=> [:aryptn, nil, nil, nil, nil] Ripper.sexp('0 in [*]')[1][0][2][1] #=> [:aryptn, nil, nil, [:var_field, nil], nil] # After the commit Ripper.sexp('0 in []')[1][0][2][1] #=> [:aryptn, nil, nil, nil, nil] Ripper.sexp('0 in [*]')[1][0][2][1] #=> [:aryptn, nil, nil, nil, nil] This commit reverts it.	2023-04-01 16:35:24 +09:00
Nobuyoshi Nakada	040fab3782	Ripper: Add keyword options to assert_parse_files	2023-03-28 17:14:34 +09:00
Kazuki Tsujimoto	d51529244f	[Bug #19175] p_kw without a sub pattern should be `assignable'	2023-03-26 18:57:34 +09:00
Kazuki Tsujimoto	6c0925ba70	[Bug #19175] p_rest should be `assignable' ... It should also check for duplicate names.	2023-03-26 18:56:21 +09:00
Nobuyoshi Nakada	fad48fefe1	[Bug #19399] Parsing invalid heredoc inside block parameter ... Although this is of course invalid as Ruby code, allow to just parse and tokenize.	2023-02-02 12:20:10 +09:00
Nobuyoshi Nakada	3becc4a105	[Bug #19291] Rewind to the previous line ... When rewinding looking ahead after newline token, also reset the last line string, the pointers to it, and the location, not only the line number.	2023-01-02 16:12:08 +09:00
Nobuyoshi Nakada	1912bf5461	Adjut indent [ci skip]	2023-01-02 16:12:08 +09:00
Nobuyoshi Nakada	ace2eee544	[Bug #18963] Separate string contents by here document terminator	2022-08-28 09:29:24 +09:00