Commit graph

36 commits

Author SHA1 Message Date
tomoya ishida
a4a6019550
[Bug #21202] Fix wrong token concat while tokenizing nested unterminated heredoc (#13000) 2025-03-29 20:46:43 +09:00
Nobuyoshi Nakada
e4ec2128ae
[Bug #20990] Reject escaped multibyte char with control/meta prefix 2024-12-28 18:40:37 +09:00
Nobuyoshi Nakada
a1f72a563b [Bug #20579] ripper: Dispatch spaces at END-OF-INPUT without newline 2024-06-14 17:54:02 +09:00
Nobuyoshi Nakada
2e59cf00cc [Bug #20578] ripper: Fix dispatching part at invalid escapes 2024-06-14 15:02:15 +09:00
Nobuyoshi Nakada
27321290d9 [Bug #20521] ripper: Clean up strterm 2024-06-06 20:43:56 +09:00
Nobuyoshi Nakada
05553cf22d
[Bug #20517] Make a multibyte character one token at meta escape 2024-06-01 19:33:12 +09:00
Yusuke Endoh
25d74b9527 Do not include a backtick in error messages and backtraces
[Feature #16495]
2024-02-15 18:42:31 +09:00
Nobuyoshi Nakada
e5e1f9813e [Bug #19838] Flush delayed token nonconsecutive with the next token 2023-12-02 02:41:39 +09:00
Nobuyoshi Nakada
6be402e172
[Bug #19736] Recover after unterminated interpolation 2023-06-20 20:10:46 +09:00
Nobuyoshi Nakada
fad48fefe1 [Bug #19399] Parsing invalid heredoc inside block parameter
Although this is of course invalid as Ruby code, allow to just parse
and tokenize.
2023-02-02 12:20:10 +09:00
Nobuyoshi Nakada
ace2eee544
[Bug #18963] Separate string contents by here document terminator 2022-08-28 09:29:24 +09:00
Nobuyoshi Nakada
844a0edbae [Bug #18962] Do not read again once reached EOF
`Ripper::Lexer#parse` re-parses the source code with syntax errors
when `raise_errors: false`.

Co-Authored-By: tompng <tomoyapenguin@gmail.com>
2022-08-12 15:58:18 +09:00
Shugo Maeda
a319c04818
Fix typo [ci skip] 2021-02-19 17:27:53 +09:00
Shugo Maeda
5de38c41ae
ripper: fix a bug of Ripper::Lexer with syntax error and heredoc [Bug #17644] 2021-02-19 16:40:29 +09:00
manga_osyo
b84b253a69 Fix Ripper with heredoc. 2021-01-17 12:58:13 +09:00
Nobuhiro IMAI
7ff0e93f96
parse.y: handle "duplicated argument name" appropriately on ripper.y
refs: 733ed1e184
2021-01-09 13:33:33 +09:00
Nobuyoshi Nakada
733ed1e184 ripper: fix bad label parameter handling [Bug #17425] 2020-12-23 09:56:35 +09:00
Nobuhiro IMAI
e33eb09b76 ripper: fix #tok on some error events [Bug 17345]
sorting alias target by event arity, and setup suitable `Elem` for error.
2020-12-19 17:32:39 +09:00
Nobuyoshi Nakada
e0bdd54348 Ripper: Refined error callbacks [Bug #17345] 2020-12-15 21:36:23 +09:00
Nobuyoshi Nakada
f5ca3ff4db
Store all kinds of syntax errors [Bug #17345] 2020-11-26 20:14:34 +09:00
Nobuyoshi Nakada
1df3896382
Removed a duplicate assertion 2020-11-26 20:03:38 +09:00
Nobuhiro IMAI
1800f3fa5c
Ripper.{lex,tokenize} return full tokens even if syntax error
yet another implements [Feature #17276]
2020-11-20 11:44:57 +09:00
Jeremy Evans
cd0877a93e
Support raise_errors keyword for Ripper.{lex,tokenize,sexp,sexp_raw}
Implements [Feature #17276]
2020-11-17 21:15:50 -08:00
Takashi Kokubun
4a7dccf44f
Add a Ripper.lex test of :on_embexpr_end
This is a weird use case of Ripper.lex which I'm not sure is supposed to
be maintained, so I'm adding this test so that we can easily notice such
changes.

If we change the behavior, this will break the behavior of hamlit.gem v1
and code like https://github.com/haml/haml/pull/1043.
2020-10-19 20:56:09 -07:00
Nobuyoshi Nakada
ea68bb914a Changed numbered parameter prefix 2019-09-24 21:57:54 +09:00
Nobuyoshi Nakada
5331932cd2
Removed unused variables 2019-06-28 17:08:21 +09:00
Nobuyoshi Nakada
92ecf58b1e
parse.y: adjust here-doc error token
* parse.y (here_document): adjust token to the here-doc identifier
  in compile_error when a here-document misses the closing
  identifier.
2019-05-30 22:03:33 +09:00
Nobuyoshi Nakada
b0e2b7a5ff
Include stack elements left after errors 2019-05-30 21:54:38 +09:00
Nobuyoshi Nakada
1da5c73932
parse.y: fix state after ivar/cvar
* parse.y (parse_atmark): return EXPR_END or EXPR_ENDFN, depending
  on the previous state, even incomplete names consistently.
2019-05-29 21:42:58 +09:00
k0kubun
0e60fdc511 test/ripper/test_lexer.rb: add test for r62743
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62744 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-13 13:03:12 +00:00
nobu
8b501f6d7f parse.y: indent at '#'
* parse.y (parser_here_document): update indent at '#', which is
  not a space.  [ruby-core:83368] [Bug #14032]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60213 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-10-19 05:10:38 +00:00
nobu
0e7936f896 lexer.rb: no dedent strings in middle
* ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): dedent only
  strings at the beginning, not strings in middle.
  [ruby-core:83343] [Bug #14027]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60212 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-10-19 04:25:30 +00:00
yui-knk
a7452a8f4f test_lexer.rb: require ripper directly
* test/ripper/test_lexer.rb: This file does not use
  DummyParser, so requiring ripper directly instead of
  requiring dummyparser.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59340 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-07-15 07:24:12 +00:00
nobu
d9d8ca0b8f lexer.rb: ignore dedented space
* ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): replace an
  empty string content because of dedentation with :on_ignored_sp.
  an empty token makes the sorted order unstable.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59269 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-07-06 04:12:51 +00:00
nobu
4671737b0f ripper/lexer.rb: nested indented heredoc
* ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): insert
  stripped leading spaces as `on_ignored_sp` elements, so that the
  original source can be reconsructed.
  [ruby-core:80977] [Bug #13536]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58584 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-06 15:14:58 +00:00
nobu
d24c4683f7 ripper/lexer.rb: nested indented heredoc
* ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): fix for
  nested indedented here documents, where `Elem`s are nested too.
  [ruby-core:80977] [Bug #13536]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58545 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-02 17:48:10 +00:00