Commit graph

18546 commits

Author SHA1 Message Date
Jemma Issroff
e3ca50b02f [PRISM] Fix compilation for NextNode
This code was almost enitrely the same as the existing compiler's
code for its NextNode.
2023-12-04 14:03:00 -05:00
Kevin Newton
ea9f89eeb6 [ruby/prism] Ripper compat docs update
5f70b32b02
2023-12-04 17:00:14 +00:00
TSUYUSATO Kitsune
2a65d83707 [ruby/prism] Check "void value expression" for array literals
Fix https://github.com/ruby/prism/pull/1978

194c997d0a
2023-12-04 14:00:23 +00:00
John Hawthorn
85bc80a51b Revert "Add missing GVL hooks for M:N threads and ractors"
This reverts commit ad54fbf281.
2023-12-03 18:37:06 -08:00
tomoya ishida
a2197466a2 [ruby/irb] Disable pager in eval_history test
(https://github.com/ruby/irb/pull/799)

ee85e84935
2023-12-03 14:05:16 +00:00
John Hawthorn
ad54fbf281 Add missing GVL hooks for M:N threads and ractors
[Bug #20019]

This fixes GVL instrumentation in three locations it was missing:
- Suspending when blocking on a Ractor
- Suspending when doing a coroutine transfer from an M:N thread
- Resuming after an M:N thread starts

Co-authored-by: Matthew Draper <matthew@trebex.net>
2023-12-02 10:06:07 -08:00
Nobuyoshi Nakada
c9c1670f7e [ruby/rdoc] Markup punctuations need to be separated with a space
83f0149fc1
2023-12-02 12:40:12 +00:00
Gary Tou
4ee1f0fb5d [ruby/irb] Implement history command
(https://github.com/ruby/irb/pull/761)

* Implement `history` command

Lists IRB input history with indices. Also aliased as `hist`.

* Add tests for `history` command

* Address feedback: `puts` with multiple arguments instead of `join`ing

* Address feedback: Handle nil from splitting an empty input string

* Refactor line truncation

* Add `-g` grep option to `history` command

* Add `history` command to README

* Remove unused `*args` parameter

* Allow spaces to be included in grep

* Allow `/` to be included in grep regex

* Handle `input` being an empty string

* Exclude "#{index}: " from matching the grep regex

* Add new line after joining

3f9eacbfa9
2023-12-02 04:32:04 +00:00
eileencodes
818813c2bd Implement paren node for defined?
Implements and adds a test for passing a parentheses node to `defined?`.
2023-12-01 14:56:25 -08:00
Kevin Newton
492c82cb41 [ruby/prism] Prism.parse_success?(source)
A lot of tools use Ripper/RubyVM::AbstractSyntaxTree to determine
if a source is valid. These tools both create an AST instead of
providing an API that will return a boolean only.

This new API only creates the C structs, but doesn't bother
reifying them into Ruby/the serialization API. Instead it only
returns true/false, which is significantly more efficient.

7014740118
2023-12-01 20:53:34 +00:00
Haldun Bayhantopcu
562d949e02 [ruby/prism] Fix parsing heredoc ends
aa8c702271
2023-12-01 20:10:58 +00:00
eileencodes
39238888bc Implements missing literals for defined?
This PR implements the following literals:

- String
- Symbols
- Integers
- Floats
- Regexs
- Ranges
- Lambdas
- Hashes

and tests for them.
2023-12-01 12:05:22 -08:00
Peter Zhu
ee0eca191f Make String#undump compaction safe 2023-12-01 15:04:31 -05:00
Nobuyoshi Nakada
a607d62d8c [Bug #20033] Dynamic regexp should not assign captures 2023-12-02 03:57:41 +09:00
Kevin Newton
ec83bd7356 [ruby/prism] Provide heredoc? queries
e148e8fe6a
2023-12-01 18:46:52 +00:00
Jemma Issroff
2a8d9c59ff [PRISM] Account for RescueNodes with no statements
We need a PUTNIL if a RescueNode has no statements.
2023-12-01 13:23:23 -05:00
Jemma Issroff
d6584a0201 [PRISM] Fix behavior of BlockParameters with only one parameter
This commit sets the ambiguous param flag if there is only one
parameter on a block node. It also fixes a small bug with a trailing
comma on params.
2023-12-01 13:07:59 -05:00
hogelog
ef466ac931 [ruby/irb] Scrub past history input before split
(https://github.com/ruby/irb/pull/795)

* Scrub past history input before split

* Don't rewrite ENV["LANG"]

0f344f66d9
2023-12-01 18:04:03 +00:00
Nobuyoshi Nakada
e5e1f9813e [Bug #19838] Flush delayed token nonconsecutive with the next token 2023-12-02 02:41:39 +09:00
Jemma Issroff
d224618bea [PRISM] Restructure parameters
Prior to this commit, we weren't accounting for hidden variables
on the locals table, so we would have inconsistencies on the stack.
This commit fixes params, and introduces a hidden_variable_count
on the scope, both of which fix parameters.
2023-12-01 12:14:54 -05:00
Kevin Newton
cdb74d74af [ruby/prism] Change numbered parameters
Previously numbered parameters were a field on blocks and lambdas
that indicated the maximum number of numbered parameters in either
the block or lambda, respectively. However they also had a
parameters field that would always be nil in these cases.

This changes it so that we introduce a NumberedParametersNode that
goes in place of parameters, which has a single uint8_t maximum
field on it. That field contains the maximum numbered parameter in
either the block or lambda.

As a part of the PR, I'm introducing a new UInt8Field type that
can be used on nodes, which is just to make it a little more
explicit what the maximum values can be (the maximum is actually 9,
since it only goes up to _9). Plus we can do a couple of nice
things in serialization like just read a single byte.

2d87303903
2023-12-01 12:03:09 -05:00
Matt Valentine-House
90d9c20a0c [PRISM] Compile RescueNode 2023-12-01 16:40:25 +00:00
Kevin Newton
64f33aea2b [ruby/prism] Update snapshots
f4c80c67dc
2023-12-01 15:18:19 +00:00
Martin Emde
cbe57caa24 [ruby/prism] Fix comments after HEREDOCs again.
The problem was deeper than just looking back a single token.
You can push the heredoc_end token way back into the list.
We need to save the last location of a heredoc end to see if
it's the last token in the file.

Fixes https://github.com/ruby/prism/pull/1954

91dfd4eecd
2023-12-01 15:17:20 +00:00
TSUYUSATO Kitsune
417d700406 [ruby/prism] Improve to handle unterminated strings
Fix https://github.com/ruby/prism/pull/1946

This fixes to set an error position for unterminated strings to the
opening delimiters. Previously, the error position was set to the end
of the delimiter.

The same fix applies to other string-like literals.

Additionally, this fixes https://github.com/ruby/prism/pull/1946; that is, it adds the last part of the
string even though the string literal does not terminate.

c1240baafd
2023-12-01 15:15:01 +00:00
Matt Valentine-House
ce5f5ca1d6 [PRISM] Remove transparent scope nodes 2023-12-01 15:04:13 +00:00
Hiroshi SHIBATA
64c4bbb86c [ruby/prism] Add test/prism/snapshots/heredoc_with_comment.txt
97b296a0f7
2023-12-01 11:33:26 +00:00
Hiroshi SHIBATA
7d142c08cb
lib/helper only needs on flori/json repo 2023-12-01 16:47:06 +09:00
Hiroshi SHIBATA
86045fca24
Manually merged from flori/json
> https://github.com/flori/json/pull/525
  > Rename escape_slash in script_safe and also escape E+2028 and E+2029

  Co-authored-by: Jean Boussier <jean.boussier@gmail.com>

  > https://github.com/flori/json/pull/454
  > Remove unnecessary initialization of create_id in JSON.parse()

  Co-authored-by: Watson <watson1978@gmail.com>
2023-12-01 16:47:06 +09:00
Jean Boussier
0dfeb17296
Rename escape_slash in script_safe and also escape E+2028 and E+2029
It is rather common to directly interpolate JSON string inside
<script> tags in HTML as to provide configuration or parameters to a
script.

However this may lead to XSS vulnerabilities, to prevent that 3
characters need to be escaped:

  - `/` (forward slash)
  - `U+2028` (LINE SEPARATOR)
  - `U+2029` (PARAGRAPH SEPARATOR)

The forward slash need to be escaped to prevent closing the script
tag early, and the other two are valid JSON but invalid Javascript
and can be used to break JS parsing.

Given that the intent of escaping forward slash is the same than escaping
U+2028 and U+2029, I chos to rename and repurpose the existing `escape_slash`
option.
2023-12-01 16:47:06 +09:00
John Hawthorn
4b770527c2
[flori/json] Fix "unexpected token" offset for Infinity
Previously in the JSON::Ext parser, when we encountered an "Infinity"
token (and weren't allowing NaN/Infinity) we would try to display the
"unexpected token" at the character before.

42ac170712
2023-12-01 16:47:06 +09:00
Lucas Kanashiro
854e6559b6
[flori/json] tests/ractor_test.rb: make assert_separately available
Require tests/lib/helper.rb to avoid:

NoMethodError: undefined method `assert_separately'

a81bcc0328
2023-12-01 16:47:06 +09:00
Nobuyoshi Nakada
e36b9760fd
Dispatch invalid hex escape content too 2023-12-01 15:04:30 +09:00
Nobuyoshi Nakada
d503e1b95a
[Bug #20030] dispatch invalid escaped character without ignoring it 2023-12-01 15:04:30 +09:00
Misaki Shioi
52c2660163 Fixup with review comment
https://github.com/ruby/ruby/pull/9088#discussion_r1411490445
2023-12-01 12:20:03 +09:00
Misaki Shioi
1bfd30a1e3 Relax test conditions to velify Socket::ResolutionError#error_code
The test for Socket::ResolutionError#error_code fails in the FreeBSD environment with this test condition. Because Socket::ResolutionError#error_code returns Socket::EAI_FAIL instead of Socket::EAI_FAMILY.

20231130T103002Z.fail.html.gz

This PR avoids the test failure by relaxing the condition.

Also changed the domain for testing to `example.com`.
2023-12-01 12:20:03 +09:00
Kevin Newton
99a147ff4f [ruby/prism] Document remaining encodings
b9510aed40
2023-11-30 21:37:56 -05:00
Kevin Newton
19a321bff0 [ruby/prism] CESU encoding
2d5b9c2b3c
2023-11-30 21:37:56 -05:00
Kevin Newton
ca26e0e34b [ruby/prism] EUC-TW encoding
edfb54f039
2023-11-30 21:37:56 -05:00
Kevin Newton
32249c2cf1 [ruby/prism] GB18030 encoding
ca3ab7ec89
2023-11-30 21:37:56 -05:00
Kevin Newton
a9162a44c5 [ruby/prism] Emacs MULE encodings
4c06b6c42e
2023-11-30 21:37:56 -05:00
Kevin Newton
10d3897e13 [PRISM] Big5 encodings 2023-11-30 21:37:56 -05:00
Kevin Newton
700e172a50 [ruby/prism] EUC-KR encodings
ba5218385a
2023-11-30 21:37:56 -05:00
Kevin Newton
ddaa073058 [ruby/prism] Support other EUC-JP encodings
d040337ce9
2023-11-30 21:37:56 -05:00
Kevin Newton
7b5bb978fb [PRISM] Alias CP51932 to EUC-JP 2023-11-30 21:37:56 -05:00
Kevin Newton
9ba92327f2 [PRISM] Consolidate SJIS encodings 2023-11-30 21:37:56 -05:00
Kevin Newton
219c3c1c09 [ruby/prism] Add other UTF8 encodings
709fb6e09f
2023-11-30 21:37:56 -05:00
Takashi Kokubun
ba1cdadfc8
YJIT: Cancel on-stack jit_return on invalidation (#9086)
* YJIT: Cancel on-stack jit_return on invalidation

Co-authored-by: Alan Wu <alansi.xingwu@shopify.com>

* Use RUBY_VM_CONTROL_FRAME_STACK_OVERFLOW_P

---------

Co-authored-by: Alan Wu <alansi.xingwu@shopify.com>
2023-11-30 21:35:55 -05:00
Alan Wu
5888a16a12 YJIT: Use stats[:live_page_count], renamed from :compiled_page_count
Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
2023-11-30 19:19:43 -05:00
Matt Valentine-House
0c7c654b4d [Prism] Fix local variable access for POST_EXECUTION_NODE 2023-11-30 21:31:57 +00:00