Commit graph

20692 commits

Author SHA1 Message Date
Nobuyoshi Nakada
d4a53b465a
Prefer the simple read/write File singleton methods 2024-03-16 23:29:06 +09:00
Koichi ITO
c9da8d67fd [ruby/prism] Fix a token incompatibility for Prism::Translation::Parser::Lexer
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser`
for the heredocs_leading_whitespace.txt test.

7d45fb1eed
2024-03-15 18:07:59 +00:00
Nobuyoshi Nakada
a83703af79
Skip warnings for duplicate keys 2024-03-16 00:09:29 +09:00
Jean Boussier
f2e96d4bd9 [ruby/prism] Add PM_STRING_FLAGS_FROZEN / PM_STRING_FLAGS_MUTABLE on PM_SOURCE_FILE_NODE
For all intent and purposes, `__FILE__` is a string literal subject
to the `# frozen_string_literal: true/false` comment and to the
global `--enable-frozen-string-literal / --disable-frozen-string-literal`
CLI flags.

7e33c92afd
2024-03-15 13:31:02 +00:00
Nobuyoshi Nakada
1d91a1ad2d Cover all warning categories by using Warning.categories 2024-03-15 21:54:55 +09:00
Nobuyoshi Nakada
03e7b266a7 Test deprecation warning with $; 2024-03-15 21:54:55 +09:00
Koichi ITO
c0b8dee95a [ruby/prism] Fix an AST and token incompatibility for Prism::Translation::Parser
This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for dstring literal:

```ruby
"foo
  #{bar}"
```

## Parser gem (Expected)

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:str, "foo\n"),
  s(:str, "  "),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
[:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
[:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
[:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
[:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
[:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
[:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the AST and tokens returned by the Parser gem were different. In this case, `dstr` node should not be nested:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:dstr,
    s(:str, "foo\n"),
    s(:str, "  ")),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
[:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
[:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
[:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
[:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
[:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
[:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:dstr,
  s(:str, "foo\n"),
  s(:str, "  "),
  s(:begin,
    s(:send, nil, :bar))), [], [[:tSTRING_BEG, ["\"", #<Parser::Source::Range example.rb 0...1>]],
[:tSTRING_CONTENT, ["foo\n", #<Parser::Source::Range example.rb 1...5>]],
[:tSTRING_CONTENT, ["  ", #<Parser::Source::Range example.rb 5...7>]],
[:tSTRING_DBEG, ["\#{", #<Parser::Source::Range example.rb 7...9>]],
[:tIDENTIFIER, ["bar", #<Parser::Source::Range example.rb 9...12>]],
[:tSTRING_DEND, ["}", #<Parser::Source::Range example.rb 12...13>]],
[:tSTRING_END, ["\"", #<Parser::Source::Range example.rb 13...14>]], [:tNL, [nil, #<Parser::Source::Range example.rb 14...15>]]]]
```

c1652a9ee7
2024-03-15 12:31:40 +00:00
Kevin Newton
c45ad17fa1 [ruby/prism] Shareable constant nodes
473cfed6d0
2024-03-15 12:31:26 +00:00
Nobuyoshi Nakada
a1ced60692 [ruby/stringio] NULL StringIO by StringIO.new(nil)
779f71359d
2024-03-15 04:34:14 +00:00
toshimaru
193ebc3c3a [ruby/rdoc] test: Add tests for RDoc::TokenStream#add_tokens, add_token ,collect_tokens and pop_token
9ed530b8f9
2024-03-15 00:10:14 +00:00
Jean Boussier
09d8c99cdc Ensure test suite is compatible with --frozen-string-literal
As preparation for https://bugs.ruby-lang.org/issues/20205
making sure the test suite is compatible with frozen string
literals is making things easier.
2024-03-14 17:56:15 +01:00
Jean Boussier
315bde5a0f Exception#set_backtrace accept arrays of Backtrace::Location
[Feature #13557]

Setting the backtrace with an array of strings is lossy. The resulting
exception will return nil on `#backtrace_locations`.

By accepting an array of `Backtrace::Location` instance, we can rebuild
a `Backtrace` instance and have a fully functioning Exception.

Co-Authored-By: Étienne Barrié <etienne.barrie@gmail.com>
2024-03-14 11:38:40 +01:00
Nobuyoshi Nakada
5326337d4f [Feature #20244] Issue a single Warning.warn call
Make the entire series of message lines a multiline string so that the
`Warning.warn` hook can receive them in a single call.
2024-03-14 18:33:28 +09:00
Nobuyoshi Nakada
4bb8f8582f [Feature #20244] Show the conflicting another chdir block 2024-03-14 18:33:28 +09:00
Nobuyoshi Nakada
1ad366134d
[Feature #20293] Add Warning.categories 2024-03-14 17:56:55 +09:00
Nobuyoshi Nakada
f36a71e269 [Bug #20307] Fix Hash#update to make frozen copy of string keys 2024-03-14 17:49:37 +09:00
Kevin Newton
f0b5d0ad54 [PRISM] Turn on more tests 2024-03-13 14:25:03 -04:00
Koichi ITO
a5ac27e9e9 [ruby/prism] Turn on heredocs nested test for Prism::Translation::Parser
This PR enables the heredocs_nested.txt that was previously skipped testing.

9c54f7f10b
2024-03-13 17:35:31 +00:00
Nobuyoshi Nakada
c843afbf6f Chomp last punctuations from descriptions for -h
The following parts will not be shown for `-h` option.  And not to
reach 80 columns.  Some terminal emulators (Windows command prompt at
least) wrap the cursor to the next line when reaching the rightmost
column, before exceeding.
2024-03-14 01:19:57 +09:00
Koichi ITO
0f076fa520 [ruby/prism] Fix an AST and token incompatibility for Prism::Translation::Parser
This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for empty xstring literal.

## Parser gem (Expected)

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
[:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the AST and tokens returned by the Parser gem were different:

```console
$ bunele exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:xstr, s(:str, "")), [], [[:tBACK_REF2, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
[:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
```

After this correction, the AST and tokens returned by the Parser gem are the same:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("/tmp/s.rb"); buf.source = "``"; p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:xstr), [], [[:tXSTRING_BEG, ["`", #<Parser::Source::Range /tmp/s.rb 0...1>]],
[:tSTRING_END, ["`", #<Parser::Source::Range /tmp/s.rb 1...2>]]]]
```

4ac89dcbb5
2024-03-13 16:02:10 +00:00
Jun Aruga
eb4082284a [ruby/openssl] test_provider.rb: Make a legacy provider test optional.
In some cases such as OpenSSL package in FreeBSD[1], the legacy provider is not
installed intentionally. So, we omit a test depending the legacy provider if the
legacy provider is not loadable.

For the test_openssl_provider_names test, we use base provider[2] instead of
legacy provider, because we would expect the base provider is always loadable
in OpenSSL 3 for now.

* [1] https://www.freshports.org/security/openssl/
* [2] https://wiki.openssl.org/index.php/OpenSSL_3.0#Providers

7223da7730
2024-03-13 15:47:27 +00:00
Jun Aruga
171791426b [ruby/openssl] Revert "omit tests related legacy provider"
This reverts commit 9b7543d588.

The case not providing the legacy provider can happen not only in FreeBSD but
also in other platforms. The fix is addressed in the next commit.

8c26fc26a1
2024-03-13 15:47:26 +00:00
Jun Aruga
dd5e625d7b [ruby/openssl] test_asn1.rb: Remove the assertions of the time string format without second.
This commit fixes the following errors in the tests.
Because the OpenSSL project changed the code to make the time string format
without second invalid. So, we drop the assertions.

```
1) Error: test_generalizedtime(OpenSSL::TestASN1): OpenSSL::ASN1::ASN1Error: generalizedtime is too short
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_asn1.rb:698:in `decode'
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_asn1.rb:698:in `decode_test'
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_asn1.rb:433:in `test_generalizedtime'
     430:       OpenSSL::ASN1::GeneralizedTime.new(Time.utc(9999, 9, 8, 23, 43, 39))
     431:     # LibreSSL 3.6.0 requires the seconds element
     432:     return if libressl?
  => 433:     decode_test B(%w{ 18 0D }) + "201612081934Z".b,
     434:       OpenSSL::ASN1::GeneralizedTime.new(Time.utc(2016, 12, 8, 19, 34, 0))
     435:     # not implemented
     436:     # decode_test B(%w{ 18 13 }) + "2016120819+0930".b,

2) Error: test_utctime(OpenSSL::TestASN1): OpenSSL::ASN1::ASN1Error: utctime is too short
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_asn1.rb:698:in `decode'
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_asn1.rb:698:in `decode_test'
/home/runner/work/ruby-openssl/ruby-openssl/test/openssl/test_asn1.rb:411:in `test_utctime'
     408:     end
     409:     # Seconds is omitted. LibreSSL 3.6.0 requires it
     410:     return if libressl?
  => 411:     decode_test B(%w{ 17 0B }) + "1609082343Z".b,
     412:       OpenSSL::ASN1::UTCTime.new(Time.utc(2016, 9, 8, 23, 43, 0))
     413:     # not implemented
     414:     # decode_test B(%w{ 17 11 }) + "5009082343+0930".b,
```

2e826d5715
2024-03-13 15:19:40 +00:00
Peter Zhu
3896f9940e Make special const and too complex shapes before T_OBJECT shapes 2024-03-13 09:55:52 -04:00
Peter Zhu
6b0434c0f7 Don't create per size pool shapes for non-T_OBJECT 2024-03-13 09:55:52 -04:00
Kevin Newton
d1eaa97ec3 [ruby/prism] Track parentheses in patterns
62db99f156
2024-03-13 13:52:13 +00:00
Nobuyoshi Nakada
3f8ef7ff7c [ruby/stringio] Define StringIO::MAX_LENGTH
0205bd1c86
2024-03-13 13:20:23 +00:00
Koichi ITO
7eea268b70 [ruby/prism] Fix an AST incompatibility for Prism::Translation::Parser
Fixes ruby/prism#2480.

This PR fixes an AST incompatibility between Parser gem and `Prism::Translation::Parser`
for xstring literal with line breaks.

The following case in ruby/prism#2480 has already been addressed in ruby/prism#2576:

```ruby
"foo
bar"
```

cf85e72c55
2024-03-13 11:34:17 +00:00
Charles Oliver Nutter
0b7487b44e [ruby/stringio] Various cleanup for JRuby ext
(https://github.com/ruby/stringio/pull/85)

* Fix compile warnings on latest Java.
* Adjust one test for JRuby erroring on bad pos= value.
* Replace hard JVM synchronization with a spin lock.
* Avoid duplicate reads of state fields where possible.
* Remove long-deprecated codepoints, chars, bytes, lines methods.
* Mark fcntl as not implemented
* Split most of the remaining variable-arity methods.

31a9d42e6c
2024-03-13 04:46:54 +00:00
Kevin Newton
e914fa0d10 [ruby/prism] Add warning for chained comparisons
f9f3620d44
2024-03-12 22:29:54 +00:00
Takashi Kokubun
c3915a0741 Skip a flaky GC.compact test for YJIT
This test has failed without YJIT too:
20240301T063003Z.fail.html.gz

However, it somehow fails more often on "YJIT Ubuntu / check RUSTC='rustc +1.58.0'" job.

Nobody is actively looking into this issue, and it's most likely not a
YJIT-specific issue, so let me silence this false positive until we
start working on this.
2024-03-12 11:02:35 -07:00
Takashi Kokubun
a9eb0400c2 Check what actually needs to be checked
RubyVM::YJIT is conditionally defined, depending on --enable-yjit.
However, RubyVM::YJIT.enabled? is always defined as long as RubyVM::YJIT
is defined, so it seems weird to check RubyVM::YJIT.enabled? instead of
RubyVM::YJIT.
2024-03-12 11:02:35 -07:00
Koichi ITO
f8cab4ef8e [ruby/prism] Fix a token incompatibility for Prism::Translation::Parser::Lexer
In practice, the `BACKTICK` is mapped either as `:tXSTRING_BEG` or `:tBACK_REF2`.
The former is used in xstrings like `` `foo` ``, while the latter is utilized as
a back reference in contexts like `` A::` ``.
This PR will make corrections to differentiate the use of `BACKTICK`.

This mistake was discovered through the investigation of xstring.txt file.
The PR will run tests from xstring.txt file except for `` `f\oo` ``, which will still fail,
hence it will be separated into xstring_with_backslash.txt file.
This separation will facilitate addressing the correction at a different time.

49ad8df40a
2024-03-12 17:33:14 +00:00
Kevin Newton
83790e5fe1 [ruby/prism] Consolidate warnings for conditional predicates
* Also add warnings for literals in predicates
* Also create flip-flops in while/until

a6b5c523c2
2024-03-12 17:32:06 +00:00
Koichi ITO
a17a33b38e [ruby/prism] Follow ruby/prism#2581
Due to ruby/prism#2581, the test case for non_alphanumeric_methods.txt has also been restored.
It also solved issues with `def self.`' not limited to constants for `tBACK_REF2` token.

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "def self.`; end"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[[:kDEF, ["def", #<Parser::Source::Range example.rb 0...3>]], [:kSELF, ["self", #<Parser::Source::Range example.rb 4...8>]],
[:tDOT, [".", #<Parser::Source::Range example.rb 8...9>]], [:tBACK_REF2, ["`", #<Parser::Source::Range example.rb 9...10>]],
[:tSEMI, [";", #<Parser::Source::Range example.rb 10...11>]], [:kEND, ["end", #<Parser::Source::Range example.rb 12...15>]]]
```

f814db8ada
2024-03-12 17:20:41 +00:00
Koichi ITO
c637f53edd [ruby/prism] Fix a token incompatibility for Prism::Translation::Parser::Lexer
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for `tBACK_REF2`:

## Parser gem (Expected)

Returns `tBACK_REF2` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]],
[:tBACK_REF2, ["`", #<Parser::Source::Range example.rb 3...4>]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the parser returned `tXSTRING_BEG` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]],
[:tXSTRING_BEG, ["`", #<Parser::Source::Range example.rb 3...4>]]]
```

After the update, the parser now returns `tBACK_REF2` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "A::`"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[[:tCONSTANT, ["A", #<Parser::Source::Range example.rb 0...1>]], [:tCOLON2, ["::", #<Parser::Source::Range example.rb 1...3>]],
[:tBACK_REF2, ["`", #<Parser::Source::Range example.rb 3...4>]]]
```

This correction enables the restoration of `constants.txt` as a test case.

7f63b28f98
2024-03-12 15:17:20 +00:00
Koichi ITO
a41649bff4 [ruby/prism] Follow #2579
Due to #2579, the test case for range_begin_open_inclusive.txt has also been restored.

da173526a1
2024-03-12 14:48:39 +00:00
Koichi ITO
1ed6b7b40d [ruby/prism] Restore some skipped heredoc tests for Prism::Translation::Parser
This PR restores some heredoc tests that have already passed
for `Prism::Translation::Parser`.

297b9e2380
2024-03-12 14:15:52 +00:00
Koichi ITO
72a57076e2 [ruby/prism] Fix a token incompatibility for Prism::Translation::Parser::Lexer
This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser` for
beginless range:

## Parser gem (Expected)

Returns `tBDOT2` token:

```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "..42"; p Parser::Ruby33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[[:tBDOT2, ["..", #<Parser::Source::Range example.rb 0...2>]], [:tINTEGER, [42, #<Parser::Source::Range example.rb 2...4>]]]
```

## `Prism::Translation::Parser` (Actual)

Previously, the parser returned `tDOT2` token when parsing the following:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "..42"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[[:tDOT2, ["..", #<Parser::Source::Range example.rb 0...2>]], [:tINTEGER, [42, #<Parser::Source::Range example.rb 2...4>]]]
```

After the update, the parser now returns `tBDOT2` token for the same input:

```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = "..42"; p Prism::Translation::Parser33.new.tokenize(buf)[2]'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[[:tBDOT2, ["..", #<Parser::Source::Range example.rb 0...2>]], [:tINTEGER, [42, #<Parser::Source::Range example.rb 2...4>]]]
```

This correction enables the restoration of `endless_range_in_conditional.txt` as a test case.

f624b99ab0
2024-03-12 14:15:34 +00:00
Nobuyoshi Nakada
ad2aa6d727
Use assert_crash_report in test_crash_report_pipe
Instead of using `assert_in_out_err` directly.
2024-03-12 20:03:05 +09:00
Koichi ITO
2af6bc26c5 [ruby/prism] Fix an AST and token incompatibility for Prism::Translation::Parser
Fixes https://github.com/ruby/prism/pull/2515.

This PR fixes an AST and token incompatibility between Parser gem and `Prism::Translation::Parser`
for string literal with line breaks.

c58466e5bf
2024-03-12 10:45:56 +00:00
Nobuyoshi Nakada
76bd586330
[Bug #20277] Remove stale String test conditionals
These instance variables for conditional execution have remained
unchanged for nearly twenty years, since YARV merger.
2024-03-12 18:46:04 +09:00
Kevin Newton
21ea290b34 [ruby/prism] Static literals inspect
4913d112da
2024-03-12 03:30:50 +00:00
Kevin Newton
2ab75bc444 [ruby/prism] Support offset
665f533373
2024-03-11 14:49:23 +00:00
Hartley McGuire
08961ce8e3 [ruby/rdoc] Allow rich definition list labels for Markdown
Previously, any sort of "rich" markup for a definition list's label
would cause the Markdown parser to not recognize a definition list:

```ruby
md = <<~md
`one`
:    This is a definition
md

doc = RDoc::Markdown.parse(md)
doc # => [doc: [para: "<code>one</code>\n: This is a definition"]]
```

This commit tweaks the grammar for Markdown definition lists so that
labels can include "rich" markup such as bold (`**`), code (```), etc:

```ruby
md = <<~md
`one`
:    This is a definition
md

doc = RDoc::Markdown.parse(md)
doc # => [doc: [list: NOTE [item: ["<code>one</code>"]; [para: "This is a definition"]]]]
```

The [PHP Markdown Extra][1] Spec does not seem to specify whether or not
this should be allowed, but it is allowed in the RDoc format:

```ruby
rdoc = <<~rdoc
+code+::
    This is a definition
rdoc

doc = RDoc::Markup.parse(rdoc)
doc # => [doc: [list: NOTE [item: ["+code+"]; [para: "This is a definition"]]]]
```

so accepting this change increases the parity of the two formats.

[1]: https://michelf.ca/projects/php-markdown/extra/#def-list

8f943bbba4
2024-03-11 20:00:51 +09:00
Hartley McGuire
5ac6194c2b [ruby/rdoc] Fix ToRdoc generating incorrect {label,name}-lists
Previously, trying to round-trip label-list and name-lists with the
ToRdoc converter was not possible:

```ruby
doc = <<~RDOC
foo ::
bar ::
  hi
RDOC

markup = RDoc::Markup.parse(doc)
markup # => [doc: [list: NOTE [item: ["foo ", "bar"]; [para: "hi"]]]]

rt = RDoc::Markup::ToRdoc.new.convert(markup)
rt # => "foo\nbar:\n  hi\n\n"

rt_markup = RDoc::Markup.parse(rt)
rt_markup # => [doc: [para: "foo ", "bar:"], [verb: "hi\n"]]
```

This commit addresses the issue by fixing ToRdoc to generate output that
can be properly reparsed by RDoc. ToRdoc tests additionally needed to be
updated for the new output.

The old implementation of `accept_list_item_start` was copied to ToBs
because those tests did not pass with the new changes and I am
unfamiliar with the `backspace` format.

After:

```ruby
doc = <<~RDOC
foo ::
bar ::
  hi
RDOC

markup = RDoc::Markup.parse(doc)
markup # => [doc: [list: NOTE [item: ["foo ", "bar"]; [para: "hi"]]]]

rt = RDoc::Markup::ToRdoc.new.convert(markup)
rt # => "foo::\nbar::\n  hi\n\n"

rt_markup = RDoc::Markup.parse(rt)
rt_markup # => [doc: [list: NOTE [item: ["foo", "bar"]; [para: "hi"], blankline]]]
```

c6c51aa900
2024-03-09 03:50:09 +00:00
Koichi ITO
dd24d88473 [ruby/prism] Fix a token incompatibility for Prism::Translation::Parser
Fixes https://github.com/ruby/prism/pull/2512.

This PR fixes a token incompatibility between Parser gem and `Prism::Translation::Parser`
for `HEREDOC_END` with a newline.

b67d1e0c6f
2024-03-08 19:07:53 +00:00
Kevin Newton
609bbad15d [ruby/prism] Fix up regexp memory leaks
4dc58a533a
2024-03-08 18:48:55 +00:00
Kevin Newton
a564f30fb8 [ruby/prism] Extract out pm_regexp_token_buffer_t
7bbe4224a3
2024-03-08 18:48:55 +00:00
Kevin Menard
4e1d19c457 [ruby/prism] Add initial implementation of Regexp validation.
6bf1b8edf0
2024-03-08 18:48:54 +00:00