This PR fixes a diagnostic incompatibility for `Prism::Translation::Parser` when using constant argument:
```ruby
def foo(A)
end
```
## Parser gem (Expected)
Displays `formal argument cannot be a constant (Parser::SyntaxError)`:
```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve 'p Parser::Ruby33.parse("def foo(A) end")'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
(string):1:9: error: formal argument cannot be a constant
(string):1: def foo(A) end
(string):1: ^
/Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/diagnostic/engine.rb:72:
in `process': formal argument cannot be a constant (Parser::SyntaxError)
from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:274:in `diagnostic'
from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/ruby33.rb:12177:in `_reduce_663'
from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/racc-1.7.3/lib/racc/parser.rb:267:in `_racc_do_parse_c'
from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/racc-1.7.3/lib/racc/parser.rb:267:in `do_parse'
from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:190:in `parse'
from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:33:in `parse'
from -e:1:in `<main>'
```
## `Prism::Translation::Parser` (Actual)
Previously, the error messages displayed by the Parser gem were different.
Before:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'Prism::Translation::Parser33.parse("def foo(A) end")'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
(string):1:9: error:
(string):1: def foo(A) end
(string):1: ^
/Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/diagnostic/engine.rb:72:
in `process': Parser::SyntaxError (Parser::SyntaxError)
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:218:in `block in unwrap'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:216:in `each'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:216:in `unwrap'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:49:in `parse'
from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:33:in `parse'
from -e:1:in `<main>'
```
After:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'Prism::Translation::Parser33.parse("def foo(A) end")'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
(string):1:9: error: formal argument cannot be a constant
(string):1: def foo(A) end
(string):1: ^
/Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/diagnostic/engine.rb:72:
in `process': formal argument cannot be a constant (Parser::SyntaxError)
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:218:in `block in unwrap'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:216:in `each'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:216:in `unwrap'
from /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:49:in `parse'
from /Users/koic/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/parser-3.3.0.5/lib/parser/base.rb:33:in `parse'
from -e:1:in `<main>'
```
4f2af88520
(https://github.com/ruby/reline/pull/614)
* Re-architecture LineEditor's internal state and rendering
* Fix test related to LineEditor re-architecture
* Bump to 0.5.0.pre.1
* Hide cursor only when updating screen. Frequent hide&show makes cursor flickering.
* Simplify rerender call from reline.rb
* Simplify handle_cleared
It only need to clear screen. line_editor.rerender will be called later.
* Add description of updating pasting_state inserts continuous_insertion_buffer
* Use meaningful block parameter
Co-authored-by: Stan Lo <stan001212@gmail.com>
* Fix use of `@cursor_y`
Fix bug updating `@cursor_y`. Do not use `@cursor_y` while updating dialog because it is not current cursor position but cursor position at last rendered time.
* Remove useless instance_variable_set in test
These instance variables are already removed from LineEditor
* Always initialize instance variables to avoid ruby 2.7 warning, remove unused instance variable
* Call update_dialogs from reline.rb before first render
* Combine state representing rendered screen information into `@rendered_screen`
* Rename editor_cursor_ to wrapped_cursor
It represents cursor position of word wrapped whole content
* Remove unused code, tweak, add comment
---------
3fa376217d
Co-authored-by: Stan Lo <stan001212@gmail.com>
[Feature #20205]
As a path toward enabling frozen string literals by default in the future,
this commit introduce "chilled strings". From a user perspective chilled
strings pretend to be frozen, but on the first attempt to mutate them,
they lose their frozen status and emit a warning rather than to raise a
`FrozenError`.
Implementation wise, `rb_compile_option_struct.frozen_string_literal` is
no longer a boolean but a tri-state of `enabled/disabled/unset`.
When code is compiled with frozen string literals neither explictly enabled
or disabled, string literals are compiled with a new `putchilledstring`
instruction. This instruction is identical to `putstring` except it marks
the String with the `STR_CHILLED (FL_USER3)` and `FL_FREEZE` flags.
Chilled strings have the `FL_FREEZE` flag as to minimize the need to check
for chilled strings across the codebase, and to improve compatibility with
C extensions.
Notes:
- `String#freeze`: clears the chilled flag.
- `String#-@`: acts as if the string was mutable.
- `String#+@`: acts as if the string was mutable.
- `String#clone`: copies the chilled flag.
Co-authored-by: Jean Boussier <byroot@ruby-lang.org>
If there are many searches in the `$LOAD_PATH` in the user environment,
require will perform unnecessary searches that are not needed.
In contrast, `require_relative` is efficient because it uses a relative path.
438ccc67bd
(https://github.com/ruby/irb/pull/901)
* Always save irb_history in HOME or XDG_CONFIG_HOME
Also split irbrc search logic from irb_history search logic as a refactor
* Remove IRB.conf[:RC_NAME_GENERATOR] because it's not configurable
This conf is used to specify which irbrc to load. Need to configure before irbrc is loaded, so it's actually not configurable.
This conf is also used for history file search, but it is configurable by conf[:HISTORY_FILE].
* remove rc_file_test because it is tested with rc_files, remove useless test setup
* Make internal irbrc searching method private
11d03a6ff7
(https://github.com/ruby/irb/pull/890)
* Convert irb.rb's document into markdown format
* Hide should-be-private top-level methods from docs
* Skip xmp.rb's docs
* Declare lib/irb.rb's markup do it works in ruby/ruby too
* Ignore docs folder
e9a175e06b
The variable ``length2`` is an Integer, call to floor returns self.
The variables ``i`` and ``prefix_bonus`` are equally incremented, no need to keep the ``i`` variable.
4408802289
This PR fixes token incompatibility for `Prism::Translation::Parser::Lexer`
when using escaped backslash in string literal:
```ruby
"\\ foo \\ bar"
```
## Parser gem (Expected)
```console
$ bundle exec ruby -Ilib -rparser/ruby33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Parser::Ruby33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #<Parser::Source::Range example.rb 0...15>]],
[:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]]
```
## `Prism::Translation::Parser` (Actual)
Previously, the tokens returned by the Parser gem were different. The escaped backslash does not match in the `tSTRING` token:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\\\ foo \\\\ bar", #<Parser::Source::Range example.rb 0...15>]],
[:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]]
```
After this correction, the AST and tokens returned by the Parser gem are the same:
```console
$ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve \
'buf = Parser::Source::Buffer.new("example.rb"); buf.source = File.read("example.rb"); p Prism::Translation::Parser33.new.tokenize(buf)'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
[s(:str, "\\ foo \\ bar"), [], [[:tSTRING, ["\\ foo \\ bar", #<Parser::Source::Range example.rb 0...15>]],
[:tNL, [nil, #<Parser::Source::Range example.rb 15...16>]]]]
```
The reproduction test is based on the following strings.txt and exists:
https://github.com/ruby/prism/blob/v0.24.0/test/prism/fixtures/strings.txt#L79
But, the restoration has not yet been performed due to remaining other issues in strings.txt.
2c44e7e307
(https://github.com/ruby/irb/pull/899)
* Rescue from exceptions raised by #name
Irb might terminate if the class overwrites `name` and raise errors. This commit rescue irb from termination.
* fix for other unknown patterns
35b87cf893
Fixesruby/prism#2480.
This PR fixes an AST incompatibility between Parser gem and `Prism::Translation::Parser`
for xstring literal with line breaks.
The following case in ruby/prism#2480 has already been addressed in ruby/prism#2576:
```ruby
"foo
bar"
```
cf85e72c55
In practice, the `BACKTICK` is mapped either as `:tXSTRING_BEG` or `:tBACK_REF2`.
The former is used in xstrings like `` `foo` ``, while the latter is utilized as
a back reference in contexts like `` A::` ``.
This PR will make corrections to differentiate the use of `BACKTICK`.
This mistake was discovered through the investigation of xstring.txt file.
The PR will run tests from xstring.txt file except for `` `f\oo` ``, which will still fail,
hence it will be separated into xstring_with_backslash.txt file.
This separation will facilitate addressing the correction at a different time.
49ad8df40a
Previously, any sort of "rich" markup for a definition list's label
would cause the Markdown parser to not recognize a definition list:
```ruby
md = <<~md
`one`
: This is a definition
md
doc = RDoc::Markdown.parse(md)
doc # => [doc: [para: "<code>one</code>\n: This is a definition"]]
```
This commit tweaks the grammar for Markdown definition lists so that
labels can include "rich" markup such as bold (`**`), code (```), etc:
```ruby
md = <<~md
`one`
: This is a definition
md
doc = RDoc::Markdown.parse(md)
doc # => [doc: [list: NOTE [item: ["<code>one</code>"]; [para: "This is a definition"]]]]
```
The [PHP Markdown Extra][1] Spec does not seem to specify whether or not
this should be allowed, but it is allowed in the RDoc format:
```ruby
rdoc = <<~rdoc
+code+::
This is a definition
rdoc
doc = RDoc::Markup.parse(rdoc)
doc # => [doc: [list: NOTE [item: ["+code+"]; [para: "This is a definition"]]]]
```
so accepting this change increases the parity of the two formats.
[1]: https://michelf.ca/projects/php-markdown/extra/#def-list8f943bbba4
By replacing `ALLOBJS` suffix with intermediate file suffixes instead
of roughly removing by wildcards. Made `cleanlibs` append `.dSYM`
suffix for each word in `TARGET_SO`, not the end of the entire list.
Previously, trying to round-trip label-list and name-lists with the
ToRdoc converter was not possible:
```ruby
doc = <<~RDOC
foo ::
bar ::
hi
RDOC
markup = RDoc::Markup.parse(doc)
markup # => [doc: [list: NOTE [item: ["foo ", "bar"]; [para: "hi"]]]]
rt = RDoc::Markup::ToRdoc.new.convert(markup)
rt # => "foo\nbar:\n hi\n\n"
rt_markup = RDoc::Markup.parse(rt)
rt_markup # => [doc: [para: "foo ", "bar:"], [verb: "hi\n"]]
```
This commit addresses the issue by fixing ToRdoc to generate output that
can be properly reparsed by RDoc. ToRdoc tests additionally needed to be
updated for the new output.
The old implementation of `accept_list_item_start` was copied to ToBs
because those tests did not pass with the new changes and I am
unfamiliar with the `backspace` format.
After:
```ruby
doc = <<~RDOC
foo ::
bar ::
hi
RDOC
markup = RDoc::Markup.parse(doc)
markup # => [doc: [list: NOTE [item: ["foo ", "bar"]; [para: "hi"]]]]
rt = RDoc::Markup::ToRdoc.new.convert(markup)
rt # => "foo::\nbar::\n hi\n\n"
rt_markup = RDoc::Markup.parse(rt)
rt_markup # => [doc: [list: NOTE [item: ["foo", "bar"]; [para: "hi"], blankline]]]
```
c6c51aa900
This PR fixes the following error for `Prism::Translation::Parser::Lexer` on the main branch:
```console
$ cat example.rb
'a' # aあ
"
#{x}
"
$ bundle exec rubocop
Parser::Source::Range: end_pos must not be less than begin_pos
/Users/koic/.rbenv/versions/3.0.4/lib/ruby/gems/3.0.0/gems/parser-3.3.0.5/lib/parser/source/range.rb:39:in `initialize'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:299:in `new'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:299:in `block in to_a'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:297:in `map'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/lexer.rb:297:in `to_a'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:263:in `build_tokens'
/Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser.rb:92:in `tokenize'
```
This change was made in https://github.com/ruby/prism/pull/2557, and it seems there was
an inconsistency in Range due to forgetting to apply `offset_cache` to `start_offset`.
Previously, using ToMarkdown on a label-list would generate output that
could not be reparsed by the RDoc::Markdown parser:
```
md = <<~MD
apple
: a red fruit
banana
: a yellow fruit
MD
doc = RDoc::Markdown.parse(md)
doc # => [doc: [list: NOTE [item: ["apple"]; [para: "a red fruit"]], [item: ["banana"]; [para: "a yellow fruit"]]]]
new_md = doc.accept(RDoc::Markup::ToMarkdown.new)
new_md # => "apple\n: a red fruit\nbanana\n: a yellow fruit\n\n"
new_doc = RDoc::Markdown.parse(new_md)
new_doc # => [doc: [list: NOTE [item: ["apple"]; [para: "a red fruit\nbanana\n: a yellow fruit"]]]]
```
The issue is that the [PHP Markdown Extra spec][1] requires a newline
after each definition list item, but ToMarkdown was not putting newlines
between label-list items.
This commit fixes the issue by properly appending a newline after each
label-list item so that the output of ToMarkdown can be reparsed by
RDoc::Markdown:
```
md = <<~MD
apple
: a red fruit
banana
: a yellow fruit
MD
doc = RDoc::Markdown.parse(mdoc)
doc # => [doc: [list: NOTE [item: ["apple"]; [para: "a red fruit"]], [item: ["banana"]; [para: "a yellow fruit"]]]]
new_md = doc.accept(RDoc::Markup::ToMarkdown.new)
new_md # => "apple\n: a red fruit\n\nbanana\n: a yellow fruit\n\n"
new_doc = RDoc::Markdown.parse(new_md)
new_doc # => [doc: [list: NOTE [item: ["apple"]; [para: "a red fruit"]], [item: ["banana"]; [para: "a yellow fruit"]]]]
```
[1]: https://michelf.ca/projects/php-markdown/extra/#def-listc65266437c
