(https://github.com/ruby/irb/pull/692)
`RubyLex` has always been a private component of IRB, so we should
explicitly discourage usages of it.
Also, it should be placed under the `IRB` module like other components.
069b5625f7
(https://github.com/ruby/irb/pull/670)
Since assignment check relies on tokenization with `Ripper`, it feels like
the responsibility of `RubyLex`. `Irb#eval_input` should simply get the result
when calling `each_top_level_statement` on `RubyLex`.
89d1adb3fd
(https://github.com/ruby/irb/pull/644)
* Avoid initialising Row at every test input
* Extract common assertion patterns into methods
* Remove unnecessary PromptRow strcut and boilerplate code
1ba586c0c6
(https://github.com/ruby/irb/pull/515)
* Implement heredoc embdoc and string indentation with bugfix
* Fix test_ruby_lex's indentation value
* Add embdoc indent test
* Add workaround for lines==[nil] passed to auto_indent when exit IRB with CTRL+d
add new features easily
(https://github.com/ruby/irb/pull/500)
* Add nesting level parser for multiple use (indent, prompt, termination check)
* Rewrite RubyLex using NestingParser
* Add nesting parser tests, fix some existing tests
* Add description comment, rename method to NestingParser
* Add comments and tweak code to RubyLex
* Update NestingParser test
* Extract list of ltype tokens to constants
(https://github.com/ruby/irb/pull/586)
Because the assertions for `dynamic_prompt` lives inside a block given to
`RubyLex`, they could be skipped unnoticed if the setup is not correct.
This commit adds a simple assertion to check if the block was actually
executed.
(https://github.com/ruby/irb/pull/576)
* Simplify each_top_level_statement, reduce instance vars
* Update lib/irb/ruby-lex.rb
Co-authored-by: Stan Lo <stan001212@gmail.com>
* Remove unused ltype from TestRubyLex#check_state response
* Remove unnecessary const path of TerminateLineInput
* Combine duplicated code state check into method
---------
172453cec4
Co-authored-by: Stan Lo <stan001212@gmail.com>
(https://github.com/ruby/irb/pull/583)
1. Make `RubyLex#set_input` simply assign the input block. This matches
the behavior of `RubyLex#set_prompt`.
2. Merge `RubyLex#set_input`'s IO configuration logic with `#set_auto_indent`
into `#configure_io`.
(https://github.com/ruby/irb/pull/484)
* Improve assert_indenting helper
Instead of putting assertions inside the `auto_indent` block, we
can just make `auto_indent` return the calculated space count, and use
it for assertion outside of the `auto_indent` block call.
This simplifies the setup code and makes the intention easier to
understand.
* Introduce assert_row_indenting helper
1. Helper users shouldn't need to write 2 assertions for the current and
the next line's indentation.
2. With this new approach, we can generate clearer error message for
both cases:
When the current line's space count doesn't match
```
Incorrect spaces calculation for line:
```
> def each_top_level_statement
```
All lines:
```
def each_top_level_statement
```
<0> expected but was
<nil>
```
When the next line's space count doesn't match
```
Incorrect spaces calculation for line after the current line:
```
def each_top_level_statement
>
```
All lines:
```
def each_top_level_statement
```
<3> expected but was
<2>
```
* Replace assert_indenting with assert_row_indenting
Some background for this refactor:
1. Through a RubyLex instance's lifetime, the context passed to its methods
should be the same.
Given that `Context` is only initialised in `Irb#initialize`,
this should be true.
2. When `RubyLex` is initialised, the context object should be accessible.
This is also true in all 3 of `RubyLex.new`'s invocations.
With the above observations, we should be able to store the context in `RubyLex`
as an instance variable. And doing so will make `RubyLex`'s instance methods
easier to use and maintain.
5c8d3df2df
(https://github.com/ruby/irb/pull/504)
* Simplify `RubyLex#set_prompt`
It's optional argument is never used by any caller.
* Remove the optional `p` argument from `RubyLex#set_input`
The argument is only used in a test case, which can be easily replaced by
a block argument.
It looks like tool/sync_default_gems.rb is not capable of cherry-picking
commits from ruby/irb. I just executed `tool/sync_default_gems.rb irb`
to fix the sync status.
I'm not sure if what's the cause. It could be related to some diff that
doesn't exist in ruby/ruby, or it might be related to non-linear history
due to merge commits. For next time, I'd like to at least exclude the
second possibility, so I disabled merge commits in ruby/irb.
conveniently
(https://github.com/ruby/irb/pull/429)
* Create a base TestIRB::TestCase class
* Save/restore encodings for tests that initializes InputMethod classes
Because `RelineInputMethod#initializes` calls `set_encoding`, which
changes stdio/out/err and Encoding's default encoding values, we need to
make sure any test that directly or indirectly (e.g. through Context)
initializes `RelineInputMethod` restores encodings.
`ReadlineInputMethod` also changes encodings but currently no tests
cover it.
* Remove unnecessary TestHelper module
Since we now have a base TestCase, without_rdoc can just live there.
c2874ec121
(https://github.com/ruby/irb/pull/428)
RubyLex is not designed to be used alone. It's usually used with an IRB
context, which requires workspace. So its tests should have access to
those components too.
608f261da4
(https://github.com/ruby/irb/pull/427)
* Make sure `RubyLex#set_input`'s context is always present in tests
In real-world scenarios, the context should always be non-nil:
https://github.com/ruby/irb/blob/master/lib/irb.rb#L489
So we should make sure our test setup reflects that.
* Make context a required keyword
Since in practice, `set_input`'s context should always be non-nil, its
parameters should reflect that.
And since `RubyLex#check_state` is only called by `#lex` and
`#set_input`, both of which now always require context, we can assume
its context should be non-nil too.
1aeeb86203
* Use local_variables for colorize, code_block_open check, nesting_level and assignment_expression check
* Check if expression is an assignment BEFORE evaluating it. evaluate might define new localvars and change result of assignment_expression?
* Add local_variables dependent code test
* pend local variable dependent test on truffleruby
code_block_open is not working on truffleruby
* Always pass context to RubyLex#lex
* Rename local_variable_assign_code generator method name
* Add assignment expression truncate test
* Add Context#local_variables and make generate_local_variables_assign_code more simple
* Update lib/irb/input-method.rb
Co-authored-by: Stan Lo <stan001212@gmail.com>
* Add a comment why assignment expression check should be done before evaluate
c8b3877281
Co-authored-by: Stan Lo <stan001212@gmail.com>
Co-authored-by: Takashi Kokubun <takashikkbn@gmail.com>
* Fix backtick method def method call handled as backtick open
* Fix handling heredoc in check_string_literal
* Sort result of lexer.parse by pos in ruby<2.7. It's not sorted when the given code includes heredoc.
* Update lib/irb/ruby-lex.rb
Co-authored-by: Stan Lo <stan001212@gmail.com>
* Update lib/irb/ruby-lex.rb
Co-authored-by: Stan Lo <stan001212@gmail.com>
* Add check_string_literal test for heredoc code that does not end with newline
44bc712460
Co-authored-by: Stan Lo <stan001212@gmail.com>
Instead of accessing the struct as an array, access it via methods. There are other places inside of this file already using this API (for example e0a5c3d2b7/lib/irb/ruby-lex.rb (L829-L830)).
This commit moves all struct array-ish calls to use their method calls instead. It is also ~1.23 faster accessing values via a method instead of as an array according to this microbenchmark:
```ruby
Elem = Struct.new(:pos, :event, :tok, :state, :message) do
def initialize(pos, event, tok, state, message = nil)
super(pos, event, tok, State.new(state), message)
end
# ...
def to_a
a = super
a.pop unless a.empty?
a
end
end
class ElemClass
attr_accessor :pos, :event, :tok, :state, :message
def initialize(pos, event, tok, state, message = nil)
@pos = pos
@event = event
@tok = tok
@state = State.new(state)
@message = message
end
def to_a
if @message
[@pos, @event, @tok, @state, @message]
else
[@pos, @event, @tok, @state]
end
end
end
# stub state class creation for now
class State; def initialize(val); end; end
```
```ruby
Benchmark.ips do |x|
x.report("struct") { struct[1] }
x.report("class ") { from_class.event }
x.compare!
end; nil
```
```
Warming up --------------------------------------
struct 1.624M i/100ms
class 1.958M i/100ms
Calculating -------------------------------------
struct 17.139M (± 2.6%) i/s - 86.077M in 5.025801s
class 21.104M (± 3.4%) i/s - 105.709M in 5.015193s
Comparison:
class : 21103826.3 i/s
struct: 17139201.5 i/s - 1.23x (± 0.00) slower
```
