This patch adds `int line_count` field to `rb_ast_body_t` structure.
Instead, we no longer cast `script_lines` to Fixnum.
## Background
Ref https://github.com/ruby/ruby/pull/10618
In the PR above, we have decoupled IMEMO from `rb_ast_t`.
This means we could lift the five-words-restriction of the structure
that forced us to unionize `rb_ast_t *` and `FIXNUM` in one field.
## Relating refactor
- Remove the second parameter of `rb_ruby_ast_new()` function
## Attention
I will remove a code that assigns -1 to line_count, in `rb_binding_add_dynavars()`
of vm.c, because I don't think it is necessary.
But I will make another PR for this so that we can atomically revert
in case I was wrong (See the comment on the code)
This patch removes the `VALUE flags` member from the `rb_ast_t` structure making `rb_ast_t` no longer an IMEMO object.
## Background
We are trying to make the Ruby parser generated from parse.y a universal parser that can be used by other implementations such as mruby.
To achieve this, it is necessary to exclude VALUE and IMEMO from parse.y, AST, and NODE.
## Summary (file by file)
- `rubyparser.h`
- Remove the `VALUE flags` member from `rb_ast_t`
- `ruby_parser.c` and `internal/ruby_parser.h`
- Use TypedData_Make_Struct VALUE which wraps `rb_ast_t` `in ast_alloc()` so that GC can manage it
- You can retrieve `rb_ast_t` from the VALUE by `rb_ruby_ast_data_get()`
- Change the return type of `rb_parser_compile_XXXX()` functions from `rb_ast_t *` to `VALUE`
- rb_ruby_ast_new() which internally `calls ast_alloc()` is to create VALUE vast outside ruby_parser.c
- `iseq.c` and `vm_core.h`
- Amend the first parameter of `rb_iseq_new_XXXX()` functions from `rb_ast_body_t *` to `VALUE`
- This keeps the VALUE of AST on the machine stack to prevent being removed by GC
- `ast.c`
- Almost all change is replacement `rb_ast_t *ast` with `VALUE vast` (sorry for the big diff)
- Fix `node_memsize()`
- Now it includes `rb_ast_local_table_link`, `tokens` and script_lines
- `compile.c`, `load.c`, `node.c`, `parse.y`, `proc.c`, `ruby.c`, `template/prelude.c.tmpl`, `vm.c` and `vm_eval.c`
- Follow-up due to the above changes
- `imemo.{c|h}`
- If an object with `imemo_ast` appears, considers it a bug
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
* Revert "Revert "YJIT: Optimize local variables when EP == BP" (#10584)"
This reverts commit c878344195.
* YJIT: Take care of GC references in ISEQ invariants
Co-authored-by: Alan Wu <alansi.xingwu@shopify.com>
---------
Co-authored-by: Alan Wu <alansi.xingwu@shopify.com>
This patch is part of universal parser work.
## Summary
- Decouple VALUE from members below:
- `(struct parser_params *)->debug_lines`
- `(rb_ast_t *)->body.script_lines`
- Instead, they are now `rb_parser_ary_t *`
- They can also be a `(VALUE)FIXNUM` as before to hold line count
- `ISEQ_BODY(iseq)->variable.script_lines` remains VALUE
- In order to do this,
- Add `VALUE script_lines` param to `rb_iseq_new_with_opt()`
- Introduce `rb_parser_build_script_lines_from()` to convert `rb_parser_ary_t *` into `VALUE`
## Other details
- Extend `rb_parser_ary_t *`. It previously could only store `rb_parser_ast_token *`, now can store script_lines, too
- Change tactics of building the top-level `SCRIPT_LINES__` in `yycompile0()`
- Before: While parsing, each line of the script is added to `SCRIPT_LINES__[path]`
- After: After `yyparse(p)`, `SCRIPT_LINES__[path]` will be built from `p->debug_lines`
- Remove the second parameter of `rb_parser_set_script_lines()` to make it simple
- Introduce `script_lines_free()` to be called from `rb_ast_free()` because the GC no longer takes care of the script_lines
- Introduce `rb_parser_string_deep_copy()` in parse.y to maintain script_lines when `rb_ruby_parser_free()` called
- With regard to this, please see *Future tasks* below
## Future tasks
- Decouple IMEMO from `rb_ast_t *`
- This lifts the five-members-restriction of Ruby object,
- So we will be able to move the ownership of the `lex.string_buffer` from parser to AST
- Then we remove `rb_parser_string_deep_copy()` to make the whole thing simple
With verbopse mode (-w), the interpreter shows a warning if
a block is passed to a method which does not use the given block.
Warning on:
* the invoked method is written in C
* the invoked method is not `initialize`
* not invoked with `super`
* the first time on the call-site with the invoked method
(`obj.foo{}` will be warned once if `foo` is same method)
[Feature #15554]
`Primitive.attr! :use_block` is introduced to declare that primitive
functions (written in C) will use passed block.
For minitest, test needs some tweak, so use
ea9caafc07
for `test-bundled-gems`.
Using rb_gc_mark_movable and a reference update function, we can make
instruction sequences movable in memory, and avoid pinning compiled iseqs.
```
require "objspace"
iseqs = []
GC.disable
50_000.times do
iseqs << RubyVM::InstructionSequence.compile("")
end
GC.enable
GC.compact
p ObjectSpace.dump_all(output: :string).lines.grep(/"pinned":true/).count
```
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
[Feature #20205]
As a path toward enabling frozen string literals by default in the future,
this commit introduce "chilled strings". From a user perspective chilled
strings pretend to be frozen, but on the first attempt to mutate them,
they lose their frozen status and emit a warning rather than to raise a
`FrozenError`.
Implementation wise, `rb_compile_option_struct.frozen_string_literal` is
no longer a boolean but a tri-state of `enabled/disabled/unset`.
When code is compiled with frozen string literals neither explictly enabled
or disabled, string literals are compiled with a new `putchilledstring`
instruction. This instruction is identical to `putstring` except it marks
the String with the `STR_CHILLED (FL_USER3)` and `FL_FREEZE` flags.
Chilled strings have the `FL_FREEZE` flag as to minimize the need to check
for chilled strings across the codebase, and to improve compatibility with
C extensions.
Notes:
- `String#freeze`: clears the chilled flag.
- `String#-@`: acts as if the string was mutable.
- `String#+@`: acts as if the string was mutable.
- `String#clone`: copies the chilled flag.
Co-authored-by: Jean Boussier <byroot@ruby-lang.org>
In preparation for https://bugs.ruby-lang.org/issues/20205.
The `frozen_string_literal` compilation option will no longer
be a boolean but a tri-state: `on/off/default`.
Before this commit, we were mixing a lot of concerns with the prism
compile between RubyVM::InstructionSequence and the general entry
points to the prism parser/compiler.
This commit makes all of the various prism-related APIs mirror
their corresponding APIs in the existing parser/compiler. This means
we now have the correct frame naming, and it's much easier to follow
where the logic actually flows. Furthermore this consolidates a lot
of the prism initialization, making it easier to see where we could
potentially be raising errors.
This flag is set when the caller has already created a new array to
handle a splat, such as for `f(*a, b)` and `f(*a, *b)`. Previously,
if `f` was defined as `def f(*a)`, these calls would create an extra
array on the callee side, instead of using the new array created
by the caller.
This modifies `setup_args_core` to set the flag whenver it would add
a `splatarray true` instruction. However, when `splatarray true` is
changed to `splatarray false` in the peephole optimizer, to avoid
unnecessary allocations on the caller side, the flag must be removed.
Add `optimize_args_splat_no_copy` and have the peephole optimizer call
that. This significantly simplifies the related peephole optimizer
code.
On the callee side, in `setup_parameters_complex`, set
`args->rest_dupped` to true if the flag is set.
This takes a similar approach for optimizing regular splats that was
previiously used for keyword splats in
d2c41b1bff (via VM_CALL_KW_SPLAT_MUT).
Make it easier to check what annotations an ISEQ has. SINGLE_NOARG_LEAF
is added automatically, so it's hard to be sure about the annotation by
just reading code. It's also unclear to me what happens to it with
Primitive.mandatory_only?, but this at least explains that LEAF
annotation is not added to the non-mandatory_only ISEQ.
This causes the Iseq file names to be wrong, which is affecting
Tracepoint events in certain cases.
because we're taking a pointer to the string and using it in
`pm_string_mapped_pointer` we also need to `RB_GC_GUARD` the relevant
Ruby object to ensure it's not moved or swept before the parser has been
free'd.
pm_scope_node_destroy frees the scope node after we're done using it to
make sure that the index_lookup_table is not leaked.
For example:
10.times do
100_000.times do
RubyVM::InstructionSequence.compile_prism("begin; 1; rescue; 2; end")
end
puts `ps -o rss= -p #{$$}`
end
Before:
33056
50304
67776
84544
101520
118448
135712
152352
169136
186656
After:
15264
15296
15408
17040
17152
17152
18320
18352
18400
18608
There is a memory leak when passing a file to
RubyVM::InstructionSequence.compile_prism because it does not free the
mapped file.
For example:
require "tempfile"
Tempfile.create(%w"test_iseq .rb") do |f|
f.puts "name = 'Prism'; puts 'hello'"
f.close
10.times do
1_000.times do
RubyVM::InstructionSequence.compile_prism(f)
end
puts `ps -o rss= -p #{$$}`
end
end
Before:
27968
44848
61408
77872
94144
110432
126640
142816
159200
175584
After:
11504
12144
12592
13072
13488
13664
14064
14368
14704
15168
Before this commit the Prism compiler would try to intern constants
every time it re-entered. This pool of constants is "constant" (there is
only one pool per parser instance), so we should do it only once: upon
the top level entry to the compiler.
This change does just that: it populates the interned constants once.
Fixes: https://github.com/ruby/prism/issues/2152
This value is only incremented when rb_iseq_translate_threaded_code is
called, which doesn't happen for iseqs which result in a syntax error.
This is easy to hit by running a debug build with RUBY_FREE_AT_EXIT=1,
but any build and options could underflow this value by running enough
evals.
when the RUBY_FREE_ON_SHUTDOWN environment variable is set, manually free memory at shutdown.
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
The operands in each instruction needs to be pinned because if
auto-compaction runs in iseq_set_sequence, then the objects could exist
on the generated_iseq buffer, which would not be reference updated which
can lead to T_MOVED (and subsequently T_NONE) objects on the iseq.
* Provide a new API compile_file_prism which mirrors compile_file
but uses prism to parse/compile.
* Provide the ability to run test-all with RUBY_ISEQ_DUMP_DEBUG set
to "prism". If it is, we'll use the new compile_file_prism API to
load iseqs during the test run.
