Since universal-parser and prism support, prelude code used functions
inaccessible from outside libruby shared library.
```
linking goruby
/usr/bin/ld: goruby.o: in function `prelude_eval':
/home/runner/work/ruby/ruby/build/golf_prelude.c:221: undefined reference to `rb_ruby_prism_ptr'
/usr/bin/ld: goruby.o: in function `pm_prelude_load':
/home/runner/work/ruby/ruby/build/golf_prelude.c:192: undefined reference to `pm_options_line_set'
/usr/bin/ld: /home/runner/work/ruby/ruby/build/golf_prelude.c:193: undefined reference to `pm_parse_string'
/usr/bin/ld: goruby.o: in function `prelude_eval':
/home/runner/work/ruby/ruby/build/golf_prelude.c:224: undefined reference to `pm_iseq_new_with_opt'
/usr/bin/ld: /home/runner/work/ruby/ruby/build/golf_prelude.c:226: undefined reference to `pm_parse_result_free'
/usr/bin/ld: goruby.o: in function `prelude_ast_value':
/home/runner/work/ruby/ruby/build/golf_prelude.c:181: undefined reference to `rb_ruby_ast_data_get'
/usr/bin/ld: goruby.o: in function `prelude_eval':
/home/runner/work/ruby/ruby/build/golf_prelude.c:231: undefined reference to `rb_ruby_ast_data_get'
/usr/bin/ld: goruby.o: in function `pm_prelude_load':
/home/runner/work/ruby/ruby/build/golf_prelude.c:196: undefined reference to `pm_parse_result_free'
collect2: error: ld returned 1 exit status
```
This patch removes the `VALUE flags` member from the `rb_ast_t` structure making `rb_ast_t` no longer an IMEMO object.
## Background
We are trying to make the Ruby parser generated from parse.y a universal parser that can be used by other implementations such as mruby.
To achieve this, it is necessary to exclude VALUE and IMEMO from parse.y, AST, and NODE.
## Summary (file by file)
- `rubyparser.h`
- Remove the `VALUE flags` member from `rb_ast_t`
- `ruby_parser.c` and `internal/ruby_parser.h`
- Use TypedData_Make_Struct VALUE which wraps `rb_ast_t` `in ast_alloc()` so that GC can manage it
- You can retrieve `rb_ast_t` from the VALUE by `rb_ruby_ast_data_get()`
- Change the return type of `rb_parser_compile_XXXX()` functions from `rb_ast_t *` to `VALUE`
- rb_ruby_ast_new() which internally `calls ast_alloc()` is to create VALUE vast outside ruby_parser.c
- `iseq.c` and `vm_core.h`
- Amend the first parameter of `rb_iseq_new_XXXX()` functions from `rb_ast_body_t *` to `VALUE`
- This keeps the VALUE of AST on the machine stack to prevent being removed by GC
- `ast.c`
- Almost all change is replacement `rb_ast_t *ast` with `VALUE vast` (sorry for the big diff)
- Fix `node_memsize()`
- Now it includes `rb_ast_local_table_link`, `tokens` and script_lines
- `compile.c`, `load.c`, `node.c`, `parse.y`, `proc.c`, `ruby.c`, `template/prelude.c.tmpl`, `vm.c` and `vm_eval.c`
- Follow-up due to the above changes
- `imemo.{c|h}`
- If an object with `imemo_ast` appears, considers it a bug
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
This patch is part of universal parser work.
## Summary
- Decouple VALUE from members below:
- `(struct parser_params *)->debug_lines`
- `(rb_ast_t *)->body.script_lines`
- Instead, they are now `rb_parser_ary_t *`
- They can also be a `(VALUE)FIXNUM` as before to hold line count
- `ISEQ_BODY(iseq)->variable.script_lines` remains VALUE
- In order to do this,
- Add `VALUE script_lines` param to `rb_iseq_new_with_opt()`
- Introduce `rb_parser_build_script_lines_from()` to convert `rb_parser_ary_t *` into `VALUE`
## Other details
- Extend `rb_parser_ary_t *`. It previously could only store `rb_parser_ast_token *`, now can store script_lines, too
- Change tactics of building the top-level `SCRIPT_LINES__` in `yycompile0()`
- Before: While parsing, each line of the script is added to `SCRIPT_LINES__[path]`
- After: After `yyparse(p)`, `SCRIPT_LINES__[path]` will be built from `p->debug_lines`
- Remove the second parameter of `rb_parser_set_script_lines()` to make it simple
- Introduce `script_lines_free()` to be called from `rb_ast_free()` because the GC no longer takes care of the script_lines
- Introduce `rb_parser_string_deep_copy()` in parse.y to maintain script_lines when `rb_ruby_parser_free()` called
- With regard to this, please see *Future tasks* below
## Future tasks
- Decouple IMEMO from `rb_ast_t *`
- This lifts the five-members-restriction of Ruby object,
- So we will be able to move the ownership of the `lex.string_buffer` from parser to AST
- Then we remove `rb_parser_string_deep_copy()` to make the whole thing simple
Introduce Universal Parser mode for the parser.
This commit includes these changes:
* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
Saves comitters' daily life by avoid #include-ing everything from
internal.h to make each file do so instead. This would significantly
speed up incremental builds.
We take the following inclusion order in this changeset:
1. "ruby/config.h", where _GNU_SOURCE is defined (must be the very
first thing among everything).
2. RUBY_EXTCONF_H if any.
3. Standard C headers, sorted alphabetically.
4. Other system headers, maybe guarded by #ifdef
5. Everything else, sorted alphabetically.
Exceptions are those win32-related headers, which tend not be self-
containing (headers have inclusion order dependencies).
The script in prelude.rb was embed in MRI to load it (eval this
script at everyboot).
This commit change the loading process of prelude.rb. MRI doesn't
eval a script, but load from compiled binary with builtin feature.
So that Init_prelude() does not load `prelude.rb` now.
Instead of reading from the files by the full-path at runtime. As
rbinc files need to be included in distributed tarballs, the
full-paths at the packaging are unavailable at compilation times.
We see several occurrence of "diagnostic push/pop" so why not
make them macros. Tested on GCC8 / Clang 6.
Note that ruby.h is intentionally left untouched because we don't
want to introduce new public macros.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64118 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* parse.y (yycompile): in some cases (warning, error, dtrace,...),
ruby_sourcefile is expected to be NUL-terminated, so ensure it.
* template/prelude.c.tmpl (prelude_name): NUL-terminate to get rid
of copying static data.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62841 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/prelude.c.tmpl: ignore missing-field-initializers on
old gcc, e.g. 4.4, which does not support pushing/popping
diagnostics.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/prelude.c.tmpl: moved diagnostic pragmas outside
prelude_eval() for very old gcc. [ruby-core:84449] [Bug #14234]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61478 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/prelude.c.tmpl (PRELUDE_STR): make string objects with
static literals not to copy them.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61460 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/prelude.c.tmpl (prelude_eval): fix optimization
options. trace_instruction member has been removed. filled
coverage_enabled and debug_level members instead.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61459 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/prelude.c.tmpl (Init_prelude): fix line number of
preludes. line of prelude_eval is an int, not a VALUE.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61042 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/prelude.c.tmpl: split prelude code into blocks so that
each elements do not exceed the string literal size limit in
C89.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60953 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/prelude.c.tmpl (translate): revert r60873 because when some errors or
warnings are shown their line numbers are shifted.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60950 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/prelude.c.tmpl (translate): empty (only LF) lines are not necessary.
so skip them, but for safety only when they are made from comment line.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60873 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This re-introduces r60485.
This reverts commit 5a176b75b1.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60488 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
NODEs in AST are no longer objects managed by GC. This change will
remove the restriction imposed by the GC. For example, a NODE can use
more than five words (this is my primary purpose; we want to store the
position data for each NODE, for coverage library), or even a NODE can
have variable length (some kinds of NODEs have unused fields).
To do this, however, we need more work, since Ripper still uses T_NODE
objects managed by the GC.
The life time of NODEs is more obvious than other kinds of objects; they
are created at parsing, and they become disused immediately after
compilation. This change releases all NODEs by a few `xfree`s after
compilation, so performance will be improved a bit. In extreme example,
`eval("x=1;" * 10000000)` runs much faster (40 sec. -> 7.8 sec. on my
machine).
The most important part of this change is `ast_t` struct, which has
three contents: (1) NODE buffer (malloc'ed memory), (2) a reference to
the root NODE, and (3) an array that contains objects that must be
marked during parsing (such as literal objects). Some functions that
had received `NODE*` arguments, must now receive `ast_t*`.
* node.c, node.h: defines `ast_t` struct and related operations.
* gc.c, internal.h: defines `imemo_ast`.
* parse.y: makes `parser_params` struct have a reference to `ast_t`.
Instead of `rb_node_newnode`, use `rb_ast_newnode` to create a NODE.
* iseq.c, load.c, ruby.c, template/prelude.c.tmpl: modifies some
functions to handle `ast_t*` instead of `NODE*`.
* test/ruby/test_gc.rb: ad-hoc fix for a failed test. The test assumes
GC eden is increased at startup by NODE object creation. However,
this change now create no NODE object, so GC eden is not necessarily
increased.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60485 b2dd03c8-39d4-4d8f-98ff-823fe69b080e