Commit graph

56 commits

Author SHA1 Message Date
Nobuyoshi Nakada
b42afa1dbc
Suppress gcc 15 unterminated-string-initialization warnings 2025-04-30 20:04:10 +09:00
Nobuyoshi Nakada
ecb58a8d08
Fix prelude to use IBF
Since universal-parser and prism support, prelude code used functions
inaccessible from outside libruby shared library.

```
linking goruby
/usr/bin/ld: goruby.o: in function `prelude_eval':
/home/runner/work/ruby/ruby/build/golf_prelude.c:221: undefined reference to `rb_ruby_prism_ptr'
/usr/bin/ld: goruby.o: in function `pm_prelude_load':
/home/runner/work/ruby/ruby/build/golf_prelude.c:192: undefined reference to `pm_options_line_set'
/usr/bin/ld: /home/runner/work/ruby/ruby/build/golf_prelude.c:193: undefined reference to `pm_parse_string'
/usr/bin/ld: goruby.o: in function `prelude_eval':
/home/runner/work/ruby/ruby/build/golf_prelude.c:224: undefined reference to `pm_iseq_new_with_opt'
/usr/bin/ld: /home/runner/work/ruby/ruby/build/golf_prelude.c:226: undefined reference to `pm_parse_result_free'
/usr/bin/ld: goruby.o: in function `prelude_ast_value':
/home/runner/work/ruby/ruby/build/golf_prelude.c:181: undefined reference to `rb_ruby_ast_data_get'
/usr/bin/ld: goruby.o: in function `prelude_eval':
/home/runner/work/ruby/ruby/build/golf_prelude.c:231: undefined reference to `rb_ruby_ast_data_get'
/usr/bin/ld: goruby.o: in function `pm_prelude_load':
/home/runner/work/ruby/ruby/build/golf_prelude.c:196: undefined reference to `pm_parse_result_free'
collect2: error: ld returned 1 exit status
```
2024-09-08 23:40:17 +09:00
Nobuyoshi Nakada
70871fa6e3
Extract rb_builtin_find
Refactor out the same code from `rb_builtin_ast_value` and
`pm_builtin_ast_value.
2024-09-08 23:16:46 +09:00
Kevin Newton
371432b2d7 [PRISM] Handle RubyVM.keep_script_lines 2024-08-29 20:27:01 -04:00
Kevin Newton
4e36abbab3 [PRISM] Support for compiling builtins 2024-05-30 15:38:02 -04:00
yui-knk
899d9f79dd Rename vast to ast_value
There is an English word "vast".
This commit changes the name to be more clear name to avoid confusion.
2024-05-03 12:40:35 +09:00
HASUMI Hitoshi
2244c58b00 [Universal parser] Decouple IMEMO from rb_ast_t
This patch removes the `VALUE flags` member from the `rb_ast_t` structure making `rb_ast_t` no longer an IMEMO object.

## Background

We are trying to make the Ruby parser generated from parse.y a universal parser that can be used by other implementations such as mruby.
To achieve this, it is necessary to exclude VALUE and IMEMO from parse.y, AST, and NODE.

## Summary (file by file)

- `rubyparser.h`
  - Remove the `VALUE flags` member from `rb_ast_t`
- `ruby_parser.c` and `internal/ruby_parser.h`
  - Use TypedData_Make_Struct VALUE which wraps `rb_ast_t` `in ast_alloc()` so that GC can manage it
    - You can retrieve `rb_ast_t` from the VALUE by `rb_ruby_ast_data_get()`
  - Change the return type of `rb_parser_compile_XXXX()` functions from `rb_ast_t *` to `VALUE`
  - rb_ruby_ast_new() which internally `calls ast_alloc()` is to create VALUE vast outside ruby_parser.c
- `iseq.c` and `vm_core.h`
  - Amend the first parameter of `rb_iseq_new_XXXX()` functions from `rb_ast_body_t *` to `VALUE`
  - This keeps the VALUE of AST on the machine stack to prevent being removed by GC
- `ast.c`
  - Almost all change is replacement `rb_ast_t *ast` with `VALUE vast` (sorry for the big diff)
  - Fix `node_memsize()`
    - Now it includes `rb_ast_local_table_link`, `tokens` and script_lines
- `compile.c`, `load.c`, `node.c`, `parse.y`, `proc.c`, `ruby.c`, `template/prelude.c.tmpl`, `vm.c` and `vm_eval.c`
  - Follow-up due to the above changes
- `imemo.{c|h}`
  - If an object with `imemo_ast` appears, considers it a bug

Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
2024-04-26 11:21:08 +09:00
HASUMI Hitoshi
9b1e97b211 [Universal parser] DeVALUE of p->debug_lines and ast->body.script_lines
This patch is part of universal parser work.

## Summary
- Decouple VALUE from members below:
  - `(struct parser_params *)->debug_lines`
  - `(rb_ast_t *)->body.script_lines`
- Instead, they are now `rb_parser_ary_t *`
  - They can also be a `(VALUE)FIXNUM` as before to hold line count
- `ISEQ_BODY(iseq)->variable.script_lines` remains VALUE
  - In order to do this,
  - Add `VALUE script_lines` param to `rb_iseq_new_with_opt()`
  - Introduce `rb_parser_build_script_lines_from()` to convert `rb_parser_ary_t *` into `VALUE`

## Other details
- Extend `rb_parser_ary_t *`. It previously could only store `rb_parser_ast_token *`, now can store script_lines, too
- Change tactics of building the top-level `SCRIPT_LINES__` in `yycompile0()`
  - Before: While parsing, each line of the script is added to `SCRIPT_LINES__[path]`
  - After: After `yyparse(p)`, `SCRIPT_LINES__[path]` will be built from `p->debug_lines`
- Remove the second parameter of `rb_parser_set_script_lines()` to make it simple
- Introduce `script_lines_free()` to be called from `rb_ast_free()` because the GC no longer takes care of the script_lines
- Introduce `rb_parser_string_deep_copy()` in parse.y to maintain script_lines when `rb_ruby_parser_free()` called
  - With regard to this, please see *Future tasks* below

## Future tasks
- Decouple IMEMO from `rb_ast_t *`
  - This lifts the five-members-restriction of Ruby object,
  - So we will be able to move the ownership of the `lex.string_buffer` from parser to AST
  - Then we remove `rb_parser_string_deep_copy()` to make the whole thing simple
2024-04-15 20:51:54 +09:00
Nobuyoshi Nakada
5ec1fc52c1
Escape non-ascii characters in prelude C comments
Non-ASCII code are often warned by localized compilers.
2023-08-24 21:12:51 +09:00
Takashi Kokubun
38be9a9b72
Clean up OPT_STACK_CACHING (#8132) 2023-07-27 17:27:05 -07:00
yui-knk
1740482d06 Fix rb_compile_option_t comments [ci skip] 2023-06-18 14:39:15 +09:00
yui-knk
b481b673d7 [Feature #19719] Universal Parser
Introduce Universal Parser mode for the parser.
This commit includes these changes:

* Introduce `UNIVERSAL_PARSER` macro. All of CRuby related functions
  are passed via `struct rb_parser_config_struct` when this macro is enabled.
* Add CI task with 'cppflags=-DUNIVERSAL_PARSER' for ubuntu.
2023-06-12 18:23:48 +09:00
Nobuyoshi Nakada
677c3228d0 Check loading built-in binaries 2023-03-08 13:59:21 +09:00
Nobuyoshi Nakada
9b462aec4a
Follow up "Rework first_lineno to be int." 2022-10-28 19:33:19 +09:00
Nobuyoshi Nakada
9faa9ced96 Support sub-library in builtin-loader
Previously, it was supported in prelude.c, but has not followed up the
builtin-loader system.
2022-09-09 15:47:24 +09:00
Nobuyoshi Nakada
be56033248
Move RDoc --/++ directives to comments 2022-09-09 14:23:39 +09:00
Takashi Kokubun
5b21e94beb Expand tabs [ci skip]
[Misc #18891]
2022-07-21 09:42:04 -07:00
Nobuyoshi Nakada
93591d31b0
Use rb_cObject instead of rb_cData 2021-12-31 10:53:54 +09:00
Peter Zhu
aeae6e2842 [Feature #18290] Remove all usages of rb_gc_force_recycle
This commit removes usages of rb_gc_force_recycle since it is a burden
to maintain and makes changes to the GC difficult.
2021-11-08 14:05:54 -05:00
Nobuyoshi Nakada
2e97240617
prelude.c.tmpl: strip comment-only or empty lines at the beginning 2021-06-06 22:45:37 +09:00
Koichi Sasada
07c03bc309 check isolated Proc more strictly
Isolated Proc prohibit to access outer local variables, but it was
violated by binding and so on, so they should be error.
2020-10-29 23:42:55 +09:00
卜部昌平
5af983af4f template/prelude.c.tmpl: suppress clang-12 warning
Clang 12 warns "suspicious concatenation of string literals in an array
initialization", which is rather annoying than useful in this context.
2020-08-11 16:51:07 +09:00
卜部昌平
acd8ee8dbc tool/prelude.c.tmpl: use RubyVM::CEscape
Do not repeat yourself.
2020-08-11 16:51:07 +09:00
卜部昌平
5e22f873ed decouple internal.h headers
Saves comitters' daily life by avoid #include-ing everything from
internal.h to make each file do so instead.  This would significantly
speed up incremental builds.

We take the following inclusion order in this changeset:

1.  "ruby/config.h", where _GNU_SOURCE is defined (must be the very
    first thing among everything).
2.  RUBY_EXTCONF_H if any.
3.  Standard C headers, sorted alphabetically.
4.  Other system headers, maybe guarded by #ifdef
5.  Everything else, sorted alphabetically.

Exceptions are those win32-related headers, which tend not be self-
containing (headers have inclusion order dependencies).
2019-12-26 20:45:12 +09:00
Koichi Sasada
2c5c60754c use compiled binary for gem_prelude.rb.
`gem_prelude.rb` is not compiled yet. This patch compile it to
compiled binary.
2019-12-11 11:24:42 +09:00
Koichi Sasada
9c1abe2a6c load prelude.rb by builtin features.
The script in prelude.rb was embed in MRI to load it (eval this
script at everyboot).
This commit change the loading process of prelude.rb. MRI doesn't
eval a script, but load from compiled binary with builtin feature.

So that Init_prelude() does not load `prelude.rb` now.
2019-11-15 03:55:46 +09:00
Nobuyoshi Nakada
dfaac2b372
Embed builtin ruby scripts in miniprelude.c
Instead of reading from the files by the full-path at runtime.  As
rbinc files need to be included in distributed tarballs, the
full-paths at the packaging are unavailable at compilation times.
2019-11-09 19:28:45 +09:00
Nobuyoshi Nakada
4dc4b18904
Fixed commit miss 2019-11-09 19:15:10 +09:00
Nobuyoshi Nakada
95aed94d2a
Fixed #line directives in miniprelude.c 2019-11-09 18:45:48 +09:00
nobu
4cfc6cedc4 template/prelude.c.tmpl: allow UTF-8 characters
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67617 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2019-04-19 10:45:16 +00:00
nobu
241dced625 Disable tailcall optimization [Bug #15303]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66349 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-12 05:46:13 +00:00
shyouhei
d83536c980 reduce copy & paste
We see several occurrence of "diagnostic push/pop" so why not
make them macros.  Tested on GCC8 / Clang 6.

Note that ruby.h is intentionally left untouched because we don't
want to introduce new public macros.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64118 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-30 07:07:48 +00:00
nobu
6916130566 parse.y: NUL-terminate ruby_sourcefile
* parse.y (yycompile): in some cases (warning, error, dtrace,...),
  ruby_sourcefile is expected to be NUL-terminated, so ensure it.

* template/prelude.c.tmpl (prelude_name): NUL-terminate to get rid
  of copying static data.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62841 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-03-19 14:12:00 +00:00
mame
92b81dc597 make rb_iseq_new* accept rb_ast_body_t instead of NODE*
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61609 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-05 08:59:22 +00:00
mame
503b858cef node.h: define rb_ast_body_t and restructure rb_ast_t
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61608 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-05 08:59:20 +00:00
nobu
d0afbff3b0 prelude.c.tmpl: get rid of warnings on old gcc
* template/prelude.c.tmpl: ignore missing-field-initializers on
  old gcc, e.g. 4.4, which does not support pushing/popping
  diagnostics.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61524 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-31 12:21:53 +00:00
nobu
de11e472b4 prelude.c.tmpl: get rid of errors with old gcc
* template/prelude.c.tmpl: moved diagnostic pragmas outside
  prelude_eval() for very old gcc.  [ruby-core:84449] [Bug #14234]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61478 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-25 23:59:56 +00:00
nobu
d66f2d4443 prelude.c.tmpl: static literals
* template/prelude.c.tmpl (PRELUDE_STR): make string objects with
  static literals not to copy them.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61460 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-25 02:42:13 +00:00
nobu
d9c76c5bd8 prelude.c.tmpl: fix optimization options
* template/prelude.c.tmpl (prelude_eval): fix optimization
  options.  trace_instruction member has been removed.  filled
  coverage_enabled and debug_level members instead.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61459 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-25 02:09:14 +00:00
nobu
9fbd48ee71 prelude.c.tmpl: fix line number
* template/prelude.c.tmpl (Init_prelude): fix line number of
  preludes.  line of prelude_eval is an int, not a VALUE.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61042 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-06 05:22:19 +00:00
nobu
a8a25b36f1 prelude.c.tmpl: escape comments
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60955 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-01 04:27:32 +00:00
nobu
b7a908f61b prelude.c.tmpl: split prelude code
* template/prelude.c.tmpl: split prelude code into blocks so that
  each elements do not exceed the string literal size limit in
  C89.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60953 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-01 03:54:50 +00:00
usa
b4070513dd revert r60873
* template/prelude.c.tmpl (translate): revert r60873 because when some errors or
  warnings are shown their line numbers are shifted.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60950 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-01 01:29:50 +00:00
usa
afa10bd125 Skip comment lines
* template/prelude.c.tmpl (translate): empty (only LF) lines are not necessary.
  so skip them, but for safety only when they are made from comment line.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60873 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-11-21 08:48:07 +00:00
ko1
85fcaf025d * node.h (ast_t): renamed to rb_ast_t.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60565 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-10-29 15:51:23 +00:00
mame
e35fe8d11b Revert "Revert "Manage AST NODEs out of GC""
This re-introduces r60485.
This reverts commit 5a176b75b1.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60488 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-10-27 16:44:57 +00:00
mame
5a176b75b1 Revert "Manage AST NODEs out of GC"
This reverts commit 620ba74778.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60486 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-10-27 16:02:34 +00:00
mame
620ba74778 Manage AST NODEs out of GC
NODEs in AST are no longer objects managed by GC.  This change will
remove the restriction imposed by the GC.  For example, a NODE can use
more than five words (this is my primary purpose; we want to store the
position data for each NODE, for coverage library), or even a NODE can
have variable length (some kinds of NODEs have unused fields).
To do this, however, we need more work, since Ripper still uses T_NODE
objects managed by the GC.

The life time of NODEs is more obvious than other kinds of objects; they
are created at parsing, and they become disused immediately after
compilation.  This change releases all NODEs by a few `xfree`s after
compilation, so performance will be improved a bit.  In extreme example,
`eval("x=1;" * 10000000)` runs much faster (40 sec. -> 7.8 sec. on my
machine).

The most important part of this change is `ast_t` struct, which has
three contents: (1) NODE buffer (malloc'ed memory), (2) a reference to
the root NODE, and (3) an array that contains objects that must be
marked during parsing (such as literal objects).  Some functions that
had received `NODE*` arguments, must now receive `ast_t*`.

* node.c, node.h: defines `ast_t` struct and related operations.
* gc.c, internal.h: defines `imemo_ast`.
* parse.y: makes `parser_params` struct have a reference to `ast_t`.
  Instead of `rb_node_newnode`, use `rb_ast_newnode` to create a NODE.
* iseq.c, load.c, ruby.c, template/prelude.c.tmpl: modifies some
  functions to handle `ast_t*` instead of `NODE*`.
* test/ruby/test_gc.rb: ad-hoc fix for a failed test.  The test assumes
  GC eden is increased at startup by NODE object creation.  However,
  this change now create no NODE object, so GC eden is not necessarily
  increased.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@60485 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-10-27 15:59:02 +00:00
kazu
72ad249e54 Update comments in prelude.c [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-10-18 15:40:31 +00:00
nobu
2c42aac9fb prelude.c.tmpl: optimize
* template/prelude.c.tmpl: enable tail call optimization.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52549 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-11-12 07:16:23 +00:00