[Universal Parser] Reduce dependence on RArray in parse.y

- Introduce `rb_parser_ary_t` structure to partly eliminate RArray from parse.y
  - In this patch, `parser_params->tokens` and `parser_params->ast->node_buffer->tokens` are now `rb_parser_ary_t *`
  - Instead, `ast_node_all_tokens()` internally creates a Ruby Array object from the `rb_parser_ary_t`
  - Also, delete `rb_ast_tokens()` and `rb_ast_set_tokens()` in node.c

- Implement `rb_parser_str_escape()`
  - This is a port of the `rb_str_escape()` function in string.c
  - `rb_parser_str_escape()` does not depend on `VALUE` (RString)
  - Instead, it uses `rb_parser_stirng_t *`
  - This function works when --dump=y option passed

- Because WIP of the universal parser, similar functions like `rb_parser_tokens_free()` exist in both node.c and parse.y. Refactoring them may be needed in some way in the future

- Although we considered redesigning the structure: `ast->node_buffer->tokens` into `ast->tokens`, we leave it as it is because `rb_ast_t` is an imemo. (We will address it in the future)
This commit is contained in:
HASUMI Hitoshi 2024-02-16 17:45:22 +09:00 committed by Yuichiro Kaneko
parent f42164e037
commit 9a19cfd4cd
7 changed files with 428 additions and 214 deletions

4
node.h
View file

@ -40,7 +40,7 @@ struct node_buffer_struct {
// - text of token
// - location info
// Array, whose entry is array
VALUE tokens;
rb_parser_ary_t *tokens;
#ifdef UNIVERSAL_PARSER
const rb_parser_config_t *config;
#endif
@ -55,7 +55,6 @@ rb_ast_t *rb_ast_new(void);
#endif
size_t rb_ast_memsize(const rb_ast_t*);
void rb_ast_dispose(rb_ast_t*);
VALUE rb_ast_tokens(rb_ast_t *ast);
#if RUBY_DEBUG
void rb_ast_node_type_change(NODE *n, enum node_type type);
#endif
@ -65,7 +64,6 @@ void rb_node_init(NODE *n, enum node_type type);
void rb_ast_mark_and_move(rb_ast_t *ast, bool reference_updating);
void rb_ast_update_references(rb_ast_t*);
void rb_ast_free(rb_ast_t*);
void rb_ast_set_tokens(rb_ast_t*, VALUE);
NODE *rb_ast_newnode(rb_ast_t*, enum node_type type, size_t size, size_t alignment);
void rb_ast_delete_node(rb_ast_t*, NODE *n);
rb_ast_id_table_t *rb_ast_new_local_table(rb_ast_t*, int);