(rb_enc_nth): don't check the return value of rb_enc_mbclen.
(rb_enc_strlen): ditto.
(rb_enc_precise_mbclen): return needmore(1) if e <= p.
(rb_enc_get_ascii): new function for extracting ASCII character.
* include/ruby/encoding.h (rb_enc_get_ascii): declared.
* include/ruby/regex.h (ismbchar): removed.
* re.c (rb_reg_expr_str): use rb_enc_get_ascii.
(unescape_escaped_nonascii): use rb_enc_precise_mbclen to determine
the termination of escaped non-ASCII character.
(unescape_nonascii): use rb_enc_precise_mbclen.
(rb_reg_quote): use rb_enc_get_ascii.
(rb_reg_regsub): use rb_enc_get_ascii.
* string.c (rb_str_reverse) don't check the return value of
rb_enc_mbclen.
(rb_str_split_m): don't call rb_enc_mbclen with e <= p.
* parse.y (is_identchar): use ISASCII.
(parser_ismbchar): removed.
(parser_precise_mbclen): new macro.
(parser_isascii): new macro.
(parser_tokadd_mbchar): use parser_precise_mbclen to check invalid
character precisely.
(parser_tokadd_string): use parser_isascii.
(parser_yylex): ditto.
(is_special_global_name): don't call is_identchar with e <= p.
(rb_enc_symname_p): ditto.
[ruby-dev:32455]
* ext/tk/sample/tkextlib/vu/canvSticker2.rb: remove coding cookie
because the encoding is not UTF-8. [ruby-dev:32475]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14131 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
[ruby-dev:31351]
* thread.c (ruby_suppress_tracing): added a new parameter, which
directs to call func always.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14104 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* re.c (rb_reg_preprocess): new function for dynamic regexp with
\u{} such as Regexp.new("\\u{6666}").
(rb_reg_prepare_re): preprocess regexp for recompiling.
(read_escaped_byte): new function.
(unescape_escaped_nonascii): new function.
(append_utf8): new function.
(unescape_unicode_list): new function.
(unescape_unicode_bmp): new function.
(unescape_nonascii): new function.
(rb_reg_initialize): preprocess regexp.
* pack.c (rb_uv_to_utf8): renamed from uv_to_utf8.
* parse.y (STR_NEW3): take func instead of has8 and hasmb.
(parser_str_new): use default coderange mechanism except for regexp.
(parser_tokadd_utf8): copy regexp source as-is.
(parser_read_escape): UTF-8 stuff removed.
(parser_tokadd_escape): has8bit and hasmb removed.
(parser_tokadd_string): fix 8-bit single byte character with \u.
(parser_parse_string): has8bit and hasmb removed.
(parser_here_document): has8bit and hasmb removed.
(parser_yylex): call parser_tokadd_utf8 instead of read_escape for
UTF-8 character.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14072 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
rename ENC_CODERANGE_SINGLE to ENC_CODERANGE_7BIT.
rename ENC_CODERANGE_MULTI to ENC_CODERANGE_8BIT.
Because single byte 8bit character, such as Shift_JIS 1byte katakana,
is represented by ENC_CODERANGE_MULTI even if it is not multi byte.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@14027 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
[ruby-dev:32250]
* parse.y: remove NEED_ASSOC that break test_parser_events.
* parse.y (parser_yylex): should not decrement line numbers at the
end of file.
* file.c (rb_find_file_ext): search .rb files first through in the
loadpath.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13966 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
bison with sed. [ruby-dev:32204]
* ruby.c (proc_options): use yydebug in cmdline_options.
* ruby.c (process_options): set yydebug flag of parser.
* parse.y (yydebug): moved into struct parser_params.
* parse.y (rb_parser_get_yydebug, rb_parser_set_yydebug): parser
generic methods.
* */Makefile.sub (parse.c): moved to common.mk.
* tool/ytab.sed: comment out yydebug definition, and substitute
yyerror with parser_yyerror.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13908 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* parse.y (parser_yylex): adjust line number for fluent interface.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13858 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
also modifications for better coderange detection
* test/ruby/test_unicode_escapes.rb: test cases
* test/ruby/test_mixed_unicode_escapes.rb: mixed encoding test cases
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13836 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
correct interning of multi-byte symbols. Without this patch
:x==:x is false when x is a multi-byte character.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13835 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
version" warning. when I added this warning, I had a plan to
reimplement the parser that is simpler than the current one.
since we abandoned the plan, warning no longer required.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13821 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
continuation from previous one to support fluent interface.
[experimental]
* misc/ruby-mode.el (ruby-calculate-indent): support fluent dot.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13777 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* parse.y (parser_yylex): fix encoding of single character literal.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13766 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
matches to current encoding.
* re.c (char_to_option, rb_char_to_option_kcode): 'n' is not kcode
option now.
* re.c (rb_reg_to_s, rb_reg_error_desc): copy encoding rather than
append as an option.
* re.c (make_regexp, rb_reg_prepare_re): use encoding of Regexp and
String instead of kcode.
* re.c (rb_reg_initialize): set fixed option if none is set.
* re.c (rb_reg_regcomp): ditto.
* re.c (rb_reg_equal): check if encodings are equal.
* re.c (rb_reg_initialize_m): encoding option is obsolete.
* re.c (rb_kcode, rb_get_kcode, rb_set_kcode): removed.
* re.c (Init_Regexp): removed Regexp#kcode method.
* ruby.c (proc_options): allow long encoding name.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13717 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
David Flanagan <david AT davidflanagan.com> [ruby-core:12629]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13689 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
/coding[:=] ?/, a la Python PEP-263, so that VIM comments like
"# vim: set fileencoding=<encoding name>" should be recognized.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@13685 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
