Commit graph

398 commits

Author SHA1 Message Date
akr
e0f064cf3c * transcode.c (econv_opts): don't use to_sym.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19363 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-15 14:38:12 +00:00
akr
c767be3039 * transcode_data.h: return output functions ssize_t.
* transcode.c (transcode_restartable0): don't need to cast the result
  of output functions.

* enc/trans/newline.trans: follow the type change.

* enc/trans/escape.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.

* enc/trans/iso2022.trans: ditto.

* enc/trans/japanese.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19351 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-14 18:35:17 +00:00
akr
a3c8c0adec * transcode_data.h: output function takes output buffer size.
* transcode.c: give output buffer size for output functions.

* enc/trans/newline.trans: follow the type change.

* enc/trans/escape.trans: ditto.

* enc/trans/utf_16_32.trans: ditto.

* enc/trans/iso2022.trans: ditto.

* enc/trans/japanese.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19350 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-14 18:06:20 +00:00
nobu
2f67b43aec * transcode.c (str_encode): returns duplicated string if nothing
changed.  [ruby-core:18578]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19349 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-14 17:37:35 +00:00
akr
0a3fc5be05 * transcode.c (rb_transcoding): align state field.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19326 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-13 09:52:02 +00:00
akr
d4c7473841 * transcode.c (DECORATOR_P): renamed from SUPPLEMENTAL_CONVERSION.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19325 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-13 09:21:34 +00:00
akr
8d7e774f5f * io.c (make_writeconv): use rb_enc_name.
(make_readconv): ditto.

* transcode.c (rb_econv_open_opts): ditto.
  (econv_args): ditto.
  (decorate_convpath): ditto.
  (econv_set_replacement): ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19324 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-13 09:06:51 +00:00
akr
2cc9b488a0 * transcode.c (rb_econv_prepare_opts): raise ArgumentError if
a broken string is specified as a replacement.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19316 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 18:45:44 +00:00
akr
7e58885f0a * transcode.c (rb_econv_prepare_opts): initialize *opt and return 0
if opthash is nil. 


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19312 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 17:53:26 +00:00
akr
41d3a01486 * enc/trans/escape.trans: transcoder name renamed to use underscore.
* transcode.c: follow the renaming.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19311 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 17:30:07 +00:00
akr
248acf0025 * transcode.c (rb_econv_init_by_convpath_i): new function.
(rb_econv_init_by_convpath): new function.
  (econv_init): use rb_econv_init_by_convpath.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19310 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 17:13:04 +00:00
akr
cf86176338 * transcode.c (econv_args): take snamev_p and dnamev_p to prevent
possible GC problem.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19309 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 16:13:02 +00:00
akr
aa423b9311 * transcode.c (enc_arg): take arg as VALUE * to prevent GC problem.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19307 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 15:39:46 +00:00
akr
c0bd38eb94 * transcode.c (rb_econv_add_transcoder_at): fix loop condition.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19306 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 14:20:27 +00:00
akr
2d80b1695d * transcode.c (decorator_names): extracted from rb_econv_open.
(rb_econv_open): use decorator_names.
  (econv_args): extracted from econv_init.
  (econv_init): use econv_args.
  (decorate_convpath): new function.
  (search_convpath_i): new function.
  (econv_s_search_convpath): new method.
  (Init_transcode): new method defined.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19305 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 13:53:42 +00:00
akr
89a5c998c7 * transcode.c (rb_econv_t): new field: num_allocated.
(rb_econv_alloc): extracted from
  rb_econv_open_by_transcoder_entries.
  (rb_econv_add_transcoder_at): extracted from rb_econv_decorate_at
  and generalized 
  (rb_econv_open_by_transcoder_entries): use rb_econv_alloc and
  rb_econv_add_transcoder_at.
  (rb_econv_add_converter): extracted from rb_econv_decorate_at.
  (rb_econv_decorate_at): use rb_econv_add_converter.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19304 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 12:58:23 +00:00
akr
19438eee8b * transcode.c (rb_econv_t): new field: started.
(rb_econv_open_by_transcoder_entries): initialize started field.
  (rb_econv_convert): set started field.
  (rb_econv_insert_output): ditto.
  (rb_econv_decorate_at): check started field instead of num_finished.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19303 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-12 12:23:25 +00:00
akr
10ddce3833 * transcode.c (make_encoding): new function.
(make_encobj): new function.
  (econv_s_asciicompat_encoding): use make_encoding.
  (rb_econv_open_exc): use SUPPLEMENTAL_CONVERSION.
  (econv_convpath): use encoding object in the result.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19288 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-10 17:26:49 +00:00
akr
4e93ace24e * transcode.c (econv_convpath): new method.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19286 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-10 17:15:08 +00:00
akr
2d26208a55 * include/ruby/encoding.h (econv_after_output): renamed from
econv_output_followed_by_input.
  (ECONV_AFTER_OUTPUT): renamed from ECONV_OUTPUT_FOLLOWED_BY_INPUT.

* transcode.c: follow the renaming.

* io.c: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19278 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-09 16:27:02 +00:00
akr
3cc7c9d0dd * transcode.c (get_replacement_character): don't return ASCII
incompatible replacements.
  (make_replacement): don't convert the result of
  get_replacement_character.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19277 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-09 16:18:29 +00:00
akr
817a623d13 * enc/trans/newline.trans (rb_universal_newline): swap src_encoding
and dst_encoding.

* transcode.c (rb_econv_decorate_at): call get_transcoder_entry only
  once.
  (rb_econv_binmode): follow universal_newline change.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19276 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-09 16:06:54 +00:00
akr
d456f7e357 * include/ruby/encoding.h (ECONV_DECORATOR_MASK): merge
ECONV_ENCODER_MASK and ECONV_DECORATOR_MASK.
  (ECONV_UNIVERSAL_NEWLINE_DECORATOR): renamed from
  ECONV_UNIVERSAL_NEWLINE_DECODER.
  (ECONV_CRLF_NEWLINE_DECORATOR): renamed from
  ECONV_CRLF_NEWLINE_ENCODER.
  (ECONV_CR_NEWLINE_DECORATOR): renamed from ECONV_CR_NEWLINE_ENCODER.
  (ECONV_XML_TEXT_DECORATOR): renamed from ECONV_XML_TEXT_ENCODER.
  (ECONV_XML_ATTR_CONTENT_DECORATOR): renamed from
  ECONV_XML_ATTR_CONTENT_ENCODER.
  (ECONV_STATEFUL_DECORATOR_MASK): renamed from
  ECONV_STATEFUL_ENCODER_MASK.
  (ECONV_XML_ATTR_QUOTE_DECORATOR): renamed from
  ECONV_XML_ATTR_CONTENT_DECORATOR.

* io.c: follow the renaming.

* transcode.c: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19271 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-09 12:22:43 +00:00
akr
8c2d4076f0 * transcode.c (rb_econv_open0): make it static.
(rb_econv_open): place decorators at last.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19269 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-09 11:57:31 +00:00
akr
40c31d5b75 * transcode.c (rb_econv_t): last_trans_index removed.
(rb_econv_open_by_transcoder_entries): follow the type change.
  (rb_econv_open0): ditto.
  (rb_econv_decorate_at): ditto.
  (rb_econv_binmode): ditto.
  (rb_econv_insert_output): simplified because there are no decorators
  at last.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19267 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-09 11:37:23 +00:00
akr
b96c3dfe67 * transcode.c (rb_econv_open): reverse the order of decorders.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19266 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-09 11:22:17 +00:00
akr
4fcb83c06c * transcode.c (rb_econv_open0): num_encoders and num_decoders removed.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19265 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-09 11:18:59 +00:00
akr
795bf7a665 * include/ruby/encoding.h (rb_econv_decorate_at_first): declared.
(rb_econv_decorate_at_last): declared.

* transcode.c (rb_econv_open_by_transcoder_entries): initialize
  replacement_enc.  allocate outbuf for the last transcoder.
  (rb_econv_open0): extracted from rb_econv_open.
  (rb_econv_open): use rb_econv_open0 and decorate the result using
  rb_econv_decorate_at_first and rb_econv_decorate_at_last.
  (rb_econv_decorate_at): new function.
  (rb_econv_decorate_at_first): ditto.
  (rb_econv_decorate_at_last): ditto.
  (rb_econv_binmode): fix iteration end condition.
  (econv_init): don't set source_encoding_name and
  destination_encoding_name because they are set in rb_econv_open0.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19262 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-08 17:23:37 +00:00
akr
65110d6464 * transcode.c (rb_econv_open): make ec a local variable.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19260 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-08 16:11:32 +00:00
akr
4106a7c067 * transcode.c (rb_econv_asciicompat_encoding): check decoder.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19253 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-08 15:21:05 +00:00
akr
dbf0487581 * transcode.c (rb_econv_t): last_error.partial_input removed.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19250 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-08 15:01:16 +00:00
akr
6270ad5b7f * include/ruby/encoding.h (rb_econv_asciicompat_encoding): renamed
from rb_econv_stateless_encoding to apply stateless ASCII
  incompatible encodings such as UTF-16BE.

* io.c (make_writeconv): use rb_econv_asciicompat_encoding.

* transcode_data.h (rb_transcoder_asciicompat_type_t): renamed from
  rb_transcoder_stateful_type_t.
  (rb_transcoder): use rb_transcoder_asciicompat_type_t.

* transcode.c: follow the type change.
  (asciicompat_encoding_i): renamed from stateless_encoding_i.
  (rb_econv_asciicompat_encoding): renamed from
  rb_econv_stateless_encoding.
  (econv_s_asciicompat_encoding): method renamed.

* tool/transcode-tblgen.rb: follow the type change.

* enc/trans/utf_16_32.trans: follow the type change.
  rb_from_UTF_16BE to UTF-8 is asciicompat_decoder.
  rb_from_UTF_16LE to UTF-8 is asciicompat_decoder.
  rb_from_UTF_32BE to UTF-8 is asciicompat_decoder.
  rb_from_UTF_32LE to UTF-8 is asciicompat_decoder.
  UTF-8 to rb_to_UTF_16BE is asciicompat_encoder.
  UTF-8 to rb_to_UTF_16LE is asciicompat_encoder.
  UTF-8 to rb_to_UTF_32BE is asciicompat_encoder.
  UTF-8 to rb_to_UTF_32LE is asciicompat_encoder.

* enc/trans/newline.trans: follow the type change.  universal newline
  decoder is asciicompat_converter.

* enc/trans/escape.trans: follow the type change.

* enc/trans/iso2022.trans: ditto.

* enc/trans/japanese.trans: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19249 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-08 14:33:17 +00:00
akr
a98bb55e01 * transcode.c (rb_econv_insert_output): "readagain" part should be
after replacement.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19247 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-08 14:08:44 +00:00
akr
6e36c4842e * include/ruby/encoding.h (rb_econv_set_replacement): renamed from
rb_econv_set_replacemenet.

* transcode.c: follow the renaming.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19227 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07 17:08:38 +00:00
akr
d4832b7aac rdoc update.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19223 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07 15:05:30 +00:00
akr
c0bec2fae1 * transcode_data.h (STR1): defined for a string up to 255 bytes.
(STR1_BYTEINDEX): defined.
  (makeSTR1): defined.

* tool/transcode-tblgen.rb: generate STR1.

* transcode.c (transcode_restartable0): interpret STR1.

* enc/trans/escape.trans (fun_so_escape_xml_chref): removed.  STR1 is
  used instead.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19214 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07 09:13:16 +00:00
akr
020e681eec * include/ruby/encoding.h (ECONV_XML_ATTR_CONTENT_ENCODER): defined.
(ECONV_STATEFUL_ENCODER_MASK): defined.
  (ECONV_XML_ATTR_QUOTE_ENCODER): defined.
  (ECONV_XML_ATTR_ENCODER): removed.

* enc/trans/escape.trans (rb_escape_xml_attr_content): defined.
  (rb_escape_xml_attr_quote): defined.
  (rb_escape_xml_attr): removed.

* io.c (NEED_WRITECONV): writeconv is required if supplemental
  converter is used.
  (make_writeconv): apply stateful encoder in writeconv.

* transcode.c: follow the constant change.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19209 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-07 03:13:29 +00:00
akr
76b3063022 * include/ruby/encoding.h (ECONV_XML_TEXT_ENCODER): renamed from
ECONV_HTML_TEXT_ENCODER.
  (ECONV_XML_ATTR_ENCODER): renamed from ECONV_HTML_ATTR_ENCODER.

* enc/trans/escape.trans: follow the renaming.

* transcode.c: ditto.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19191 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 14:05:10 +00:00
akr
ca3148059b * transcode.c (econv_description): show ECONV_HTML_TEXT_ENCODER and
ECONV_HTML_ATTR_ENCODER.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19188 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 11:11:21 +00:00
akr
e4b70ffe43 * transcode.c (rb_econv_binmode): check actual transcoders.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19187 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 11:07:01 +00:00
akr
f88c9f03f7 * transcode.c (rb_econv_open): fix last_tc.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19186 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 10:54:59 +00:00
akr
f10af82d25 * transcode.c (str_transcode0): check ECONV_HTML_TEXT_ENCODER and
ECONV_HTML_ATTR_ENCODER.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19184 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 09:48:32 +00:00
akr
987ec6b787 * transcode.c (allocate_converted_string): fix overflow condition.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19183 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 09:39:12 +00:00
akr
e490d92548 * transcode.c (sym_html): new variable.
(sym_text): ditto.
  (sym_attr): ditto.
  (econv_opts): check :html=>:text and :html=>:attr.
  (Init_transcode): initialize the above variables.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19180 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 06:07:34 +00:00
akr
34be4c1ce5 * include/ruby/encoding.h (ECONV_HTML_TEXT_ENCODER): new constant.
(ECONV_HTML_ATTR_ENCODER): ditto.

* transcode.c (rb_econv_open): check ECONV_HTML_TEXT_ENCODER and
  ECONV_HTML_ATTR_ENCODER.
  (Init_transcode): Encoding::Converter::HTML_TEXT_ENCODER and
  Encoding::Converter::HTML_ATTR_ENCODER defined.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19179 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 05:47:30 +00:00
akr
5117020d9e * transcode.c (struct trans_open_t): defined to pass num_additional.
(trans_open_i): use struct trans_open_t.
  (rb_econv_open): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19178 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 05:16:40 +00:00
akr
27e9c48ccc * transcode.c (rb_econv_open): needless branch removed.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19176 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 03:44:31 +00:00
akr
073bced367 * transcode.c (rb_econv_open): test condition simplified.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19175 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 03:38:59 +00:00
akr
393f57670f * transcode.c (stateless_encoding_i): ignore supplemental conversions.
Encoding::Converter.stateless_encoding("html-attr-escaped") should be
  nil.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19174 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-06 03:26:35 +00:00
akr
0be34e2a49 * transcode.c (rb_econv_open): fail for ASCII incompatible with
newline conversion.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@19167 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2008-09-05 22:27:46 +00:00