Commit graph

621 commits

Author SHA1 Message Date
nobu
8b4448e2e1 unicode.c: off-by-one error
* enc/unicode.c (CodePointListValidP): fix off-by-one error.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54091 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-03-12 01:35:54 +00:00
nobu
d48f923648 unicode.c: boundary check
* enc/unicode.c (CodePointListValidP): add pathological boundary
  check, for gcc 4.9.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54090 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-03-12 01:15:31 +00:00
duerst
59766643db * enc/unicode/case-folding.rb, casefold.h: Streamlining approach to
case mapping data not available from case folding by unifying all
  three cases (special title, special upper, special lower).
* enc/unicode.c: Adjust macro names for above (macros are currently inactive).
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54085 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-03-11 07:11:27 +00:00
duerst
c4e6964141 * enc/unicode/case-folding.rb, casefold.h: Reducing size of TitleCase
table by eliminating duplicates.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53957 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-27 08:06:17 +00:00
duerst
7feb182a08 * enc/unicode/case-folding.rb: Adding possibility for debugging output
for TitleCase table in casefold.h.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53930 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-25 10:04:59 +00:00
duerst
f1f48e6103 * include/ruby/oniguruma.h: Rearranging flag assignments and making
space for titlecase indices; adding additional macros to add or
  extract titlecase index; adding comments for better documentation.
* enc/unicode.c: Moving some macros to include/ruby/oniguruma.h;
  activating use of titlecase indices.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53915 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-24 13:32:01 +00:00
duerst
1cc579cb00 * enc/unicode/case-folding.rb, casefold.h: Outputting actual titlecase
data (new table, with indices from other tables).
* enc/unicode.c: Ignoring titlecase data indices for the moment.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53906 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-23 12:53:10 +00:00
duerst
8aa8847b7c * enc/unicode/case-folding.rb, casefold.h: Reading casing data from
SpecialCasing.txt.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53904 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-23 06:21:55 +00:00
duerst
4ca9138bac * enc/unicode/case-folding.rb, casefold.h: Adding flag for title-case,
not yet operational.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53891 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-22 09:34:34 +00:00
duerst
5470ce8206 * enc/unicode/case-folding.rb, casefold.h: Fixed bug that avoided inclusion
of compatibility characters in uppper-/lower-case mappings.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53890 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-22 09:17:43 +00:00
duerst
6286ff6301 * enc/unicode.c: Activated use of case mapping data in CaseUnfold_11 array.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53870 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-19 03:45:32 +00:00
duerst
6a808bda64 * enc/unicode/case-folding.rb, casefold.h: Used only first element
(rather than all) of target in CaseUnfold_11 array.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53843 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-16 10:10:37 +00:00
duerst
c3554cdea6 * enc/unicode/case-folding.rb: Added debugging option
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53833 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-15 05:43:55 +00:00
svn
60c7061770 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53780 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-08 12:26:36 +00:00
duerst
73ab88994f * enc/unicode/case-folding.rb, enc/unicode/casefold.h: Flags for
upper/lower conversion added (titlecase and SpecialCasing still missing)
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53779 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-08 12:26:35 +00:00
duerst
2ca7569c6d * string.c, enc/unicode.c: Disassociating ONIGENC_CASE_FOLD flag from
ONIGENC_CASE_DOWNCASE.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53778 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-08 11:44:12 +00:00
nobu
584f9e51d6 unicode.c: magic numbers
* enc/unicode.c (I_WITH_DOT_ABOVE, DOTLESS_i, DOT_ABOVE): name
  magic numbers.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53776 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-08 05:01:00 +00:00
duerst
8f10a72d90 * enc/unicode.c: Shortened macros for enc/unicode/casefold.h to
single-letter; use flags in casefold.h for logic.
* enc/unicode/case-folding.rb: Added flag for case folding.
  Changed parameter passing.
* enc/unicode/casefold.h: New flags added.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53775 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-08 04:00:31 +00:00
svn
3e82668e62 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53768 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 13:10:21 +00:00
duerst
49ca434bcf * common.mk: Added two more precondition files for enc/unicode/casefold.h
* enc/unicode.c: Added shortening macros for enc/unicode/casefold.h
* enc/unicode/case-folding.rb: Fixed file encoding for CaseFolding.txt
  to ASCII-8BIT (should fix some ci errors). Clarified usage. Created
  class MapItem. Partially implemented class CaseMapping.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53767 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 13:10:20 +00:00
duerst
e40d117339 * enc/unicode/case-folding.rb: Fixing parameter passing.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53765 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 05:12:44 +00:00
duerst
5cbe9fe5c4 * enc/unicode/case-folding.rb: New classes CaseMapping/CaseMappingDummy
to pass as parameters; not yet implemented or used.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53764 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 02:44:14 +00:00
duerst
ee696372a8 * common.mk: using new option in recipe for enc/unicode/casefold.h
* enc/unicode/case-folding.rb: Correctly specify argument to new option.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53762 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 02:16:12 +00:00
svn
b0b7cb5a46 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53760 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-07 01:39:27 +00:00
duerst
07ad033850 git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53759 b2dd03c8-39d4-4d8f-98ff-823fe69b080e 2016-02-07 01:39:26 +00:00
duerst
e8dde46b60 * test/ruby/enc/test_regex_casefold.rb: Added data-based testing for
String#downcase :fold.
* enc/unicode.c: Fixed a range error (lowest non-ASCII character affected
  by case operations is U+00B5, MICRO SIGN)
* test/ruby/enc/test_case_mapping.rb: Explicit test for case folding of
  MICRO SIGN to Greek mu.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53749 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-06 06:18:38 +00:00
duerst
81515b2381 * enc/unicode.c, test/ruby/enc/test_case_mapping.rb: Implemented :fold
option for String#downcase by using case folding data from
  regular expression engine, and added a few simple tests.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53747 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-06 05:37:29 +00:00
duerst
b658249cef * enc/unicode.c: Activated :ascii flag for ASCII-only case conversion
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53740 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-04 12:05:23 +00:00
nobu
7373320615 enc: workdir timestamps
* enc/depend: make timestamps for each work directory, instead of
  making for each compilation and link.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53714 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-02 04:39:38 +00:00
duerst
a7c987968d * enc/unicode.c: Fixed bit mask in macro OnigCodePointCount
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53670 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-27 09:54:38 +00:00
duerst
415949faba * enc/unicode.c: Protect code point count by macro, in order to
be able to use the remaining bits for flags.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53669 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-27 08:55:40 +00:00
duerst
f307d1fe21 * enc/unicode.c: Fixed a logical error and some comments.
* test/ruby/enc/test_case_mapping.rb: Made tests more general.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53564 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-17 11:10:45 +00:00
nobu
39f44f0113 get rid of non-ascii chars
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53563 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-17 09:03:11 +00:00
duerst
959bbb6f72 * enc/unicode.c: Removed artificial expansion for Turkic,
added hand-coded support for Turkic, fixed logic for swapcase.
* string.c: Made use of new case mapping code possible from upcase,
  capitalize, and swapcase (with :lithuanian as a guard).
* test/ruby/enc/test_case_mapping.rb: Adjusted for above.
  (with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53562 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-17 08:42:16 +00:00
duerst
c12af76763 * enc/unicode.c: Artificial mapping to test buffer expansion code.
* string.c: Fixed buffer expansion logic.
* test/ruby/enc/test_case_mapping.rb: Tests for above.
(with Kimihito Matsui)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53554 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-16 08:24:58 +00:00
hsbt
219467abde * enc/unicode.c: fix implicit conversion error with clang. fixup r53548.
* string.c: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53552 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-16 01:51:58 +00:00
duerst
be897c2507 * string.c, enc/unicode.c: New code path as a preparation for Unicode-wide
case mapping. The code path is currently guarded by the :lithuanian
  option to avoid accidental problems in daily use.
* test/ruby/enc/test_case_mapping.rb: Test for above.
* string.c: function 'check_case_options': fixed logical errors

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53548 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-16 01:24:03 +00:00
nobu
fabb8b45cd enc/Makefile.in: get rid of nmake bug
* enc/Makefile.in (ECHO1): expand NULLCMD by configured value to
  get rid of a bug of nmake, that it can expand bare single name
  variable but cannot in substition.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53437 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-05 00:48:04 +00:00
hsbt
701bfa0fed * enc/x_emoji.h: fix dead-link.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53345 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-27 11:00:36 +00:00
nobu
bff89a030a depend: version dependency
* enc/depend (enc, trans): fix version dependency, let encoding
  and transcoding shared object files depend on config.status,
  instead of enc.mk which is regenerated at each build, for the
  RUBY_SO_NAME value used at runtime link.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53325 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-27 01:51:50 +00:00
nobu
c7b2450e89 depend: version dependency
* enc/depend (enc, trans): fix version dependency, shared object
  files depend on the RUBY_SO_NAME value for runtime link.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53324 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-27 01:03:16 +00:00
ngoto
a2bf609304 * enc/windows_1250.c: Should not use C++ style comments (C99 feature).
[Bug #11843]


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53202 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-19 11:09:28 +00:00
duerst
6973c8d51e enc/iso_8859_2.c, enc/windows_1250.c: separate Windows-1250
from ISO-8859-2 to fix 0x80..0x9e range (from Kimihito Matsui)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53198 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-19 01:33:38 +00:00
naruse
abfc03c6cf follow the change of the name
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53128 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-15 13:11:33 +00:00
duerst
1db69bf3ef reverting accidental commit at r53124 by re-committing version from r52612
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53127 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-15 10:25:44 +00:00
duerst
81e7be08ed enc/ebcdic.h, enc/trans/ebcdic.trans,
test/ruby/test_transcode.rb: Fixed encoding name
to the correct one in the IANA registry (IBM037)
and added an alias (ebcdic-cp-us)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53124 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-15 08:57:58 +00:00
svn
3473c9e7d4 * remove trailing spaces.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53113 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-14 13:11:59 +00:00
duerst
83304b75c1 * enc/ebcdic.h: new dummy encoding EBCDIC-US
* enc/trans/ebcdic.trans: transcodings between EBCDIC-US
  and iso-8859-1 [with code from Andrea Ribuoli]
* test/ruby/test_transcode.rb: tests for above
* tool/transcode_tablegen.rb: additional argument for
  method transcode_tblgen

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53112 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-14 13:11:31 +00:00
hsbt
43ba3280c3 * enc/iso_2022_jp.h: fix typos.
[ci skip][fix GH-1140] Patch by @jutaz
* enc/utf_16_32.h: ditto.
* enc/utf_7.h: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53102 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-14 02:50:21 +00:00
nobu
3ee57230b3 enc/windows_1252.c: new
* enc/windows_1252.c: separate from ISO-8859-1 to fix 0x80..0x9e
  range.  [ruby-core:64049] [Bug #10097]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53046 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-12-11 22:05:31 +00:00