Commit graph

72 commits

Author SHA1 Message Date
duerst
13c15beb68 avoid passing unnecessary options to download method
The option --unicode-beta for tool/downloader.rb introduced in
r66448 must not be passed as an option to actual download machinery.
Thanks to MSP-Greg (Greg L.) for bug report and patch.
This closes issue #15434.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66449 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-19 04:08:46 +00:00
duerst
eb2e931f6f add option to deal with Unicode beta data files
Unicode uses file names with explicit versions for beta publication.
This commit introduces a variable on the makefile level to distinguish
between beta and regular versions of file names.
 
common.mk: Define new variable UNICODE_BETA, usually set to NO,
but would be YES during tests with beta data files. Pass the value
of this variable to tool/downloader.rb with option --unicode-beta.

tool/downloader.rb: Receive and store value of --unicode-beta.
Raise an exception if value is YES, because we don't yet actually
deal with this case. Continue as usual if value is not YES.

This completes the changes needed in the makefile. Upcomming changes
will only affect tool/downloader.rb.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66448 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-19 00:26:31 +00:00
k0kubun
57005046a1 tool/downloader.rb: retry Net::OpenTimeout
to prevent random failure
461667474

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@66110 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-12-01 00:37:54 +00:00
k0kubun
c312111c7c tool/downloader.rb: retry Net::ReadTimeout
as failed in
19540849/job/9p9pq0b7hfrb2r8q

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65097 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-10-16 09:29:23 +00:00
k0kubun
288e27899b tool/downloader.rb: retry 500 from GitHub
Unfortunately, GitHub may return 500 for temporary failure:
417246523

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64425 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-18 02:04:44 +00:00
k0kubun
4fe8308e0c tool/downloader.rb: increase retries
GitHub download failed on 13:50:36
https://ci.appveyor.com/project/ruby/ruby/build/1.0.9221 and it also
failed on 13:51:35 (all builds between them failed too).

It means that we need to expect GitHub 502 that continues 1 minute.
So I configured 6 retries, that will sleep at most 91s in total.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64317 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-12 06:00:31 +00:00
k0kubun
92c03be888 tool/downloader.rb: retry on 502 and 503 error
rubyci was failed by download 503
20180811T021706Z.fail.html.gz

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64282 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-08-11 04:22:14 +00:00
duerst
abb7438849 tool/downloader.rb:
* remove initial "./" from destdir to make downloading
  ./enc/unicode/data/11.0.0/ucd/auxiliary/GraphemeBreakProperty.txt
  work properly
* tweak a comment

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63978 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-07-16 10:20:24 +00:00
k0kubun
349e7fcb66 tool/downloader.rb: retry SocketError
I would like to stabilize RubyCI.

20180205T123003Z.fail.html.gz

downloading did_you_mean-1.2.0.gem ... /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:212:in `rescue in download': failed to download did_you_mean-1.2.0.gem (RuntimeError)
SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Name or service not known): https://rubygems.org/downloads/did_you_mean-1.2.0.gem
	from /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:123:in `download'
	from /home/hsbt/chkbuild/tmp/build/20180205T123003Z/tmp/ruby-snapshot20180205-23226-3201ha/ruby-2.6.0-r62226/tool/downloader.rb:66:in `download'
	from -e:4:in `<main>'
make[1]: *** [Makefile:1685: update-gems] Error 1

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@62231 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-02-05 13:24:00 +00:00
k0kubun
35e631273a tool/downloader.rb: retry against Errno::ETIMEDOUT
instead of Errno::ECONNREFUSED.

As I commented in r61498, at that moment I was not sure if the actual
exception is really `Errno::ECONNREFUSED` or not.

In https://ci.appveyor.com/project/ruby/ruby/build/1.0.6974, I could
confirm that the download can fail with `Errno::ETIMEDOUT`. So I want to
retry that.

Let's add `Errno::ECONNREFUSED` too if it fails with the exception.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61762 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2018-01-10 15:44:52 +00:00
k0kubun
57e95f38b1 tool/downloader.rb: don't suppress exception
on the end of retry failure.

Fixed mistake in r61498

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61499 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-27 13:35:25 +00:00
k0kubun
ff6c540adb tool/downloader.rb: retry downloads
because it's randomly failing on CI like
https://ci.appveyor.com/project/ruby/ruby/build/1.0.6724

Actually I'm not sure whether the exception class is Errno::ECONNREFUSED
or not. Please change the rescued exception to the correct one if it's
wrong. I changed to log exception class too in this commit.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61498 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-27 13:32:59 +00:00
hsbt
774f650bbe Revert "switch from http to https for Unicode data file downloads"
This reverts commit 90de118a75.

  The environment of mswinci couldn't handle https download.
  (It's certificates problem, We should upgrade its environment)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61169 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-12 14:10:43 +00:00
duerst
90de118a75 switch from http to https for Unicode data file downloads
(patch from MSP-Greg (Greg L), this closes issue #13962)

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61145 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-12-12 09:00:20 +00:00
nobu
b391708718 downloader.rb: get rid of symlinks in dist files
* tool/downloader.rb: download to the target file directly if
  CACHE_SAVE=no, to get rid of symlinks in dist files.
  [ruby-core:81887] [Bug #13709]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59292 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-07-09 04:49:15 +00:00
nobu
5522d79803 downloader.rb: get rid of symlinks
* tool/downloader.rb: do not save downloaded files to cache.
  [ruby-core:81780] [Bug #13684]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59185 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-06-27 17:10:56 +00:00
nobu
c45a52af36 downloader.rb: true symlink on cygwin
* tool/downloader.rb (Downloader.link_cache): on Cygwin, make true
  symlink only, which is provided by the OS.  as mingw/mswin ruby
  can't follow cygwin's pseudo symlink, it will fail on the same
  source tree.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58843 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-22 04:10:51 +00:00
nobu
7a14ffbf5d fix r58833
* tool/downloader.rb (Downloader.download): just link to the cache
  when downloaded to the cache but not the target file.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58841 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-22 03:18:17 +00:00
nobu
48070cef94 downloader cache
* tool/downloader.rb (Downloader.download): manage download cache.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58833 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21 16:45:35 +00:00
nobu
2ee57d5ff1 remove verify option
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58832 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21 14:29:00 +00:00
nobu
70b9957689 downloader.rb: utilize dir argument
* tool/downloader.rb (Downloader::RubyGems.download): utilize
  `dir` argument of Downloader.download.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58831 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-05-21 08:30:50 +00:00
nobu
81ab413288 fix GraphemeBreakProperty.txt
* tool/downloader.rb: download to the file given in ARGV.

* tool/enc-unicode.rb (parse_GraphemeBreakProperty): fix data file
  path as $(UNICODE_PROPERTY_FILES) in common.mk.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58069 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-03-23 15:49:10 +00:00
nobu
3805c4442b tool/downloader.rb: dryrun option [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@58043 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2017-03-21 07:19:23 +00:00
naruse
c11e648799 Regexp supports Unicoe 9.0.0's \X
* meta character \X matches Unicode 9.0.0 characters with some workarounds
  for UTR #51 Unicode Emoji, Version 4.0 emoji zwj sequences.
  [Feature #12831] [ruby-core:77586]

The term "character" can have many meanings bytes, codepoints, combined
characters, and so on. "grapheme cluster" is highest one of such words,
which means user-perceived characters.
Unicode Standard Annex #29 UNICODE TEXT SEGMENTATION specifies how to
handle grapheme clusters (extended grapheme cluster).
But some specs aren't updated to current situation because Unicode Emoji
is rapidly extended without well definition.
It breaks the precondition of UTR#29 "Grapheme cluster boundaries can be
easily tested by looking at immediately adjacent characters". (the
sentence will be removed in the next version)
Though some of its detail are described in Unicode Technical Report #51
UNICODE EMOJI but it is not merged into UTR#29 yet.

http://unicode.org/reports/tr29/
http://unicode.org/reports/tr51/
http://unicode.org/Public/emoji/4.0/

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56949 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-11-30 17:29:19 +00:00
hsbt
2e5fffa7cf * tool/downloader.rb: Removed verification of gem certification.
Because signed gem is not working on rubygems ecosystem.
* tool/gem-unpack.rb: ditto.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@56399 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-10-12 06:24:59 +00:00
normal
f8d0bdedf1 tool: add descriptions and fix typos
* tool/asm_parse.rb: add description
* tool/change_maker.rb: ditto
* tool/downloader.rb: ditto
* tool/eval.rb: ditto
* tool/expand-config.rb: ditto
* tool/extlibs.rb: ditto
* tool/fake.rb: ditto
* tool/file2lastrev.rb: ditto
* tool/gem-unpack.rb: ditto
* tool/gen_dummy_probes.rb: ditto
* tool/gen_ruby_tapset.rb: ditto
* tool/generic_erb.rb: ditto
* tool/id2token.rb: ditto
* tool/ifchange: ditto
* tool/insns2vm.rb: ditto
* tool/instruction.rb: ditto
* tool/jisx0208.rb: ditto
* tool/merger.rb: ditto
* tool/mkrunnable.rb: ditto
* tool/node_name.rb: ditto
* tool/parse.rb: ditto
* tool/rbinstall.rb: ditto
* tool/rbuninstall.rb: ditto
* tool/rmdirs: ditto
* tool/runruby.rb: ditto
* tool/strip-rdoc.rb: ditto
* tool/vcs.rb: ditto
* tool/vtlh.rb: ditto
* tool/ytab.sed: ditto
* tool/enc-unicode.rb: fix typo
* tool/mk_call_iseq_optimized.rb: ditto
* tool/update-deps: ditto
  [ruby-core:76215] [Bug #12539]
  by Noah Gibbs <the.codefolio.guy@gmail.com>

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55564 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-07-02 21:01:04 +00:00
nobu
37beb43b55 downloader.rb: disable verify if rubygems is old
* tool/downloader.rb (Downloader::RubyGems.download): verify gems
  only if RubyGems is 2.4 or later.  old RubyGems fails to verify
  almost all of bundled gems.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55055 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-05-18 07:04:55 +00:00
usa
3f1fc5e42b * tool/downloader.rb (RubyGems.download): follow the change of the
rubygems ssl_certs directory tree introduced by previous commit.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@54502 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-04-06 08:40:02 +00:00
nobu
38c0bdccdb downloader.rb: defer warning
* tool/downloader.rb (Downloader.download): warn using http only
  when really downloading.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53865 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-18 07:53:13 +00:00
nobu
df64c27ef8 downloader.rb: https setter
* tool/downloader.rb (Downloader.https): define class variable
  setter for 1.8, class_variable_set was private.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53717 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-02-02 07:08:13 +00:00
duerst
39988ff0ac * tool/downloader.rb: Fixed a logical error, improved documentation
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53631 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-23 07:30:32 +00:00
nobu
f78ab819bf common.mk: get rid of repeating same names
* common.mk (download-unicode-data): strip directory paths from
  the target names, to get rid of repeating same file names.

* tool/downloader.rb: add -p option for prefix.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53574 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2016-01-18 10:21:03 +00:00
naruse
8f0268991c Use more tough server: GitHub or repo.or.cz
https://gcc.gnu.org/wiki/GitMirror

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52761 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-11-26 15:40:58 +00:00
nobu
a3ddead991 downloader.rb: Downloader.https
* tool/downloader.rb (Downloader.https): extract a method instead
  of a global variable.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@52758 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-11-26 09:09:49 +00:00
kazu
894507c5f4 fix a typo [ci skip]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50175 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-04-06 15:00:00 +00:00
naruse
886d8e3f40 * tool/downloader.rb (http_options): prevent content auto decoding
because this is a downloader.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@50170 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-04-06 02:28:44 +00:00
usa
665ee6cd03 * tool/downloader.rb (Downloader::RubyGems.download): Gem::Security::*#ui= is
very new feature.



git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49459 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-01-31 11:40:43 +00:00
naruse
e7317dde41 Use master instead of HEAD
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49340 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-01-20 02:25:58 +00:00
nobu
3842ebb43d suppress alert warnings
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49295 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-01-17 06:55:20 +00:00
nobu
a9e033b104 downloader.rb: verify gems
* tool/downloader.rb (RubyGems.download): verify downloaded gem
  packages.  LowSecurity to allow untrusted certificates now.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49251 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2015-01-14 07:45:28 +00:00
usa
e7011a3ec9 * tool/downloader.rb: support old versions of ruby.
* tool/downloader.rb: now can download gems by http if openssl is not
  available (this may be danger!)


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@49014 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-25 16:25:57 +00:00
nobu
5b4ff7d6f9 downloader.rb: fix exception arguments
* tool/downloader.rb (Downloader.download): fix the exception to
  re-raise.  initialize methods of subclasses of Exception may
  have different parameters.  [ruby-core:67086] [Bug #10639]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48963 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-24 04:00:27 +00:00
naruse
a270e96a14 * tool/downloader.rb: use config.guess in gcc repo.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48961 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-24 03:07:14 +00:00
naruse
7979a5ad08 url is not required here
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48960 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-24 03:06:59 +00:00
usa
84c39fb2fa * tool/downloader.rb: typo.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48956 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-24 01:44:51 +00:00
naruse
943a05d234 support ruby 1.8
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48955 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-24 01:29:03 +00:00
naruse
7a0676e8a5 * tool/downloader.rb: support ruby 1.8.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48953 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-23 20:41:51 +00:00
usa
7e9175e3d9 * lib/open-uri.rb (OpenURI.open_http): accept multiple certs path in
ssl_ca_certs.

* tool/downloader.rb: use certs of rubygems for downloading gems.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48941 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-12-23 15:06:40 +00:00
nobu
4daa0fce20 common.mk: ims=nil option
* common.mk (update-gems): use ims=nil option.

* tool/downloader.rb (Downloader.download): show the message if no
  need to download in verbose mode.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48497 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-19 15:06:04 +00:00
naruse
501656c4d8 * tool/downloader.rb (RubyGems.download): Don't download gem if the
version is already downloaded. A gem file is versioned and
  it must be identical if the version is the same.

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48494 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
2014-11-19 09:15:04 +00:00