This commit introduces new core class Enumerator::ArithmeticSequence.
Enumerator::ArithmeticSequence is a subclass of Enumerator, and
represents a number generator of an arithmetic sequence.
After this commit, Numeric#step and Range#step without blocks
returned an ArithmeticSequence object instead of an Enumerator.
This class introduces the following incompatibilities:
- You can create a zero-step ArithmeticSequence,
and its size is not ArgumentError, but Infinity.
- You can create a negative-step ArithmeticSequence from a range.
[ruby-core:82816] [Feature #13904]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64205 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Unfortunately, MJIT goes over the 100M RLIMIT_AS for this test
on x86-64 (Debian 9)
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64204 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Trying to diagnose CI failures from i686-linux on Debian 7.
This also fixes a potential GC problem with Thread.start or
IO#read potentially closing the write-end of the pipe due
to allocation.
cf. 20180805T080500Z.fail.html.gz
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64199 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
MJIT way generate an extra SIGCHLD, so we'll have to deal with it.
Any realistic Ruby program may hit unexpected SIGCHLD, too,
since any spawned subprocess could have extra grandchildren
which get reaped by the main Ruby process, and SIGCHLD may
be sent spuriously by an external process using kill(2) syscall
(via kill(1) or Process.kill).
cf. http://ci.rvm.jp/results/trunk-mjit@silicon-docker/1194620
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64194 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
We can reuse the sub-thread and exception with Thread#raise to
reproduce the old memory leak with less overhead. This allows
us to to run more iterations and improve reliability of the
actual test, particularly on platforms without USE_THREAD_CACHE.
For glibc and jemalloc, also limit arena count to avoid
inadvertant growth.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64187 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Use Thread.pass to let the sub-thread finish. This should avoid
running out of memory on resource-constrained systems.
(We don't want to use Thread#join to hit the exception)
This results in a significant memory usage reduction in this test:
make test-all TESTS='../test/ruby/test_thread.rb -v -n test_thread_interrupt_for_killed_thread'
Before:
3.46user 2.02system 0:05.29elapsed 103%CPU (0avgtext+0avgdata 88672maxresident)k
0inputs+0outputs (0major+143071minor)pagefaults 0swaps
After:
3.40user 1.83system 0:05.20elapsed 100%CPU (0avgtext+0avgdata 9368maxresident)k
0inputs+0outputs (0major+3059minor)pagefaults 0swaps
cf. 20180803T231706Z.fail.html.gz
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64186 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Still looking into CI failures on P895 (and seemingly no other boxes):
http://ci.rvm.jp/results/trunk@P895/1190369
:<
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64184 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ast.c (Init_ast): undefine allocator of Node, as a method call
on an uninitialized Node causes segfault.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64168 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Actually unload_units is working on MinGW, but putiseq is behaving
badly.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64163 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
It looks like we need to retry test_timestampns in addition
to test_timestamp; so share some code while we're at it.
cf. http://ci.rvm.jp/results/trunk-test@frontier/1153126
[ruby-core:88104] [Bug #14898]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64157 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
when $MJIT_SEARCH_BUILD_DIR is set.
If prefix path is owned by root, `make install` needs to be run by root.
But in general we don't want to run `make test-all`, and also running
`make test-all` currently fails due to permission tests of rdoc and rubygems.
Thus, prior to this commit, specifying a prefix like "/usr/local" could
mean there was no way to pass test-all.
So we should not depend on `make install` for `make test-all`. Thus I
reverted r64104 and r64103, and applied this workaround to pass `make
test-all` without `make install`.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64143 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/ruby/test_thread_{cv,queue}.rb: Move under the test/ruby directory.
and rename TestThread* from Test*.
* test/test_sync.rb: Move toplevel of test diretory because sync is still
standard library.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64137 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
I can't reproduce the test failure and I'm still not sure what's
wrong, but maybe we can get a core dump from P895:
http://ci.rvm.jp/results/trunk@P895/1174672
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64135 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
To reduce resource use and reduce CI failure; remove
timer-thread. Single-threaded Ruby processes (including forked
children) will never see extra thread overhead. This prevents
glibc and jemalloc from going into multi-threaded mode and
initializing locks or causing fragmentation via arena explosion.
The GVL is implements its own wait-queue as a ccan/list to
permit controlling wakeup order. Timeslice under contention is
handled by a designated timer thread (similar to choosing a
"patrol_thread" for current deadlock checking).
There is only one self-pipe, now, as wakeups for timeslice are
done independently using condition variables. This reduces FD
pressure slightly.
Signal handling is handled directly by a Ruby Thread (instead
of timer-thread) by exposing signal self-pipe to callers of
rb_thread_fd_select, native_sleep, rb_wait_for_single_fd, etc...
Acquiring, using, and releasing the self-pipe is exposed via 4
new internal functions:
1) rb_sigwait_fd_get - exclusively acquire timer_thread_pipe.normal[0]
2) rb_sigwait_fd_sleep - sleep and wait for signal (and no other FDs)
3) rb_sigwait_fd_put - release acquired result from rb_sigwait_fd_get
4) rb_sigwait_fd_migrate - migrate signal handling to another thread
after calling rb_sigwait_fd_put.
rb_sigwait_fd_migrate is necessary for waitpid callers because
only one thread can wait on self-pipe at a time, otherwise a
deadlock will occur if threads fight over the self-pipe.
TRAP_INTERRUPT_MASK is now set for the main thread directly in
signal handler via rb_thread_wakeup_timer_thread.
Originally, I wanted to use POSIX timers
(timer_create/timer_settime) for this. Unfortunately, this
proved unfeasible as Mutex#sleep resumes on spurious wakeups and
test/thread/test_cv.rb::test_condvar_timed_wait failed. Using
pthread_sigmask to mask out SIGVTALRM fixed that test, but
test/fiddle/test_function.rb::test_nogvl_poll proved there'd be
some unavoidable (and frequent) incompatibilities from that
approach.
Finally, this allows us to drop thread_destruct_lock and
interrupt current ec directly.
We don't need to rely on vm->thread_destruct_lock or a coherent
vm->running_thread on any platform. Separate timer-thread for
time slice and signal handling is relegated to thread_win32.c,
now.
[ruby-core:88088] [Misc #14937]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64107 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This should fix CI failure under MJIT, 100ms wait may not be
enough for signal handler to fire:
http://ci.rvm.jp/results/trunk-mjit@silicon-docker/1169472
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64105 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
for the ease of MSP-Greg's CI. I'll track this failure as Bug#14948 and
fix it later.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64099 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
When --jit-wait is specified, `unit_queue.length` is always 0 and it's
not a good metric.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64097 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
When all compilation finishes or the number of JIT-ed code reaches
--jit-max-cache, this compacts all generated code to a single .so file
and re-loads all methods from it.
In the future, it may trigger compaction more frequently and/or limit
the maximum times of compaction to prevent unlimited memory usage.
So the current behavior is experimental, but at least the performance
improvement in this commit won't be removed.
=== Benchmark ===
In this benchmark, I'll compare following four conditions:
* trunk: r64082
* trunk JIT: r64082 w/ --jit
* single-so JIT: This commit w/ --jit
* objfcn JIT: This branch https://github.com/k0kubun/ruby/tree/objfcn w/ --jit,
which is shinh's objfcn https://github.com/shinh/ruby/tree/objfcn rebased from this commit
```
$ uname -a
Linux bionic 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
```
* Micro benchmark
Using this script https://gist.github.com/k0kubun/10e6d3387c9ab1b134622b2c9d76ef51,
calls some amount of different methods that just return `nil`. The following tables
are its average duration seconds of 3 measurements.
Smaller is better.
** 1 method (seconds)
| | trunk | trunk JIT | single-so JIT | objfcn JIT |
|:------|:------------------|:------------------|:------------------|:------------------|
| Time | 5.576067774333296 | 5.915551971666446 | 5.833641665666619 | 5.845915191666639 |
| Ratio | 1.00x | 1.06x | 1.05x | 1.05x |
** 50 methods (seconds)
| | trunk | trunk JIT | single-so JIT | objfcn JIT |
|:------|:------------------|:------------------|:------------------|:------------------|
| Time | 3.1661167996666677| 6.125825928333342 | 4.135432743666665 | 3.750358728333348 |
| Ratio | 1.00x | 1.93x | 1.31x | 1.18x |
** 1500 methods (seconds)
| | trunk | trunk JIT | single-so JIT | objfcn JIT |
|:------|:------------------|:------------------|:------------------|:------------------|
| Time | 5.971650823666664 | 19.579182102999994| 10.511108153999961| 10.854653588999932|
| Ratio | 1.00x | 3.28x | 1.76x | 1.82x |
* Discourse
Using the same benchmark strategy as https://bugs.ruby-lang.org/issues/14490 with
this branch https://github.com/k0kubun/discourse/commits/benchmark2 forked from discourse
v1.8.11 to support running trunk.
1. Run ruby script/bench.rb to warm up profiling database
2. Run RUBYOPT='--jit-verbose=1 --jit-max-cache=10000' RAILS_ENV=profile bin/puma -e production
3. WAIT 5-15 or so minutes for all jitting to stop so we have no cross talk
4. Run ab -n 100 http://localhost:9292/
5. Wait for all new jitting to finish
6. Run ab -n 100 http://localhost:9292/
** Response time (ms)
Here is the response time milliseconds for each percentile.
Skipping 99%ile because it's the same as 100%ile in 100 calls.
| | trunk| trunk|single|objfcn|
| | | JIT|so JIT| JIT|
|:----|:-----|:-----|:-----|:-----|
| 50% | 38 | 45 | 41 | 43 |
| 66% | 39 | 50 | 44 | 44 |
| 75% | 47 | 51 | 46 | 45 |
| 80% | 49 | 52 | 47 | 47 |
| 90% | 50 | 63 | 50 | 52 |
| 95% | 60 | 79 | 52 | 55 |
| 98% | 91 | 114 | 91 | 91 |
|100% | 97 | 133 | 96 | 99 |
** Ratio (smaller is better)
Here is the response time increase ratio against no-JIT trunk's one.
| | trunk| trunk|single|objfcn|
| | | JIT|so JIT| JIT|
|:----|:-----|:-----|:-----|:-----|
| 50% | 1.00x| 1.18x| 1.08x| 1.13x|
| 66% | 1.00x| 1.28x| 1.13x| 1.13x|
| 75% | 1.00x| 1.09x| 0.98x| 0.96x|
| 80% | 1.00x| 1.06x| 0.96x| 0.96x|
| 90% | 1.00x| 1.26x| 1.00x| 1.04x|
| 95% | 1.00x| 1.32x| 0.87x| 0.92x|
| 98% | 1.00x| 1.25x| 1.00x| 1.00x|
|100% | 1.00x| 1.37x| 0.99x| 1.02x|
While 50 and 60 %ile are still worse than no-JIT trunk, 75, 80, 90, 95,
98 and 100% are not slower than that.
So now it's a little harder to say "MJIT slows down Rails applications".
Probably I can close [Bug #14490] now. Let's start improving it.
Close https://github.com/ruby/ruby/pull/1921
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64094 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: Fix the range check for trailing
Hangul jamo characters in Unicode normalization. Different from
leading or vowel jamos, where LBASE and VBASE are actual characters,
a value equal to TBASE expresses the absence of a trailing jamo.
This fix is technically correct, but there was no bug because
the regular expressions in lib/unicode_normalize/tables.rb
eliminate jamos equal to TBASE from normalization processing.
* test/test_unicode_normalize.rb: Add preventive test
test_no_trailing_jamo based on
d134809cd3
just for the case we ever get a regression.
This closes issue #14934, thanks to MaLin (Lin Ma) for reporting.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/test_unicode_normalize.rb: extract tests that do not depend
on NormalizationTest.txt data file from conditionally constructed
part of TestUnicodeNormalize class, to always run them even if
the data file isn't found.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/test_unicode_normalize.rb: search ucd directory
first if it exists. This change follows r61415.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64085 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
and lazily delete them on termination.
This will be needed to create a large so file later.
The large number of .o files will be probably compacted before the large so
file is created.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64077 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* ruby.c (external_str_new_cstr): strings come from the external
should be tainted. [ruby-dev:50596] [Bug #14941]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64071 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This adds a password_hash keyword argument to
WEBrick::HTTPAuth::Htpasswd#initialize. If set to :bcrypt, it
will create bcrypt hashes instead of crypt hashes, and will
raise an exception if the .htpasswd file uses crypt hashes.
If :bcrypt is used, then instead of calling
BasicAuth.make_passwd (which uses crypt),
WEBrick::HTTPAuth::Htpasswd#set_passwd will set the bcrypt
password directly. It isn't possible to change the
make_passwd API to accept the password hash format, as that
would break configurations who use Htpasswd#auth_type= to set
a custom auth_type.
This modifies WEBrick::HTTPAuth::BasicAuth to handle checking
both crypt and bcrypt hashes.
There are commented out requires for 'string/crypt', to handle
when String#crypt is deprecated and the undeprecated version is
moved to a gem.
There is also a commented out warning for the case when
the password_hash keyword is not specified and 'string/crypt'
cannot be required. I think the warning makes sense to nudge
users to using bcrypt.
I've updated the tests to test nil, :crypt, and :bcrypt values
for the password_hash keyword, skipping the bcrypt tests if the
bcrypt library cannot be required.
[ruby-core:88111] [Feature #14940]
From: Jeremy Evans <code@jeremyevans.net>
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64060 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
in a short term, and add retries to prevent random CI failures by it.
I remember this and will address it later for sure.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64032 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/ruby/test_io.rb (test_select_leak): increase timeout seconds
to pass this test on a high-load machine.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64019 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* object.c (rb_f_integer): prefer `base` optional argument over
keyword arguments. this issue should be resolved more generally
by separating keyword arguments from hashes in the future.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64015 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
with --jit.
`./ruby --jit -Itest/lib test/ruby/test_rubyoptions.rb -n "TestRubyOptions#test_verbose"`
has failed without -DMJIT_FORCE_ENABLE like:
1) Failure:
TestRubyOptions#test_verbose [/home/travis/build/k0kubun/mjit-test/test/ruby/test_rubyoptions.rb:118]:
<"ruby 2.6.0dev (2018-07-21 trunk 64010) +JIT [x86_64-linux]"> expected but was
<"ruby 2.6.0dev (2018-07-21 trunk 64010) [x86_64-linux]">.
but the test should pass even without -DMJIT_FORCE_ENABLE.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64011 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
When do_select is interrupted and raise happens from
RUBY_VM_CHECK_INTS_BLOCKING, the original FD sets we copied
do not get freed, leading to a memory leak. Wrap up all the
FD sets into a Ruby object to ensure the GC can release an
allocations made for rb_fdset_t.
This leak existed since Ruby 2.0.0 (r36430)
[Bug #14929]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64007 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Spurious wakeup is unavoidable with MJIT; and any real
code must be able to deal with spurious wakeup anyways.
[ruby-core:87882] https://bugs.ruby-lang.org/issues/14901
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64005 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* doc/*: Move entry to default gems category from standard library.
* lib/shell/*: Added `Shell::VERSION` and re-used it with @RELEASE_VERSION.
* test/shell/test_command_processor.rb: added missing require for test library.
* tool/sync_default_gems.rb: Support shell library. We need to ignore shellwords.rb when
syncing shell* files.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64004 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
by promoting jit to feature flag.
mjit.h: update comment about mjit_opts.on
test_rubyoptions.rb: add test for switching JIT enablement
"--jit" flag usage may be deprecated later, but not discussed yet.
[Feature #14878]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@63995 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
