```
==> memprof.after.txt <==
Total allocated: 1.13 MB (2352 objects)
Total retained: 10.08 kB (78 objects)
==> memprof.before.txt <==
Total allocated: 46.27 MB (38439 objects)
Total retained: 9.94 kB (75 objects)
```
Yes, we were allocating 45MB of arrays in `dependencies_installed?`,
it was accidentally cubic.
13ab874388
Following up on https://github.com/rubygems/rubygems/pull/6355, which
turned a crash into a nicer error message, this commit auto-heals the
corrupt lockfile instead.
In this particular case (a corrupt Gemfile.lock with missing
dependencies) the LazySpecification will not have accurate dependency
information, we have to materialize the SpecSet to determine there are
missing dependencies. We've already got a way to handle this, via
`SpecSet#incomplete_specs`, but it wasn't quite working for this case
because we'd get to `@incomplete_specs += lookup[name]` and
`lookup[name]` would be empty for the dependency.
With this commit we catch it a bit earlier, marking the parent spec
containing the missing dependency as incomplete.
486ecb8f20
I did a bad thing (script that edits the Gemfile.lock directly) and
ended up with a Gemfile.lock that was completely missing some indirect
dependencies. While this is my fault and an error is reasonable, I
noticed that the error got progressively less friendly in recent
versions of bundler.
Something similar came up in https://github.com/rubygems/rubygems/issues/6210,
and this commit would have helped with that case as well
(although we've already handled this a different way with #6219).
Details:
---
Back on Bundler 2.2.23, a corrupt lockfile like this would cause a helpful error:
```
Unable to find a spec satisfying minitest (>= 5.1) in the set. Perhaps the lockfile is corrupted?
```
Bundler 2.3.26 gave a helpful warning:
```
Warning:
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time,
instead of installing 16 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle
install, and then removing the gems from your Gemfile.
The missing gems are:
* minitest depended upon by activesupport
```
But then continued on and crashed while trying to report the unmet
dependency:
```
--- ERROR REPORT TEMPLATE -------------------------------------------------------
NoMethodError: undefined method `full_name' for nil:NilClass
lib/bundler/installer/parallel_installer.rb:127:in `block (2 levels) in check_for_unmet_dependencies'
...
```
Bundler 2.4.0 and up crash as above when jobs=1, but crash
even harder when run in parallel:
```
--- ERROR REPORT TEMPLATE -------------------------------------------------------
fatal: No live threads left. Deadlock?
3 threads, 3 sleeps current:0x00007fa6b6704660 main thread:0x00007fa6b6704660
* #<Thread:0x000000010833b130 sleep_forever>
rb_thread_t:0x00007fa6b6704660 native:0x0000000108985600 int:0
* #<Thread:0x0000000108dea630@Parallel Installer Worker #0 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
rb_thread_t:0x00007fa6b67f67c0 native:0x0000700009a62000 int:0
* #<Thread:0x0000000108dea4a0@Parallel Installer Worker #1 tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:90 sleep_forever>
rb_thread_t:0x00007fa6b67f63c0 native:0x0000700009c65000 int:0
<internal:thread_sync>:18:in `pop'
tmp/1/gems/system/gems/bundler-2.5.0.dev/lib/bundler/worker.rb:42:in `deq'
...
```
Changes
---
This commit fixes the confusing thread deadlock crash by detecting if
dependencies are missing such that we'll never be able to enqueue. When
that happens we treat it as a failure so the install can finish.
That gets us back to the `NoMethodError`, which this commit fixes by
using a different warning in the case where no spec is found.
d73001a21d
`trying to manually editing` doesn't seem quite grammatically
correct. We could change it to `trying to manually edit` (is that a
split infinitive?), but I don't think `trying to` adds much here so
I've removed it instead so `editing` is the verb.
For the list of dependencies, the wording before this commit seemed to
reverse the dependency. "B, depended on A" sounds like B depends on A
(or did in the past but doesn't anymore?), but that's not correct. I
think there's a missing word: "B, depended on by A", but I find "B,
dependency of A" a bit nicer.
49a31257e3
We have a check for a corrupt lockfile right before installing. However,
the check accounted for locked specs not satisfying locked dependencies,
but not for locked specs missing for some locked dependencies.
Instead of fixing this check, I decided to remove it in favor of
automatically detecting the situation and re-resolve to automatically
fix the lockfile rather than printing a warning but leave the problem
there.
4a7a584252
The main thread may detect that installation has finished and thus
abort, while the thread responsible for installing the spec that failed
has not yet set the error message.
In this case, the install process will abort with a misterious "empty"
exception. I can be force the issue to reproduce by applying the
following patch:
```diff
diff --git a/bundler/lib/bundler/installer/parallel_installer.rb b/bundler/lib/bundler/installer/parallel_installer.rb
index 3dee9f4664..7827a11d11 100644
--- a/bundler/lib/bundler/installer/parallel_installer.rb
+++ b/bundler/lib/bundler/installer/parallel_installer.rb
@@ -166,6 +166,7 @@ module Bundler
spec_install.post_install_message = message unless message.nil?
else
spec_install.state = :failed
+ sleep 1
spec_install.error = "#{message}\n\n#{require_tree_for_spec(spec_install.spec)}"
end
Plugin.hook(Plugin::Events::GEM_AFTER_INSTALL, spec_install)
@@ -183,6 +184,7 @@ module Bundler
end
def finished_installing?
+ sleep 0.5
@specs.all? do |spec|
return true if spec.failed?
spec.installed?
diff --git a/bundler/lib/bundler/rubygems_gem_installer.rb b/bundler/lib/bundler/rubygems_gem_installer.rb
index 8ce33c3953..c585cd517b 100644
--- a/bundler/lib/bundler/rubygems_gem_installer.rb
+++ b/bundler/lib/bundler/rubygems_gem_installer.rb
@@ -42,6 +42,7 @@ module Bundler
return true unless source = @package.instance_variable_get(:@gem)
return true unless source.respond_to?(:with_read_io)
digest = source.with_read_io do |io|
+ raise BundlerError, "asdafss"
digest = SharedHelpers.digest(:SHA256).new
digest << io.read(16_384) until io.eof?
io.rewind
```
and running `bin/rspec spec/install/gems/compact_index_spec.rb:892` will
result in
```
Run options:
include {:locations=>{"./spec/install/gems/compact_index_spec.rb"=>[892]}}
exclude {:jruby=>true, :readline=>false, :permissions=>false, :no_color_tty=>false, :ruby_repo=>false, :bundler=>"!= 2", :git=>"!= 2.26.2", :rubygems=>"!= 3.2.0.pre1", :realworld=>true, :sudo=>true}
Randomized with seed 59277
F
Retried examples: 0
Failures:
1) compact index api checksum validation raises when the checksum is the wrong length
Failure/Error: expect(err).to include("The given checksum for rack-1.0.0 (\"checksum!\") is not a valid SHA256 hexdigest nor base64digest")
expected "" to include "The given checksum for rack-1.0.0 (\"checksum!\") is not a valid SHA256 hexdigest nor base64digest"
Commands:
$ /home/deivid/.rbenv/versions/2.7.1/bin/ruby -I/home/deivid/Code/rubygems/rubygems/bundler/spec -r/home/deivid/Code/rubygems/rubygems/bundler/spec/support/artifice/compact_index_wrong_gem_checksum.rb -r/home/deivid/Code/rubygems/rubygems/bundler/spec/support/hax.rb /home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/bin/bundle install --verbose --retry 0
Running `bundle install --retry 0 --verbose` with bundler 2.2.0.dev
Found changes from the lockfile, re-resolving dependencies because the list of sources changed, the dependencies in your gemfile changed, you added a new platform to your gemfile
HTTP GET http://localgemserver.test/versions
HTTP 200 OK http://localgemserver.test/versions
Fetching gem metadata from http://localgemserver.test/
Looking up gems ["rack"]
HTTP GET http://localgemserver.test/info/rack
HTTP 200 OK http://localgemserver.test/info/rack
Resolving dependencies...
Using bundler 2.2.0.dev
0: bundler (2.2.0.dev) from /home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/specifications/bundler-2.2.0.dev.gemspec
Fetching rack 1.0.0
Installing rack 1.0.0
Bundler::InstallError:
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer/parallel_installer.rb:199:in `handle_error'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer/parallel_installer.rb:102:in `call'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer/parallel_installer.rb:78:in `call'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer.rb:271:in `install_in_parallel'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer.rb:197:in `install'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer.rb:92:in `block in run'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/process_lock.rb:12:in `block in lock'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/process_lock.rb:9:in `open'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/process_lock.rb:9:in `lock'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer.rb:73:in `run'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/installer.rb:25:in `install'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/cli/install.rb:66:in `run'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/cli.rb:261:in `block in install'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/settings.rb:121:in `temporary'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/cli.rb:260:in `install'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/cli.rb:30:in `dispatch'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/cli.rb:24:in `start'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/exe/bundle:49:in `block in <top (required)>'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/gems/bundler-2.2.0.dev/exe/bundle:37:in `<top (required)>'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/bin/bundle:23:in `load'
/home/deivid/Code/rubygems/rubygems/bundler/tmp/1/gems/system/bin/bundle:23:in `<main>'
# $? => 5
# ./spec/install/gems/compact_index_spec.rb:892:in `block (3 levels) in <top (required)>'
# ./spec/spec_helper.rb:104:in `block (4 levels) in <top (required)>'
# ./spec/spec_helper.rb:104:in `block (3 levels) in <top (required)>'
# ./spec/support/helpers.rb:352:in `block in with_gem_path_as'
# ./spec/support/helpers.rb:366:in `without_env_side_effects'
# ./spec/support/helpers.rb:348:in `with_gem_path_as'
# ./spec/spec_helper.rb:101:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:73:in `block (2 levels) in <top (required)>'
# ./spec/support/rubygems_ext.rb:90:in `load'
# ./spec/support/rubygems_ext.rb:90:in `gem_load_and_activate'
# ./spec/support/rubygems_ext.rb:18:in `gem_load'
Finished in 3.01 seconds (files took 0.14209 seconds to load)
1 example, 1 failure
Failed examples:
rspec ./spec/install/gems/compact_index_spec.rb:886 # compact index api checksum validation raises when the checksum is the wrong length
Randomized with seed 59277
```
Without any mention to `BundlerError` and the original "asdafss" message.
Fix the issue by making sure the error message is set before the ":failed"
status.
83c8feb2c4
* bin/*, lib/bundler/*, lib/bundler.rb, spec/bundler, man/*:
Merge from latest stable branch of bundler/bundler repository and
added workaround patches. I will backport them into upstream.
* common.mk, defs/gmake.mk: Added `test-bundler` task for test suite
of bundler.
* tool/sync_default_gems.rb: Added sync task for bundler.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65509 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
I faced a big issue about Bundler with ruby core.
I have no time to resolve it issue before 2.5 final release.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@61416 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
rubygems 2.7.x depends bundler-1.15.x. This is preparation for
rubygems and bundler migration.
* lib/bundler.rb, lib/bundler/*: files of bundler-1.15.4
* spec/bundler/*: rspec examples of bundler-1.15.4. I applied patches.
* https://github.com/bundler/bundler/pull/6007
* Exclude not working examples on ruby repository.
* Fake ruby interpriter instead of installed ruby.
* Makefile.in: Added test task named `test-bundler`. This task is only
working macOS/linux yet. I'm going to support Windows environment later.
* tool/sync_default_gems.rb: Added sync task for bundler.
[Feature #12733][ruby-core:77172]
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@59779 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
