If two platform specific variants have different dependencies, then
resolution may fallback to the non platform specific variant. However,
the platform specific variants that have the same dependencies as the
non specific one can still be kept.
Do a pass to complete those after resolution.
If a platform specific variant would not match the current Ruby, we would still be
considering it compatible with the initial resolution and adding its
platform to the lockfile, but we would later fail to materialize it for
installation due to not really being compatible.
Fix is to only add platforms for variants that are also compatible with
current Ruby and RubyGems versions.
75d1290843
Since we started locking the specific platform in the lockfile, that has
created an annoying situation for users that don't develop on Linux.
They will create a lockfile on their machines, locking their local
platform, for example, darwin. But then that lockfile won't work
automatically when deploying to Heroku for example, because the lockfile
is frozen and the Linux platform is not included.
There's the chance though that resolving against two platforms (Linux +
the local platform) won't succeed while resolving for just the current
platform will. So, instead, we check other platform specific variants
available for the resolution we initially found, and lock those
platforms and specs too if they satisfy the resolution.
This is only done when generating new lockfiles from scratch, existing
lockfiles should keep working as before, and it's only done for "ruby
platforms", i.e., not Java or Windows which have their own complexities,
and so are excluded.
With this change, we expect that MacOS users can bundle locally and
deploy to Heroku without needing to do anything special.
5f24f06bc5
While working on locking multiple platforms by default, I got an
infinite resolution loop in one of our resolver specs.
The culprit ended up being that when dealing with lockfile specs with
incomplete dependencies (spec appears in lockfile, but its dependencies
don't), those specs were not being properly expired and that tripped up
resolution.
The issue for some reason only manifests when dealing with multiple
lockfile platforms, that's why it only manifested when working on
locking multiple platforms by default.
4ca72913bb
Recent bugs fixed made me realize we were relying on state too much
here. We only need to keep incomplete specs to be able to expire them
and retry resolution without them locked. If we use a separate class, we
can do that more transparently and handle them just like we handle
"missing specs".
Following up on https://github.com/rubygems/rubygems/pull/6355, which
turned a crash into a nicer error message, this commit auto-heals the
corrupt lockfile instead.
In this particular case (a corrupt Gemfile.lock with missing
dependencies) the LazySpecification will not have accurate dependency
information, we have to materialize the SpecSet to determine there are
missing dependencies. We've already got a way to handle this, via
`SpecSet#incomplete_specs`, but it wasn't quite working for this case
because we'd get to `@incomplete_specs += lookup[name]` and
`lookup[name]` would be empty for the dependency.
With this commit we catch it a bit earlier, marking the parent spec
containing the missing dependency as incomplete.
486ecb8f20
This is a regression from a change intended to raise errors when user
puts a gem under an incorrect source in the Gemfile by mistake. To fix
the issue, we revert the change that caused it and implement it in a
different way that restores the resolver independency from real
specifications. Now it deals only with names and versions and does not
try to materialize anything into real specifications before resolving.
d2bf1b86eb
Do dependency filtering and materialization in one step. Before,
dependency filtering would not consider ruby metadata so it would
discard variants that end up not being materializable in the end.
0c0d40d417
Co-authored-by: Ian Ker-Seymer <ian.kerseymer@shopify.com>
I was looking at (yet another) flamegraph in speedscope, and used the
'left hand heavy' and was shocked to realize that 0.5s of the 1.7s
is spent in DepProxy#name. This method _only_ delegates the name to an
underlying spec, so it's not complex at all.
It seems to be of how often this line ends up calling it:
next if handled.any?{|d| d.name == dep.name && (match_current_platform || d.__platform == dep.__platform) } || dep.name == "bundler"
The `handled` array is built up as dependencies are handled, so this get
slower as more dependencies are installed.
This change changes how `handled` is track. Instead of just an array, I've
tried using a Hash, with the key being a dep's name, and the value being
a list of deps with that name. This means it's constant time to find
the dependencies with the same name.
I saw a drop from 1.7s to 1.0s against master, and from 0.95s to 0.24s
when used with https://github.com/rubygems/rubygems/pull/5533844dac30d4
On a different patch, it was noticed Ngam Pham that we are calling
`LazySpecification#hash` many times, and simply memoizing that led to a
very considerable performance improvement in his app.
I noticed though that we shouldn't be calling `LazySpecification#hash`
that many times, and I located the culprit at `SpecSet#for` where we
were deduplicating the partial aggregated result on every iteration. It
is enough to do it just once at the end.
This leads on a 12% speedup on Rails repository Gemfile vs the previous
8% I was getting from memoizing `LazySpecification#hash`. Also, after
this patch memoizing `LazySpecification#hash` has no effect in
performance anymore.
68d00a9edd
Co-authored-by: Ngan Pham <ngan@users.noreply.github.com>
