18 Commits (3e3d8f6764dcfac8f3bb68bd4e1820cf98bb7bb6)

Author	SHA1	Message	Date
Martin Atkins	958ea4f7d1	internal/providercache: Handle built-in providers ... Built-in providers are special providers that are distributed as part of Terraform CLI itself, rather than being installed separately. They always live in the terraform.io/builtin/... namespace so it's easier to see that they are special, and currently there is only one built-in provider named "terraform". Previous commits established the addressing scheme for built-in providers. This commit makes the installer aware of them to the extent that it knows not to try to install them the usual way and it's able to report an error if the user requests a built-in provider that doesn't exist or tries to impose a particular version constraint for a built-in provider. For the moment the tests for this are the ones in the "command" package because that's where the existing testing infrastructure for this functionality lives. A later commit should add some more focused unit tests here in the internal/providercache package, too.	6 years ago
Martin Atkins	f35ebe2d65	internal/providercache: Fix incorrect logic in Installer.SetGlobalCacheDir ... Due to some incomplete rework of this function in an earlier commit, the safety check for using the same directory as both the target and the cache was inverted and was raising an error _unless_ they matched, rather than _if_ they matched. This change is verified by the e2etest TestInitProviders_pluginCache, which is also updated to use the new-style cache directory layout as part of this commit.	6 years ago
Martin Atkins	ff55e1a1cd	internal/providercache: installFromLocalDir ... We previously skipped this one because it wasn't strictly necessary for replicating the old "terraform init" behavior, but we do need it to work so that things like the -plugin-dir option can behave correctly. Linking packages from other cache directories and installing from unpacked directories are fundamentally the same operation because a cache directory is really just a collection of unpacked packages, so here we refactor the LinkFromOtherCache functionality to actually be in installFromLocalDir, and LinkFromOtherCache becomes a wrapper for the installFromLocalDir function that just calculates the source and target directories automatically and invalidates the metaCache.	6 years ago
Martin Atkins	2ff4582be2	internal/providercache: Fix positions on selections file/dir ... On Unix-derived systems a directory must be marked as "executable" in order to be accessible, so our previous mode of 0660 here was unsufficient and would cause a failure if it happened to be the installer that was creating the plugins directory for the first time here. Now we'll make it executable and readable for all but only writable by the same user/group. For consistency, we also make the selections file itself readable by everyone. In both cases, the umask we are run with may further constrain these modes.	6 years ago
Martin Atkins	ae080481c0	internal/providercache: Installer records its selections in a file ... Just as with the old installer mechanism, our goal is that explicit provider installation is the only way that new provider versions can be selected. To achieve that, we conclude each call to EnsureProviderVersions by writing a selections lock file into the target directory. A later caller can then recall the selections from that file by calling SelectedPackages, which both ensures that it selects the same set of versions and also verifies that the checksums recorded by the installer still match. This new selections.json file has a different layout than our old plugins.json lock file. Not only does it use a different hashing algorithm than before, we also record explicitly which version of each provider was selected. In the old model, we'd repeat normal discovery when reloading the lock file and then fail with a confusing error message if discovery happened to select a different version, but now we'll be able to distinguish between a package that's gone missing since installation (which could previously have then selected a different available version) from a package that has been modified.	6 years ago
Martin Atkins	f6a7a4868b	internal/providercache: Hashing of contents of cached packages ... For the old-style provider cache directory model we hashed the individual executable file for each provider. That's no longer appropriate because we're giving each provider package a whole directory to itself where it can potentially have many files. This therefore introduces a new directory-oriented hashing algorithm, and it's just using the Go Modules directory hashing algorithm directly because that's already had its cross-platform quirks and other wrinkles addressed during the Go Modules release process, and is now used prolifically enough in Go codebases that breaking changes to the upstream algorithm would be very expensive to the Go ecosystem. This is also a bit of forward planning, anticipating that later we'll use hashes in a top-level lock file intended to be checked in to user version control, and then use those hashes also to verify packages _during_ installation, where we'd need to be able to hash unpacked zip files. The Go Modules hashing algorithm is already implemented to consistently hash both a zip file and an unpacked version of that zip file.	6 years ago
Martin Atkins	079b4cf7be	internal/providercache: Clear the metadata cache during package install ... This was previously happening during linking from another cache, but not when installing an entirely new provider.	6 years ago
Martin Atkins	4061cbed38	internal/getproviders: A new shared model for provider requirements ... We've been using the models from the "moduledeps" package to represent our provider dependencies everywhere since the idea of provider dependencies was introduced in Terraform 0.10, but that model is not convenient to use for any use-case other than the "terraform providers" command that needs individual-module-level detail. To make things easier for new codepaths working with the new-style provider installer, here we introduce a new model type getproviders.Requirements which is based on the type the new installer was already taking as its input. We have new methods in the states, configs, and earlyconfig packages to produce values of this type, and a helper to merge Requirements together so we can combine config-derived and state-derived requirements together during installation. The advantage of this new model over the moduledeps one is that all of recursive module walking is done up front and we produce a simple, flat structure that is more convenient for the main use-cases of selecting providers for installation and then finding providers in the local cache to use them for other operations. This new model is _not_ suitable for implementing "terraform providers" because it does not retain module-specific requirement details. Therefore we will likely keep using moduledeps for "terraform providers" for now, and then possibly at a later time consider specializing the moduledeps logic for only what "terraform providers" needs, because it seems to be the only use-case that needs to retain that level of detail.	6 years ago
Martin Atkins	537c1bedcf	internal/providercache: LinkFromOtherCache removes target, not source ... This was incorrectly removing the _source_ entry prior to creating the symlink, therefore ending up with a dangling symlink and no source file. This wasn't obvious before because the test case for LinkFromOtherCache was also incorrectly named and therefore wasn't running. Fixing the name of that test made this problem apparent. The TestLinkFromOtherCache test case now ends up seeing the final resolved directory rather than the symlink target, because of upstream changes to the internal/getproviders filesystem scanning logic to handle symlinks properly.	6 years ago
Martin Atkins	ad15459468	internal/{getproviders,providercache}: improved trace logging ... There's a lot going on in these functions that can be hard to follow from the outside, so we'll add some additional trace logging so that we can more easily understand why things are behaving the way they are.	6 years ago
Martin Atkins	391ca0c991	internal/providercache: add windows test fixture ... This was accidentally left out of an earlier commit due to our top-level .gitignore file containing *.exe as an ignore pattern.	6 years ago
Martin Atkins	807267d1b5	internal/providercache: Installation from HTTP URLs and local archives ... When a provider source produces an HTTP URL location we'll expect it to resolve to a zip file, which we'll first download to a temporary directory and then treat it like a local archive. When a provider source produces a local archive path we'll expect it to be a zip file and extract it into the target directory. This does not yet include an implementation of installing from an already-unpacked local directory. That will follow in a subsequent commit, likely following a similar principle as in Dir.LinkFromOtherCache.	6 years ago
Martin Atkins	754b7ebb65	command: Expose providercache package objects for use elsewhere ... These new functions allow command implementations to get hold of the providercache objects and installation source object derived from the current CLI configuration.	6 years ago
Martin Atkins	18dd0a396d	internal/providercache: First pass of the actual install process ... This is not tested yet, but it's a compilable strawman implementation of the necessary sequence of events to coordinate all of the moving parts of running a provider installation operation. This will inevitably see more iteration in later commits as we complete the surrounding parts and wire it up to be used by "terraform init". So far, it's just dead code not called by any other package.	6 years ago
Martin Atkins	03155daf98	internal/providercache: Start to stub Installer type ... The Installer type will encapsulate the logic for running an entire provider installation request: given a set of providers to install, it will determine a method to obtain each of them (or detect that they are already installed) and then take the necessary actions. So far it doesn't do anything, but this stubs out an interface by which the caller can request ongoing notifications during an installation operation.	6 years ago
Martin Atkins	e4d14324e0	internal/providercache: Stub of Dir.InstallPackage method ... This will eventually be responsible for actually retrieving a package from a source and then installing it into the cache directory, but for the moment it's just a stub to complete the proposed API, which I intend to test in a subsequent commit by writing the full "Installer" API that will encapsulate the full installation logic.	6 years ago
Martin Atkins	67ca067910	internal/providercache: Linking from one cache to another ... When a system-wide shared plugin cache is configured, we'll want to make use of entries already in the shared cache when populating a local (configuration-specific) cache. This new method LinkFromOtherCache encapsulates the work of placing a link from one cache to another. If possible it will create a symlink, therefore retaining a key advantage of configuring a shared plugin cache, but otherwise we'll do a deep copy of the package directory from one cache to the other. Our old provider installer would always skip trying to create symlinks on Windows because Go standard library support for os.Symlink on Windows was inconsistent in older versions. However, os.Symlink can now create symlinks using a new API introduced in a Windows 10 update and cleanly fail if symlink creation is impossible, so it's safe for us to just try to create the symlink and react if that produces an error, just as we used to do on non-Windows systems when possibly creating symlinks on filesystems that cannot support them.	6 years ago
Martin Atkins	d13001830b	providercache: A package to encapsulate management of provider cache dirs ... Historically our logic to handle discovering and installing providers has been spread across several different packages. This package is intended to become the home of all logic related to what is now called "provider cache directories", which means directories on local disk where Terraform caches providers in a form that is ready to run. That includes both logic related to interrogating items already in a cache (included in this commit) and logic related to inserting new items into the cache from upstream provider sources (to follow in later commits). These new codepaths are focused on providers and do not include other plugin types (provisioners and credentials helpers), because providers are the only plugin type that is represented by a heirarchical, decentralized namespace and the only plugin type that has an auto-installation protocol defined. The existing codepaths will remain to support the handling of the other plugin types that require manual installation and that use only a flat, locally-defined namespace.	6 years ago