78 Commits (f-providerinst-command-tests)

Author	SHA1	Message	Date
Martin Atkins	a79340e7fd	internal/providercache: Fix incorrect logic in Installer.SetGlobalCacheDir ... Due to some incomplete rework of this function in an earlier commit, the safety check for using the same directory as both the target and the cache was inverted and was raising an error _unless_ they matched, rather than _if_ they matched. This change is verified by the e2etest TestInitProviders_pluginCache, which is also updated to use the new-style cache directory layout as part of this commit.	6 years ago
Martin Atkins	ca9b5c515e	internal/providercache: installFromLocalDir ... We previously skipped this one because it wasn't strictly necessary for replicating the old "terraform init" behavior, but we do need it to work so that things like the -plugin-dir option can behave correctly. Linking packages from other cache directories and installing from unpacked directories are fundamentally the same operation because a cache directory is really just a collection of unpacked packages, so here we refactor the LinkFromOtherCache functionality to actually be in installFromLocalDir, and LinkFromOtherCache becomes a wrapper for the installFromLocalDir function that just calculates the source and target directories automatically and invalidates the metaCache.	6 years ago
Martin Atkins	abb486d8e5	internal/getproviders: Implement MultiSource ... We previously had only a stub implementation for a totally-empty MultiSource. Here we have an initial implementation of the full functionality, which we'll need to support "terraform init -plugin-dir=..." in a subsequent commit.	6 years ago
Martin Atkins	5cec09977f	internal/providercache: Fix positions on selections file/dir ... On Unix-derived systems a directory must be marked as "executable" in order to be accessible, so our previous mode of 0660 here was unsufficient and would cause a failure if it happened to be the installer that was creating the plugins directory for the first time here. Now we'll make it executable and readable for all but only writable by the same user/group. For consistency, we also make the selections file itself readable by everyone. In both cases, the umask we are run with may further constrain these modes.	6 years ago
Martin Atkins	bc0f85e85e	internal/getproviders: MockSource and mock package metadata ... These are some helpers to support unit testing in other packages, allowing callers to exercise provider installation mechanisms without hitting any real upstream source or having to prepare local package directories. MockSource is a Source implementation that just scans over a provided static list of packages and returns whatever matches. FakePackageMeta is a shorthand for concisely constructing a realistic-looking but uninstallable PackageMeta, probably for use with MockSource. FakeInstallablePackageMeta is similar to FakePackageMeta but also goes to the trouble of creating a real temporary archive on local disk so that the resulting package meta is pointing to something real on disk. This makes the result more useful to the caller, but in return they get the responsibility to clean up the temporary file once the test is over. Nothing is using these yet.	6 years ago
Martin Atkins	3cd48dec16	internal/providercache: Installer records its selections in a file ... Just as with the old installer mechanism, our goal is that explicit provider installation is the only way that new provider versions can be selected. To achieve that, we conclude each call to EnsureProviderVersions by writing a selections lock file into the target directory. A later caller can then recall the selections from that file by calling SelectedPackages, which both ensures that it selects the same set of versions and also verifies that the checksums recorded by the installer still match. This new selections.json file has a different layout than our old plugins.json lock file. Not only does it use a different hashing algorithm than before, we also record explicitly which version of each provider was selected. In the old model, we'd repeat normal discovery when reloading the lock file and then fail with a confusing error message if discovery happened to select a different version, but now we'll be able to distinguish between a package that's gone missing since installation (which could previously have then selected a different available version) from a package that has been modified.	6 years ago
Martin Atkins	7891454d10	internal/providercache: Hashing of contents of cached packages ... For the old-style provider cache directory model we hashed the individual executable file for each provider. That's no longer appropriate because we're giving each provider package a whole directory to itself where it can potentially have many files. This therefore introduces a new directory-oriented hashing algorithm, and it's just using the Go Modules directory hashing algorithm directly because that's already had its cross-platform quirks and other wrinkles addressed during the Go Modules release process, and is now used prolifically enough in Go codebases that breaking changes to the upstream algorithm would be very expensive to the Go ecosystem. This is also a bit of forward planning, anticipating that later we'll use hashes in a top-level lock file intended to be checked in to user version control, and then use those hashes also to verify packages _during_ installation, where we'd need to be able to hash unpacked zip files. The Go Modules hashing algorithm is already implemented to consistently hash both a zip file and an unpacked version of that zip file.	6 years ago
Martin Atkins	039d2c28a2	internal/providercache: Clear the metadata cache during package install ... This was previously happening during linking from another cache, but not when installing an entirely new provider.	6 years ago
Martin Atkins	4061cbed38	internal/getproviders: A new shared model for provider requirements ... We've been using the models from the "moduledeps" package to represent our provider dependencies everywhere since the idea of provider dependencies was introduced in Terraform 0.10, but that model is not convenient to use for any use-case other than the "terraform providers" command that needs individual-module-level detail. To make things easier for new codepaths working with the new-style provider installer, here we introduce a new model type getproviders.Requirements which is based on the type the new installer was already taking as its input. We have new methods in the states, configs, and earlyconfig packages to produce values of this type, and a helper to merge Requirements together so we can combine config-derived and state-derived requirements together during installation. The advantage of this new model over the moduledeps one is that all of recursive module walking is done up front and we produce a simple, flat structure that is more convenient for the main use-cases of selecting providers for installation and then finding providers in the local cache to use them for other operations. This new model is _not_ suitable for implementing "terraform providers" because it does not retain module-specific requirement details. Therefore we will likely keep using moduledeps for "terraform providers" for now, and then possibly at a later time consider specializing the moduledeps logic for only what "terraform providers" needs, because it seems to be the only use-case that needs to retain that level of detail.	6 years ago
Martin Atkins	537c1bedcf	internal/providercache: LinkFromOtherCache removes target, not source ... This was incorrectly removing the _source_ entry prior to creating the symlink, therefore ending up with a dangling symlink and no source file. This wasn't obvious before because the test case for LinkFromOtherCache was also incorrectly named and therefore wasn't running. Fixing the name of that test made this problem apparent. The TestLinkFromOtherCache test case now ends up seeing the final resolved directory rather than the symlink target, because of upstream changes to the internal/getproviders filesystem scanning logic to handle symlinks properly.	6 years ago
Martin Atkins	eb25fe8b24	internal/getproviders: SearchLocalDirectory can handle symlinks ... Previously this was failing to treat symlinks to directories as unpacked layout, because our file info was only an Lstat result, not a full Stat. Now we'll resolve the symlink first, allowing us to handle a symlink to a directory. That's important because our internal/providercache behavior is to symlink from one cache to another where possible.	6 years ago
Martin Atkins	ad15459468	internal/{getproviders,providercache}: improved trace logging ... There's a lot going on in these functions that can be hard to follow from the outside, so we'll add some additional trace logging so that we can more easily understand why things are behaving the way they are.	6 years ago
Martin Atkins	391ca0c991	internal/providercache: add windows test fixture ... This was accidentally left out of an earlier commit due to our top-level .gitignore file containing *.exe as an ignore pattern.	6 years ago
Martin Atkins	807267d1b5	internal/providercache: Installation from HTTP URLs and local archives ... When a provider source produces an HTTP URL location we'll expect it to resolve to a zip file, which we'll first download to a temporary directory and then treat it like a local archive. When a provider source produces a local archive path we'll expect it to be a zip file and extract it into the target directory. This does not yet include an implementation of installing from an already-unpacked local directory. That will follow in a subsequent commit, likely following a similar principle as in Dir.LinkFromOtherCache.	6 years ago
Martin Atkins	754b7ebb65	command: Expose providercache package objects for use elsewhere ... These new functions allow command implementations to get hold of the providercache objects and installation source object derived from the current CLI configuration.	6 years ago
Martin Atkins	e4d7a71d91	internal/getproviders: Exit early if MultiSource has no sources ... The MultiSource isn't actually properly implemented yet, but this is a minimal implementation just for the case where there are no underlying sources at all, because we use an empty MultiSource as a placeholder when a test in the "command" package fails to explicitly populate a ProviderSource.	6 years ago
Martin Atkins	18dd0a396d	internal/providercache: First pass of the actual install process ... This is not tested yet, but it's a compilable strawman implementation of the necessary sequence of events to coordinate all of the moving parts of running a provider installation operation. This will inevitably see more iteration in later commits as we complete the surrounding parts and wire it up to be used by "terraform init". So far, it's just dead code not called by any other package.	6 years ago
Martin Atkins	03155daf98	internal/providercache: Start to stub Installer type ... The Installer type will encapsulate the logic for running an entire provider installation request: given a set of providers to install, it will determine a method to obtain each of them (or detect that they are already installed) and then take the necessary actions. So far it doesn't do anything, but this stubs out an interface by which the caller can request ongoing notifications during an installation operation.	6 years ago
Martin Atkins	e4d14324e0	internal/providercache: Stub of Dir.InstallPackage method ... This will eventually be responsible for actually retrieving a package from a source and then installing it into the cache directory, but for the moment it's just a stub to complete the proposed API, which I intend to test in a subsequent commit by writing the full "Installer" API that will encapsulate the full installation logic.	6 years ago
Martin Atkins	67ca067910	internal/providercache: Linking from one cache to another ... When a system-wide shared plugin cache is configured, we'll want to make use of entries already in the shared cache when populating a local (configuration-specific) cache. This new method LinkFromOtherCache encapsulates the work of placing a link from one cache to another. If possible it will create a symlink, therefore retaining a key advantage of configuring a shared plugin cache, but otherwise we'll do a deep copy of the package directory from one cache to the other. Our old provider installer would always skip trying to create symlinks on Windows because Go standard library support for os.Symlink on Windows was inconsistent in older versions. However, os.Symlink can now create symlinks using a new API introduced in a Windows 10 update and cleanly fail if symlink creation is impossible, so it's safe for us to just try to create the symlink and react if that produces an error, just as we used to do on non-Windows systems when possibly creating symlinks on filesystems that cannot support them.	6 years ago
Martin Atkins	514184cc9d	internal/getproviders: Functions to determine installation directories ... The existing functionality in this package deals with finding packages that are either available for installation or already installed. In order to support installation we also need to determine the location where a package should be installed. This lives in the getproviders package because that way all of the logic related to the filesystem layout for local provider directories lives together here where they can be maintained together more easily in future.	6 years ago
Martin Atkins	072c6d9aed	internal/copydir: Factor out our recursive directory copy for reuse ... We've previously been copying this function around so it could remain unexported while being used in various packages. However, it's a non-trivial function with lots of specific assumptions built into it, so here we'll put it somewhere that other packages can depend on it _and_ document the assumptions it seems to be making for future reference. As a bonus, this now uses os.SameFile to detect when two paths point to the same physical file, instead of the slightly buggy local implementation we had before which only worked on Unix systems and did not correctly handle when the paths were on different physical devices. The copy of the function I extracted here is the one from internal/initwd, so this commit also includes the removal of that unexported version and updating the callers in that package to use at at this new location.	6 years ago
Martin Atkins	d13001830b	providercache: A package to encapsulate management of provider cache dirs ... Historically our logic to handle discovering and installing providers has been spread across several different packages. This package is intended to become the home of all logic related to what is now called "provider cache directories", which means directories on local disk where Terraform caches providers in a form that is ready to run. That includes both logic related to interrogating items already in a cache (included in this commit) and logic related to inserting new items into the cache from upstream provider sources (to follow in later commits). These new codepaths are focused on providers and do not include other plugin types (provisioners and credentials helpers), because providers are the only plugin type that is represented by a heirarchical, decentralized namespace and the only plugin type that has an auto-installation protocol defined. The existing codepaths will remain to support the handling of the other plugin types that require manual installation and that use only a flat, locally-defined namespace.	6 years ago
Martin Atkins	283b4d4cad	internal/getproviders: Make local directory search a public API ... Previously this was available by instantiating a throwaway FilesystemMirrorSource, but that's pretty counter-intuitive for callers that just want to do a one-off scan without retaining any ongoing state. Now we expose SearchLocalDirectory as an exported function, and the FilesystemMirrorSource then uses it as part of its implementation too. Callers that just want to know what's available in a directory can call SearchLocalDirectory directly.	6 years ago
Paddy	e6592dc710	Add support for provider metadata to modules. (#22583) ... Implement a new provider_meta block in the terraform block of modules, allowing provider-keyed metadata to be communicated from HCL to provider binaries. Bundled in this change for minimal protocol version bumping is the addition of markdown support for attribute descriptions and the ability to indicate when an attribute is deprecated, so this information can be shown in the schema dump. Co-authored-by: Paul Tyng <paul@paultyng.net>	6 years ago
Kim Ngo	8f5159ad54	Merge pull request #24260 from findkim/registry-retry ... registry: retryable discovery requests	6 years ago
Martin Atkins	43c1ec69e2	build: Use Go 1.14 ... This implies some notable changes that will have a visible impact to end-users of official Terraform releases: - Terraform is no longer compatible with MacOS 10.10 Yosemite, and requires at least 10.11 El Capitan. (Relatedly, Go 1.14 is planned to be the last release to support El Capitan, so while that remains supported for now, it's notable that Terraform 0.13 is likely to be the last major release of Terraform supporting it, with 0.14 likely to further require MacOS 10.12 Sierra.) - Terraform is no longer compatible with FreeBSD 10.x, which has reached end-of-life. Terraform now requires FreeBSD 11.2 or later. - Terraform now supports TLS 1.3 when it makes connections to remote services such as backends and module registries. Although TLS 1.3 is backward-compatible in principle, some legacy systems reportedly work incorrectly when attempting to negotiate it. (This change does not affect outgoing requests made by provider plugins, though they will see a similar change in behavior once built with Go 1.13 or later.) - Ed25519 certificates are now supported for TLS 1.2 and 1.3 connections. - On UNIX systems where "use-vc" is set in resolv.conf, TCP will now be used for DNS resolution. This is unlikely to cause issues in practice because a system set up in this way can presumably already reach its nameservers over TCP (or else other applications would misbehave), but could potentially lead to lookup failures in unusual situations where a system only runs Terraform, has historically had "use-vc" in its configuration, but yet is blocked from reaching its configured nameservers over TCP. - Some parts of Terraform now support Unicode 12.0 when working with strings. However, notably the Terraform Language itself continues to use the text segmentation tables from Unicode 9.0, which means it lacks up-to-date support for recognizing modern emoji combining forms as single characters. (We may wish to upgrade the text segmentation tables to Unicode 12.0 tables in a later commit, to restore consistency.) This also includes some changes to the contents of "vendor", and particularly to the format of vendor/modules.txt, per the changes to vendoring in the Go 1.14 toolchain. This new syntax is activated by the specification of "go 1.14" in the go.mod file. Finally, the exact format of error messages from the net/http library has changed since Go 1.12, and so a couple of our tests needed updates to their expected error messages to match that.	6 years ago
Martin Atkins	8eff19e48f	internal/getproviders: Initial implementation of FilesystemMirrorSource ... This is a basic implementation of FilesystemMirrorSource for now aimed only at the specific use-case of scanning the cache of provider plugins Terraform will keep under the ".terraform" directory, as part of our interim provider installer implementation for Terraform 0.13. The full functionality of this will grow out in later work when we implement explicit local filesystem mirrors, but for now the goal is to use this just to inspect the work done by the automatic installer once we switch it to the new provider-FQN-aware directory structure. The various FIXME comments in this are justified by the limited intended scope of this initial implementation, and they should be resolved by later work to use FilesystemMirrorSource explicitly for user-specified provider package mirrors.	6 years ago
Martin Atkins	c073db09ea	internal/getproviders: Sorting and filtering for lists of PackageMeta ... These are utility functions to ease processing of lists of PackageMeta elsewhere, once we have functionality that works with multiple packages at once. The local filesystem mirror source will be the first example of this, so these methods are motivated mainly by its needs.	6 years ago
Martin Atkins	d82a36b6f5	internal/getproviders: ParsePlatform method ... This is just to have a centralized set of logic for converting from a platform string (like "linux_amd64") to a Platform object, so we can do normalization and validation consistently.	6 years ago
Martin Atkins	3078d21c5f	internal/getproviders: Include Provider and Version in PackageMeta ... Although we tend to return these in contexts where at least one of these values is implied, being explicit means that PackageMeta values are self-contained and less reliant on such external context.	6 years ago
Pam Selle	c249943360	Module Expansion: Part 2 (#24154) ... * WIP: dynamic expand * WIP: add variable and local support * WIP: outputs * WIP: Add referencer * String representation, fixing tests it impacts * Fixes TestContext2Apply_outputOrphanModule * Fix TestContext2Apply_plannedDestroyInterpolatedCount * Update DestroyOutputTransformer and associated types to reflect PlannableOutputs * Remove comment about locals * Remove module count enablement * Removes allowing count for modules, and reverts the test, while adding a Skip()'d test that works when you re-enable the config * update TargetDownstream signature to match master * remove unnecessary method Co-authored-by: James Bardin <j.bardin@gmail.com>	6 years ago
Martin Atkins	86f0b5191c	addrs: Stronger validation and normalization of provider namespace/type ... The provider FQN is becoming our primary identifier for a provider, so it's important that we are clear about the equality rules for these addresses and what characters are valid within them. We previously had a basic regex permitting ASCII letters and digits for validation and no normalization at all. We need to do at least case folding and UTF-8 normalization because these names will appear in file and directory names in case-insensitive filesystems and in repository names such as on GitHub. Since we're already using DNS-style normalization and validation rules for the hostname part, rather than defining an entirely new set of rules here we'll just treat the provider namespace and type as if they were single labels in a DNS name. Aside from some internal consistency, that also works out nicely because systems like GitHub use organization and repository names as part of hostnames (e.g. with GitHub Pages) and so tend to apply comparable constraints themselves. This introduces the possibility of names containing letters from alphabets other than the latin alphabet, and for latin letters with diacritics. That's consistent with our introduction of similar support for identifiers in the language in Terraform 0.12, and is intended to be more friendly to Terraform users throughout the world that might prefer to name their products using a different alphabet. This is also a further justification for using the DNS normalization rules: modern companies tend to choose product names that make good domain names, and now such names will be usable as Terraform provider names too.	6 years ago
findkim	d0e13320d5	registry: configurable retry client	6 years ago
Kristin Laemmert	7eed30595a	moduledeps: replace ProviderInstance with addrs.Provider (#24017) ... * addrs: add ParseProviderSourceString function to parse fqns from tfconfig-inspect * moduledeps: use addrs.Provider instead of ProviderInstance	6 years ago
Pam Selle	9f374eb1fc	Merge pull request #23865 from loslocostacos/fix-win-tfe-pathing ... Fix compatibility issue between Windows CLI and remote plan and apply	6 years ago
Martin Atkins	4d7122a0dd	internal/getproviders: LookupLegacyProvider ... This is a temporary helper so that we can potentially ship the new provider installer without making a breaking change by relying on the old default namespace lookup API on the default registry to find a proper FQN for a legacy provider provider address during installation. If it's given a non-legacy provider address then it just returns the given address verbatim, so any codepath using it will also correctly handle explicit full provider addresses. This also means it will automatically self-disable once we stop using addrs.NewLegacyProvider in the config loader, because there will therefore no longer be any legacy provider addresses in the config to resolve. (They'll be "default" provider addresses instead, assumed to be under registry.terraform.io/hashicorp/* ) It's not decided yet whether we will actually introduce the new provider in a minor release, but even if we don't this API function will likely be useful for a hypothetical automatic upgrade tool to introduce explicit full provider addresses into existing modules that currently rely on the equivalent to this lookup in the current provider installer. This is dead code for now, but my intent is that it would either be called as part of new provider installation to produce an address suitable to pass to Source.AvailableVersions, or it would be called from the aforementioned hypothetical upgrade tool. Whatever happens, these functions can be removed no later than one whole major release after the new provider installer is introduced, when everyone's had the opportunity to update their legacy unqualified addresses.	6 years ago
Steve Burns	26305a83a3	Remove unecessary conditions for module manifest read and write functions	6 years ago
Steve Burns	43b855743b	Uniformity for module manifest dir read/write	6 years ago
Steven Burns	98e612ee50	Add condition for conversion of module path in manifest	6 years ago
Steve Burns	8bc0ef8fdc	Attempt to change the output of module manifest	6 years ago
Martin Atkins	2aac8cf812	internal/getproviders: Distinguished packed vs. unpacked local packages ... Our local filesystem mirror mechanism will allow provider packages to be given either in packed form as an archive directly downloaded to disk or in an unpacked form where the archive is extracted. Distinguishing these two cases in the concrete Location types will allow callers to reliably select the mode chosen by the selected installation source and handle it appropriately, rather than resorting to out-of-band heuristics like checking whether the object is a directory or a file.	6 years ago
Martin Atkins	92a58ec438	internal/getproviders: Stub out the two mirror sources ... In a future commit, these implementations of Source will allow finding and retrieving provider packages via local mirrors, both in the local filesystem and over the network using an HTTP-based protocol.	6 years ago
Martin Atkins	773e38b49c	internal/getproviders: Stub of MultiSource ... This is an API stub for a component that will be added in a future commit to support considering a number of different installation sources for each provider. These will eventually be configurable in the CLI configuration, allowing users to e.g. mirror certain providers within their own infrastructure while still being able to go upstream for those that aren't mirrored, or permit locally-mirrored providers only, etc.	6 years ago
Martin Atkins	d695956061	internal/getproviders: MemoizeSource for local caching of source responses ... Some sources make network requests that are likely to be slow, so this wrapper type can cache previous responses for its lifetime in order to speed up repeated requests for the same information.	6 years ago
Martin Atkins	a77bc59c44	internal/getproviders: Package URL should always be absolute ... Registries backed by static files are likely to use relative paths to their archives for simplicity's sake, but we'll normalize them to be absolute before returning because the caller wouldn't otherwise know what to resolve the URLs relative to.	6 years ago
Martin Atkins	c8f7223adb	internal/getproviders: Source interface for generalization ... We intend to support installation both directly from origin registries and from mirrors in the local filesystem or over the network. This Source interface will serve as our abstraction over those three options, allowing calling code to treat them all the same.	6 years ago
Martin Atkins	c76260e957	internal/getproviders: Query a provider registry ... Our existing provider installer was originally built to work with releases.hashicorp.com and later retrofitted to talk to the official Terraform Registry. It also assumes a flat namespace of providers. We're starting a new one here, copying and adapting code from the old one as necessary, so that we can build out this new API while retaining all of the existing functionality and then cut over to this new implementation in a later step. Here we're creating a foundational component for the new installer, which is a mechanism to query for the available versions and download locations of a particular provider. Subsequent commits in this package will introduce other Source implementations for installing from network and filesystem mirrors.	6 years ago
Kristin Laemmert	18dd1bb4d6	Mildwonkey/tfconfig upgrade (#23670) ... * deps: bump terraform-config-inspect library * configs: parse `version` in new required_providers block With the latest version of `terraform-config-inspect`, the required_providers attribute can now be a string or an object with attributes "source" and "version". This change allows parsing the version constraint from the new object while ignoring any given source attribute.	6 years ago
Kristin Laemmert	4045a6e5d0	initwd: cache registry responses for module versions and download URL (#23727) ... * initwd: cache registry responses for module versions and download URL Closes #23544	6 years ago