60 Commits (6dcf00aefc56b48bae54822a6e62eacffd2bc3ea)

Author	SHA1	Message	Date
Martin Atkins	23395a1022	providercache: Discard lock entries for unused providers ... Previously we would only ever add new lock entries or update existing ones. However, it's possible that over time a module may _cease_ using a particular provider, at which point we ought to remove it from the lock file so that operations won't fail when seeing that the provider cache directory is inconsistent with the lock file. Now the provider installer (EnsureProviderVersions) will remove any lock file entries that relate to providers not included in the given requirements, which therefore makes the resulting lock file properly match the set of packages the installer wrote into the cache. This does potentially mean that someone could inadvertently defeat the lock by removing a provider dependency, running "terraform init", then undoing that removal, and finally running "terraform init" again. However, that seems relatively unlikely compared to the likelihood of removing a provider and keeping it removed, and in the event it _did_ happen the changes to the lock entry for that provider would be visible in the diff of the provider lock file as usual, and so could be noticed in code review just as for any other change to dependencies.	4 years ago
Kristin Laemmert	5a48530f47	tools: remove terraform-bundle. (#28876) ... * tools: remove terraform-bundle. terraform-bundle is no longer supported in the main branch of terraform. Users can build terraform-bundle from terraform tagged v0.15 and older. * add a README pointing users to the v0.15 branch	5 years ago
Martin Atkins	e5f52e56f8	addrs: Rename DefaultRegistryHost to DefaultProviderRegistryHost ... As the comment notes, this hostname is the default for provide source addresses. We'll shortly be adding some address types to represent module source addresses too, and so we'll also have DefaultModuleRegistryHost for that situation. (They'll actually both contain the the same hostname, but that's a coincidence rather than a requirement.)	5 years ago
Martin Atkins	b9a93a0fe7	Move addrs/ to internal/addrs/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	4c254cc2be	Move httpclient/ to internal/httpclient/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	7f78342953	command: Experimental "terraform test" command ... This is just a prototype to gather some feedback in our ongoing research on integration testing of Terraform modules. The hope is that by having a command integrated into Terraform itself it'll be easier for interested module authors to give it a try, and also easier for us to iterate quickly based on feedback without having to coordinate across multiple codebases. Everything about this is subject to change even in future patch releases. Since it's a CLI command rather than a configuration language feature it's not using the language experiments mechanism, but generates a warning similar to the one language experiments generate in order to be clear that backward compatibility is not guaranteed.	5 years ago
Pam Selle	aa24bfec47	Emit ProviderAlreadyInstalled when provider installed ... Emit the ProviderAlreadyInstalled event when we successfully verify that we've already installed this provider and are skipping installation	5 years ago
Pam Selle	aedca597dd	Reuse installed target dir providers in init ... In init, we can check to see if the target dir already has the provider we are seeking and skip further querying/installing of that provider.	5 years ago
Martin Atkins	4b3e237668	command/init: Hint about providers in other namespaces ... If a user forgets to specify the source address for a provider, Terraform will assume they meant a provider in the registry.terraform.io/hashicorp/ namespace. If that ultimately doesn't exist, we'll now try to see if there's some other provider source address recorded in the registry's legacy provider lookup table, and suggest it if so. The error message here is a terse one addressed primarily to folks who are already somewhat familiar with provider source addresses and how to specify them. Terraform v0.13 had a more elaborate version of this error message which directed the user to try the v0.13 automatic upgrade tool, but we no longer have that available in v0.14 and later so the user must make the fix themselves.	5 years ago
James Bardin	6529659455	internal/providercache: staticcheck	5 years ago
James Bardin	276dfe634f	internal/providercache: staticcheck	5 years ago
James Bardin	7eb491719f	normalize temp dir paths in tests ... The temporary directory on some systems (most notably MacOS) contains symlinks, which would not be recorded by the installer. In order to make these paths comparable in the tests we need to eval the symlinks in the paths before giving them to the installer.	5 years ago
Alisdair McDiarmid	90c4f11b61	go get github.com/hashicorp/go-getter@v1.5.0 ... Includes fix for breaking upstream API change.	5 years ago
Martin Atkins	e6e0b6ee46	providercache: verify locked hashes for local package dirs ... Previously we were only verifying locked hashes for local archive zip files, but if we have non-ziphash hashes available then we can and should also verify that a local directory matches at least one of them. This does mean that folks using filesystem mirrors but yet also running Terraform across multiple platforms will need to take some extra care to ensure the hashes pass on all relevant platforms, which could mean using "terraform providers lock" to pre-seed their lock files with hashes across all platforms, or could mean using the "packed" directory layout for the filesystem mirror so that Terraform will end up in the install-from-archive codepath instead of this install-from-directory codepath, and can thus verify ziphash too. (There's no additional documentation about the above here because there's already general information about this in the lock file documentation due to some similar -- though not identical -- situations with network mirrors.)	5 years ago
Martin Atkins	24d32e9ca2	providercache: More exhaustive testing of the main installer ... We previously had some tests for some happy paths and a few specific failures into an empty directory with no existing locks, but we didn't have tests for the installer respecting existing lock file entries. This is a start on a more exhaustive set of tests for the installer, aiming to visit as many of the possible codepaths as we can reasonably test using this mocking strategy. (Some other codepaths require different underlying source implementations, etc, so we'll have to visit those in other tests separately.)	5 years ago
Martin Atkins	2611e08430	command/init: Mention using the lock file for provider selection ... This probably isn't the best UI we could do here, but it's a placeholder for now just to avoid making it seem like we're ignoring the lock file and checking for new versions anyway.	5 years ago
Martin Atkins	b3f5c7f1e6	command/init: Read, respect, and update provider dependency locks ... This changes the approach used by the provider installer to remember between runs which selections it has previously made, using the lock file format implemented in internal/depsfile. This means that version constraints in the configuration are considered only for providers we've not seen before or when -upgrade mode is active.	5 years ago
Kristin Laemmert	04be220f5f	deprecate helper/copy ... helper/copy CopyDir was used heavily in tests. It differes from internal/copydir in a few ways, the main one being that it creates the dst directory while the internal version expected the dst to exist (there are other differences, which is why I did not just switch tests to using internal's CopyDir). I moved the CopyDir func from helper/copy into command_test.go; I could also have moved it into internal/copy and named it something like CreateDirAndCopy so if that seems like a better option please let me know. helper/copy/CopyFile was used in a couple of spots so I moved it into internal, at which point I thought it made more sense to rename the package copy (instead of copydir). There's also a `go mod tidy` included.	5 years ago
Martin Atkins	0b734a2803	command: Make provider installation interruptible ... In earlier commits we started to make the installation codepath context-aware so that it could be canceled in the event of a SIGINT, but we didn't complete wiring that through the API of the getproviders package. Here we make the getproviders.Source interface methods, along with some other functions that can make network requests, take a context.Context argument and act appropriately if that context is cancelled. The main providercache.Installer.EnsureProviderVersions method now also has some context-awareness so that it can abort its work early if its context reports any sort of error. That avoids waiting for the process to wind through all of the remaining iterations of the various loops, logging each request failure separately, and instead returns just a single aggregate "canceled" error. We can then set things up in the "terraform init" and "terraform providers mirror" commands so that the context will be cancelled if we get an interrupt signal, allowing provider installation to abort early while still atomically completing any local-side effects that may have started.	6 years ago
Martin Atkins	6694cfaa0e	getproviders: Add a real type Hash for package hashes ... The logic for what constitutes a valid hash and how different hash schemes are represented was starting to get sprawled over many different files and packages. Consistently with other cases where we've used named types to gather the definition of a particular string into a single place and have the Go compiler help us use it properly, this introduces both getproviders.Hash representing a hash value and getproviders.HashScheme representing the idea of a particular hash scheme. Most of this changeset is updating existing uses of primitive strings to uses of getproviders.Hash. The new type definitions are in internal/getproviders/hash.go.	6 years ago
Alisdair McDiarmid	5587509bcf	internal: Fix providercache test failures on macOS ... For reasons that are unclear, these two tests just started failing on macOS very recently. The failure looked like: PackageDir: strings.Join({ "/", + "private/", "var/folders/3h/foobar/T/terraform-test-p", "rovidercache655312854/registry.terraform.io/hashicorp/null/2.0.0", "/windows_amd64", }, Speculating that the macOS temporary directory moved into the /private directory, I added a couple of EvalSymlinks calls and the tests pass again. No other unit tests appear to be affected by this at the moment.	6 years ago
Alisdair McDiarmid	915f53af23	internal: Clean up package install temp file ... The installFromHTTPURL function downloads a package to a temporary file, then delegates to installFromLocalArchive to install it. We were previously not deleting the temporary file afterwards. This commit fixes that.	6 years ago
Alisdair McDiarmid	440543f427	internal/providercache: Fix bug when symlink fails ... When installing a provider which is already cached, we attempt to create a symlink from the install directory targeting the cache. If symlinking fails due to missing OS/filesystem support, we instead want to copy the cached provider. The fallback code to do this would always fail, due to a missing target directory. This commit fixes that. I was unable to find a way to add automated tests around this, but I have manually verified the fix on Windows 8.1.	6 years ago
Alisdair McDiarmid	3b1347ac1a	providercache: Validate provider executable file ... At the end of the EnsureProviderVersions process, we generate a lockfile of the selected and installed provider versions. This includes a hash of the unpacked provider directory. When calculating this hash and generating the lockfile, we now also verify that the provider directory contains a valid executable file. If not, we return an error for this provider and trigger the installer's HashPackageFailure event. Note that this event is not yet processed by terraform init; that comes in the next commit.	6 years ago
Alisdair McDiarmid	8e87ccb689	providercache: Lazily detect executable file ... Instead of searching the installed provider package directory for a binary as we install it, we can lazily detect the executable as it is required. Doing so allows us to separately report an invalid unpacked package, giving the user more actionable error messages.	6 years ago
Kristin Laemmert	47e657c611	internal/getproviders: decode and return any registry warnings (#25337) ... * internal/getproviders: decode and return any registry warnings The public registry may include a list of warnings in the "versions" response for any given provider. This PR adds support for warnings from the registry and an installer event to return those warnings to the user.	6 years ago
Martin Atkins	9489672d54	internal/getproviders: Package hashing for local filesystem packages ... We previously had this functionality available for cached packages in the providercache package. This moves the main implementation of this over to the getproviders package and then implements it also for PackageMeta, allowing us to compute hashes in a consistent way across both of our representations of a provider package. The new methods on PackageMeta will only be effective for packages in the local filesystem because we need direct access to the contents in order to produce the hash. Hopefully in future the registry protocol will be able to also provide hashes using this content-based (rather than archive-based) algorithm and then we'll be able to make this work for PackageMeta referring to a package obtained from a registry too, but hashes for local packages only are still useful for some cases right now, such as generating mirror directories in the "terraform providers mirror" command.	6 years ago
Alisdair McDiarmid	ef28671b34	Merge pull request #24932 from hashicorp/signing-language ... Modify language for reporting signing state	6 years ago
Paddy	5127f1ef8b	command: Unmanaged providers ... This adds supports for "unmanaged" providers, or providers with process lifecycles not controlled by Terraform. These providers are assumed to be started before Terraform is launched, and are assumed to shut themselves down after Terraform has finished running. To do this, we must update the go-plugin dependency to v1.3.0, which added support for the "test mode" plugin serving that powers all this. As a side-effect of not needing to manage the process lifecycle anymore, Terraform also no longer needs to worry about the provider's binary, as it won't be used for anything anymore. Because of this, we can disable the init behavior that concerns itself with downloading that provider's binary, checking its version, and otherwise managing the binary. This is all managed on a per-provider basis, so managed providers that Terraform downloads, starts, and stops can be used in the same commands as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable is added, and is a JSON encoding of the provider's address to the information we need to connect to it. This change enables two benefits: first, delve and other debuggers can now be attached to provider server processes, and Terraform can connect. This allows for attaching debuggers to provider processes, which before was difficult to impossible. Second, it allows the SDK test framework to host the provider in the same process as the test driver, while running a production Terraform binary against the provider. This allows for Go's built-in race detector and test coverage tooling to work as expected in provider tests. Unmanaged providers are expected to work in the exact same way as managed providers, with one caveat: Terraform kills provider processes and restarts them once per graph walk, meaning multiple times during most Terraform CLI commands. As unmanaged providers can't be killed by Terraform, and have no visibility into graph walks, unmanaged providers are likely to have differences in how their global mutable state behaves when compared to managed providers. Namely, unmanaged providers are likely to retain global state when managed providers would have reset it. Developers relying on global state should be aware of this.	6 years ago
Paul Tyng	22ef5cc99c	Modify language for reporting signing state ... Be more explicit about the signing status of fetched plugins and provide documentation about the different signing options.	6 years ago
Kristin Laemmert	0d620018fe	provider cache: log errors and validate dir exists (#24993) ... * providercache: add logging for errors from getproviders.SearchLocalDirectory providercache.fillMetaCache() was silently swallowing errors when searching the cache directory. This commit logs the error without changing the behavior otherwise. * command/cliconfig: validate plugin cache dir exists The plugin cache directory must exist for terraform to use it, so we will add a check at the begining.	6 years ago
Alisdair McDiarmid	070c3018f8	internal/providercache: Remove unused retry events	6 years ago
Kristin Laemmert	60321b41e8	getproviders: move protocol compatibility functions into registry client (#24846) ... * internal/registry source: return error if requested provider version protocols are not supported * getproviders: move responsibility for protocol compatibility checks into the registry client The original implementation had the providercache checking the provider metadata for protocol compatibility, but this is only relevant for the registry source so it made more sense to move the logic into getproviders. This also addresses an issue where we were pulling the metadata for every provider version until we found one that was supported. I've extended the registry client to unmarshal the protocols in `ProviderVersions` so we can filter through that list, instead of pulling each version's metadata.	6 years ago
Kristin Laemmert	cca0526705	providercache: actually break out of the loop when a matching version is found (#24823)	6 years ago
Kristin Laemmert	ce03f1255f	internal/providercache: fix error message for protocol mismatch (#24818) ... There was a bug in the installer trying to pass a nil error.	6 years ago
Kristin Laemmert	84e9d86c25	Update installer_test.go ... update now-exported function (fix bad PR)	6 years ago
Kristin Laemmert	21b9da5a02	internal/providercache: verify that the provider protocol version is compatible (#24737) ... * internal/providercache: verify that the provider protocol version is compatible The public registry includes a list of supported provider protocol versions for each provider version. This change adds verification of support and adds a specific error message pointing users to the closest matching version.	6 years ago
Alisdair McDiarmid	54abb87fb4	Fix broken test due to function rename	6 years ago
Kristin Laemmert	a43f141f9e	tools/terraform-bundle: refactor to use new provider installer and provider directory layouts (#24629) ... * tools/terraform-bundle: refactor to use new provider installer and provider directory layouts terraform-bundle now supports a "source" attribute for providers, uses the new provider installer, and the archive it creates preserves the new (required) directory hierarchy for providers, under a "plugins" directory. This is a breaking change in many ways: source is required for any non-HashiCorp provider, locally-installed providers must be given a source (can be arbitrary, see docs) and placed in the expected directory hierarchy, and the unzipped archive is no longer flat; there is a new "plugins" directory created with providers in the new directory layout. This PR also extends the existing test to check the contents of the zip file. TODO: Re-enable e2e tests (currently suppressed with a t.Skip) This commit includes an update to our travis configuration, so the terraform-bundle e2e tests run. It also turns off the e2e tests, which will fail until we have a terraform 0.13.* release under releases.hashicorp.com. We decided it was better to merge this now instead of waiting when we started seeing issues opened from users who built terraform-bundle from 0.13 and found it didn't work with 0.12 - better that they get an immediate error message from the binary directing them to build from the appropriate release.	6 years ago
Alisdair McDiarmid	a5b3d497cc	internal: Verify provider signatures on install ... Providers installed from the registry are accompanied by a list of checksums (the "SHA256SUMS" file), which is cryptographically signed to allow package authentication. The process of verifying this has multiple steps: - First we must verify that the SHA256 hash of the package archive matches the expected hash. This could be done for local installations too, in the future. - Next we ensure that the expected hash returned as part of the registry API response matches an entry in the checksum list. - Finally we verify the cryptographic signature of the checksum list, using the public keys provided by the registry. Each of these steps is implemented as a separate PackageAuthentication type. The local archive installation mechanism uses only the archive checksum authenticator, and the HTTP installation uses all three in the order given. The package authentication system now also returns a result value, which is used by command/init to display the result of the authentication process. There are three tiers of signature, each of which is presented differently to the user: - Signatures from the embedded HashiCorp public key indicate that the provider is officially supported by HashiCorp; - If the signing key is not from HashiCorp, it may have an associated trust signature, which indicates that the provider is from one of HashiCorp's trusted partners; - Otherwise, if the signature is valid, this is a community provider.	6 years ago
Kristin Laemmert	f09ae6f862	provider source tests: added test suite to exercise hyphenated providers (#24685)	6 years ago
Martin Atkins	0ad4c1be2f	internal/getproviders: Tidy up some confusion about package hashes ... Earlier on in the stubbing of this package we realized that it wasn't going to be possible to populate the authentication-related bits for all packages because the relevant metadata just isn't available for packages that are already local. However, we just moved ahead with that awkward design at the time because we needed to get other work done, and so we've been mostly producing PackageMeta values with all-zeros hashes and just ignoring them entirely as a temporary workaround. This is a first step towards what is hopefully a more intuitive model: authentication is an optional thing in a PackageMeta that is currently populated only for packages coming from a registry. So far this still just models checking a SHA256 hash, which is not a sufficient set of checks for a real release but hopefully the "real" implementation is a natural iteration of this starting point, and if not then at least this interim step is a bit more honest about the fact that Authentication will not be populated on every PackageMeta.	6 years ago
Martin Atkins	958ea4f7d1	internal/providercache: Handle built-in providers ... Built-in providers are special providers that are distributed as part of Terraform CLI itself, rather than being installed separately. They always live in the terraform.io/builtin/... namespace so it's easier to see that they are special, and currently there is only one built-in provider named "terraform". Previous commits established the addressing scheme for built-in providers. This commit makes the installer aware of them to the extent that it knows not to try to install them the usual way and it's able to report an error if the user requests a built-in provider that doesn't exist or tries to impose a particular version constraint for a built-in provider. For the moment the tests for this are the ones in the "command" package because that's where the existing testing infrastructure for this functionality lives. A later commit should add some more focused unit tests here in the internal/providercache package, too.	6 years ago
Martin Atkins	f35ebe2d65	internal/providercache: Fix incorrect logic in Installer.SetGlobalCacheDir ... Due to some incomplete rework of this function in an earlier commit, the safety check for using the same directory as both the target and the cache was inverted and was raising an error _unless_ they matched, rather than _if_ they matched. This change is verified by the e2etest TestInitProviders_pluginCache, which is also updated to use the new-style cache directory layout as part of this commit.	6 years ago
Martin Atkins	ff55e1a1cd	internal/providercache: installFromLocalDir ... We previously skipped this one because it wasn't strictly necessary for replicating the old "terraform init" behavior, but we do need it to work so that things like the -plugin-dir option can behave correctly. Linking packages from other cache directories and installing from unpacked directories are fundamentally the same operation because a cache directory is really just a collection of unpacked packages, so here we refactor the LinkFromOtherCache functionality to actually be in installFromLocalDir, and LinkFromOtherCache becomes a wrapper for the installFromLocalDir function that just calculates the source and target directories automatically and invalidates the metaCache.	6 years ago
Martin Atkins	2ff4582be2	internal/providercache: Fix positions on selections file/dir ... On Unix-derived systems a directory must be marked as "executable" in order to be accessible, so our previous mode of 0660 here was unsufficient and would cause a failure if it happened to be the installer that was creating the plugins directory for the first time here. Now we'll make it executable and readable for all but only writable by the same user/group. For consistency, we also make the selections file itself readable by everyone. In both cases, the umask we are run with may further constrain these modes.	6 years ago
Martin Atkins	ae080481c0	internal/providercache: Installer records its selections in a file ... Just as with the old installer mechanism, our goal is that explicit provider installation is the only way that new provider versions can be selected. To achieve that, we conclude each call to EnsureProviderVersions by writing a selections lock file into the target directory. A later caller can then recall the selections from that file by calling SelectedPackages, which both ensures that it selects the same set of versions and also verifies that the checksums recorded by the installer still match. This new selections.json file has a different layout than our old plugins.json lock file. Not only does it use a different hashing algorithm than before, we also record explicitly which version of each provider was selected. In the old model, we'd repeat normal discovery when reloading the lock file and then fail with a confusing error message if discovery happened to select a different version, but now we'll be able to distinguish between a package that's gone missing since installation (which could previously have then selected a different available version) from a package that has been modified.	6 years ago
Martin Atkins	f6a7a4868b	internal/providercache: Hashing of contents of cached packages ... For the old-style provider cache directory model we hashed the individual executable file for each provider. That's no longer appropriate because we're giving each provider package a whole directory to itself where it can potentially have many files. This therefore introduces a new directory-oriented hashing algorithm, and it's just using the Go Modules directory hashing algorithm directly because that's already had its cross-platform quirks and other wrinkles addressed during the Go Modules release process, and is now used prolifically enough in Go codebases that breaking changes to the upstream algorithm would be very expensive to the Go ecosystem. This is also a bit of forward planning, anticipating that later we'll use hashes in a top-level lock file intended to be checked in to user version control, and then use those hashes also to verify packages _during_ installation, where we'd need to be able to hash unpacked zip files. The Go Modules hashing algorithm is already implemented to consistently hash both a zip file and an unpacked version of that zip file.	6 years ago
Martin Atkins	079b4cf7be	internal/providercache: Clear the metadata cache during package install ... This was previously happening during linking from another cache, but not when installing an entirely new provider.	6 years ago
Martin Atkins	4061cbed38	internal/getproviders: A new shared model for provider requirements ... We've been using the models from the "moduledeps" package to represent our provider dependencies everywhere since the idea of provider dependencies was introduced in Terraform 0.10, but that model is not convenient to use for any use-case other than the "terraform providers" command that needs individual-module-level detail. To make things easier for new codepaths working with the new-style provider installer, here we introduce a new model type getproviders.Requirements which is based on the type the new installer was already taking as its input. We have new methods in the states, configs, and earlyconfig packages to produce values of this type, and a helper to merge Requirements together so we can combine config-derived and state-derived requirements together during installation. The advantage of this new model over the moduledeps one is that all of recursive module walking is done up front and we produce a simple, flat structure that is more convenient for the main use-cases of selecting providers for installation and then finding providers in the local cache to use them for other operations. This new model is _not_ suitable for implementing "terraform providers" because it does not retain module-specific requirement details. Therefore we will likely keep using moduledeps for "terraform providers" for now, and then possibly at a later time consider specializing the moduledeps logic for only what "terraform providers" needs, because it seems to be the only use-case that needs to retain that level of detail.	6 years ago