70 Commits (7bee77bdd3dc67de0973e1c19bd5a1685dc805c0)

Author	SHA1	Message	Date
hashicorp-copywrite[bot]	325d18262e	[COMPLIANCE] Add Copyright and License Headers	3 years ago
Martin Atkins	e2380b1038	cliconfig: Allow forcing use of the plugin cache despite the lock file ... Currently Terraform will use an entry from the global plugin cache only if it matches a checksum already recorded in the dependency lock file. This allows Terraform to produce a complete lock file entry on the first encounter with a new provider, whereas using the cache in that case would cause the lock file to only cover the single package in the cache and thereefore be unusable on any other operating system or CPU architecture. This temporary CLI config option is a pragmatic exception to support those who cannot currently correctly use the dependency lock file but who still want to benefit from the plugin cache. With this setting enabled, Terraform has permission to produce a dependency lock file that is only suitable for the current system if that would allow use of an existing entry in the plugin cache. We are introducing this option to resolve a conflict between the needs of folks who are using the dependency lock file as expected and the needs of folks who cannot use the dependency lock file for some reason. The hope then is to give respite to those who need this exception in the meantime while we understand better why they cannot use the dependency lock file and improve its design so that everyone will be able to use it successfully in a future version of Terraform. This option will become a silent no-op in a future version of Terraform, once the dependency lock file behavior is sufficient for all supported Terraform development workflows.	3 years ago
Martin Atkins	d0a35c60a7	providercache: Ignore lock-mismatching global cache entries ... When we originally introduced the trust-on-first-use checksum locking mechanism in v0.14, we had to make some tricky decisions about how it should interact with the pre-existing optional read-through global cache of provider packages: The global cache essentially conflicts with the checksum locking because if the needed provider is already in the cache then Terraform skips installing the provider from upstream and therefore misses the opportunity to capture the signed checksums published by the provider developer. We can't use the signed checksums to verify a cache entry because the origin registry protocol is still using the legacy ziphash scheme and that is only usable for the original zipped provider packages and not for the unpacked-layout cache directory. Therefore we decided to prioritize the existing cache directory behavior at the expense of the lock file behavior, making Terraform produce an incomplete lock file in that case. Now that we've had some real-world experience with the lock file mechanism, we can see that the chosen compromise was not ideal because it causes "terraform init" to behave significantly differently in its lock file update behavior depending on whether or not a particular provider is already cached. By robbing Terraform of its opportunity to fetch the official checksums, Terraform must generate a lock file that is inherently non-portable, which is problematic for any team which works with the same Terraform configuration on multiple different platforms. This change addresses that problem by essentially flipping the decision so that we'll prioritize the lock file behavior over the provider cache behavior. Now a global cache entry is eligible for use if and only if the lock file already contains a checksum that matches the cache entry. This means that the first time a particular configuration sees a new provider it will always be fetched from the configured installation source (typically the origin registry) and record the checksums from that source. On subsequent installs of the same provider version already locked, Terraform will then consider the cache entry to be eligible and skip re-downloading the same package. This intentionally makes the global cache mechanism subordinate to the lock file mechanism: the lock file must be populated in order for the global cache to be effective. For those who have many separate configurations which all refer to the same provider version, they will need to re-download the provider once for each configuration in order to gather the information needed to populate the lock file, whereas before they would have only downloaded it for the _first_ configuration using that provider. This should therefore remove the most significant cause of folks ending up with incomplete lock files that don't work for colleagues using other platforms, and the expense of bypassing the cache for the first use of each new package with each new configuration. This tradeoff seems reasonable because otherwise such users would inevitably need to run "terraform providers lock" separately anyway, and that command _always_ bypasses the cache. Although this change does decrease the hit rate of the cache, if we subtract the never-cached downloads caused by "terraform providers lock" then this is a net benefit overall, and does the right thing by default without the need to run a separate command.	3 years ago
Radek Simko	196ff4c5c5	providercache: Improve installation error message (#31898)	3 years ago
Radek Simko	c1fc7fd744	providercache: include host in provider installation error (#31524)	4 years ago
Liam Cervante	ad5ac89461	Fix tests failing on main (#31481)	4 years ago
Liam Cervante	9f0d1d0630	terraform init: add link to documentation when a checksum is missing from the lock file (#31408) ... * terraform init: add suggested fix for when a checksum is missing from the lock file * improve error message * add link to the documentation * cleanup leftovers from previous attempt * fix tests * s/,/; * fix imports	4 years ago
Liam Cervante	83e84e5477	terraform init: add warning and guidance when lock file is incomplete (#31399) ... * terraform init: add warning and guidance when lock file is incomplete * make the provider list in the warning deterministic * create installer event for tracking provider lock hashes (#31406) * create installer event for tracking provider lock hashes * address comments * fix tests * improve error message * Update internal/command/init.go Co-authored-by: Martin Atkins <mart@degeneration.co.uk> Co-authored-by: Martin Atkins <mart@degeneration.co.uk>	4 years ago
Liam Cervante	224728879d	Ignore existing package hashes for `providers lock` command (#31389) ... * Ignore existing package hashes for command * missing new line * Fix incorrect logic when deciding change message * fix imports	4 years ago
Eng Zer Jun	fedd315275	test: use `T.TempDir` to create temporary test directory (#30803) ... This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The directory created by `t.TempDir` is automatically removed when the test and all its subtests complete. Prior to this commit, temporary directory created using `ioutil.TempDir` needs to be removed manually by calling `os.RemoveAll`, which is omitted in some tests. The error handling boilerplate e.g. defer func() { if err := os.RemoveAll(dir); err != nil { t.Fatal(err) } } is also tedious, but `t.TempDir` handles this for us nicely. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	4 years ago
Martin Atkins	23395a1022	providercache: Discard lock entries for unused providers ... Previously we would only ever add new lock entries or update existing ones. However, it's possible that over time a module may _cease_ using a particular provider, at which point we ought to remove it from the lock file so that operations won't fail when seeing that the provider cache directory is inconsistent with the lock file. Now the provider installer (EnsureProviderVersions) will remove any lock file entries that relate to providers not included in the given requirements, which therefore makes the resulting lock file properly match the set of packages the installer wrote into the cache. This does potentially mean that someone could inadvertently defeat the lock by removing a provider dependency, running "terraform init", then undoing that removal, and finally running "terraform init" again. However, that seems relatively unlikely compared to the likelihood of removing a provider and keeping it removed, and in the event it _did_ happen the changes to the lock entry for that provider would be visible in the diff of the provider lock file as usual, and so could be noticed in code review just as for any other change to dependencies.	4 years ago
Kristin Laemmert	5a48530f47	tools: remove terraform-bundle. (#28876) ... * tools: remove terraform-bundle. terraform-bundle is no longer supported in the main branch of terraform. Users can build terraform-bundle from terraform tagged v0.15 and older. * add a README pointing users to the v0.15 branch	5 years ago
Martin Atkins	e5f52e56f8	addrs: Rename DefaultRegistryHost to DefaultProviderRegistryHost ... As the comment notes, this hostname is the default for provide source addresses. We'll shortly be adding some address types to represent module source addresses too, and so we'll also have DefaultModuleRegistryHost for that situation. (They'll actually both contain the the same hostname, but that's a coincidence rather than a requirement.)	5 years ago
Martin Atkins	b9a93a0fe7	Move addrs/ to internal/addrs/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	4c254cc2be	Move httpclient/ to internal/httpclient/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	7f78342953	command: Experimental "terraform test" command ... This is just a prototype to gather some feedback in our ongoing research on integration testing of Terraform modules. The hope is that by having a command integrated into Terraform itself it'll be easier for interested module authors to give it a try, and also easier for us to iterate quickly based on feedback without having to coordinate across multiple codebases. Everything about this is subject to change even in future patch releases. Since it's a CLI command rather than a configuration language feature it's not using the language experiments mechanism, but generates a warning similar to the one language experiments generate in order to be clear that backward compatibility is not guaranteed.	5 years ago
Pam Selle	aa24bfec47	Emit ProviderAlreadyInstalled when provider installed ... Emit the ProviderAlreadyInstalled event when we successfully verify that we've already installed this provider and are skipping installation	5 years ago
Pam Selle	aedca597dd	Reuse installed target dir providers in init ... In init, we can check to see if the target dir already has the provider we are seeking and skip further querying/installing of that provider.	5 years ago
Martin Atkins	4b3e237668	command/init: Hint about providers in other namespaces ... If a user forgets to specify the source address for a provider, Terraform will assume they meant a provider in the registry.terraform.io/hashicorp/ namespace. If that ultimately doesn't exist, we'll now try to see if there's some other provider source address recorded in the registry's legacy provider lookup table, and suggest it if so. The error message here is a terse one addressed primarily to folks who are already somewhat familiar with provider source addresses and how to specify them. Terraform v0.13 had a more elaborate version of this error message which directed the user to try the v0.13 automatic upgrade tool, but we no longer have that available in v0.14 and later so the user must make the fix themselves.	5 years ago
James Bardin	6529659455	internal/providercache: staticcheck	5 years ago
James Bardin	276dfe634f	internal/providercache: staticcheck	5 years ago
James Bardin	7eb491719f	normalize temp dir paths in tests ... The temporary directory on some systems (most notably MacOS) contains symlinks, which would not be recorded by the installer. In order to make these paths comparable in the tests we need to eval the symlinks in the paths before giving them to the installer.	5 years ago
Alisdair McDiarmid	90c4f11b61	go get github.com/hashicorp/go-getter@v1.5.0 ... Includes fix for breaking upstream API change.	5 years ago
Martin Atkins	e6e0b6ee46	providercache: verify locked hashes for local package dirs ... Previously we were only verifying locked hashes for local archive zip files, but if we have non-ziphash hashes available then we can and should also verify that a local directory matches at least one of them. This does mean that folks using filesystem mirrors but yet also running Terraform across multiple platforms will need to take some extra care to ensure the hashes pass on all relevant platforms, which could mean using "terraform providers lock" to pre-seed their lock files with hashes across all platforms, or could mean using the "packed" directory layout for the filesystem mirror so that Terraform will end up in the install-from-archive codepath instead of this install-from-directory codepath, and can thus verify ziphash too. (There's no additional documentation about the above here because there's already general information about this in the lock file documentation due to some similar -- though not identical -- situations with network mirrors.)	5 years ago
Martin Atkins	24d32e9ca2	providercache: More exhaustive testing of the main installer ... We previously had some tests for some happy paths and a few specific failures into an empty directory with no existing locks, but we didn't have tests for the installer respecting existing lock file entries. This is a start on a more exhaustive set of tests for the installer, aiming to visit as many of the possible codepaths as we can reasonably test using this mocking strategy. (Some other codepaths require different underlying source implementations, etc, so we'll have to visit those in other tests separately.)	5 years ago
Martin Atkins	2611e08430	command/init: Mention using the lock file for provider selection ... This probably isn't the best UI we could do here, but it's a placeholder for now just to avoid making it seem like we're ignoring the lock file and checking for new versions anyway.	5 years ago
Martin Atkins	b3f5c7f1e6	command/init: Read, respect, and update provider dependency locks ... This changes the approach used by the provider installer to remember between runs which selections it has previously made, using the lock file format implemented in internal/depsfile. This means that version constraints in the configuration are considered only for providers we've not seen before or when -upgrade mode is active.	5 years ago
Kristin Laemmert	04be220f5f	deprecate helper/copy ... helper/copy CopyDir was used heavily in tests. It differes from internal/copydir in a few ways, the main one being that it creates the dst directory while the internal version expected the dst to exist (there are other differences, which is why I did not just switch tests to using internal's CopyDir). I moved the CopyDir func from helper/copy into command_test.go; I could also have moved it into internal/copy and named it something like CreateDirAndCopy so if that seems like a better option please let me know. helper/copy/CopyFile was used in a couple of spots so I moved it into internal, at which point I thought it made more sense to rename the package copy (instead of copydir). There's also a `go mod tidy` included.	5 years ago
Martin Atkins	0b734a2803	command: Make provider installation interruptible ... In earlier commits we started to make the installation codepath context-aware so that it could be canceled in the event of a SIGINT, but we didn't complete wiring that through the API of the getproviders package. Here we make the getproviders.Source interface methods, along with some other functions that can make network requests, take a context.Context argument and act appropriately if that context is cancelled. The main providercache.Installer.EnsureProviderVersions method now also has some context-awareness so that it can abort its work early if its context reports any sort of error. That avoids waiting for the process to wind through all of the remaining iterations of the various loops, logging each request failure separately, and instead returns just a single aggregate "canceled" error. We can then set things up in the "terraform init" and "terraform providers mirror" commands so that the context will be cancelled if we get an interrupt signal, allowing provider installation to abort early while still atomically completing any local-side effects that may have started.	5 years ago
Martin Atkins	6694cfaa0e	getproviders: Add a real type Hash for package hashes ... The logic for what constitutes a valid hash and how different hash schemes are represented was starting to get sprawled over many different files and packages. Consistently with other cases where we've used named types to gather the definition of a particular string into a single place and have the Go compiler help us use it properly, this introduces both getproviders.Hash representing a hash value and getproviders.HashScheme representing the idea of a particular hash scheme. Most of this changeset is updating existing uses of primitive strings to uses of getproviders.Hash. The new type definitions are in internal/getproviders/hash.go.	5 years ago
Alisdair McDiarmid	5587509bcf	internal: Fix providercache test failures on macOS ... For reasons that are unclear, these two tests just started failing on macOS very recently. The failure looked like: PackageDir: strings.Join({ "/", + "private/", "var/folders/3h/foobar/T/terraform-test-p", "rovidercache655312854/registry.terraform.io/hashicorp/null/2.0.0", "/windows_amd64", }, Speculating that the macOS temporary directory moved into the /private directory, I added a couple of EvalSymlinks calls and the tests pass again. No other unit tests appear to be affected by this at the moment.	6 years ago
Alisdair McDiarmid	915f53af23	internal: Clean up package install temp file ... The installFromHTTPURL function downloads a package to a temporary file, then delegates to installFromLocalArchive to install it. We were previously not deleting the temporary file afterwards. This commit fixes that.	6 years ago
Alisdair McDiarmid	440543f427	internal/providercache: Fix bug when symlink fails ... When installing a provider which is already cached, we attempt to create a symlink from the install directory targeting the cache. If symlinking fails due to missing OS/filesystem support, we instead want to copy the cached provider. The fallback code to do this would always fail, due to a missing target directory. This commit fixes that. I was unable to find a way to add automated tests around this, but I have manually verified the fix on Windows 8.1.	6 years ago
Alisdair McDiarmid	3b1347ac1a	providercache: Validate provider executable file ... At the end of the EnsureProviderVersions process, we generate a lockfile of the selected and installed provider versions. This includes a hash of the unpacked provider directory. When calculating this hash and generating the lockfile, we now also verify that the provider directory contains a valid executable file. If not, we return an error for this provider and trigger the installer's HashPackageFailure event. Note that this event is not yet processed by terraform init; that comes in the next commit.	6 years ago
Alisdair McDiarmid	8e87ccb689	providercache: Lazily detect executable file ... Instead of searching the installed provider package directory for a binary as we install it, we can lazily detect the executable as it is required. Doing so allows us to separately report an invalid unpacked package, giving the user more actionable error messages.	6 years ago
Kristin Laemmert	47e657c611	internal/getproviders: decode and return any registry warnings (#25337) ... * internal/getproviders: decode and return any registry warnings The public registry may include a list of warnings in the "versions" response for any given provider. This PR adds support for warnings from the registry and an installer event to return those warnings to the user.	6 years ago
Martin Atkins	9489672d54	internal/getproviders: Package hashing for local filesystem packages ... We previously had this functionality available for cached packages in the providercache package. This moves the main implementation of this over to the getproviders package and then implements it also for PackageMeta, allowing us to compute hashes in a consistent way across both of our representations of a provider package. The new methods on PackageMeta will only be effective for packages in the local filesystem because we need direct access to the contents in order to produce the hash. Hopefully in future the registry protocol will be able to also provide hashes using this content-based (rather than archive-based) algorithm and then we'll be able to make this work for PackageMeta referring to a package obtained from a registry too, but hashes for local packages only are still useful for some cases right now, such as generating mirror directories in the "terraform providers mirror" command.	6 years ago
Alisdair McDiarmid	ef28671b34	Merge pull request #24932 from hashicorp/signing-language ... Modify language for reporting signing state	6 years ago
Paddy	5127f1ef8b	command: Unmanaged providers ... This adds supports for "unmanaged" providers, or providers with process lifecycles not controlled by Terraform. These providers are assumed to be started before Terraform is launched, and are assumed to shut themselves down after Terraform has finished running. To do this, we must update the go-plugin dependency to v1.3.0, which added support for the "test mode" plugin serving that powers all this. As a side-effect of not needing to manage the process lifecycle anymore, Terraform also no longer needs to worry about the provider's binary, as it won't be used for anything anymore. Because of this, we can disable the init behavior that concerns itself with downloading that provider's binary, checking its version, and otherwise managing the binary. This is all managed on a per-provider basis, so managed providers that Terraform downloads, starts, and stops can be used in the same commands as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable is added, and is a JSON encoding of the provider's address to the information we need to connect to it. This change enables two benefits: first, delve and other debuggers can now be attached to provider server processes, and Terraform can connect. This allows for attaching debuggers to provider processes, which before was difficult to impossible. Second, it allows the SDK test framework to host the provider in the same process as the test driver, while running a production Terraform binary against the provider. This allows for Go's built-in race detector and test coverage tooling to work as expected in provider tests. Unmanaged providers are expected to work in the exact same way as managed providers, with one caveat: Terraform kills provider processes and restarts them once per graph walk, meaning multiple times during most Terraform CLI commands. As unmanaged providers can't be killed by Terraform, and have no visibility into graph walks, unmanaged providers are likely to have differences in how their global mutable state behaves when compared to managed providers. Namely, unmanaged providers are likely to retain global state when managed providers would have reset it. Developers relying on global state should be aware of this.	6 years ago
Paul Tyng	22ef5cc99c	Modify language for reporting signing state ... Be more explicit about the signing status of fetched plugins and provide documentation about the different signing options.	6 years ago
Kristin Laemmert	0d620018fe	provider cache: log errors and validate dir exists (#24993) ... * providercache: add logging for errors from getproviders.SearchLocalDirectory providercache.fillMetaCache() was silently swallowing errors when searching the cache directory. This commit logs the error without changing the behavior otherwise. * command/cliconfig: validate plugin cache dir exists The plugin cache directory must exist for terraform to use it, so we will add a check at the begining.	6 years ago
Alisdair McDiarmid	070c3018f8	internal/providercache: Remove unused retry events	6 years ago
Kristin Laemmert	60321b41e8	getproviders: move protocol compatibility functions into registry client (#24846) ... * internal/registry source: return error if requested provider version protocols are not supported * getproviders: move responsibility for protocol compatibility checks into the registry client The original implementation had the providercache checking the provider metadata for protocol compatibility, but this is only relevant for the registry source so it made more sense to move the logic into getproviders. This also addresses an issue where we were pulling the metadata for every provider version until we found one that was supported. I've extended the registry client to unmarshal the protocols in `ProviderVersions` so we can filter through that list, instead of pulling each version's metadata.	6 years ago
Kristin Laemmert	cca0526705	providercache: actually break out of the loop when a matching version is found (#24823)	6 years ago
Kristin Laemmert	ce03f1255f	internal/providercache: fix error message for protocol mismatch (#24818) ... There was a bug in the installer trying to pass a nil error.	6 years ago
Kristin Laemmert	84e9d86c25	Update installer_test.go ... update now-exported function (fix bad PR)	6 years ago
Kristin Laemmert	21b9da5a02	internal/providercache: verify that the provider protocol version is compatible (#24737) ... * internal/providercache: verify that the provider protocol version is compatible The public registry includes a list of supported provider protocol versions for each provider version. This change adds verification of support and adds a specific error message pointing users to the closest matching version.	6 years ago
Alisdair McDiarmid	54abb87fb4	Fix broken test due to function rename	6 years ago
Kristin Laemmert	a43f141f9e	tools/terraform-bundle: refactor to use new provider installer and provider directory layouts (#24629) ... * tools/terraform-bundle: refactor to use new provider installer and provider directory layouts terraform-bundle now supports a "source" attribute for providers, uses the new provider installer, and the archive it creates preserves the new (required) directory hierarchy for providers, under a "plugins" directory. This is a breaking change in many ways: source is required for any non-HashiCorp provider, locally-installed providers must be given a source (can be arbitrary, see docs) and placed in the expected directory hierarchy, and the unzipped archive is no longer flat; there is a new "plugins" directory created with providers in the new directory layout. This PR also extends the existing test to check the contents of the zip file. TODO: Re-enable e2e tests (currently suppressed with a t.Skip) This commit includes an update to our travis configuration, so the terraform-bundle e2e tests run. It also turns off the e2e tests, which will fail until we have a terraform 0.13.* release under releases.hashicorp.com. We decided it was better to merge this now instead of waiting when we started seeing issues opened from users who built terraform-bundle from 0.13 and found it didn't work with 0.12 - better that they get an immediate error message from the binary directing them to build from the appropriate release.	6 years ago
Alisdair McDiarmid	a5b3d497cc	internal: Verify provider signatures on install ... Providers installed from the registry are accompanied by a list of checksums (the "SHA256SUMS" file), which is cryptographically signed to allow package authentication. The process of verifying this has multiple steps: - First we must verify that the SHA256 hash of the package archive matches the expected hash. This could be done for local installations too, in the future. - Next we ensure that the expected hash returned as part of the registry API response matches an entry in the checksum list. - Finally we verify the cryptographic signature of the checksum list, using the public keys provided by the registry. Each of these steps is implemented as a separate PackageAuthentication type. The local archive installation mechanism uses only the archive checksum authenticator, and the HTTP installation uses all three in the order given. The package authentication system now also returns a result value, which is used by command/init to display the result of the authentication process. There are three tiers of signature, each of which is presented differently to the user: - Signatures from the embedded HashiCorp public key indicate that the provider is officially supported by HashiCorp; - If the signing key is not from HashiCorp, it may have an associated trust signature, which indicates that the provider is from one of HashiCorp's trusted partners; - Otherwise, if the signature is valid, this is a community provider.	6 years ago