20 Commits (b5658a46a2aacb3e96bd10722027f179f3900a8d)

Author	SHA1	Message	Date
hashicorp-copywrite[bot]	325d18262e	[COMPLIANCE] Add Copyright and License Headers	3 years ago
Martin Atkins	e2380b1038	cliconfig: Allow forcing use of the plugin cache despite the lock file ... Currently Terraform will use an entry from the global plugin cache only if it matches a checksum already recorded in the dependency lock file. This allows Terraform to produce a complete lock file entry on the first encounter with a new provider, whereas using the cache in that case would cause the lock file to only cover the single package in the cache and thereefore be unusable on any other operating system or CPU architecture. This temporary CLI config option is a pragmatic exception to support those who cannot currently correctly use the dependency lock file but who still want to benefit from the plugin cache. With this setting enabled, Terraform has permission to produce a dependency lock file that is only suitable for the current system if that would allow use of an existing entry in the plugin cache. We are introducing this option to resolve a conflict between the needs of folks who are using the dependency lock file as expected and the needs of folks who cannot use the dependency lock file for some reason. The hope then is to give respite to those who need this exception in the meantime while we understand better why they cannot use the dependency lock file and improve its design so that everyone will be able to use it successfully in a future version of Terraform. This option will become a silent no-op in a future version of Terraform, once the dependency lock file behavior is sufficient for all supported Terraform development workflows.	3 years ago
Martin Atkins	d0a35c60a7	providercache: Ignore lock-mismatching global cache entries ... When we originally introduced the trust-on-first-use checksum locking mechanism in v0.14, we had to make some tricky decisions about how it should interact with the pre-existing optional read-through global cache of provider packages: The global cache essentially conflicts with the checksum locking because if the needed provider is already in the cache then Terraform skips installing the provider from upstream and therefore misses the opportunity to capture the signed checksums published by the provider developer. We can't use the signed checksums to verify a cache entry because the origin registry protocol is still using the legacy ziphash scheme and that is only usable for the original zipped provider packages and not for the unpacked-layout cache directory. Therefore we decided to prioritize the existing cache directory behavior at the expense of the lock file behavior, making Terraform produce an incomplete lock file in that case. Now that we've had some real-world experience with the lock file mechanism, we can see that the chosen compromise was not ideal because it causes "terraform init" to behave significantly differently in its lock file update behavior depending on whether or not a particular provider is already cached. By robbing Terraform of its opportunity to fetch the official checksums, Terraform must generate a lock file that is inherently non-portable, which is problematic for any team which works with the same Terraform configuration on multiple different platforms. This change addresses that problem by essentially flipping the decision so that we'll prioritize the lock file behavior over the provider cache behavior. Now a global cache entry is eligible for use if and only if the lock file already contains a checksum that matches the cache entry. This means that the first time a particular configuration sees a new provider it will always be fetched from the configured installation source (typically the origin registry) and record the checksums from that source. On subsequent installs of the same provider version already locked, Terraform will then consider the cache entry to be eligible and skip re-downloading the same package. This intentionally makes the global cache mechanism subordinate to the lock file mechanism: the lock file must be populated in order for the global cache to be effective. For those who have many separate configurations which all refer to the same provider version, they will need to re-download the provider once for each configuration in order to gather the information needed to populate the lock file, whereas before they would have only downloaded it for the _first_ configuration using that provider. This should therefore remove the most significant cause of folks ending up with incomplete lock files that don't work for colleagues using other platforms, and the expense of bypassing the cache for the first use of each new package with each new configuration. This tradeoff seems reasonable because otherwise such users would inevitably need to run "terraform providers lock" separately anyway, and that command _always_ bypasses the cache. Although this change does decrease the hit rate of the cache, if we subtract the never-cached downloads caused by "terraform providers lock" then this is a net benefit overall, and does the right thing by default without the need to run a separate command.	4 years ago
Liam Cervante	ad5ac89461	Fix tests failing on main (#31481)	4 years ago
Liam Cervante	9f0d1d0630	terraform init: add link to documentation when a checksum is missing from the lock file (#31408) ... * terraform init: add suggested fix for when a checksum is missing from the lock file * improve error message * add link to the documentation * cleanup leftovers from previous attempt * fix tests * s/,/; * fix imports	4 years ago
Liam Cervante	83e84e5477	terraform init: add warning and guidance when lock file is incomplete (#31399) ... * terraform init: add warning and guidance when lock file is incomplete * make the provider list in the warning deterministic * create installer event for tracking provider lock hashes (#31406) * create installer event for tracking provider lock hashes * address comments * fix tests * improve error message * Update internal/command/init.go Co-authored-by: Martin Atkins <mart@degeneration.co.uk> Co-authored-by: Martin Atkins <mart@degeneration.co.uk>	4 years ago
Liam Cervante	224728879d	Ignore existing package hashes for `providers lock` command (#31389) ... * Ignore existing package hashes for command * missing new line * Fix incorrect logic when deciding change message * fix imports	4 years ago
Eng Zer Jun	fedd315275	test: use `T.TempDir` to create temporary test directory (#30803) ... This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The directory created by `t.TempDir` is automatically removed when the test and all its subtests complete. Prior to this commit, temporary directory created using `ioutil.TempDir` needs to be removed manually by calling `os.RemoveAll`, which is omitted in some tests. The error handling boilerplate e.g. defer func() { if err := os.RemoveAll(dir); err != nil { t.Fatal(err) } } is also tedious, but `t.TempDir` handles this for us nicely. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	4 years ago
Martin Atkins	23395a1022	providercache: Discard lock entries for unused providers ... Previously we would only ever add new lock entries or update existing ones. However, it's possible that over time a module may _cease_ using a particular provider, at which point we ought to remove it from the lock file so that operations won't fail when seeing that the provider cache directory is inconsistent with the lock file. Now the provider installer (EnsureProviderVersions) will remove any lock file entries that relate to providers not included in the given requirements, which therefore makes the resulting lock file properly match the set of packages the installer wrote into the cache. This does potentially mean that someone could inadvertently defeat the lock by removing a provider dependency, running "terraform init", then undoing that removal, and finally running "terraform init" again. However, that seems relatively unlikely compared to the likelihood of removing a provider and keeping it removed, and in the event it _did_ happen the changes to the lock entry for that provider would be visible in the diff of the provider lock file as usual, and so could be noticed in code review just as for any other change to dependencies.	4 years ago
Martin Atkins	b9a93a0fe7	Move addrs/ to internal/addrs/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Pam Selle	aa24bfec47	Emit ProviderAlreadyInstalled when provider installed ... Emit the ProviderAlreadyInstalled event when we successfully verify that we've already installed this provider and are skipping installation	5 years ago
Pam Selle	aedca597dd	Reuse installed target dir providers in init ... In init, we can check to see if the target dir already has the provider we are seeking and skip further querying/installing of that provider.	5 years ago
James Bardin	7eb491719f	normalize temp dir paths in tests ... The temporary directory on some systems (most notably MacOS) contains symlinks, which would not be recorded by the installer. In order to make these paths comparable in the tests we need to eval the symlinks in the paths before giving them to the installer.	6 years ago
Martin Atkins	e6e0b6ee46	providercache: verify locked hashes for local package dirs ... Previously we were only verifying locked hashes for local archive zip files, but if we have non-ziphash hashes available then we can and should also verify that a local directory matches at least one of them. This does mean that folks using filesystem mirrors but yet also running Terraform across multiple platforms will need to take some extra care to ensure the hashes pass on all relevant platforms, which could mean using "terraform providers lock" to pre-seed their lock files with hashes across all platforms, or could mean using the "packed" directory layout for the filesystem mirror so that Terraform will end up in the install-from-archive codepath instead of this install-from-directory codepath, and can thus verify ziphash too. (There's no additional documentation about the above here because there's already general information about this in the lock file documentation due to some similar -- though not identical -- situations with network mirrors.)	6 years ago
Martin Atkins	24d32e9ca2	providercache: More exhaustive testing of the main installer ... We previously had some tests for some happy paths and a few specific failures into an empty directory with no existing locks, but we didn't have tests for the installer respecting existing lock file entries. This is a start on a more exhaustive set of tests for the installer, aiming to visit as many of the possible codepaths as we can reasonably test using this mocking strategy. (Some other codepaths require different underlying source implementations, etc, so we'll have to visit those in other tests separately.)	6 years ago
Martin Atkins	b3f5c7f1e6	command/init: Read, respect, and update provider dependency locks ... This changes the approach used by the provider installer to remember between runs which selections it has previously made, using the lock file format implemented in internal/depsfile. This means that version constraints in the configuration are considered only for providers we've not seen before or when -upgrade mode is active.	6 years ago
Alisdair McDiarmid	3b1347ac1a	providercache: Validate provider executable file ... At the end of the EnsureProviderVersions process, we generate a lockfile of the selected and installed provider versions. This includes a hash of the unpacked provider directory. When calculating this hash and generating the lockfile, we now also verify that the provider directory contains a valid executable file. If not, we return an error for this provider and trigger the installer's HashPackageFailure event. Note that this event is not yet processed by terraform init; that comes in the next commit.	6 years ago
Kristin Laemmert	60321b41e8	getproviders: move protocol compatibility functions into registry client (#24846) ... * internal/registry source: return error if requested provider version protocols are not supported * getproviders: move responsibility for protocol compatibility checks into the registry client The original implementation had the providercache checking the provider metadata for protocol compatibility, but this is only relevant for the registry source so it made more sense to move the logic into getproviders. This also addresses an issue where we were pulling the metadata for every provider version until we found one that was supported. I've extended the registry client to unmarshal the protocols in `ProviderVersions` so we can filter through that list, instead of pulling each version's metadata.	6 years ago
Kristin Laemmert	84e9d86c25	Update installer_test.go ... update now-exported function (fix bad PR)	6 years ago
Kristin Laemmert	21b9da5a02	internal/providercache: verify that the provider protocol version is compatible (#24737) ... * internal/providercache: verify that the provider protocol version is compatible The public registry includes a list of supported provider protocol versions for each provider version. This change adds verification of support and adds a specific error message pointing users to the closest matching version.	6 years ago