17 Commits (b5658a46a2aacb3e96bd10722027f179f3900a8d)

Author	SHA1	Message	Date
hashicorp-copywrite[bot]	325d18262e	[COMPLIANCE] Add Copyright and License Headers	3 years ago
Martin Atkins	d0a35c60a7	providercache: Ignore lock-mismatching global cache entries ... When we originally introduced the trust-on-first-use checksum locking mechanism in v0.14, we had to make some tricky decisions about how it should interact with the pre-existing optional read-through global cache of provider packages: The global cache essentially conflicts with the checksum locking because if the needed provider is already in the cache then Terraform skips installing the provider from upstream and therefore misses the opportunity to capture the signed checksums published by the provider developer. We can't use the signed checksums to verify a cache entry because the origin registry protocol is still using the legacy ziphash scheme and that is only usable for the original zipped provider packages and not for the unpacked-layout cache directory. Therefore we decided to prioritize the existing cache directory behavior at the expense of the lock file behavior, making Terraform produce an incomplete lock file in that case. Now that we've had some real-world experience with the lock file mechanism, we can see that the chosen compromise was not ideal because it causes "terraform init" to behave significantly differently in its lock file update behavior depending on whether or not a particular provider is already cached. By robbing Terraform of its opportunity to fetch the official checksums, Terraform must generate a lock file that is inherently non-portable, which is problematic for any team which works with the same Terraform configuration on multiple different platforms. This change addresses that problem by essentially flipping the decision so that we'll prioritize the lock file behavior over the provider cache behavior. Now a global cache entry is eligible for use if and only if the lock file already contains a checksum that matches the cache entry. This means that the first time a particular configuration sees a new provider it will always be fetched from the configured installation source (typically the origin registry) and record the checksums from that source. On subsequent installs of the same provider version already locked, Terraform will then consider the cache entry to be eligible and skip re-downloading the same package. This intentionally makes the global cache mechanism subordinate to the lock file mechanism: the lock file must be populated in order for the global cache to be effective. For those who have many separate configurations which all refer to the same provider version, they will need to re-download the provider once for each configuration in order to gather the information needed to populate the lock file, whereas before they would have only downloaded it for the _first_ configuration using that provider. This should therefore remove the most significant cause of folks ending up with incomplete lock files that don't work for colleagues using other platforms, and the expense of bypassing the cache for the first use of each new package with each new configuration. This tradeoff seems reasonable because otherwise such users would inevitably need to run "terraform providers lock" separately anyway, and that command _always_ bypasses the cache. Although this change does decrease the hit rate of the cache, if we subtract the never-cached downloads caused by "terraform providers lock" then this is a net benefit overall, and does the right thing by default without the need to run a separate command.	4 years ago
Radek Simko	196ff4c5c5	providercache: Improve installation error message (#31898)	4 years ago
Radek Simko	c1fc7fd744	providercache: include host in provider installation error (#31524)	4 years ago
Liam Cervante	9f0d1d0630	terraform init: add link to documentation when a checksum is missing from the lock file (#31408) ... * terraform init: add suggested fix for when a checksum is missing from the lock file * improve error message * add link to the documentation * cleanup leftovers from previous attempt * fix tests * s/,/; * fix imports	4 years ago
Martin Atkins	4c254cc2be	Move httpclient/ to internal/httpclient/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
James Bardin	6529659455	internal/providercache: staticcheck	5 years ago
Alisdair McDiarmid	90c4f11b61	go get github.com/hashicorp/go-getter@v1.5.0 ... Includes fix for breaking upstream API change.	6 years ago
Martin Atkins	e6e0b6ee46	providercache: verify locked hashes for local package dirs ... Previously we were only verifying locked hashes for local archive zip files, but if we have non-ziphash hashes available then we can and should also verify that a local directory matches at least one of them. This does mean that folks using filesystem mirrors but yet also running Terraform across multiple platforms will need to take some extra care to ensure the hashes pass on all relevant platforms, which could mean using "terraform providers lock" to pre-seed their lock files with hashes across all platforms, or could mean using the "packed" directory layout for the filesystem mirror so that Terraform will end up in the install-from-archive codepath instead of this install-from-directory codepath, and can thus verify ziphash too. (There's no additional documentation about the above here because there's already general information about this in the lock file documentation due to some similar -- though not identical -- situations with network mirrors.)	6 years ago
Martin Atkins	b3f5c7f1e6	command/init: Read, respect, and update provider dependency locks ... This changes the approach used by the provider installer to remember between runs which selections it has previously made, using the lock file format implemented in internal/depsfile. This means that version constraints in the configuration are considered only for providers we've not seen before or when -upgrade mode is active.	6 years ago
Kristin Laemmert	04be220f5f	deprecate helper/copy ... helper/copy CopyDir was used heavily in tests. It differes from internal/copydir in a few ways, the main one being that it creates the dst directory while the internal version expected the dst to exist (there are other differences, which is why I did not just switch tests to using internal's CopyDir). I moved the CopyDir func from helper/copy into command_test.go; I could also have moved it into internal/copy and named it something like CreateDirAndCopy so if that seems like a better option please let me know. helper/copy/CopyFile was used in a couple of spots so I moved it into internal, at which point I thought it made more sense to rename the package copy (instead of copydir). There's also a `go mod tidy` included.	6 years ago
Martin Atkins	0b734a2803	command: Make provider installation interruptible ... In earlier commits we started to make the installation codepath context-aware so that it could be canceled in the event of a SIGINT, but we didn't complete wiring that through the API of the getproviders package. Here we make the getproviders.Source interface methods, along with some other functions that can make network requests, take a context.Context argument and act appropriately if that context is cancelled. The main providercache.Installer.EnsureProviderVersions method now also has some context-awareness so that it can abort its work early if its context reports any sort of error. That avoids waiting for the process to wind through all of the remaining iterations of the various loops, logging each request failure separately, and instead returns just a single aggregate "canceled" error. We can then set things up in the "terraform init" and "terraform providers mirror" commands so that the context will be cancelled if we get an interrupt signal, allowing provider installation to abort early while still atomically completing any local-side effects that may have started.	6 years ago
Alisdair McDiarmid	915f53af23	internal: Clean up package install temp file ... The installFromHTTPURL function downloads a package to a temporary file, then delegates to installFromLocalArchive to install it. We were previously not deleting the temporary file afterwards. This commit fixes that.	6 years ago
Alisdair McDiarmid	440543f427	internal/providercache: Fix bug when symlink fails ... When installing a provider which is already cached, we attempt to create a symlink from the install directory targeting the cache. If symlinking fails due to missing OS/filesystem support, we instead want to copy the cached provider. The fallback code to do this would always fail, due to a missing target directory. This commit fixes that. I was unable to find a way to add automated tests around this, but I have manually verified the fix on Windows 8.1.	6 years ago
Alisdair McDiarmid	a5b3d497cc	internal: Verify provider signatures on install ... Providers installed from the registry are accompanied by a list of checksums (the "SHA256SUMS" file), which is cryptographically signed to allow package authentication. The process of verifying this has multiple steps: - First we must verify that the SHA256 hash of the package archive matches the expected hash. This could be done for local installations too, in the future. - Next we ensure that the expected hash returned as part of the registry API response matches an entry in the checksum list. - Finally we verify the cryptographic signature of the checksum list, using the public keys provided by the registry. Each of these steps is implemented as a separate PackageAuthentication type. The local archive installation mechanism uses only the archive checksum authenticator, and the HTTP installation uses all three in the order given. The package authentication system now also returns a result value, which is used by command/init to display the result of the authentication process. There are three tiers of signature, each of which is presented differently to the user: - Signatures from the embedded HashiCorp public key indicate that the provider is officially supported by HashiCorp; - If the signing key is not from HashiCorp, it may have an associated trust signature, which indicates that the provider is from one of HashiCorp's trusted partners; - Otherwise, if the signature is valid, this is a community provider.	6 years ago
Martin Atkins	ff55e1a1cd	internal/providercache: installFromLocalDir ... We previously skipped this one because it wasn't strictly necessary for replicating the old "terraform init" behavior, but we do need it to work so that things like the -plugin-dir option can behave correctly. Linking packages from other cache directories and installing from unpacked directories are fundamentally the same operation because a cache directory is really just a collection of unpacked packages, so here we refactor the LinkFromOtherCache functionality to actually be in installFromLocalDir, and LinkFromOtherCache becomes a wrapper for the installFromLocalDir function that just calculates the source and target directories automatically and invalidates the metaCache.	6 years ago
Martin Atkins	807267d1b5	internal/providercache: Installation from HTTP URLs and local archives ... When a provider source produces an HTTP URL location we'll expect it to resolve to a zip file, which we'll first download to a temporary directory and then treat it like a local archive. When a provider source produces a local archive path we'll expect it to be a zip file and extract it into the target directory. This does not yet include an implementation of installing from an already-unpacked local directory. That will follow in a subsequent commit, likely following a similar principle as in Dir.LinkFromOtherCache.	6 years ago