74 Commits (f-debugger)

Author	SHA1	Message	Date
Pam Selle	242f319638	Give suggestions & remind users to use required_providers when provider not in registry (#28014) ... * Add helper suggestion when failed registry err When someone has a failed registry error on init, remind them that they should have required_providers in every module * Give suggestion for a provider based on reqs Suggest another provider on a registry error, from the list of requirements we have on init. This skips the legacy lookup process if there is a similar provider existing in requirements.	5 years ago
Alex Chan	4ec80fa18d	Correct the spelling of heirarchy/hierarchy throughout	5 years ago
Romain Philibert	5ae5c99ba0	fix: do not fail when provider symlink is broken. Only print a warning. ... fix https://github.com/hashicorp/terraform/issues/27445	5 years ago
Martin Atkins	4b3e237668	command/init: Hint about providers in other namespaces ... If a user forgets to specify the source address for a provider, Terraform will assume they meant a provider in the registry.terraform.io/hashicorp/ namespace. If that ultimately doesn't exist, we'll now try to see if there's some other provider source address recorded in the registry's legacy provider lookup table, and suggest it if so. The error message here is a terse one addressed primarily to folks who are already somewhat familiar with provider source addresses and how to specify them. Terraform v0.13 had a more elaborate version of this error message which directed the user to try the v0.13 automatic upgrade tool, but we no longer have that available in v0.14 and later so the user must make the fix themselves.	6 years ago
James Bardin	f0c3ddf5a1	internal/getproviders: staticcheck	6 years ago
Alisdair McDiarmid	b1bc0e5d92	getproviders: Normalize versions before dedupe ... When rendering a set of version constraints to a string, we normalize partially-constrained versions. This means converting a version like 2.68.* to 2.68.0. Prior to this commit, this normalization was done after deduplication. This could result in a version constraints string with duplicate entries, if multiple partially-constrained versions are equivalent. This commit fixes this by normalizing before deduplicating and sorting.	6 years ago
Martin Atkins	430318e262	getproviders: Consistent ordering of terms in VersionConstraintsString ... An earlier commit made this remove duplicates, which set the precedent that this function is trying to canonically represent the _meaning_ of the version constraints rather than exactly how they were expressed in the configuration. Continuing in that vein, now we'll also apply a consistent (though perhaps often rather arbitrary) ordering to the terms, so that it doesn't change due to irrelevant details like declarations being written in a different order in the configuration. The ordering here is intended to be reasonably intuitive for simple cases, but constraint strings with many different constraints are hard to interpret no matter how we order them so the main goal is consistency, so those watching how the constraints change over time (e.g. in logs of Terraform output, or in the dependency log file) will see fewer noisy changes that don't actually mean anything.	6 years ago
Alisdair McDiarmid	fe9a9fadfb	internal: Suppress duplicate version constraints ... A set of version constraints can contain duplicates. This can happen if multiple identical constraints are specified throughout a configuration. When rendering the set, it is confusing to display redundant constraints. This commit changes the string renderer to only show the first instance of a given constraint, and adds unit tests for this function to cover this change. This also fixes a bug with the locks file generation: previously, a configuration with redundant constraints would result in this error on second init: Error: Invalid provider version constraints on .terraform.lock.hcl line 6: (source code not available) The recorded version constraints for provider registry.terraform.io/hashicorp/random must be written in normalized form: "3.0.0".	6 years ago
Alisdair McDiarmid	7a31e56cb7	Merge pull request #26637 from hashicorp/alisdair/fix-locksfile-unconstrained-versions ... internal: Fix lockfile constraint output for 1.2.*	6 years ago
Martin Atkins	1fe314b70b	getproviders: Don't log if EvalSymlinks changes nothing ... Previously this codepath was generating a confusing message in the absense of any symlinks, because filepath.EvalSymlinks returns a successful result if the target isn't a symlink. Now we'll emit the log line only if filepath.EvalSymlinks returns a result that's different in a way that isn't purely syntactic (which filepath.Clean would "fix"). The new message is a little more generic because technically we've not actually ensured that a difference here was caused by a symlink and so we shouldn't over-promise and generate something potentially misleading.	6 years ago
Alisdair McDiarmid	9576a5b2d8	internal: Fix lockfile constraint output for 1.2.* ... If a configuration requires a partial provider version (with some parts unspecified), Terraform considers this as a constrained-to-zero version. For example, a version constraint of 1.2 will result in an attempt to install version 1.2.0, even if 1.2.1 is available. When writing the dependency locks file, we previously would write 1.2.*, as this is the in-memory representation of 1.2. This would then cause an error on re-reading the locks file, as this is not a valid constraint format. Instead, we now explicitly convert the constraint to its zero-filled representation before writing the locks file. This ensures that it correctly round-trips. Because this change is made in getproviders.VersionConstraintsString, it also affects the output of the providers sub-command.	6 years ago
James Bardin	0b31ffa587	use a single log writer ... Use a single log writer instance for all std library logging. Setup the std log writer in the logging package, and remove boilerplate from test packages.	6 years ago
James Bardin	6ca477f042	move helper/logging to internal ... remove a dead code file too	6 years ago
Martin Atkins	b3f5c7f1e6	command/init: Read, respect, and update provider dependency locks ... This changes the approach used by the provider installer to remember between runs which selections it has previously made, using the lock file format implemented in internal/depsfile. This means that version constraints in the configuration are considered only for providers we've not seen before or when -upgrade mode is active.	6 years ago
Martin Atkins	59b116f7bf	command/init: Remove support for legacy provider addresses ... We no longer need to support 0.12-and-earlier-style provider addresses because users should've upgraded their existing configurations and states on Terraform 0.13 already. For now this is only checked in the "init" command, because various test shims are still relying on the idea of legacy providers the core layer. However, rejecting these during init is sufficient grounds to avoid supporting legacy provider addresses in the new dependency lock file format, and thus sets the stage for a more severe removal of legacy provider support in a later commit.	6 years ago
Martin Atkins	0b734a2803	command: Make provider installation interruptible ... In earlier commits we started to make the installation codepath context-aware so that it could be canceled in the event of a SIGINT, but we didn't complete wiring that through the API of the getproviders package. Here we make the getproviders.Source interface methods, along with some other functions that can make network requests, take a context.Context argument and act appropriately if that context is cancelled. The main providercache.Installer.EnsureProviderVersions method now also has some context-awareness so that it can abort its work early if its context reports any sort of error. That avoids waiting for the process to wind through all of the remaining iterations of the various loops, logging each request failure separately, and instead returns just a single aggregate "canceled" error. We can then set things up in the "terraform init" and "terraform providers mirror" commands so that the context will be cancelled if we get an interrupt signal, allowing provider installation to abort early while still atomically completing any local-side effects that may have started.	6 years ago
Martin Atkins	ef64df950c	getproviders: Prepare for having multiple valid hashes per package ... As we continue iterating towards saving valid hashes for a package in a depsfile lock file after installation and verifying them on future installation, this prepares getproviders for the possibility of having multiple valid hashes per package. This will arise in future commits for two reasons: - We will need to support both the legacy "zip hash" hashing scheme and the new-style content-based hashing scheme because currently the registry protocol is only able to produce the legacy scheme, but our other installation sources prefer the content-based scheme. Therefore packages will typically have a mixture of hashes of both types. - Installing from an upstream registry will save the hashes for the packages across all supported platforms, rather than just the current platform, and we'll consider all of those valid for future installation if we see both successful matching of the current platform checksum and a signature verification for the checksums file as a whole. This also includes some more preparation for the second case above in that signatureAuthentication now supports AcceptableHashes and returns all of the zip-based hashes it can find in the checksums file. This is a bit of an abstraction leak because previously that authenticator considered its "document" to just be opaque bytes, but we want to make sure that we can only end up trusting _all_ of the hashes if we've verified that the document is signed. Hopefully we'll make this better in a future commit with some refactoring, but that's deferred for now in order to minimize disruption to existing codepaths while we work towards a provider locking MVP.	6 years ago
Martin Atkins	6694cfaa0e	getproviders: Add a real type Hash for package hashes ... The logic for what constitutes a valid hash and how different hash schemes are represented was starting to get sprawled over many different files and packages. Consistently with other cases where we've used named types to gather the definition of a particular string into a single place and have the Go compiler help us use it properly, this introduces both getproviders.Hash representing a hash value and getproviders.HashScheme representing the idea of a particular hash scheme. Most of this changeset is updating existing uses of primitive strings to uses of getproviders.Hash. The new type definitions are in internal/getproviders/hash.go.	6 years ago
Martin Atkins	264a3cf031	depsfile: Flatten the "hashes" locks to a single set of strings ... Although origin registries return specific [filename, hash] pairs, our various different installation methods can't produce a structured mapping from platform to hash without breaking changes. Therefore, as a compromise, we'll continue to do platform-specific checks against upstream data in the cases where that's possible (installation from origin registry or network mirror) but we'll treat the lock file as just a flat set of equally-valid hashes, at least one of which must match after we've completed whatever checks we've made against the upstream-provided checksums/signatures. This includes only the minimal internal/getproviders updates required to make this compile. A subsequent commit will update that package to actually support the idea of verifying against multiple hashes.	6 years ago
Martin Atkins	b2c0ccdf96	internal/getproviders: Allow PackageMeta to carry acceptable hashes ... The "acceptable hashes" for a package is a set of hashes that the upstream source considers to be good hashes for checking whether future installs of the same provider version are considered to match this one. Because the acceptable hashes are a package authentication concern and they already need to be known (at least in part) to implement the authenticators, here we add AcceptableHashes as an optional extra method that an authenticator can implement. Because these are hashes chosen by the upstream system, the caller must make its own determination about their trustworthiness. The result of authentication is likely to be an input to that, for example by distrusting hashes produced by an authenticator that succeeds but doesn't report having validated anything.	6 years ago
Martin Atkins	e843097e52	internal/getproviders: Formalize the "ziphash" hashing scheme ... This is the pre-existing hashing scheme that was initially built for releases.hashicorp.com and then later reused for the provider registry protocol, which takes a SHA256 hash of the official distribution .zip file and formats it as lowercase hex. This is a non-ideal hash scheme because it works only for PackageLocalArchive locations, and so we can't verify package directories on local disk against such hashes. However, the registry protocol is now a compatibility constraint and so we're going to need to support this hashing scheme for the foreseeable future.	6 years ago
Martin Atkins	92723661d0	internal/depsfile: Loading locks from HCL files on disk ... This is the initial implementation of the parser/decoder portion of the new dependency lock file handler. It's currently dead code because the caller isn't written yet. We'll continue to build out this functionality here until we have the basic level of both load and save functionality before introducing this into the provider installer codepath.	6 years ago
Martin Atkins	6993ecb0a6	internal/getproviders: VersionConstraintsString for "~> 2" input ... The version constraint parser allows "~> 2", but it behavior is identical to "~> 2.0". Due to a quirk of the constraint parser (caused by the fact that it supports both Ruby-style and npm/cargo-style constraints), it ends up returning "~> 2" with the minor version marked as "unconstrained" rather than as zero, but that means the same thing as zero in this context anyway and so we'll prefer to stringify as "~> 2.0" so that we can be clearer about how Terraform is understanding that version constraint.	6 years ago
Alisdair McDiarmid	fc7e467d19	command: Add redirect support to 0.13upgrade ... If a provider changes namespace in the registry, we can detect this when running the 0.13upgrade command. As long as there is a version matching the user's constraints, we now use the provider's new source address. Otherwise, warn the user that the provider has moved and a version upgrade is necessary to move to it.	6 years ago
Martin Atkins	2bd2a9a923	internal/getproviders: HTTPMirrorSource implementation ... We previously had this just stubbed out because it was a stretch goal for the v0.13.0 release and it ultimately didn't make it in. Here we fill out the existing stub -- with a minor change to its interface so it can access credentials -- with a client implementation that is compatible with the directory structure produced by the "terraform providers mirror" subcommand, were the result to be published on a static file server.	6 years ago
Martin Atkins	146e983c36	internal/getproviders: package authenticator for our new-style hashes ... Earlier we introduced a new package hashing mechanism that is compatible with both packed and unpacked packages, because it's a hash of the contents of the package rather than of the archive it's delivered in. However, we were using that only for the local selections file and not for any remote package authentication yet. The provider network mirrors protocol includes new-style hashes as a step towards transitioning over to the new hash format in all cases, so this new authenticator is here in preparation for verifying the checksums of packages coming from network mirrors, for mirrors that support them. For now this leaves us in a kinda confusing situation where we have both NewPackageHashAuthentication for the new style and NewArchiveChecksumAuthentication for the old style, which for the moment is represented only by a doc comment on the latter. Hopefully we can remove NewArchiveChecksumAuthentication in a future commit, if we can get the registry updated to use the new hashing format.	6 years ago
Martin Atkins	ce67a818db	internal/getproviders: Allow basedir for local search to be symlink ... The SearchLocalDirectory function was intentionally written to only support symlinks at the leaves so that it wouldn't risk getting into an infinite loop traversing intermediate symlinks, but that rule was also applying to the base directory itself. It's pretty reasonable to put your local plugins in some location Terraform wouldn't normally search (e.g. because you want to get them from a shared filesystem mounted somewhere) and creating a symlink from one of the locations Terraform _does_ search is a convenient way to help Terraform find those without going all in on the explicit provider installation methods configuration that is intended for more complicated situations. To allow for that, here we make a special exception for the base directory, resolving that first before we do any directory walking. In order to help with debugging a situation where there are for some reason symlinks at intermediate levels inside the search tree, we also now emit a WARN log line in that case to be explicit that symlinks are not supported there and to hint to put the symlink at the top-level if you want to use symlinks at all. (The support for symlinks at the deepest level of search is not mentioned in this message because we allow it primarily for our own cache linking behavior.)	6 years ago
Alisdair McDiarmid	a18b531b14	getproviders: FakeInstallablePackageMeta filename ... Add an optional execFilename argument to the test helper function FakeInstallablePackageMeta, which allows the creation of invalid packages.	6 years ago
Kristin Laemmert	47e657c611	internal/getproviders: decode and return any registry warnings (#25337) ... * internal/getproviders: decode and return any registry warnings The public registry may include a list of warnings in the "versions" response for any given provider. This PR adds support for warnings from the registry and an installer event to return those warnings to the user.	6 years ago
Martin Atkins	85af77386c	internal/getproviders: PackageFilePathForPackage ... This is the equivalent of UnpackedDirectoryPathForPackage when working with the packed directory layout. It returns a path to a .zip file with a name that would be detected by SearchLocalDirectory as a PackageLocalArchive package.	6 years ago
Martin Atkins	9489672d54	internal/getproviders: Package hashing for local filesystem packages ... We previously had this functionality available for cached packages in the providercache package. This moves the main implementation of this over to the getproviders package and then implements it also for PackageMeta, allowing us to compute hashes in a consistent way across both of our representations of a provider package. The new methods on PackageMeta will only be effective for packages in the local filesystem because we need direct access to the contents in order to produce the hash. Hopefully in future the registry protocol will be able to also provide hashes using this content-based (rather than archive-based) algorithm and then we'll be able to make this work for PackageMeta referring to a package obtained from a registry too, but hashes for local packages only are still useful for some cases right now, such as generating mirror directories in the "terraform providers mirror" command.	6 years ago
Paul Tyng	22ef5cc99c	Modify language for reporting signing state ... Be more explicit about the signing status of fetched plugins and provide documentation about the different signing options.	6 years ago
Kristin Laemmert	eead4c49fe	command/init: add e2e tests for provider not found messages	6 years ago
Kristin Laemmert	8d28d73de3	getproviders: add a registry-specific error and modify output when a ... provider is not found. Previously a user would see the following error even if terraform was only searching the local filesystem: "provider registry registry.terraform.io does not have a provider named ...." This PR adds a registry-specific error type and modifies the MultiSource installer to check for registry errors. It will return the registry-specific error message if there is one, but if not the error message will list all locations searched.	6 years ago
Kristin Laemmert	d350818126	internal/getproviders: fix panic with invalid path parts (#24940) ... * internal/getproviders: fix panic with invalid path parts If the search path is missing a directory, the provider installer would try to create an addrs.Provider with the wrong parts. For example if the hostname was missing (as in the test case), it would call addrs.NewProvider with (namespace, typename, version). This adds a validation step for each part before calling addrs.NewProvider to avoid the panic.	6 years ago
Alisdair McDiarmid	e27a36cafd	internal/getproviders: Retry failed HTTP requests ... This is a port of the retry/timeout logic added in #24260 and #24259, using the same environment variables to configure the retry and timeout settings.	6 years ago
Kristin Laemmert	60321b41e8	getproviders: move protocol compatibility functions into registry client (#24846) ... * internal/registry source: return error if requested provider version protocols are not supported * getproviders: move responsibility for protocol compatibility checks into the registry client The original implementation had the providercache checking the provider metadata for protocol compatibility, but this is only relevant for the registry source so it made more sense to move the logic into getproviders. This also addresses an issue where we were pulling the metadata for every provider version until we found one that was supported. I've extended the registry client to unmarshal the protocols in `ProviderVersions` so we can filter through that list, instead of pulling each version's metadata.	6 years ago
Alisdair McDiarmid	62b0cbed12	internal: Fix LookupLegacyProvider ... When looking up the namespace for a legacy provider source, we need to use the /v1/providers/-/{name}/versions endpoint. For non-HashiCorp providers, the /v1/providers/-/{name} endpoint returns a 404. This commit updates the LegacyProviderDefaultNamespace method and the mock registry servers accordingly.	6 years ago
Alisdair McDiarmid	ae98bd12a7	command: Rework 0.13upgrade sub-command ... This commit implements most of the intended functionality of the upgrade command for rewriting configurations. For a given module, it makes a list of all providers in use. Then it attempts to detect the source address for providers without an explicit source. Once this step is complete, the tool rewrites the relevant configuration files. This results in a single "required_providers" block for the module, with a source for each provider. Any providers for which the source cannot be detected (for example, unofficial providers) will need a source to be defined by the user. The tool writes an explanatory comment to the configuration to help with this.	6 years ago
Kristin Laemmert	320fcf4942	internal/getproviders: apply case normalizations in ParseMultiSourceMatchingPatterns (#24753) ... * internal/getproviders: apply case normalizations in ParseMultiSourceMatchingPatterns This is a very minor refactor which takes advantage of addrs.ParseProviderPart case normalization to normalize non-wildcard sources.	6 years ago
Martin Atkins	1ce3c60693	command/cliconfig: Explicit provider installation method configuration ... This set of commits allows explicit configuration of provider installation methods in the CLI config, overriding the implicit method selections.	6 years ago
Martin Atkins	8b75d1498f	command/cliconfig: Use existing HTTP mirror source rather than new stub ... An earlier commit added a redundant stub for a new network mirror source that was already previously stubbed as HTTPMirrorSource. This commit removes the unnecessary extra stub and changes the CLI config handling to use it instead. Along the way this also switches to using a full base URL rather than just a hostname for the mirror, because using the usual "Terraform-native service discovery" protocol here doesn't isn't as useful as in the places we normally use it (the mirror mechanism is already serving as an indirection over the registry protocol) and using a direct base URL will make it easier to deploy an HTTP mirror under a path prefix on an existing static file server.	6 years ago
Kristin Laemmert	21b9da5a02	internal/providercache: verify that the provider protocol version is compatible (#24737) ... * internal/providercache: verify that the provider protocol version is compatible The public registry includes a list of supported provider protocol versions for each provider version. This change adds verification of support and adds a specific error message pointing users to the closest matching version.	6 years ago
Martin Atkins	c5bd783eba	internal/getproviders: Stub NetworkMirrorSource ... This is a placeholder for later implementation of a mirror source that talks to a particular remote HTTP server and expects it to implement the provider mirror protocol.	6 years ago
Alisdair McDiarmid	a5b3d497cc	internal: Verify provider signatures on install ... Providers installed from the registry are accompanied by a list of checksums (the "SHA256SUMS" file), which is cryptographically signed to allow package authentication. The process of verifying this has multiple steps: - First we must verify that the SHA256 hash of the package archive matches the expected hash. This could be done for local installations too, in the future. - Next we ensure that the expected hash returned as part of the registry API response matches an entry in the checksum list. - Finally we verify the cryptographic signature of the checksum list, using the public keys provided by the registry. Each of these steps is implemented as a separate PackageAuthentication type. The local archive installation mechanism uses only the archive checksum authenticator, and the HTTP installation uses all three in the order given. The package authentication system now also returns a result value, which is used by command/init to display the result of the authentication process. There are three tiers of signature, each of which is presented differently to the user: - Signatures from the embedded HashiCorp public key indicate that the provider is officially supported by HashiCorp; - If the signing key is not from HashiCorp, it may have an associated trust signature, which indicates that the provider is from one of HashiCorp's trusted partners; - Otherwise, if the signature is valid, this is a community provider.	6 years ago
Kristin Laemmert	f09ae6f862	provider source tests: added test suite to exercise hyphenated providers (#24685)	6 years ago
Martin Atkins	6ec5e0fc88	internal/getproviders: Tests for MemoizeSource ... Due to other pressures at the time this was implemented, it was tested only indirectly through integration tests in other packages. This now introduces tests for the two main entry points on MemoizeSource.	6 years ago
Martin Atkins	c6535a0fa4	internal/getproviders: Tests for MultiSource ... Due to other pressures at the time this was implemented, it was tested only indirectly through integration tests in other packages. This now introduces tests for the two main entry points on the MultiSource, along with its provider-address pattern matching logic. This does not yet include thorough tests for ParseMultiSourceMatchingPatterns, because that function still needs some adjustments to do the same case folding as for normal provider address parsing, which will follow in a latter commit along with suitable tests. With that said, the tests added here do _indirectly_ test the happy path of ParseMultiSourceMatchingPatterns, so we have some incomplete testing of that function in the meantime.	6 years ago
Martin Atkins	0ad4c1be2f	internal/getproviders: Tidy up some confusion about package hashes ... Earlier on in the stubbing of this package we realized that it wasn't going to be possible to populate the authentication-related bits for all packages because the relevant metadata just isn't available for packages that are already local. However, we just moved ahead with that awkward design at the time because we needed to get other work done, and so we've been mostly producing PackageMeta values with all-zeros hashes and just ignoring them entirely as a temporary workaround. This is a first step towards what is hopefully a more intuitive model: authentication is an optional thing in a PackageMeta that is currently populated only for packages coming from a registry. So far this still just models checking a SHA256 hash, which is not a sufficient set of checks for a real release but hopefully the "real" implementation is a natural iteration of this starting point, and if not then at least this interim step is a bit more honest about the fact that Authentication will not be populated on every PackageMeta.	6 years ago
Alisdair McDiarmid	0f5a38b384	internal: Fix init provider lockfile test ... The fake installable package meta used a ZIP archive which gave different checksums between macOS and Linux targets. This commit removes the target from the contents of this archive, and updates the golden hash value in the test to match. This test should now pass on both platforms.	6 years ago