126 Commits (v0.13)

Author	SHA1	Message	Date
James Bardin	5ebfcc7a17	update getproviders tests for new key	5 years ago
James Bardin	1d28780973	update HC PublicKey	5 years ago
Alisdair McDiarmid	fc7e467d19	command: Add redirect support to 0.13upgrade ... If a provider changes namespace in the registry, we can detect this when running the 0.13upgrade command. As long as there is a version matching the user's constraints, we now use the provider's new source address. Otherwise, warn the user that the provider has moved and a version upgrade is necessary to move to it.	6 years ago
Alan D. Salewski	f63c38d6a1	internal/initwd: allow tests to pass when $PWD contains symlinks (#26015) ... Fixes #26014	6 years ago
Martin Atkins	2bd2a9a923	internal/getproviders: HTTPMirrorSource implementation ... We previously had this just stubbed out because it was a stretch goal for the v0.13.0 release and it ultimately didn't make it in. Here we fill out the existing stub -- with a minor change to its interface so it can access credentials -- with a client implementation that is compatible with the directory structure produced by the "terraform providers mirror" subcommand, were the result to be published on a static file server.	6 years ago
Martin Atkins	146e983c36	internal/getproviders: package authenticator for our new-style hashes ... Earlier we introduced a new package hashing mechanism that is compatible with both packed and unpacked packages, because it's a hash of the contents of the package rather than of the archive it's delivered in. However, we were using that only for the local selections file and not for any remote package authentication yet. The provider network mirrors protocol includes new-style hashes as a step towards transitioning over to the new hash format in all cases, so this new authenticator is here in preparation for verifying the checksums of packages coming from network mirrors, for mirrors that support them. For now this leaves us in a kinda confusing situation where we have both NewPackageHashAuthentication for the new style and NewArchiveChecksumAuthentication for the old style, which for the moment is represented only by a doc comment on the latter. Hopefully we can remove NewArchiveChecksumAuthentication in a future commit, if we can get the registry updated to use the new hashing format.	6 years ago
Alisdair McDiarmid	915f53af23	internal: Clean up package install temp file ... The installFromHTTPURL function downloads a package to a temporary file, then delegates to installFromLocalArchive to install it. We were previously not deleting the temporary file afterwards. This commit fixes that.	6 years ago
Martin Atkins	ce67a818db	internal/getproviders: Allow basedir for local search to be symlink ... The SearchLocalDirectory function was intentionally written to only support symlinks at the leaves so that it wouldn't risk getting into an infinite loop traversing intermediate symlinks, but that rule was also applying to the base directory itself. It's pretty reasonable to put your local plugins in some location Terraform wouldn't normally search (e.g. because you want to get them from a shared filesystem mounted somewhere) and creating a symlink from one of the locations Terraform _does_ search is a convenient way to help Terraform find those without going all in on the explicit provider installation methods configuration that is intended for more complicated situations. To allow for that, here we make a special exception for the base directory, resolving that first before we do any directory walking. In order to help with debugging a situation where there are for some reason symlinks at intermediate levels inside the search tree, we also now emit a WARN log line in that case to be explicit that symlinks are not supported there and to hint to put the symlink at the top-level if you want to use symlinks at all. (The support for symlinks at the deepest level of search is not mentioned in this message because we allow it primarily for our own cache linking behavior.)	6 years ago
Alisdair McDiarmid	440543f427	internal/providercache: Fix bug when symlink fails ... When installing a provider which is already cached, we attempt to create a symlink from the install directory targeting the cache. If symlinking fails due to missing OS/filesystem support, we instead want to copy the cached provider. The fallback code to do this would always fail, due to a missing target directory. This commit fixes that. I was unable to find a way to add automated tests around this, but I have manually verified the fix on Windows 8.1.	6 years ago
Alisdair McDiarmid	3b1347ac1a	providercache: Validate provider executable file ... At the end of the EnsureProviderVersions process, we generate a lockfile of the selected and installed provider versions. This includes a hash of the unpacked provider directory. When calculating this hash and generating the lockfile, we now also verify that the provider directory contains a valid executable file. If not, we return an error for this provider and trigger the installer's HashPackageFailure event. Note that this event is not yet processed by terraform init; that comes in the next commit.	6 years ago
Alisdair McDiarmid	a18b531b14	getproviders: FakeInstallablePackageMeta filename ... Add an optional execFilename argument to the test helper function FakeInstallablePackageMeta, which allows the creation of invalid packages.	6 years ago
Alisdair McDiarmid	8e87ccb689	providercache: Lazily detect executable file ... Instead of searching the installed provider package directory for a binary as we install it, we can lazily detect the executable as it is required. Doing so allows us to separately report an invalid unpacked package, giving the user more actionable error messages.	6 years ago
Kristin Laemmert	47e657c611	internal/getproviders: decode and return any registry warnings (#25337) ... * internal/getproviders: decode and return any registry warnings The public registry may include a list of warnings in the "versions" response for any given provider. This PR adds support for warnings from the registry and an installer event to return those warnings to the user.	6 years ago
Kristin Laemmert	1b8f4566fa	internal/initwd: fix panics with relative submodules in DirFromModule (#25250) ... * internal/initwd: fix panics with relative submodules in DirFromModule There were two related issues here: 1. panic with any local module with submodules 1. panic with a relative directory that was above the workdir ("../") The first panic was caused by the local installer looking up the root module with the (nonexistant) key "root.", instead of "". The second panic was caused by the installer trying to determine the relative path from ".". This was fixed by detecting "." as the source path and using the absolute path for the call to filepath.Rel. Added test cases for both panics and updated the existing e2e tests with the correct install paths.	6 years ago
Lars Lehtonen	3ddfa66ca4	internal/modsdir: Fix Dropped Error (#24600) ... * internal/modsdir: fix dropped error * fix typo to unmarshalling Co-authored-by: Daniel Dreier <danieldreier@users.noreply.github.com>	6 years ago
Martin Atkins	85af77386c	internal/getproviders: PackageFilePathForPackage ... This is the equivalent of UnpackedDirectoryPathForPackage when working with the packed directory layout. It returns a path to a .zip file with a name that would be detected by SearchLocalDirectory as a PackageLocalArchive package.	6 years ago
Martin Atkins	9489672d54	internal/getproviders: Package hashing for local filesystem packages ... We previously had this functionality available for cached packages in the providercache package. This moves the main implementation of this over to the getproviders package and then implements it also for PackageMeta, allowing us to compute hashes in a consistent way across both of our representations of a provider package. The new methods on PackageMeta will only be effective for packages in the local filesystem because we need direct access to the contents in order to produce the hash. Hopefully in future the registry protocol will be able to also provide hashes using this content-based (rather than archive-based) algorithm and then we'll be able to make this work for PackageMeta referring to a package obtained from a registry too, but hashes for local packages only are still useful for some cases right now, such as generating mirror directories in the "terraform providers mirror" command.	6 years ago
Alisdair McDiarmid	ef28671b34	Merge pull request #24932 from hashicorp/signing-language ... Modify language for reporting signing state	6 years ago
Paddy	5127f1ef8b	command: Unmanaged providers ... This adds supports for "unmanaged" providers, or providers with process lifecycles not controlled by Terraform. These providers are assumed to be started before Terraform is launched, and are assumed to shut themselves down after Terraform has finished running. To do this, we must update the go-plugin dependency to v1.3.0, which added support for the "test mode" plugin serving that powers all this. As a side-effect of not needing to manage the process lifecycle anymore, Terraform also no longer needs to worry about the provider's binary, as it won't be used for anything anymore. Because of this, we can disable the init behavior that concerns itself with downloading that provider's binary, checking its version, and otherwise managing the binary. This is all managed on a per-provider basis, so managed providers that Terraform downloads, starts, and stops can be used in the same commands as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable is added, and is a JSON encoding of the provider's address to the information we need to connect to it. This change enables two benefits: first, delve and other debuggers can now be attached to provider server processes, and Terraform can connect. This allows for attaching debuggers to provider processes, which before was difficult to impossible. Second, it allows the SDK test framework to host the provider in the same process as the test driver, while running a production Terraform binary against the provider. This allows for Go's built-in race detector and test coverage tooling to work as expected in provider tests. Unmanaged providers are expected to work in the exact same way as managed providers, with one caveat: Terraform kills provider processes and restarts them once per graph walk, meaning multiple times during most Terraform CLI commands. As unmanaged providers can't be killed by Terraform, and have no visibility into graph walks, unmanaged providers are likely to have differences in how their global mutable state behaves when compared to managed providers. Namely, unmanaged providers are likely to retain global state when managed providers would have reset it. Developers relying on global state should be aware of this.	6 years ago
Paul Tyng	22ef5cc99c	Modify language for reporting signing state ... Be more explicit about the signing status of fetched plugins and provide documentation about the different signing options.	6 years ago
Alisdair McDiarmid	62d826e066	command/init: Use full config for provider reqs ... Relying on the early config for provider requirements was necessary in Terraform 0.12, to allow the 0.12upgrade command to run after init installs providers. However in 0.13, the same restrictions do not apply, and the detection of provider requirements has changed. As a result, the early config loader gives incorrect provider requirements in some circumstances, such as those in the new test in this commit. Therefore we are changing the init command to use the requirements found by the full configuration loader. This also means that we can remove the internal initwd CheckCoreVersionRequirements function.	6 years ago
Kristin Laemmert	eead4c49fe	command/init: add e2e tests for provider not found messages	6 years ago
Kristin Laemmert	8d28d73de3	getproviders: add a registry-specific error and modify output when a ... provider is not found. Previously a user would see the following error even if terraform was only searching the local filesystem: "provider registry registry.terraform.io does not have a provider named ...." This PR adds a registry-specific error type and modifies the MultiSource installer to check for registry errors. It will return the registry-specific error message if there is one, but if not the error message will list all locations searched.	6 years ago
Kristin Laemmert	0d620018fe	provider cache: log errors and validate dir exists (#24993) ... * providercache: add logging for errors from getproviders.SearchLocalDirectory providercache.fillMetaCache() was silently swallowing errors when searching the cache directory. This commit logs the error without changing the behavior otherwise. * command/cliconfig: validate plugin cache dir exists The plugin cache directory must exist for terraform to use it, so we will add a check at the begining.	6 years ago
Kristin Laemmert	d350818126	internal/getproviders: fix panic with invalid path parts (#24940) ... * internal/getproviders: fix panic with invalid path parts If the search path is missing a directory, the provider installer would try to create an addrs.Provider with the wrong parts. For example if the hostname was missing (as in the test case), it would call addrs.NewProvider with (namespace, typename, version). This adds a validation step for each part before calling addrs.NewProvider to avoid the panic.	6 years ago
Alisdair McDiarmid	070c3018f8	internal/providercache: Remove unused retry events	6 years ago
Alisdair McDiarmid	e27a36cafd	internal/getproviders: Retry failed HTTP requests ... This is a port of the retry/timeout logic added in #24260 and #24259, using the same environment variables to configure the retry and timeout settings.	6 years ago
Kristin Laemmert	60321b41e8	getproviders: move protocol compatibility functions into registry client (#24846) ... * internal/registry source: return error if requested provider version protocols are not supported * getproviders: move responsibility for protocol compatibility checks into the registry client The original implementation had the providercache checking the provider metadata for protocol compatibility, but this is only relevant for the registry source so it made more sense to move the logic into getproviders. This also addresses an issue where we were pulling the metadata for every provider version until we found one that was supported. I've extended the registry client to unmarshal the protocols in `ProviderVersions` so we can filter through that list, instead of pulling each version's metadata.	6 years ago
Alisdair McDiarmid	62b0cbed12	internal: Fix LookupLegacyProvider ... When looking up the namespace for a legacy provider source, we need to use the /v1/providers/-/{name}/versions endpoint. For non-HashiCorp providers, the /v1/providers/-/{name} endpoint returns a 404. This commit updates the LegacyProviderDefaultNamespace method and the mock registry servers accordingly.	6 years ago
Alisdair McDiarmid	ae98bd12a7	command: Rework 0.13upgrade sub-command ... This commit implements most of the intended functionality of the upgrade command for rewriting configurations. For a given module, it makes a list of all providers in use. Then it attempts to detect the source address for providers without an explicit source. Once this step is complete, the tool rewrites the relevant configuration files. This results in a single "required_providers" block for the module, with a source for each provider. Any providers for which the source cannot be detected (for example, unofficial providers) will need a source to be defined by the user. The tool writes an explanatory comment to the configuration to help with this.	6 years ago
Kristin Laemmert	cca0526705	providercache: actually break out of the loop when a matching version is found (#24823)	6 years ago
Kristin Laemmert	ce03f1255f	internal/providercache: fix error message for protocol mismatch (#24818) ... There was a bug in the installer trying to pass a nil error.	6 years ago
Kristin Laemmert	320fcf4942	internal/getproviders: apply case normalizations in ParseMultiSourceMatchingPatterns (#24753) ... * internal/getproviders: apply case normalizations in ParseMultiSourceMatchingPatterns This is a very minor refactor which takes advantage of addrs.ParseProviderPart case normalization to normalize non-wildcard sources.	6 years ago
Martin Atkins	1ce3c60693	command/cliconfig: Explicit provider installation method configuration ... This set of commits allows explicit configuration of provider installation methods in the CLI config, overriding the implicit method selections.	6 years ago
Martin Atkins	8b75d1498f	command/cliconfig: Use existing HTTP mirror source rather than new stub ... An earlier commit added a redundant stub for a new network mirror source that was already previously stubbed as HTTPMirrorSource. This commit removes the unnecessary extra stub and changes the CLI config handling to use it instead. Along the way this also switches to using a full base URL rather than just a hostname for the mirror, because using the usual "Terraform-native service discovery" protocol here doesn't isn't as useful as in the places we normally use it (the mirror mechanism is already serving as an indirection over the registry protocol) and using a direct base URL will make it easier to deploy an HTTP mirror under a path prefix on an existing static file server.	6 years ago
Kristin Laemmert	84e9d86c25	Update installer_test.go ... update now-exported function (fix bad PR)	6 years ago
Kristin Laemmert	21b9da5a02	internal/providercache: verify that the provider protocol version is compatible (#24737) ... * internal/providercache: verify that the provider protocol version is compatible The public registry includes a list of supported provider protocol versions for each provider version. This change adds verification of support and adds a specific error message pointing users to the closest matching version.	6 years ago
Alisdair McDiarmid	54abb87fb4	Fix broken test due to function rename	6 years ago
Martin Atkins	c5bd783eba	internal/getproviders: Stub NetworkMirrorSource ... This is a placeholder for later implementation of a mirror source that talks to a particular remote HTTP server and expects it to implement the provider mirror protocol.	6 years ago
Kristin Laemmert	a43f141f9e	tools/terraform-bundle: refactor to use new provider installer and provider directory layouts (#24629) ... * tools/terraform-bundle: refactor to use new provider installer and provider directory layouts terraform-bundle now supports a "source" attribute for providers, uses the new provider installer, and the archive it creates preserves the new (required) directory hierarchy for providers, under a "plugins" directory. This is a breaking change in many ways: source is required for any non-HashiCorp provider, locally-installed providers must be given a source (can be arbitrary, see docs) and placed in the expected directory hierarchy, and the unzipped archive is no longer flat; there is a new "plugins" directory created with providers in the new directory layout. This PR also extends the existing test to check the contents of the zip file. TODO: Re-enable e2e tests (currently suppressed with a t.Skip) This commit includes an update to our travis configuration, so the terraform-bundle e2e tests run. It also turns off the e2e tests, which will fail until we have a terraform 0.13.* release under releases.hashicorp.com. We decided it was better to merge this now instead of waiting when we started seeing issues opened from users who built terraform-bundle from 0.13 and found it didn't work with 0.12 - better that they get an immediate error message from the binary directing them to build from the appropriate release.	6 years ago
Alisdair McDiarmid	a5b3d497cc	internal: Verify provider signatures on install ... Providers installed from the registry are accompanied by a list of checksums (the "SHA256SUMS" file), which is cryptographically signed to allow package authentication. The process of verifying this has multiple steps: - First we must verify that the SHA256 hash of the package archive matches the expected hash. This could be done for local installations too, in the future. - Next we ensure that the expected hash returned as part of the registry API response matches an entry in the checksum list. - Finally we verify the cryptographic signature of the checksum list, using the public keys provided by the registry. Each of these steps is implemented as a separate PackageAuthentication type. The local archive installation mechanism uses only the archive checksum authenticator, and the HTTP installation uses all three in the order given. The package authentication system now also returns a result value, which is used by command/init to display the result of the authentication process. There are three tiers of signature, each of which is presented differently to the user: - Signatures from the embedded HashiCorp public key indicate that the provider is officially supported by HashiCorp; - If the signing key is not from HashiCorp, it may have an associated trust signature, which indicates that the provider is from one of HashiCorp's trusted partners; - Otherwise, if the signature is valid, this is a community provider.	6 years ago
Kristin Laemmert	f09ae6f862	provider source tests: added test suite to exercise hyphenated providers (#24685)	6 years ago
Martin Atkins	6ec5e0fc88	internal/getproviders: Tests for MemoizeSource ... Due to other pressures at the time this was implemented, it was tested only indirectly through integration tests in other packages. This now introduces tests for the two main entry points on MemoizeSource.	6 years ago
Martin Atkins	c6535a0fa4	internal/getproviders: Tests for MultiSource ... Due to other pressures at the time this was implemented, it was tested only indirectly through integration tests in other packages. This now introduces tests for the two main entry points on the MultiSource, along with its provider-address pattern matching logic. This does not yet include thorough tests for ParseMultiSourceMatchingPatterns, because that function still needs some adjustments to do the same case folding as for normal provider address parsing, which will follow in a latter commit along with suitable tests. With that said, the tests added here do _indirectly_ test the happy path of ParseMultiSourceMatchingPatterns, so we have some incomplete testing of that function in the meantime.	6 years ago
Martin Atkins	0ad4c1be2f	internal/getproviders: Tidy up some confusion about package hashes ... Earlier on in the stubbing of this package we realized that it wasn't going to be possible to populate the authentication-related bits for all packages because the relevant metadata just isn't available for packages that are already local. However, we just moved ahead with that awkward design at the time because we needed to get other work done, and so we've been mostly producing PackageMeta values with all-zeros hashes and just ignoring them entirely as a temporary workaround. This is a first step towards what is hopefully a more intuitive model: authentication is an optional thing in a PackageMeta that is currently populated only for packages coming from a registry. So far this still just models checking a SHA256 hash, which is not a sufficient set of checks for a real release but hopefully the "real" implementation is a natural iteration of this starting point, and if not then at least this interim step is a bit more honest about the fact that Authentication will not be populated on every PackageMeta.	6 years ago
Alisdair McDiarmid	0f5a38b384	internal: Fix init provider lockfile test ... The fake installable package meta used a ZIP archive which gave different checksums between macOS and Linux targets. This commit removes the target from the contents of this archive, and updates the golden hash value in the test to match. This test should now pass on both platforms.	6 years ago
Martin Atkins	958ea4f7d1	internal/providercache: Handle built-in providers ... Built-in providers are special providers that are distributed as part of Terraform CLI itself, rather than being installed separately. They always live in the terraform.io/builtin/... namespace so it's easier to see that they are special, and currently there is only one built-in provider named "terraform". Previous commits established the addressing scheme for built-in providers. This commit makes the installer aware of them to the extent that it knows not to try to install them the usual way and it's able to report an error if the user requests a built-in provider that doesn't exist or tries to impose a particular version constraint for a built-in provider. For the moment the tests for this are the ones in the "command" package because that's where the existing testing infrastructure for this functionality lives. A later commit should add some more focused unit tests here in the internal/providercache package, too.	6 years ago
Martin Atkins	7caf0b9246	addrs: ImpliedProviderForUnqualifiedType function ... This encapsulates the logic for selecting an implied FQN for an unqualified type name, which could either come from a local name used in a module without specifying an explicit source for it or from the prefix of a resource type on a resource that doesn't explicitly set "provider". This replaces the previous behavior of just directly calling NewDefaultProvider everywhere so that we can use a different implication for the local name "terraform", to refer to the built-in terraform provider rather than the stale one that's on registry.terraform.io for compatibility with other Terraform versions.	6 years ago
Martin Atkins	f35ebe2d65	internal/providercache: Fix incorrect logic in Installer.SetGlobalCacheDir ... Due to some incomplete rework of this function in an earlier commit, the safety check for using the same directory as both the target and the cache was inverted and was raising an error _unless_ they matched, rather than _if_ they matched. This change is verified by the e2etest TestInitProviders_pluginCache, which is also updated to use the new-style cache directory layout as part of this commit.	6 years ago
Martin Atkins	ff55e1a1cd	internal/providercache: installFromLocalDir ... We previously skipped this one because it wasn't strictly necessary for replicating the old "terraform init" behavior, but we do need it to work so that things like the -plugin-dir option can behave correctly. Linking packages from other cache directories and installing from unpacked directories are fundamentally the same operation because a cache directory is really just a collection of unpacked packages, so here we refactor the LinkFromOtherCache functionality to actually be in installFromLocalDir, and LinkFromOtherCache becomes a wrapper for the installFromLocalDir function that just calculates the source and target directories automatically and invalidates the metaCache.	6 years ago