99 Commits (2167f92096e280fd7924d874eeecd8d760fee585)

Author	SHA1	Message	Date
Alisdair McDiarmid	9f824c53a5	command: Better in-house provider install errors ... When init attempts to install a legacy provider required by state and fails, but another provider with the same type is successfully installed, this almost definitely means that the user is migrating an in-house provider. The solution here is to use the `terraform state replace-provider` subcommand. This commit makes that next step clearer, by detecting this specific case, and displaying a list of commands to fix the existing state provider references.	6 years ago
Alisdair McDiarmid	45437721c9	command: Add test for backend config validation	6 years ago
Alisdair McDiarmid	677aabc767	command: Fix backend config override validation ... When loading a backend config override file, init was doing two things wrong: - First, if the file failed to parse, we accidentally didn't return, which caused a panic due to the parsed body being nil; - Secondly, we were overzealous with the validation of the file, allowing only attributes. While most backend configs are attributes only, the enhanced remote backend body also contains a `workspaces` block, which we need to support here. This commit fixes the first bug with an early return and adds test cases for missing file and intentionally-blank filename (to clear the config). We also add a schema validation for the backend block, based on the backend schema itself. This requires constructing an HCL body schema so that we can call `Content` and check for diagnostic errors. The result is more useful errors when an invalid backend config override file is used, while also supporting the enhanced remote backend config fully. Does not include tests specific to the remote backend, because the mocking involved to allow the backend to fully initialize is too involved to be worth it.	6 years ago
Alisdair McDiarmid	c98f352dc8	terraform: Fix required version constraint diags ... If a module has multiple terraform.required_version constraints, any failures would point at the last constraint in the error diagnostics. If an earlier constraint was the actual problem, this leads to confusing errors like this: Error: Unsupported Terraform Core version on main.tf line 6, in terraform: 6: required_version = ">= 0.13.0" This configuration does not support Terraform version 0.13.0. The error was due to storing the declaration range of the constraint as a pointer to the contents of a loop variable, which was later overwritten in later iterations of the loop. Instead we now use HCL's handy Ptr() method to create a direct pointer to the range struct.	6 years ago
Kristin Laemmert	6621501ae3	state: remove deprecated state package (#25490) ... Most of the state package has been deprecated by the states package. This PR replaces all the references to the old state package that can be done simply - the low-hanging fruit. * states: move state.Locker to statemgr The state.Locker interface was a wrapper around a statemgr.Full, so moving this was relatively straightforward. * command: remove unnecessary use of state package for writing local terraform state files * move state.LocalState into terraform package state.LocalState is responsible for managing terraform.States, so it made sense (to me) to move it into the terraform package. * slight change of heart: move state.LocalState into clistate instead of terraform	6 years ago
Alisdair McDiarmid	53e587e1a6	Merge pull request #25504 from hashicorp/alisdair/post-install-cache-validate ... Add post-install provider cache validation and error reporting	6 years ago
Martin Atkins	80ab867e57	command/init: Remove special 0.12upgrade heuristic ... For Terraform v0.12 we introduced a special loading mode where we would use the 0.11-syntax-compatible "earlyconfig" package as a heuristic to identify situations where it was likely that the user was trying to use 0.11-only syntax that the upgrade tool might help with. However, as the language has moved on that is no longer a suitable heuristic in Terraform 0.13 and later: other new additions to the language can cause the main loader to disagree with earlyconfig, which would lead us to give poor advice about how to respond. Instead, we'll now return the same generic "there are errors" message in all syntax error cases. We have an extra message for errors in this case (as compared to other commands) because "terraform init" is usually the first command a new user interacts with and so this message gives some extra explanation about what "terraform init" will do with the configuration once it's valid. This also includes a reset control character in the output of the message as part of our ongoing mission to stop Terraform printing out whole paragraphs of colored text, which can often be hard to read for various reasons.	6 years ago
Alisdair McDiarmid	87d1fb4006	command/init: Display provider validation errors ... After installing providers, we validate the presence of an executable file, and generate a selected versions lockfile. If this process fails, notify the user. One possible cause for this is an invalid provider package with a missing or misnamed executable file.	6 years ago
Alisdair McDiarmid	a18b531b14	getproviders: FakeInstallablePackageMeta filename ... Add an optional execFilename argument to the test helper function FakeInstallablePackageMeta, which allows the creation of invalid packages.	6 years ago
Alisdair McDiarmid	8e87ccb689	providercache: Lazily detect executable file ... Instead of searching the installed provider package directory for a binary as we install it, we can lazily detect the executable as it is required. Doing so allows us to separately report an invalid unpacked package, giving the user more actionable error messages.	6 years ago
Kristin Laemmert	df244b87c2	command/init: return an error with invalid -backend-config files (#25411) ... * command/init: return an error with invalid -backend-config files The -backend-config flag expects a set of key-value pairs or a file containing key-value pairs. If the file instead contains a full backend configuration block, it was silently ignored. This commit adds a check for blocks in the file and returns an error if they are encountered. Fixes #24845 * emphasize backend configuration file in docs	6 years ago
Kristin Laemmert	47e657c611	internal/getproviders: decode and return any registry warnings (#25337) ... * internal/getproviders: decode and return any registry warnings The public registry may include a list of warnings in the "versions" response for any given provider. This PR adds support for warnings from the registry and an installer event to return those warnings to the user.	6 years ago
Alisdair McDiarmid	820ed48813	command: Fix init flags silent exit bug ... When using `-flag=value` with Powershell, unquoted values are broken into separate arguments. This means that the following command: terraform init -backend-config=./backend.conf is interpreted by Terraform as: terraform init -backend-config= ./backend.conf This results in an empty backend-config setting (which is semantically valid!) followed by a custom configuration path (pointing at a file). Due to a bug where we could exit without printing diagnostics, this would result in a silent failure that was very difficult to diagnose.	6 years ago
Alisdair McDiarmid	9263b28e99	command/init: Improve diags for legacy providers ... When initializing a configuration which refers to re-namespaced legacy providers, we attempt to detect this and display a diagnostic message. Previously this message would direct the user to run the 0.13upgrade command, but without specifying in which directories. This commit detects which modules are using the providers in question, and for local modules displays a list of upgrade commands which specify the source directories of these modules. For remote modules, we display a separate list noting that they need to be upgraded elsewhere, providing both the local module call name and the module source address.	6 years ago
Alisdair McDiarmid	ca40107066	command/init: Better diagnostics for provider 404s ... Fetching a default namespace provider from the public registry can result in 404 Not Found error. This might be caused by a previously- default provider moving to a new namespace, which means that the configuration needs to be upgraded to use an explicit provider source. This commit adds a more detailed diagnostic for this situation, suggesting that the intended provider might be in a new namespace. The recommended course of action is to run the 0.13upgrade command to generate the correct required_providers configuration.	6 years ago
Alisdair McDiarmid	62d826e066	command/init: Use full config for provider reqs ... Relying on the early config for provider requirements was necessary in Terraform 0.12, to allow the 0.12upgrade command to run after init installs providers. However in 0.13, the same restrictions do not apply, and the detection of provider requirements has changed. As a result, the early config loader gives incorrect provider requirements in some circumstances, such as those in the new test in this commit. Therefore we are changing the init command to use the requirements found by the full configuration loader. This also means that we can remove the internal initwd CheckCoreVersionRequirements function.	6 years ago
Kristin Laemmert	21b9da5a02	internal/providercache: verify that the provider protocol version is compatible (#24737) ... * internal/providercache: verify that the provider protocol version is compatible The public registry includes a list of supported provider protocol versions for each provider version. This change adds verification of support and adds a specific error message pointing users to the closest matching version.	6 years ago
Alisdair McDiarmid	a5b3d497cc	internal: Verify provider signatures on install ... Providers installed from the registry are accompanied by a list of checksums (the "SHA256SUMS" file), which is cryptographically signed to allow package authentication. The process of verifying this has multiple steps: - First we must verify that the SHA256 hash of the package archive matches the expected hash. This could be done for local installations too, in the future. - Next we ensure that the expected hash returned as part of the registry API response matches an entry in the checksum list. - Finally we verify the cryptographic signature of the checksum list, using the public keys provided by the registry. Each of these steps is implemented as a separate PackageAuthentication type. The local archive installation mechanism uses only the archive checksum authenticator, and the HTTP installation uses all three in the order given. The package authentication system now also returns a result value, which is used by command/init to display the result of the authentication process. There are three tiers of signature, each of which is presented differently to the user: - Signatures from the embedded HashiCorp public key indicate that the provider is officially supported by HashiCorp; - If the signing key is not from HashiCorp, it may have an associated trust signature, which indicates that the provider is from one of HashiCorp's trusted partners; - Otherwise, if the signature is valid, this is a community provider.	6 years ago
Kristin Laemmert	f09ae6f862	provider source tests: added test suite to exercise hyphenated providers (#24685)	6 years ago
Alisdair McDiarmid	0f5a38b384	internal: Fix init provider lockfile test ... The fake installable package meta used a ZIP archive which gave different checksums between macOS and Linux targets. This commit removes the target from the contents of this archive, and updates the golden hash value in the test to match. This test should now pass on both platforms.	6 years ago
Martin Atkins	958ea4f7d1	internal/providercache: Handle built-in providers ... Built-in providers are special providers that are distributed as part of Terraform CLI itself, rather than being installed separately. They always live in the terraform.io/builtin/... namespace so it's easier to see that they are special, and currently there is only one built-in provider named "terraform". Previous commits established the addressing scheme for built-in providers. This commit makes the installer aware of them to the extent that it knows not to try to install them the usual way and it's able to report an error if the user requests a built-in provider that doesn't exist or tries to impose a particular version constraint for a built-in provider. For the moment the tests for this are the ones in the "command" package because that's where the existing testing infrastructure for this functionality lives. A later commit should add some more focused unit tests here in the internal/providercache package, too.	6 years ago
Martin Atkins	de6c9ccec1	command/init: Move "vendored provider" test to e2etests ... In the new design the ProviderSource is decided by package main, not by the "command" package, and so making sure the vendor directory is included is the responsibility of that package instead. Therefore we can no longer test this at the "command" package level, but we'll retain a test for it in e2etests to record that it isn't currently working, so that we have a prompt to fix it before releasing.	6 years ago
Martin Atkins	fe9105abfe	command: Fix TestInit_getProviderMissing ... This one just needs a tweak for the new wording of the error message.	6 years ago
Martin Atkins	add7006de6	command: Fix TestInit_pluginDirProviders and _pluginDirProvidersDoesNotGet ... Both of these are attempting to test -plugin-dir, which means we need some additional help to populate some suitable directories for -plugin-dir to refer to. The new installFakeProviderPackagesElsewhere helper generalizes the earlier installFakeProviderPackages to allow installing fake provider packages to an arbitrary other directory.	6 years ago
Martin Atkins	c81eebe0ac	command/init: Fix TestInit_providerSource ... This test is focused on making sure that the required_providers syntax is working, so the rewritten version does not include any special handling of pre-installed packages or "vendored" packages. Pre-installed plugins are tested in other tests such as TestInit_getUpgradePlugins.	6 years ago
Martin Atkins	14701b8300	command/init: Fix TestInit_getUpgradePlugins ... This test now requires a bit of a different approach because it was previously directly constructing a cache directory but we now use a different directory layout. Rather than manually constructing the new heirarchical directory layout (which would've required a lot more inline code), this introduces a helper function installFakeProviderPackages that installs a fake provider package directly into the local cache directory associated with a Meta object, with the correct directory layout.	6 years ago
Martin Atkins	d40085f374	command: Make the tests compile again ... They still aren't passing, but this is just enough updating to make the test program compile successfully after the refactoring related to provider installation. They are now using the mock provider source offered by the getproviders package, which is similar but not totally identical to the idea of mocking the entire installer as these tests used to do, and so many of them need further adjustment to still be testing what they intended to test under this new architecture. Subsequent commits will gradually repair the failing tests.	6 years ago
Kristin Laemmert	4f141ae365	rename provider-specific functions (#24417) ... missingPlugins was hard-coded to work only with provider plugins, so I renamed it to clarify the usage. Also renamed a test provider from greater_than to greater-than as the underscore is an invalid provider name character and this will become a hard error in the near future.	6 years ago
Kristin Laemmert	5f313a65ad	command: remove 0.12upgrade (#24403) ... * command: remove 0.12upgrade and related `configupgrade` library * leave deprecation warning for 0.12upgrade to point users to v0.12	6 years ago
Kristin Laemmert	5901952882	command: tests should not leave dirs behind thank you (#24340)	6 years ago
Kristin Laemmert	92f427779d	command/show: fix issue with show and aliased provider (#23848) ... The formatter in `command/format/state.go`, when formatting a resource with an aliased provider, was looking for a schema with the alias (ie, test.foo), but the schemas are only listed by provider type (test). Update the state formatter to lookup schemas by provider type only. Some of the show tests (and a couple others) were not properly cleaning up the created tmpdirs, so I fixed those. Also, the show tests are using a statefile named `state.tfstate`, but were not passing that path to the show command, so we were getting some false positives (a `show` command that returns `no state` exits 0). Fixes #21462	6 years ago
Kristin Laemmert	18dd1bb4d6	Mildwonkey/tfconfig upgrade (#23670) ... * deps: bump terraform-config-inspect library * configs: parse `version` in new required_providers block With the latest version of `terraform-config-inspect`, the required_providers attribute can now be a string or an object with attributes "source" and "version". This change allows parsing the version constraint from the new object while ignoring any given source attribute.	6 years ago
Kristin Laemmert	a16e1fc0a1	command/init: omit a warning if -backend-config is used with no backend (#22164) ... * command/init: omit a warning if -backend-config is used with no backend block Terraform would silently accept - and swallow - `-backend-config` on the CLI when there was no `backend` block. Since it is mostly expected to override existing backend configuration, terraform should omit a warning if there is no backend configuration to override. If the user intended to override the default (local) backend configuration, they can first add a `backend` block to the `terraform` block to silence the warning (or just ignore it): ```hcl terraform { backend "local" {} } ```	7 years ago
James Bardin	06dfc4abd8	allow setting -backend-config='' to unset override ... There is currently no way to unset -backend-config during init, since not setting that option assumes the user will use the saved config. Allow setting `-backend-config=""` to specify no overrides.	7 years ago
James Bardin	c017149b31	don't store prepared backend config ... The backend gets to "prepare" the configuration before Configure is called, in order to validate the values and insert defaults. We don't want to store this value in the "config state", because it will often not match the raw config after it is prepared, forcing unecessary backend migrations during init. Since PrepareConfig is always called before Configure, we can store the config value directly, and assume that it will be prepared in the same manner each time.	7 years ago
James Bardin	ee9a618369	don't migrate backend during init without override ... If the backend config hashes match during init, and there are no new backend override options, then we assume the existing config is OK. Since init should be idempotent, we should be able to run init with no options or config changes, and not effect the backends at all.	7 years ago
James Bardin	cefc927e48	failing test for backend re-init	7 years ago
Martin Atkins	63fa1ac418	vendor: go get github.com/hashicorp/hcl2@master ... This includes a small fix to ensure the parser doesn't produce an invalid body for block parsing syntax errors, and instead produces an incomplete result that calling applications like Terraform can still analyze. The problem here was affecting our version-constraint-sniffing code, which intentionally tried to find a core version constraint even if there's a syntax error so that it can report that a new version of Terraform is a likely cause of the syntax error. It was working in most cases, unless it was the "terraform" block itself that contained the error, because then we'd try to analyze a broken hcl.Block with a nil body. This includes a new test for "terraform init" that exercises this recovery codepath.	7 years ago
Justin Campbell	24e13d8ec1	plugin/discovery: Return tfdiags from Get ... Allows us to surface warnings to the user using the tfdiags interfaces.	7 years ago
Radek Simko	37f397dded	command/test: Clarify failure (add full path)	7 years ago
Martin Atkins	86c02d5c35	command: "terraform init" can partially initialize for 0.12upgrade ... There are a few constructs from 0.11 and prior that cause 0.12 parsing to fail altogether, which previously created a chicken/egg problem because we need to install the providers in order to run "terraform 0.12upgrade" and thus fix the problem. This changes "terraform init" to use the new "early configuration" loader for module and provider installation. This is built on the more permissive parser in the terraform-config-inspect package, and so it allows us to read out the top-level blocks from the configuration while accepting legacy HCL syntax. In the long run this will let us do version compatibility detection before attempting a "real" config load, giving us better error messages for any future syntax additions, but in the short term the key thing is that it allows us to install the dependencies even if the configuration isn't fully valid. Because backend init still requires full configuration, this introduces a new mode of terraform init where it detects heuristically if it seems like we need to do a configuration upgrade and does a partial init if so, before finally directing the user to run "terraform 0.12upgrade" before running any other commands. The heuristic here is based on two assumptions: - If the "early" loader finds no errors but the normal loader does, the configuration is likely to be valid for Terraform 0.11 but not 0.12. - If there's already a version constraint in the configuration that excludes Terraform versions prior to v0.12 then the configuration is probably _already_ upgraded and so it's just a normal syntax error, even if the early loader didn't detect it. Once the upgrade process is removed in 0.13.0 (users will be required to go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can be simplified to remove that special mode, but the idea of doing the dependency version checks against the liberal parser will remain valuable to increase our chances of reporting version-based incompatibilities rather than syntax errors as we add new features in future.	7 years ago
James Bardin	149ccd929e	missing commits from 19688	7 years ago
Martin Atkins	8c54da0ad2	command: TestInit_fromModule_explicitDest guard against other tests ... Some other test is leaving behind a terraform.tfstate after it concludes, which can cause this test to fail in a strange way due to picking up extra provider requirements from that state. This check doesn't fix that problem, but it at least makes the test fail in a more helpful way to avoid time wasted trying to debug this test when it's some other test that actually has the bug.	7 years ago
Martin Atkins	544c2932ce	command: Fix TestInit_checkRequiredVersion ... In prior refactoring we lost the required core version check from "terraform init", which we restore here. Additionally, this test used to have an incorrect name that suggested it was testing something in the "getProvider" codepath, but version checking happens regardless of what other options are selected.	7 years ago
Martin Atkins	9ba399bca8	command: Fix TestInit_getProvider ... After all of the refactoring we were no longer checking the Terraform version field in a state file, causing this test to fail. This restores that check, though with a slightly different error message.	7 years ago
Martin Atkins	f6d468ffd5	command: Fix TestInit_inputFalse ... This test was using old-style state files as its input, differing only by lineage. Since lineages are now managed within the state manager itself, the test can't use that to distinguish the two files and so we put a different output in each one instead. This also introduces some TRACE logging to the migration codepaths. There's some hard-to-follow control flow here and so this extra logging helps to understand the reason for a particular outcome, and since this codepath is visited only in "terraform init" anyway it doesn't hurt to be a bit more verbose here.	7 years ago
Martin Atkins	297b3b8830	command: Fix TestInit_backendReinitConfigToExtra ... This test was re-using the same InitCommand value to run multiple times, which is not realistic. Since we now cache configuration source code inside command.Meta on load, it's important that we use a fresh InitCommand instance here so it'll see the modified configuration file we've left on disk.	7 years ago
Martin Atkins	7b77e20bdc	command: Fix several "terraform init" tests	8 years ago
Martin Atkins	a3403f2766	terraform: Ugly huge change to weave in new State and Plan types ... Due to how often the state and plan types are referenced throughout Terraform, there isn't a great way to switch them out gradually. As a consequence, this huge commit gets us from the old world to a _compilable_ new world, but still has a large number of known test failures due to key functionality being stubbed out. The stubs here are for anything that interacts with providers, since we now need to do the follow-up work to similarly replace the old terraform.ResourceProvider interface with its replacement in the new "providers" package. That work, along with work to fix the remaining failing tests, will follow in subsequent commits. The aim here was to replace all references to terraform.State and its downstream types with states.State, terraform.Plan with plans.Plan, state.State with statemgr.State, and switch to the new implementations of the state and plan file formats. However, due to the number of times those types are used, this also ended up affecting numerous other parts of core such as terraform.Hook, the backend.Backend interface, and most of the CLI commands. Just as with 5861dbf3fc before, I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.	8 years ago
Martin Atkins	ebafa51723	command: Various updates for the new backend package API ... This is a rather-messy, complex change to get the "command" package building again against the new backend API that was updated for the new configuration loader. A lot of this is mechanical rewriting to the new API, but meta_config.go and meta_backend.go in particular saw some major changes to interface with the new loader APIs and to deal with the change in order of steps in the backend API.	8 years ago