65 Commits (3c8a4e6e05afefb7639bf30efa852cf59f14ade6)

Author	SHA1	Message	Date
Martin Atkins	3c8a4e6e05	command+backend/local: -refresh-only and drift detection ... This is a light revamp of our plan output to make use of Terraform core's new ability to report both the previous run state and the refreshed state, allowing us to explicitly report changes made outside of Terraform. Because whether a plan has "changes" or not is no longer such a straightforward matter, this now merges views.Operation.Plan with views.Operation.PlanNoChanges to produce a single function that knows how to report all of the various permutations. This was also an opportunity to fill some holes in our previous logic which caused it to produce some confusing messages, including a new tailored message for when "terraform destroy" detects that nothing needs to be destroyed. This also allows users to request the refresh-only planning mode using a new -refresh-only command line option. In that case, Terraform _only_ performs drift detection, and so applying a refresh-only plan only involves writing a new state snapshot, without changing any real infrastructure objects.	5 years ago
Pam Selle	242f319638	Give suggestions & remind users to use required_providers when provider not in registry (#28014) ... * Add helper suggestion when failed registry err When someone has a failed registry error on init, remind them that they should have required_providers in every module * Give suggestion for a provider based on reqs Suggest another provider on a registry error, from the list of requirements we have on init. This skips the legacy lookup process if there is a similar provider existing in requirements.	5 years ago
Kristin Laemmert	7b7273e3aa	Add support for plugin protocol v6 (#27826) ... * Add support for plugin protocol v6 This PR turns on support for plugin protocol v6. A provider can advertise itself as supporting protocol version 6 and terraform will use the correct client. Todo: The "unmanaged" providers functionality does not support protocol version, so at the moment terraform will continue to assume that "unmanaged" providers are on protocol v5. This will require some upstream work on go-plugin (I believe). I would like to convert the builtin providers to use protocol v6 in a future PR; however it is not necessary until we remove protocol v6. * add e2e test for using both plugin protocol versions - copied grpcwrap and made a version that returns protocol v6 provider - copied the test provider, provider-simple, and made a version that's using protocol v6 with the above fun - added an e2etest	5 years ago
Brian Flad	70aac9c551	command: Use different warning messaging in init command and move above potentially confusing output (#27514) ... Reference: https://github.com/hashicorp/terraform/issues/27459 Reference: https://github.com/hashicorp/terraform/pull/27507 This follows the change in the plan command to move the development overrides warning above other output from the command that may exhibit unexpected behavior.	5 years ago
Katy Moe	51c687c2db	command: no visual warning hierarchy in -no-color ... Commit e865faf adds visual indentation for diagnostic messages using various vertical line characters. The present commit disables this behaviour when running with colourised output disabled. While the contents of stderr are not intended to be part of the Terraform API, this is currently how the hashicorp/terraform-exec library detects certain error types in order to present them as well-known Go errors to the user. Such detection is complicated when vertical lines are added to the CLI output at unpredictable points. I expect this change will also be helpful for screen reader users.	5 years ago
James Bardin	21896f74af	builtin provisioner e2e test	5 years ago
Martin Atkins	e865faf318	command: Better visual hierarchy for diagnostics ... I frequently see people attempting to ask questions about Terraform's error and warning messages but either only copying part of the message or accidentally copying a surrounding paragraph that isn't part of the message. While I'm sure some of these are just "careless" mistakes, I've also noticed that this has sometimes overlapped with someone asking a question whose answer is written directly in the part of the message they didn't include when copying, and so I have a theory that our current output doesn't create a good enough visual hierarchy for sighted users to understand where the diagnostic messages start and end when we show them in close proximity to other content, or to other diagnostic messages. As a result, some folks fail to notice the relevant message that might've answered their question. I tried a few different experiments for different approaches here, such as adding more horizontal rules to the output and coloring the detail text differently, but the approach that felt like the nicest compromise to me was what's implemented here, which is to add a vertical line along the left edge of each diagnostic message, colored to match with the typical color we use for each diagnostic severity. This means that the diagnostics end up slightly indented from what's around them, and the vertical line seems to help subtly signal how we intended the content to be grouped together.	5 years ago
Martin Atkins	e6a516d87e	backend/local: Use terminal properties to tweak the plan output ... We now require the output to accept UTF-8 and we can determine how wide the terminal (if any) is, so here we begin to make use of that for the "terraform plan" command. The horizontal rule is now made of box drawing characters instead of hyphens and fills the whole terminal width. The paragraphs of text in the output are now also wrapped to fill the terminal width, instead of the hard-wrapping we did before. This is just a start down the road of making better use of the terminal capabilities. Lots of other commands could benefit from updates like these too.	5 years ago
James Bardin	a1d41504f2	e2etest staticcheck	5 years ago
James Bardin	f521fcca97	cleanup error handling and some for loops	5 years ago
James Bardin	f8ccbcd3bb	Add e2e test for provisioner plugin ... Ensure we can still execute a 3rd party provisioner binary, using the legacy plugin discovery location of the configuration root directory.	5 years ago
James Bardin	76cb40005a	add grpcwrap.Provisioner ... Rename grpcwrap.New() to grpcwrap.Provider() Add a grpcwrap function to create a test proivisioner plugin.	5 years ago
James Bardin	dc9ded8618	remove old version call site	5 years ago
James Bardin	e4c72015a3	remove old test provider from e2e tests	5 years ago
James Bardin	811a3a81e9	add test provider bin for e2etest ... One e2etest still requires the bin, so add that back temporarily.	5 years ago
James Bardin	75bbf0b62b	udpate e2etest to use internal/legacy ... The use of this provider will be factored out, but just change the import for now.	5 years ago
James Bardin	a75dcd4be0	warning are now in stdout	5 years ago
James Bardin	bc1a841d65	Merge pull request #26665 from hashicorp/jbardin/logging ... Restore "crash.log" behavior and remove prefixedio	6 years ago
James Bardin	ef4fee5acb	warnings are now on stderr	6 years ago
Kristin Laemmert	9d623290f4	command/e2etest: do not leave bad directories behind ... If you run the e2etests locally and use a configured plugin_cache_dir, the test will leave a bad directory behind in your cache dir that causes later `init`s to fail. To circumvent this, pass an explicity-empty CLI config file. This is a nicety for local developers and not necessarily required, but it happens to me often enough that I'd like to fix it. It's probably not a *bad* idea to pass an explicit cli config to all e2etests, honestly, but this is the only one that causes active problems so I limited this PR to that one test. Here's the error which occurs on subsequent `init` if this test is run on a machine that uses a plugin cache dir: 2020/10/13 10:41:05 [TRACE] providercache.fillMetaCache: error while scanning directory /Users/mildwonkey/.terraform.d/plugin-cache: failed to read metadata about /Users/mildwonkey/.terraform.d/plugin-cache/example.com/awesomecorp/happycloud/1.2.0/darwin_amd64: stat /Users/mildwonkey/.terraform.d/plugin-cache/example.com/awesomecorp/happycloud/1.2.0/darwin_amd64: no such file or directory	6 years ago
Martin Atkins	30204ecded	command/cliconfig: Allow development overrides for providers ... For normal provider installation we want to associate each provider with a selected version number and find a suitable package for that version that conforms to the official hashes for that release. Those requirements are very onerous for a provider developer currently testing a not-yet-released build, though. To allow for that case this new CLI configuration feature allows overriding specific providers to refer to give local filesystem directories. Any provider overridden in this way is not subject to the usual restrictions about selected versions or checksum conformance, and activating an override won't cause any changes to the selections recorded in the lock file because it's intended to be a temporary setting for one developer only. This is, in a sense, a spiritual successor of an old capability we had to override specific plugins in the CLI configuration file. There were some vestiges of that left in the main package and CLI config package but nothing has actually been honoring them for several versions now and so this commit removes them to avoid confusion with the new mechanism.	6 years ago
Martin Atkins	e70ab09bf1	command: new cache directory .terraform/providers for providers ... Terraform v0.10 introduced .terraform/plugins as a cache directory for automatically-installed plugins, Terraform v0.13 later reorganized the directory structure inside but retained its purpose as a cache. The local cache used to also serve as a record of specifically which packages were selected in a particular working directory, with the intent that a second run of "terraform init" would always select the same packages again. That meant that in some sense it behaved a bit like a local filesystem mirror directory, even though that wasn't its intended purpose. Due to some unfortunate miscommunications, somewhere a long the line we published some documentation that _recommended_ using the cache directory as if it were a filesystem mirror directory when working with Terraform Cloud. That was really only working as an accident of implementation details, and Terraform v0.14 is now going to break that because the source of record for the currently-selected provider versions is now the public-facing dependency lock file rather than the contents of an existing local cache directory on disk. After some consideration of how to move forward here, this commit implements a compromise that tries to avoid silently doing anything surprising while still giving useful guidance to folks who were previously using the unsupported strategy. Specifically: - The local cache directory will now be .terraform/providers rather than .terraform/plugins, because .terraform/plugins is effectively "poisoned" by the incorrect usage that we can't reliably distinguish from prior version correct usage. - The .terraform/plugins directory is now the "legacy cache directory". It is intentionally _not_ now a filesystem mirror directory, because that would risk incorrectly interpreting providers automatically installed by Terraform v0.13 as if they were a local mirror, and thus upgrades and checksum fetches from the origin registry would be blocked. - Because of the previous two points, someone who _was_ trying to use the legacy cache directory as a filesystem mirror would see installation fail for any providers they manually added to the legacy directory. To avoid leaving that user stumped as to what went wrong, there's a heuristic for the case where a non-official provider fails installation and yet we can see it in the legacy cache directory. If that heuristic matches then we'll produce a warning message hinting to move the provider under the terraform.d/plugins directory, which is a _correct_ location for "bundled" provider plugins that belong only to a single configuration (as opposed to being installed globally on a system). This does unfortunately mean that anyone who was following the incorrectly-documented pattern will now encounter an error (and the aforementioned warning hint) after upgrading to Terraform v0.14. This seems like the safest compromise because Terraform can't automatically infer the intent of files it finds in .terraform/plugins in order to decide automatically how best to handle them. The internals of the .terraform directory are always considered implementation detail for a particular Terraform version and so switching to a new directory for the _actual_ cache directory fits within our usual set of guarantees, though it's definitely non-ideal in isolation but okay when taken in the broader context of this problem, where the alternative would be silent misbehavior when upgrading.	6 years ago
Martin Atkins	b3f5c7f1e6	command/init: Read, respect, and update provider dependency locks ... This changes the approach used by the provider installer to remember between runs which selections it has previously made, using the lock file format implemented in internal/depsfile. This means that version constraints in the configuration are considered only for providers we've not seen before or when -upgrade mode is active.	6 years ago
Pam Selle	1817c8ac3c	Merge pull request #26412 from OwenTuz/issue-26411-fix-failing-e2e-test-provider-not-found ... Fix bad string match that was causing TestInitProviderNotFound to fail	6 years ago
James Bardin	c51104fb7c	Merge pull request #26435 from hashicorp/jbardin/races ... Fix race conditions	6 years ago
James Bardin	59110a2ca5	e2etest server was unsynchronized	6 years ago
Martin Atkins	59b116f7bf	command/init: Remove support for legacy provider addresses ... We no longer need to support 0.12-and-earlier-style provider addresses because users should've upgraded their existing configurations and states on Terraform 0.13 already. For now this is only checked in the "init" command, because various test shims are still relying on the idea of legacy providers the core layer. However, rejecting these during init is sufficient grounds to avoid supporting legacy provider addresses in the new dependency lock file format, and thus sets the stage for a more severe removal of legacy provider support in a later commit.	6 years ago
Owen Tuz	5453147381	Fix bad string match that was causing TestInitProviderNotFound to fail on Macs ... The tmp directory path is longer on Macs than other systems and was wrapping across lines when printed, breaking the string match in the test. Fix suggested by @apparentlymart is to add two spaces before the leading '-' when printing to prevent the diagnostic renderer wrapping the line.	6 years ago
James Bardin	ab06f0c9f8	we can roll back the e2e tests ... the data sources no longer show up in the tests	6 years ago
James Bardin	86dd8938c9	data sources now show up in the initial plan	6 years ago
Martin Atkins	efe78b2910	main: new global option -chdir ... This new option is intended to address the previous inconsistencies where some older subcommands supported partially changing the target directory (where Terraform would use the new directory inconsistently) where newer commands did not support that override at all. Instead, now Terraform will accept a -chdir command at the start of the command line (before the subcommand) and will interpret it as a request to direct all actions that would normally be taken in the current working directory into the target directory instead. This is similar to options offered by some other similar tools, such as the -C option in "make". The new option is only accepted at the start of the command line (before the subcommand) as a way to reflect that it is a global command (not specific to a particular subcommand) and that it takes effect _before_ executing the subcommand. This also means it'll be forced to appear before any other command-specific arguments that take file paths, which hopefully communicates that those other arguments are interpreted relative to the overridden path. As a measure of pragmatism for existing uses, the path.cwd object in the Terraform language will continue to return the _original_ working directory (ignoring -chdir), in case that is important in some exceptional workflows. The path.root object gives the root module directory, which will always match the overriden working directory unless the user simultaneously uses one of the legacy directory override arguments, which is not a pattern we intend to support in the long run. As a first step down the deprecation path, this commit adjusts the documentation to de-emphasize the inconsistent old command line arguments, including specific guidance on what to use instead for the main three workflow commands, but all of those options remain supported in the same way as they were before. In a later commit we'll make those arguments produce a visible deprecation warning in Terraform's output, and then in an even later commit we'll remove them entirely so that -chdir is the single supported way to run Terraform from a directory other than the one containing the root module configuration.	6 years ago
Kristin Laemmert	47e657c611	internal/getproviders: decode and return any registry warnings (#25337) ... * internal/getproviders: decode and return any registry warnings The public registry may include a list of warnings in the "versions" response for any given provider. This PR adds support for warnings from the registry and an installer event to return those warnings to the user.	6 years ago
James Bardin	731b19ab46	e2e test for remote state read	6 years ago
Martin Atkins	49e2e00231	command: terraform providers mirror ... This new command is intended to make it easy to create or update a mirror directory containing suitable providers for the current configuration, producing a layout that is appropriate both for a filesystem mirror or, if copied into the document root of an HTTP server, a network mirror. This initial version is not customizable aside from being able to select multiple platforms to install packages for. Future iterations of this could include commands to turn the JSON index generation on and off, or to instruct it to produce the unpacked directory layout instead of the packed directory layout as it currently does. Both of those options would make the generated directory unsuitable to be a network mirror, but it would still work as a filesystem mirror. In the long run this will hopefully form part of a replacement workflow to terraform-bundle as a way to put copies of providers somewhere so we don't need to re-download them every time, but some other changes will be needed outside of just this command before that'd be true, such as adding support for network and/or filesystem mirrors in Terraform Enterprise.	6 years ago
Kristin Laemmert	020084f6d0	update e2etests for windows compatibility	6 years ago
Alisdair McDiarmid	ca40107066	command/init: Better diagnostics for provider 404s ... Fetching a default namespace provider from the public registry can result in 404 Not Found error. This might be caused by a previously- default provider moving to a new namespace, which means that the configuration needs to be upgraded to use an explicit provider source. This commit adds a more detailed diagnostic for this situation, suggesting that the intended provider might be in a new namespace. The recommended course of action is to run the 0.13upgrade command to generate the correct required_providers configuration.	6 years ago
Paddy	5127f1ef8b	command: Unmanaged providers ... This adds supports for "unmanaged" providers, or providers with process lifecycles not controlled by Terraform. These providers are assumed to be started before Terraform is launched, and are assumed to shut themselves down after Terraform has finished running. To do this, we must update the go-plugin dependency to v1.3.0, which added support for the "test mode" plugin serving that powers all this. As a side-effect of not needing to manage the process lifecycle anymore, Terraform also no longer needs to worry about the provider's binary, as it won't be used for anything anymore. Because of this, we can disable the init behavior that concerns itself with downloading that provider's binary, checking its version, and otherwise managing the binary. This is all managed on a per-provider basis, so managed providers that Terraform downloads, starts, and stops can be used in the same commands as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable is added, and is a JSON encoding of the provider's address to the information we need to connect to it. This change enables two benefits: first, delve and other debuggers can now be attached to provider server processes, and Terraform can connect. This allows for attaching debuggers to provider processes, which before was difficult to impossible. Second, it allows the SDK test framework to host the provider in the same process as the test driver, while running a production Terraform binary against the provider. This allows for Go's built-in race detector and test coverage tooling to work as expected in provider tests. Unmanaged providers are expected to work in the exact same way as managed providers, with one caveat: Terraform kills provider processes and restarts them once per graph walk, meaning multiple times during most Terraform CLI commands. As unmanaged providers can't be killed by Terraform, and have no visibility into graph walks, unmanaged providers are likely to have differences in how their global mutable state behaves when compared to managed providers. Namely, unmanaged providers are likely to retain global state when managed providers would have reset it. Developers relying on global state should be aware of this.	6 years ago
Kristin Laemmert	eead4c49fe	command/init: add e2e tests for provider not found messages	6 years ago
Martin Atkins	6b2050f42a	main: Properly handle provider installation method exclusions ... Previously we were incorrectly using the Include configuration for both the include and exclude list, making the include portion totally ineffective.	6 years ago
Martin Atkins	3167067029	command/e2etest: provider installation with explicit install methods ... This exercises the ability to customize the installation methods used by the provider plugin installer, in this case forcing the use of a custom local directory with a result essentially the same as what happens when you pass -plugin-dir to "terraform init".	6 years ago
Martin Atkins	92d6a30bb4	main: skip direct provider installation for providers available locally ... This more closely replicates the 0.12-and-earlier behavior, where having at least one version of a provider installed locally would totally disable any attempt to look for newer versions remotely. This is just for the implicit default behavior. Assumption is that later we'll have an explicit configuration mechanism that will allow the user to specify exactly where to look for what, and thus avoid tricky heuristics like this.	6 years ago
Martin Atkins	a6f63c4891	command/e2etest: update "init" tests for abbreviated provider addresses ... We're now longer showing the default registry hostname as part of addresses coming from that registry.	6 years ago
Martin Atkins	8c928e8358	main: Consult local directories as potential mirrors of providers ... This restores some of the local search directories we used to include when searching for provider plugins in Terraform 0.12 and earlier. The directory structures we are expecting in these are different than before, so existing directory contents will not be compatible without restructuring, but we need to retain support for these local directories so that users can continue to sideload third-party provider plugins until the explicit, first-class provider mirrors configuration (in CLI config) is implemented, at which point users will be able to override these to whatever directories they want. This also includes some new search directories that are specific to the operating system where Terraform is running, following the documented layout conventions of that platform. In particular, this follows the XDG Base Directory specification on Unix systems, which has been a somewhat-common request to better support "sideloading" of packages via standard Linux distribution package managers and other similar mechanisms. While it isn't strictly necessary to add that now, it seems ideal to do all of the changes to our search directory layout at once so that our documentation about this can cleanly distinguish "0.12 and earlier" vs. "0.13 and later", rather than having to document a complex sequence of smaller changes. Because this behavior is a result of the integration of package main with package command, this behavior is verified using an e2etest rather than a unit test. That test, TestInitProvidersVendored, is also fixed here to create a suitable directory structure for the platform where the test is being run. This fixes TestInitProvidersVendored.	6 years ago
Martin Atkins	de6c9ccec1	command/init: Move "vendored provider" test to e2etests ... In the new design the ProviderSource is decided by package main, not by the "command" package, and so making sure the vendor directory is included is the responsibility of that package instead. Therefore we can no longer test this at the "command" package level, but we'll retain a test for it in e2etests to record that it isn't currently working, so that we have a prompt to fix it before releasing.	6 years ago
Martin Atkins	f35ebe2d65	internal/providercache: Fix incorrect logic in Installer.SetGlobalCacheDir ... Due to some incomplete rework of this function in an earlier commit, the safety check for using the same directory as both the target and the cache was inverted and was raising an error _unless_ they matched, rather than _if_ they matched. This change is verified by the e2etest TestInitProviders_pluginCache, which is also updated to use the new-style cache directory layout as part of this commit.	6 years ago
Martin Atkins	14d456372a	command/e2etest: Update expected output for new plugin installer ... These tests make assertions against specific user-oriented output from the "terraform init" command, but we've intentionally changed some of these messages as part of introducing support for the decentralized provider namespace.	6 years ago
Martin Atkins	cf43663e85	command/e2etest: Fix TestInitProviders ... The canonical location of the "template" provider is now in the hashicorp namespace rather than the terraform-providers namespace, so the output has changed to reflect that.	7 years ago
Radek Simko	5b9f2fafc8	Standardise directory name for test data	7 years ago
Martin Atkins	eed605ac05	[WIP] Re-enable the end-to-end tests (#20044) ... * internal/initwd: Allow deprecated relative module paths In Terraform 0.11 we deprecated this form but didn't have any explicit warning for it. Now we'll still accept it but generate a warning. In a future major release we will drop this form altogether, since it is ambiguous with registry module source addresses. This codepath is covered by the command/e2etest suite. * e2e: Skip copying .exists file, if present We use this only in the "empty" test fixture in order to let git know that the directory exists. We need to skip copying it so that we can test "terraform init -from-module=...", which expects to find an empty directory. * command/e2etests: Re-enable and fix up the e2etest "acctests" We disabled all of the tests that accessed remote services like the Terraform Registry while they were being updated to support the new protocols we now expect. With those services now in place, we can re-enable these tests. Some details of exactly what output we print, etc, have intentionally changed since these tests were last updated. * e2e: refactor for modern states and plans * command/e2etest: re-enable e2etests and update for tf 0.12 compatibility plugin/discovery: mkdirAll instead of mkdir when creating cache dir	7 years ago
Martin Atkins	73c9521a04	command/e2etest: Temporarily disable tests that access network ... Several of these tests rely on external services (e.g. Terraform Registry) that have not yet been updated to support the needs of Terraform v0.12.0, so for now we'll skip all of these tests and wait until those systems have been updated. This should be removed before Terraform v0.12.0 final to enable these tests to be used as part of pre-release smoke testing.	7 years ago