49 Commits (ccf19fbf5e2b61ba62efd1ac47a3a3fd3917d494)

Author	SHA1	Message	Date
James Bardin	a75dcd4be0	warning are now in stdout	5 years ago
James Bardin	bc1a841d65	Merge pull request #26665 from hashicorp/jbardin/logging ... Restore "crash.log" behavior and remove prefixedio	6 years ago
James Bardin	ef4fee5acb	warnings are now on stderr	6 years ago
Kristin Laemmert	9d623290f4	command/e2etest: do not leave bad directories behind ... If you run the e2etests locally and use a configured plugin_cache_dir, the test will leave a bad directory behind in your cache dir that causes later `init`s to fail. To circumvent this, pass an explicity-empty CLI config file. This is a nicety for local developers and not necessarily required, but it happens to me often enough that I'd like to fix it. It's probably not a *bad* idea to pass an explicit cli config to all e2etests, honestly, but this is the only one that causes active problems so I limited this PR to that one test. Here's the error which occurs on subsequent `init` if this test is run on a machine that uses a plugin cache dir: 2020/10/13 10:41:05 [TRACE] providercache.fillMetaCache: error while scanning directory /Users/mildwonkey/.terraform.d/plugin-cache: failed to read metadata about /Users/mildwonkey/.terraform.d/plugin-cache/example.com/awesomecorp/happycloud/1.2.0/darwin_amd64: stat /Users/mildwonkey/.terraform.d/plugin-cache/example.com/awesomecorp/happycloud/1.2.0/darwin_amd64: no such file or directory	6 years ago
Martin Atkins	30204ecded	command/cliconfig: Allow development overrides for providers ... For normal provider installation we want to associate each provider with a selected version number and find a suitable package for that version that conforms to the official hashes for that release. Those requirements are very onerous for a provider developer currently testing a not-yet-released build, though. To allow for that case this new CLI configuration feature allows overriding specific providers to refer to give local filesystem directories. Any provider overridden in this way is not subject to the usual restrictions about selected versions or checksum conformance, and activating an override won't cause any changes to the selections recorded in the lock file because it's intended to be a temporary setting for one developer only. This is, in a sense, a spiritual successor of an old capability we had to override specific plugins in the CLI configuration file. There were some vestiges of that left in the main package and CLI config package but nothing has actually been honoring them for several versions now and so this commit removes them to avoid confusion with the new mechanism.	6 years ago
Martin Atkins	e70ab09bf1	command: new cache directory .terraform/providers for providers ... Terraform v0.10 introduced .terraform/plugins as a cache directory for automatically-installed plugins, Terraform v0.13 later reorganized the directory structure inside but retained its purpose as a cache. The local cache used to also serve as a record of specifically which packages were selected in a particular working directory, with the intent that a second run of "terraform init" would always select the same packages again. That meant that in some sense it behaved a bit like a local filesystem mirror directory, even though that wasn't its intended purpose. Due to some unfortunate miscommunications, somewhere a long the line we published some documentation that _recommended_ using the cache directory as if it were a filesystem mirror directory when working with Terraform Cloud. That was really only working as an accident of implementation details, and Terraform v0.14 is now going to break that because the source of record for the currently-selected provider versions is now the public-facing dependency lock file rather than the contents of an existing local cache directory on disk. After some consideration of how to move forward here, this commit implements a compromise that tries to avoid silently doing anything surprising while still giving useful guidance to folks who were previously using the unsupported strategy. Specifically: - The local cache directory will now be .terraform/providers rather than .terraform/plugins, because .terraform/plugins is effectively "poisoned" by the incorrect usage that we can't reliably distinguish from prior version correct usage. - The .terraform/plugins directory is now the "legacy cache directory". It is intentionally _not_ now a filesystem mirror directory, because that would risk incorrectly interpreting providers automatically installed by Terraform v0.13 as if they were a local mirror, and thus upgrades and checksum fetches from the origin registry would be blocked. - Because of the previous two points, someone who _was_ trying to use the legacy cache directory as a filesystem mirror would see installation fail for any providers they manually added to the legacy directory. To avoid leaving that user stumped as to what went wrong, there's a heuristic for the case where a non-official provider fails installation and yet we can see it in the legacy cache directory. If that heuristic matches then we'll produce a warning message hinting to move the provider under the terraform.d/plugins directory, which is a _correct_ location for "bundled" provider plugins that belong only to a single configuration (as opposed to being installed globally on a system). This does unfortunately mean that anyone who was following the incorrectly-documented pattern will now encounter an error (and the aforementioned warning hint) after upgrading to Terraform v0.14. This seems like the safest compromise because Terraform can't automatically infer the intent of files it finds in .terraform/plugins in order to decide automatically how best to handle them. The internals of the .terraform directory are always considered implementation detail for a particular Terraform version and so switching to a new directory for the _actual_ cache directory fits within our usual set of guarantees, though it's definitely non-ideal in isolation but okay when taken in the broader context of this problem, where the alternative would be silent misbehavior when upgrading.	6 years ago
Martin Atkins	b3f5c7f1e6	command/init: Read, respect, and update provider dependency locks ... This changes the approach used by the provider installer to remember between runs which selections it has previously made, using the lock file format implemented in internal/depsfile. This means that version constraints in the configuration are considered only for providers we've not seen before or when -upgrade mode is active.	6 years ago
Pam Selle	1817c8ac3c	Merge pull request #26412 from OwenTuz/issue-26411-fix-failing-e2e-test-provider-not-found ... Fix bad string match that was causing TestInitProviderNotFound to fail	6 years ago
James Bardin	c51104fb7c	Merge pull request #26435 from hashicorp/jbardin/races ... Fix race conditions	6 years ago
James Bardin	59110a2ca5	e2etest server was unsynchronized	6 years ago
Martin Atkins	59b116f7bf	command/init: Remove support for legacy provider addresses ... We no longer need to support 0.12-and-earlier-style provider addresses because users should've upgraded their existing configurations and states on Terraform 0.13 already. For now this is only checked in the "init" command, because various test shims are still relying on the idea of legacy providers the core layer. However, rejecting these during init is sufficient grounds to avoid supporting legacy provider addresses in the new dependency lock file format, and thus sets the stage for a more severe removal of legacy provider support in a later commit.	6 years ago
Owen Tuz	5453147381	Fix bad string match that was causing TestInitProviderNotFound to fail on Macs ... The tmp directory path is longer on Macs than other systems and was wrapping across lines when printed, breaking the string match in the test. Fix suggested by @apparentlymart is to add two spaces before the leading '-' when printing to prevent the diagnostic renderer wrapping the line.	6 years ago
James Bardin	ab06f0c9f8	we can roll back the e2e tests ... the data sources no longer show up in the tests	6 years ago
James Bardin	86dd8938c9	data sources now show up in the initial plan	6 years ago
Martin Atkins	efe78b2910	main: new global option -chdir ... This new option is intended to address the previous inconsistencies where some older subcommands supported partially changing the target directory (where Terraform would use the new directory inconsistently) where newer commands did not support that override at all. Instead, now Terraform will accept a -chdir command at the start of the command line (before the subcommand) and will interpret it as a request to direct all actions that would normally be taken in the current working directory into the target directory instead. This is similar to options offered by some other similar tools, such as the -C option in "make". The new option is only accepted at the start of the command line (before the subcommand) as a way to reflect that it is a global command (not specific to a particular subcommand) and that it takes effect _before_ executing the subcommand. This also means it'll be forced to appear before any other command-specific arguments that take file paths, which hopefully communicates that those other arguments are interpreted relative to the overridden path. As a measure of pragmatism for existing uses, the path.cwd object in the Terraform language will continue to return the _original_ working directory (ignoring -chdir), in case that is important in some exceptional workflows. The path.root object gives the root module directory, which will always match the overriden working directory unless the user simultaneously uses one of the legacy directory override arguments, which is not a pattern we intend to support in the long run. As a first step down the deprecation path, this commit adjusts the documentation to de-emphasize the inconsistent old command line arguments, including specific guidance on what to use instead for the main three workflow commands, but all of those options remain supported in the same way as they were before. In a later commit we'll make those arguments produce a visible deprecation warning in Terraform's output, and then in an even later commit we'll remove them entirely so that -chdir is the single supported way to run Terraform from a directory other than the one containing the root module configuration.	6 years ago
Kristin Laemmert	47e657c611	internal/getproviders: decode and return any registry warnings (#25337) ... * internal/getproviders: decode and return any registry warnings The public registry may include a list of warnings in the "versions" response for any given provider. This PR adds support for warnings from the registry and an installer event to return those warnings to the user.	6 years ago
James Bardin	731b19ab46	e2e test for remote state read	6 years ago
Martin Atkins	49e2e00231	command: terraform providers mirror ... This new command is intended to make it easy to create or update a mirror directory containing suitable providers for the current configuration, producing a layout that is appropriate both for a filesystem mirror or, if copied into the document root of an HTTP server, a network mirror. This initial version is not customizable aside from being able to select multiple platforms to install packages for. Future iterations of this could include commands to turn the JSON index generation on and off, or to instruct it to produce the unpacked directory layout instead of the packed directory layout as it currently does. Both of those options would make the generated directory unsuitable to be a network mirror, but it would still work as a filesystem mirror. In the long run this will hopefully form part of a replacement workflow to terraform-bundle as a way to put copies of providers somewhere so we don't need to re-download them every time, but some other changes will be needed outside of just this command before that'd be true, such as adding support for network and/or filesystem mirrors in Terraform Enterprise.	6 years ago
Kristin Laemmert	020084f6d0	update e2etests for windows compatibility	6 years ago
Alisdair McDiarmid	ca40107066	command/init: Better diagnostics for provider 404s ... Fetching a default namespace provider from the public registry can result in 404 Not Found error. This might be caused by a previously- default provider moving to a new namespace, which means that the configuration needs to be upgraded to use an explicit provider source. This commit adds a more detailed diagnostic for this situation, suggesting that the intended provider might be in a new namespace. The recommended course of action is to run the 0.13upgrade command to generate the correct required_providers configuration.	6 years ago
Paddy	5127f1ef8b	command: Unmanaged providers ... This adds supports for "unmanaged" providers, or providers with process lifecycles not controlled by Terraform. These providers are assumed to be started before Terraform is launched, and are assumed to shut themselves down after Terraform has finished running. To do this, we must update the go-plugin dependency to v1.3.0, which added support for the "test mode" plugin serving that powers all this. As a side-effect of not needing to manage the process lifecycle anymore, Terraform also no longer needs to worry about the provider's binary, as it won't be used for anything anymore. Because of this, we can disable the init behavior that concerns itself with downloading that provider's binary, checking its version, and otherwise managing the binary. This is all managed on a per-provider basis, so managed providers that Terraform downloads, starts, and stops can be used in the same commands as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable is added, and is a JSON encoding of the provider's address to the information we need to connect to it. This change enables two benefits: first, delve and other debuggers can now be attached to provider server processes, and Terraform can connect. This allows for attaching debuggers to provider processes, which before was difficult to impossible. Second, it allows the SDK test framework to host the provider in the same process as the test driver, while running a production Terraform binary against the provider. This allows for Go's built-in race detector and test coverage tooling to work as expected in provider tests. Unmanaged providers are expected to work in the exact same way as managed providers, with one caveat: Terraform kills provider processes and restarts them once per graph walk, meaning multiple times during most Terraform CLI commands. As unmanaged providers can't be killed by Terraform, and have no visibility into graph walks, unmanaged providers are likely to have differences in how their global mutable state behaves when compared to managed providers. Namely, unmanaged providers are likely to retain global state when managed providers would have reset it. Developers relying on global state should be aware of this.	6 years ago
Kristin Laemmert	eead4c49fe	command/init: add e2e tests for provider not found messages	6 years ago
Martin Atkins	6b2050f42a	main: Properly handle provider installation method exclusions ... Previously we were incorrectly using the Include configuration for both the include and exclude list, making the include portion totally ineffective.	6 years ago
Martin Atkins	3167067029	command/e2etest: provider installation with explicit install methods ... This exercises the ability to customize the installation methods used by the provider plugin installer, in this case forcing the use of a custom local directory with a result essentially the same as what happens when you pass -plugin-dir to "terraform init".	6 years ago
Martin Atkins	92d6a30bb4	main: skip direct provider installation for providers available locally ... This more closely replicates the 0.12-and-earlier behavior, where having at least one version of a provider installed locally would totally disable any attempt to look for newer versions remotely. This is just for the implicit default behavior. Assumption is that later we'll have an explicit configuration mechanism that will allow the user to specify exactly where to look for what, and thus avoid tricky heuristics like this.	6 years ago
Martin Atkins	a6f63c4891	command/e2etest: update "init" tests for abbreviated provider addresses ... We're now longer showing the default registry hostname as part of addresses coming from that registry.	6 years ago
Martin Atkins	8c928e8358	main: Consult local directories as potential mirrors of providers ... This restores some of the local search directories we used to include when searching for provider plugins in Terraform 0.12 and earlier. The directory structures we are expecting in these are different than before, so existing directory contents will not be compatible without restructuring, but we need to retain support for these local directories so that users can continue to sideload third-party provider plugins until the explicit, first-class provider mirrors configuration (in CLI config) is implemented, at which point users will be able to override these to whatever directories they want. This also includes some new search directories that are specific to the operating system where Terraform is running, following the documented layout conventions of that platform. In particular, this follows the XDG Base Directory specification on Unix systems, which has been a somewhat-common request to better support "sideloading" of packages via standard Linux distribution package managers and other similar mechanisms. While it isn't strictly necessary to add that now, it seems ideal to do all of the changes to our search directory layout at once so that our documentation about this can cleanly distinguish "0.12 and earlier" vs. "0.13 and later", rather than having to document a complex sequence of smaller changes. Because this behavior is a result of the integration of package main with package command, this behavior is verified using an e2etest rather than a unit test. That test, TestInitProvidersVendored, is also fixed here to create a suitable directory structure for the platform where the test is being run. This fixes TestInitProvidersVendored.	6 years ago
Martin Atkins	de6c9ccec1	command/init: Move "vendored provider" test to e2etests ... In the new design the ProviderSource is decided by package main, not by the "command" package, and so making sure the vendor directory is included is the responsibility of that package instead. Therefore we can no longer test this at the "command" package level, but we'll retain a test for it in e2etests to record that it isn't currently working, so that we have a prompt to fix it before releasing.	6 years ago
Martin Atkins	f35ebe2d65	internal/providercache: Fix incorrect logic in Installer.SetGlobalCacheDir ... Due to some incomplete rework of this function in an earlier commit, the safety check for using the same directory as both the target and the cache was inverted and was raising an error _unless_ they matched, rather than _if_ they matched. This change is verified by the e2etest TestInitProviders_pluginCache, which is also updated to use the new-style cache directory layout as part of this commit.	6 years ago
Martin Atkins	14d456372a	command/e2etest: Update expected output for new plugin installer ... These tests make assertions against specific user-oriented output from the "terraform init" command, but we've intentionally changed some of these messages as part of introducing support for the decentralized provider namespace.	6 years ago
Martin Atkins	cf43663e85	command/e2etest: Fix TestInitProviders ... The canonical location of the "template" provider is now in the hashicorp namespace rather than the terraform-providers namespace, so the output has changed to reflect that.	7 years ago
Radek Simko	5b9f2fafc8	Standardise directory name for test data	7 years ago
Martin Atkins	eed605ac05	[WIP] Re-enable the end-to-end tests (#20044) ... * internal/initwd: Allow deprecated relative module paths In Terraform 0.11 we deprecated this form but didn't have any explicit warning for it. Now we'll still accept it but generate a warning. In a future major release we will drop this form altogether, since it is ambiguous with registry module source addresses. This codepath is covered by the command/e2etest suite. * e2e: Skip copying .exists file, if present We use this only in the "empty" test fixture in order to let git know that the directory exists. We need to skip copying it so that we can test "terraform init -from-module=...", which expects to find an empty directory. * command/e2etests: Re-enable and fix up the e2etest "acctests" We disabled all of the tests that accessed remote services like the Terraform Registry while they were being updated to support the new protocols we now expect. With those services now in place, we can re-enable these tests. Some details of exactly what output we print, etc, have intentionally changed since these tests were last updated. * e2e: refactor for modern states and plans * command/e2etest: re-enable e2etests and update for tf 0.12 compatibility plugin/discovery: mkdirAll instead of mkdir when creating cache dir	7 years ago
Martin Atkins	73c9521a04	command/e2etest: Temporarily disable tests that access network ... Several of these tests rely on external services (e.g. Terraform Registry) that have not yet been updated to support the needs of Terraform v0.12.0, so for now we'll skip all of these tests and wait until those systems have been updated. This should be removed before Terraform v0.12.0 final to enable these tests to be used as part of pre-release smoke testing.	7 years ago
Laura Martin	6e1e614a56	Change -force to -auto-approve when destroying ... Since an early version of Terraform, the `destroy` command has always had the `-force` flag to allow an auto approval of the interactive prompt. 0.11 introduced `-auto-approve` as default to `false` when using the `apply` command. The `-auto-approve` flag was introduced to reduce ambiguity of it's function, but the `-force` flag was never updated for a destroy. People often use wrappers when automating commands in Terraform, and the inconsistency between `apply` and `destroy` means that additional logic must be added to the wrappers to do similar functions. Both commands are more or less able to run with similar syntax, and also heavily share their code. This commit updates the command in `destroy` to use the `-auto-approve` flag making working with the Terraform CLI a more consistent experience. We leave in `-force` in `destroy` for the time-being and flag it as deprecated to ensure a safe switchover period.	8 years ago
Martin Atkins	d4ee58ce59	Re-integrate the "terraform" provider into the main binary ... As part of the 0.10 core/provider split we moved this provider, along with all the others, out into its own repository. In retrospect, the "terraform" provider doesn't really make sense to be separated since it's just a thin wrapper around some core code anyway, and so re-integrating it into core avoids the confusion that results when Terraform Core and the terraform provider have inconsistent versions of the backend code and dependencies. There is no good reason to use a different version of the backend code in the provider than in core, so this new "internal provider" mechanism is stricter than the old one: it's not possible to use an external build of this provider at all, and version constraints for it are rejected as a result. This provider is also run in-process rather than in a child process, since again it's just a very thin wrapper around code that's already running in Terraform core anyway, and so the process barrier between the two does not create enough advantage to warrant the additional complexity.	9 years ago
Martin Atkins	400038eda4	command: "terraform apply" uses interactive confirmation by default ... In the 0.10 release we added an opt-in mode where Terraform would prompt interactively for confirmation during apply. We made this opt-in to give those who wrap Terraform in automation some time to update their scripts to explicitly opt out of this behavior where appropriate. Here we switch the default so that a "terraform apply" with no arguments will -- if it computes a non-empty diff -- display the diff and wait for the user to type "yes" in similar vein to the "terraform destroy" command. This makes the commonly-used "terraform apply" a safe workflow for interactive use, so "terraform plan" is now mainly for use in automation where a separate planning step is used. The apply command remains non-interactive when given an explicit plan file. The previous behavior -- though not recommended -- can be obtained by explicitly setting the -auto-approve option on the apply command line, and indeed that is how all of the tests are updated here so that they can continue to run non-interactively.	9 years ago
James Bardin	53c8c1e208	e2e test for `init -from-module` ... Pull down the hashicorp/vault/aws module into the current directory with init.	9 years ago
Martin Atkins	5347f82f9a	command: Include provider versions in "terraform version" ... We encourage users to share the "terraform version" output as part of filing an issue, but previously it only printed the core Terraform version and this left provider maintainers with no information about which _provider_ version an issue relates to. Here we make a best effort to show versions for providers, though we will omit some or all of them if either "terraform init" hasn't been run (and so no providers were selected yet) or if there are other inconsistencies that would cause Terraform to object on startup and require a re-run of "terraform init".	9 years ago
Martin Atkins	34cecfa839	command/e2etest: fix incorrect "terraform version" test ... Two different errors here caused this test to pass even though it was incorrect: the wanted version string was incorrect, but the test for it was also inverted, and so together this made the test pass even though it was actually not testing the output at all.	9 years ago
Martin Atkins	30e6f4e066	command/e2etest: test installation of plugins from cache ... This is a tough one to unit tests because the behavior is tangled up in the code that hits releases.hashicorp.com, so we'll add this e2etest as some extra insurance that this works end-to-end.	9 years ago
Martin Atkins	73d1298572	command/e2etest: test the "running in automation" workflow ... Since we now have a guide that recommends some specific ways to run Terraform in automation, we can mimic those suggestions in an e2e test and thus ensure they keep working. Here we test the three different approaches suggested in the guide: - init, plan, apply (main case) - init, apply (e.g. for deploying to a QA/staging environment) - init, plan (e.g. for verifying a pull request)	9 years ago
Martin Atkins	cb6d4e5f20	command/e2etest: fix TestPrimarySeparatePlan test ... In 6712192724 we stopped counting data source destroys in the destroy tally since they are an implementation detail. This caused this test to start failing, though since the new behavior is correct here we just update the test to match.	9 years ago
Martin Atkins	c12d64f340	Use t.Helper() in our test helpers ... Go 1.9 adds this new function which, when called, marks the caller as being a "helper function". Helper function stack frames are then skipped when trying to find a line of test code to blame for a test failure, so that the code in the main test function appears in the test failure output rather than a line within the helper function itself. This covers many -- but probaly not all -- of our test helpers across various packages.	9 years ago
Radek Simko	9e7e4ff4fb	e2e: Decouple logic for running e2e tests	9 years ago
Martin Atkins	23f9c8785e	command/e2etest: an initial test for the primary workflow ... This e2etest runs an init, plan, apply, destroy sequence against a test configuration using the real template and null providers downloaded from the official repository. This test _does_ trample a bit on the scope of some already-existing tests, but this is mainly just to check our assumptions about how Terraform behaves to ensure that we can reach our main conclusion here: that the main Terraform workflow commands interact correctly with each other in real use and we can complete the full workflow.	9 years ago
Martin Atkins	52df81ee49	command/e2etest: test that we can install provider plugins ... We already have good tests for the business logic around provider installation, but the existing tests all stub out the main repository server. This test completes that coverage by verifying that the installer is able to run against the real repository and install an official release of the template provider.	9 years ago
Martin Atkins	0e0b0d125a	command/e2etest: "terraform version" test ... This basic test is here primarily because it's one of the few that can run without reaching out to external services, and so it means our usual test runs will catch situations where the main executable build is somehow broken. The version command itself is not very interesting to test, but it's convenient in that its behavior is very predictable and self-contained.	9 years ago
Martin Atkins	fee61a44b4	command/e2etest: end-to-end testing harness ... Previously we had no automated testing of whether we can produce a Terraform executable that actually works. Our various functional tests have good coverage of specific Terraform features and whole operations, but we lacked end-to-end testing of actual usage of the generated binary, without any stubbing. This package is intended as a vehicle for such end-to-end testing. When run normally under "go test" it will produce a build of the main Terraform binary and make it available for tests to execute. The harness exposes a flag for whether tests are allowed to reach out to external network services, controlled with our standard TF_ACC environment variable, so that basic local tests can be safely run as part of "make test" while more elaborate tests can be run easily when desired. It also provides a separate mode of operation where the included script make-archive.sh can be used to produce a self-contained test archive that can be copied to another system to run the tests there. This is intended to allow testing of cross-compiled binaries, by shipping them over to the target OS and architecture to run without requiring a full Go compiler installation on the target system. The goal here is not to test again functionality that's already well-covered by our existing tests, but rather to test chains of normal operations against the build binary that are not otherwise tested together.	9 years ago