155 Commits (6440c839474f052d8997bb5cec19c2743e5108e6)

Author	SHA1	Message	Date
Martin Atkins	0b734a2803	command: Make provider installation interruptible ... In earlier commits we started to make the installation codepath context-aware so that it could be canceled in the event of a SIGINT, but we didn't complete wiring that through the API of the getproviders package. Here we make the getproviders.Source interface methods, along with some other functions that can make network requests, take a context.Context argument and act appropriately if that context is cancelled. The main providercache.Installer.EnsureProviderVersions method now also has some context-awareness so that it can abort its work early if its context reports any sort of error. That avoids waiting for the process to wind through all of the remaining iterations of the various loops, logging each request failure separately, and instead returns just a single aggregate "canceled" error. We can then set things up in the "terraform init" and "terraform providers mirror" commands so that the context will be cancelled if we get an interrupt signal, allowing provider installation to abort early while still atomically completing any local-side effects that may have started.	5 years ago
Martin Atkins	efe78b2910	main: new global option -chdir ... This new option is intended to address the previous inconsistencies where some older subcommands supported partially changing the target directory (where Terraform would use the new directory inconsistently) where newer commands did not support that override at all. Instead, now Terraform will accept a -chdir command at the start of the command line (before the subcommand) and will interpret it as a request to direct all actions that would normally be taken in the current working directory into the target directory instead. This is similar to options offered by some other similar tools, such as the -C option in "make". The new option is only accepted at the start of the command line (before the subcommand) as a way to reflect that it is a global command (not specific to a particular subcommand) and that it takes effect _before_ executing the subcommand. This also means it'll be forced to appear before any other command-specific arguments that take file paths, which hopefully communicates that those other arguments are interpreted relative to the overridden path. As a measure of pragmatism for existing uses, the path.cwd object in the Terraform language will continue to return the _original_ working directory (ignoring -chdir), in case that is important in some exceptional workflows. The path.root object gives the root module directory, which will always match the overriden working directory unless the user simultaneously uses one of the legacy directory override arguments, which is not a pattern we intend to support in the long run. As a first step down the deprecation path, this commit adjusts the documentation to de-emphasize the inconsistent old command line arguments, including specific guidance on what to use instead for the main three workflow commands, but all of those options remain supported in the same way as they were before. In a later commit we'll make those arguments produce a visible deprecation warning in Terraform's output, and then in an even later commit we'll remove them entirely so that -chdir is the single supported way to run Terraform from a directory other than the one containing the root module configuration.	6 years ago
Alisdair McDiarmid	35125717b2	Merge pull request #25847 from zachwhaley/fix-tf-cli-args-no-color ... Fix error when multiple -no-color arguments are used	6 years ago
Zachary Whaley	af8d5a69d8	Fix error when TF_CLI_ARGS adds -no-color argument ... Fixes #25845	6 years ago
Alisdair McDiarmid	b239570abb	command: Always validate workspace name ... The workspace name can be overridden by setting a TF_WORKSPACE environment variable. If this is done, we should still validate the resulting workspace name; otherwise, we could end up with an invalid and unselectable workspace. This change updates the Meta.Workspace function to return an error, and handles that error wherever necessary.	6 years ago
Paddy	5127f1ef8b	command: Unmanaged providers ... This adds supports for "unmanaged" providers, or providers with process lifecycles not controlled by Terraform. These providers are assumed to be started before Terraform is launched, and are assumed to shut themselves down after Terraform has finished running. To do this, we must update the go-plugin dependency to v1.3.0, which added support for the "test mode" plugin serving that powers all this. As a side-effect of not needing to manage the process lifecycle anymore, Terraform also no longer needs to worry about the provider's binary, as it won't be used for anything anymore. Because of this, we can disable the init behavior that concerns itself with downloading that provider's binary, checking its version, and otherwise managing the binary. This is all managed on a per-provider basis, so managed providers that Terraform downloads, starts, and stops can be used in the same commands as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable is added, and is a JSON encoding of the provider's address to the information we need to connect to it. This change enables two benefits: first, delve and other debuggers can now be attached to provider server processes, and Terraform can connect. This allows for attaching debuggers to provider processes, which before was difficult to impossible. Second, it allows the SDK test framework to host the provider in the same process as the test driver, while running a production Terraform binary against the provider. This allows for Go's built-in race detector and test coverage tooling to work as expected in provider tests. Unmanaged providers are expected to work in the exact same way as managed providers, with one caveat: Terraform kills provider processes and restarts them once per graph walk, meaning multiple times during most Terraform CLI commands. As unmanaged providers can't be killed by Terraform, and have no visibility into graph walks, unmanaged providers are likely to have differences in how their global mutable state behaves when compared to managed providers. Namely, unmanaged providers are likely to retain global state when managed providers would have reset it. Developers relying on global state should be aware of this.	6 years ago
Martin Atkins	549aede792	Remove terraform.ResourceProvider, use providercache.Installer instead ... Back when we first introduced provider versioning in Terraform 0.10, we did the provider version resolution in terraform.NewContext because we weren't sure yet how exactly our versioning model was going to play out (whether different versions could be selected per provider configuration, for example) and because we were building around the limitations of our existing filesystem-based plugin discovery model. However, the new installer codepath is new able to do all of the selections up front during installation, so we don't need such a heavy inversion of control abstraction to get this done: the command package can select the exact provider versions and pass their factories directly to terraform.NewContext as a simple static map. The result of this commit is that CLI commands other than "init" are now able to consume the local cache directory and selections produced by the installation process in "terraform init", passing all of the selected providers down to the terraform.NewContext function for use in implementing the main operations. This commit is just enough to get the providers passing into the terraform.Context. There's still plenty more to do here, including to repair all of the tests this change has additionally broken.	6 years ago
Alisdair McDiarmid	67203dade8	command: Simplify Meta.process helper method ... After some refactoring, this helper method had an unused argument (vars) and an always-nil error return value. This commit cleans this up.	6 years ago
Martin Atkins	e9d0822b2a	command: Accept a "provider source" from the main package ... Following the same approach we use for other CLI-Config-able objects like the service discovery system, the main package is responsible for producing a suitable implementation of this interface which the command package can then use. When unit testing in the command package we can then substitute mocks as necessary, following the dependency inversion principle.	6 years ago
Martin Atkins	c06675c616	command: New -compact-warnings option ... When warnings appear in isolation (not accompanied by an error) it's reasonable to want to defer resolving them for a while because they are not actually blocking immediate work. However, our warning messages tend to be long by default in order to include all of the necessary context to understand the implications of the warning, and that can make them overwhelming when combined with other output. As a compromise, this adds a new CLI option -compact-warnings which is supported for all the main operation commands and which uses a more compact format to print out warnings as long as they aren't also accompanied by errors. The default remains unchanged except that the threshold for consolidating warning messages is reduced to one so that we'll now only show one of each distinct warning summary. Full warning messages are always shown if there's at least one error included in the diagnostic set too, because in that case the warning message could contain additional context to help understand the error.	6 years ago
Martin Atkins	7081c26e54	command: Don't show more than two of the same warning ... Some of our warnings are produced in response to particular configuration constructs which might appear many times across a Terraform configuration. To avoid the warning output dwarfing all of the other output, we'll use ConsolidateWarnings to limit each distinct warning summary to appear at most twice, and annotate the final one in the sequence with an additional paragraph noting that some number of them have been hidden. This is intended as a compromise to ensure that these warnings are still seen and noted but to help ensure that we won't produce so many of them as to distract from other output that appears alongside them. This applies only to warnings relating to specific configuration ranges; errors will continue to be shown individually, and sourceless warnings (which are rare in Terraform today) will likewise remain ungrouped because they are less likely to be repeating the same statement about different instances of the same problem throughout the configuration.	6 years ago
Radek Simko	32f9722d9d	Replace import paths & set UA string where necessary	6 years ago
Martin Atkins	e21f0fa61e	backend/local: Handle interactive prompts for variables in UI layer ... During the 0.12 work we intended to move all of the variable value collection logic into the UI layer (command package and backend packages) and present them all together as a unified data structure to Terraform Core. However, we didn't quite succeed because the interactive prompts for unset required variables were still being handled _after_ calling into Terraform Core. Here we complete that earlier work by moving the interactive prompts for variables out into the UI layer too, thus allowing us to handle final validation of the variables all together in one place and do so in the UI layer where we have the most context still available about where all of these values are coming from. This allows us to fix a problem where previously disabling input with -input=false on the command line could cause Terraform Core to receive an incomplete set of variable values, and fail with a bad error message. As a consequence of this refactoring, the scope of terraform.Context.Input is now reduced to only gathering provider configuration arguments. Ideally that too would move into the UI layer somehow in a future commit, but that's a problem for another day.	6 years ago
Martin Atkins	7ccd6204c4	command: Swappable implementation of launching web browsers ... For unit testing in particular we can't launch a real browser for testing, so this indirection is primarily to allow us to substitute a mock when testing a command that can launch a browser. This includes a simple mock implementation that expects to interact with a running web server directly.	7 years ago
Martin Atkins	6bba3ceb42	command: "terraform login" command	7 years ago
Kristin Laemmert	c9d62bb2f6	command: discard output from flags package and return errs directly (#22373) ... Any command using meta.defaultFlagSet *might* occasionally exit before the flag package's output got written. This caused flag error messages to get lost. This PR discards the flag package output in favor of directly returning the error to the end user.	7 years ago
Alex Pilon	d5997e05db	prune dead code from command/meta	7 years ago
Masayuki Morita	ca9b643c73	docs: Fix typo in (*Meta) RunOperation comments	7 years ago
Sander van Harmelen	973e2a7cf9	core: add a context to the UIInput interface	7 years ago
Martin Atkins	2cf63d068f	command: Always normalize config path before operations ... Previously we were doing this rather inconsistently: some commands would do it and others would not. By doing it here we ensure we always apply the same normalization, regardless of which operation we're running. This normalization is mostly for cosmetic purposes in error messages, but it also ends up being used to populate path.module and path.root and so it's important that we always produce consistent results here so that we don't produce flappy changes as users work with different commands. The fact that thus mutates a data structure as a side-effect is not ideal but this is the best place to ensure it always gets applied without doing any significant refactoring, since everything after this point happens in the backend package where the normalizePath method is not available.	7 years ago
Sander van Harmelen	ef9054562e	commands: make sure the correct flagset is used ... A lot of commands used `c.Meta.flagSet()` to create the initial flagset for the command, while quite a few of them didn’t actually use or support the flags that are then added. So I updated a few commands to use `flag.NewFlagSet()` instead to only add the flags that are actually needed/supported. Additionally this prevents a few commands from using locking while they actually don’t need locking (as locking is enabled as a default in `c.Meta.flagSet()`.	7 years ago
Martin Atkins	e20346bf4f	command: fix TestMeta_process ... The mission of this process method used to include dealing with auto-loaded tfvars files, but it doesn't do that anymore. It does still deal with the -no-color option, but the test wasn't exercising that part before. Now the test here focuses on the -no-color behavior. The process method still has a "vars" flag argument which is no longer used. Since this is an unexported method we could potentially address this but this commit is intentionally limited only to fixing the test.	7 years ago
Sander van Harmelen	52a1b22f7a	Implement the remote enhanced backend ... This is a refactored version of the `remote` backend that was initially added to Terraform v0.11.8 which should now be compatible with v0.12.0.	7 years ago
Sander van Harmelen	178ec8f7b4	Remove support for the -module-depth flag ... # Conflicts: # backend/backend.go	7 years ago
Martin Atkins	541952bb8f	Revert some work that happened since v0.12-dev branched ... This work was done against APIs that were already changed in the branch before work began, and so it doesn't apply to the v0.12 development work. To allow v0.12 to merge down to master, we'll revert this work out for now and then re-introduce equivalent functionality in later commits that works against the new APIs.	7 years ago
Martin Atkins	d9051be66d	command: Use new provider and provisioner types in test helpers ... This moves our build errors from the test helpers out into the tests themselves. We'll update the tests in subsequent commits.	7 years ago
Martin Atkins	44bc7519a6	terraform: More wiring in of new provider types ... This doesn't actually work yet, but it builds and then panics in a pretty satisfying way.	7 years ago
Martin Atkins	a3403f2766	terraform: Ugly huge change to weave in new State and Plan types ... Due to how often the state and plan types are referenced throughout Terraform, there isn't a great way to switch them out gradually. As a consequence, this huge commit gets us from the old world to a _compilable_ new world, but still has a large number of known test failures due to key functionality being stubbed out. The stubs here are for anything that interacts with providers, since we now need to do the follow-up work to similarly replace the old terraform.ResourceProvider interface with its replacement in the new "providers" package. That work, along with work to fix the remaining failing tests, will follow in subsequent commits. The aim here was to replace all references to terraform.State and its downstream types with states.State, terraform.Plan with plans.Plan, state.State with statemgr.State, and switch to the new implementations of the state and plan file formats. However, due to the number of times those types are used, this also ended up affecting numerous other parts of core such as terraform.Hook, the backend.Backend interface, and most of the CLI commands. Just as with 5861dbf3fc before, I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.	7 years ago
Martin Atkins	0742e756e5	tfdiags: Sort order for diagnostics ... Because we gather together diagnostics from many different parts of the codebase, the list often ends up being in a non-ideal order. Here we define a partial ordering for diagnostics that should hopefully make them easier to scan when many are present, by grouping together diagnostics that are of the same severity and belong to the same file. We use sort.Stable here because we have a partial order and so we need to make sure that diagnostics that do not have a relative ordering will remain in their original order. This sorting is applied just in time before rendering the diagnostics in command.Meta.showDiagnostics.	7 years ago
Martin Atkins	c937c06a03	terraform: ugly huge change to weave in new HCL2-oriented types ... Due to how deeply the configuration types go into Terraform Core, there isn't a great way to switch out to HCL2 gradually. As a consequence, this huge commit gets us from the old state to a _compilable_ new state, but does not yet attempt to fix any tests and has a number of known missing parts and bugs. We will continue to iterate on this in forthcoming commits, heading back towards passing tests and making Terraform fully-functional again. The three main goals here are: - Use the configuration models from the "configs" package instead of the older models in the "config" package, which is now deprecated and preserved only to help us write our migration tool. - Do expression inspection and evaluation using the functionality of the new "lang" package, instead of the Interpolator type and related functionality in the main "terraform" package. - Represent addresses of various objects using types in the addrs package, rather than hand-constructed strings. This is not critical to support the above, but was a big help during the implementation of these other points since it made it much more explicit what kind of address is expected in each context. Since our new packages are built to accommodate some future planned features that are not yet implemented (e.g. the "for_each" argument on resources, "count"/"for_each" on modules), and since there's still a fair amount of functionality still using old-style APIs, there is a moderate amount of shimming here to connect new assumptions with old, hopefully in a way that makes it easier to find and eliminate these shims later. I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.	7 years ago
Martin Atkins	ebafa51723	command: Various updates for the new backend package API ... This is a rather-messy, complex change to get the "command" package building again against the new backend API that was updated for the new configuration loader. A lot of this is mechanical rewriting to the new API, but meta_config.go and meta_backend.go in particular saw some major changes to interface with the new loader APIs and to deal with the change in order of steps in the backend API.	7 years ago
Martin Atkins	fd5b7b42b8	command: new Meta methods for accessing the new config loader ... We need to share a single config loader across all callers because that allows us to maintain the source code cache we'll use for snippets in error messages. Nothing calls this yet. Callers will be gradually updated away from Module and Config in subsequent commits.	7 years ago
Martin Atkins	bd10b84a8e	command/format: include source snippets in diagnostics ... If we get a diagnostic message that references a source range, and if the source code for the referenced file is available, we'll show a snippet of the source code with the source range highlighted. At the moment we have no cache of source code, so in practice this codepath can never be visited. Callers to format.Diagnostic will be gradually updated in subsequent commits.	7 years ago
Sander van Harmelen	179b32d426	Add a `CredentialsForHost` method to disco.Disco ... By adding this method you now only have to pass a `*disco.Disco` object around in order to do discovery and use any configured credentials for the discovered hosts. Of course you can also still pass around both a `*disco.Disco` and a `auth.CredentialsSource` object if there is a need or a reason for that!	8 years ago
James Bardin	67a6152091	move backend operation cancellation into meta ... Create a single command method for running and operation with cancellation.	8 years ago
James Bardin	7d5f7cb22f	Merge pull request #16961 from hashicorp/jbardin/mock-provider-race ... minor race issue in mockResourceProvider	8 years ago
James Bardin	e63a3474d5	kill the flag error writer after 2 seconds ... There's no point in trying to track these, they're lost after each test. Kill them after a short delay so we don't have goroutines from every single command test to wade through if we have a stack dump.	8 years ago
James Bardin	885e4cde81	don't loop indefinitely in confirm method ... Only check for input twice in the meta.confirm method. This prevents an errant newline from aborting the run while allowing Terraform to exit if there is no input available. We don't just check for a tty, since we still rely on being able to pipe input in for testing. Remove the redundant confirmation loops in the migration code, and only use the confirm method.	8 years ago
James Bardin	2941ed464c	replace the testShutdownHook with a check for Stop ... Now that the local backend can be cancelled during plan and refresh, we don't really need the testShutdownHook. Simplify the tests by just checking for Stop being called on the provider.	8 years ago
James Bardin	e2501d7830	make apply shutdown test completely deterministic ... Add a shutdown hook to verify that a context has been correctly cancelled, so we can remove the sleep and stop guessing. Add a plan version of the shutdown test as well.	8 years ago
James Bardin	3aaa1e9d04	make plans cancellable ... There was no cancellation context for a plan, so it would always have to run to completion as SIGINT was being swallowed. Move the shutdown channel to the command Meta since it's used in multiple commands.	8 years ago
Martin Atkins	3da5fefdc1	command: Allow TF_DATA_DIR env var to override data directory ... This allows the user to customize the location where Terraform stores the files normally placed in the ".terraform" subdirectory, if e.g. the current working directory is not writable.	8 years ago
James Bardin	5203c66116	pass command credentials into module.Storage	8 years ago
James Bardin	3a495ffe56	rename ModuleStorage to Storage ... get rid of stutter and use module.Storage	8 years ago
James Bardin	f2a7b94692	use the new ModuleStorage in the command package ... Update the command package to use the new module storage. Move the old command output strings into the module storage itself. This could be moved back later either by using ui callbacks, or designing a module storage interface once we know what the final requirements will look like.	8 years ago
Martin Atkins	865e61b4ea	main+command: provide service disco and creds to commands ... The command package is the main place we need access to these, so that we can use them during init (to install packages, for example) and so that we can use them to configure remote backends. For the moment we're just providing an empty credentials object, which will start to include both statically-configured and helper-program-provided credentials sources in subsequent commits.	8 years ago
Martin Atkins	ea81e75a4e	command: utility for rendering tfdiag diagnostics ... This new method showDiagnostics takes any value that would be accepted by tfdiags.Append and renders it to the UI. This is intended to encourage consistent handling of the different kinds of errors and diagnostics that can be produced, and allow richer error objects like the HCL2 diagnostics to be easily unwrapped and shown in their full-fidelity.	9 years ago
Martin Atkins	5c0670fdf4	command: allow callers to activate a plugin cache ... Here we add a new caller-settable field to command.Meta that activates a read-through cache directory for plugin installation.	9 years ago
Jack Bruno	3f2136d7ee	cli: terraform import -ignore-missing-config ... This new option allows importing without configuration present. Configuration is required by default as a confirmation that the provided resource name is correct, but it can be useful to override this in tools that wrap Terraform to do more involved operations.	9 years ago
Martin Atkins	0fe43c8977	cli: allow disabling "next steps" message in terraform plan ... In #15884 we adjusted the plan output to give an explicit command to run to apply a plan, whereas before this command was just alluded to in the prose. Since releasing that, we've got good feedback that it's confusing to include such instructions when Terraform is running in a workflow automation tool, because such tools usually abstract away exactly what commands are run and require users to take different actions to proceed through the workflow. To accommodate such environments while retaining helpful messages for normal CLI usage, here we introduce a new environment variable TF_IN_AUTOMATION which, when set to a non-empty value, is a hint to Terraform that it isn't being run in an interactive command shell and it should thus tone down the "next steps" messaging. The documentation for this setting is included as part of the "...in automation" guide since it's not generally useful in other cases. We also intentionally disclaim comprehensive support for this since we want to avoid creating an extreme number of "if running in automation..." codepaths that would increase the testing matrix and hurt maintainability. The focus is specifically on the output of the three commands we give in the automation guide, which at present means the following two situations: * "terraform init" does not include the final paragraphs that suggest running "terraform plan" and tell you in what situations you might need to re-run "terraform init". * "terraform plan" does not include the final paragraphs that either warn about not specifying "-out=..." or instruct to run "terraform apply" with the generated plan file.	9 years ago