75 Commits (ccf19fbf5e2b61ba62efd1ac47a3a3fd3917d494)

Author	SHA1	Message	Date
James Bardin	0c3bb316ea	redirect warnigns to stdout ... The default cli Warn calls always write to the error writer (stderr by default), however the output is intended to be viewed in the UI by the user, rather than in a separate stream. Terraform also generally does not consider warnings to be errors from the cli point of view, and does not need to output the warning text to stderr. By redirecting Warn calls to Output calls at the lowest level in the main package, we can eliminate the chance that Warn and Output messages are interleaved, while still allowing the internal `cli.Ui` implementations to format `Warn` and `Output` calls separately.	5 years ago
Martin Atkins	28d2cb55fd	main: Special error message for invalid top-level command ... Previously Terraform would react to an invalid top-level command the same way as for typing no command at all: just printing out the long top-level help directory. If someone's tried to type a command, it's more helpful to respond to that request by explaining directly that the command is invalid, rather than leaving the user to puzzle that out themselves by referring to the help text. As a bonus, this also allows us to use our "didyoumean" package to suggest possible alternatives if it seems like the user made a typo.	5 years ago
James Bardin	3225d9ac11	record all plugin panics, and print on main exit ... Create a logger that will record any apparent crash output for later processing. If the cli command returns with a non-zero exit status, check for any recorded crashes and add those to the output.	6 years ago
James Bardin	f8893785f0	separate core and provider loggers ... Now that hclog can independently set levels on related loggers, we can separate the log levels for different subsystems in terraform. This adds the new environment variables, `TF_LOG_CORE` and `TF_LOG_PROVIDER`, which each take the same set of log level arguments, and only applies to logs from that subsystem. This means that setting `TF_LOG_CORE=level` will not show logs from providers, and `TF_LOG_PROVIDER=level` will not show logs from core. The behavior of `TF_LOG` alone does not change. While it is not necessarily needed since the default is to disable logs, there is also a new level argument of `off`, which reflects the associated level in hclog.	6 years ago
James Bardin	c2af5333e8	use a log sink to capture logs for panicwrap ... Use a separate log sink to always capture trace logs for the panicwrap handler to write out in a crash log. This requires creating a log file in the outer process and passing that path to the child process to log to.	6 years ago
James Bardin	fd4f7eb0b9	remove prefixed io ... The main process is now handling what output to print, so it doesn't do any good to try and run it through prefixedio, which is only adding extra coordination to echo the same data.	6 years ago
James Bardin	1d9d82973b	move panicwrap handler to logging package	6 years ago
James Bardin	211edf5d75	use hclog as the default logger ... Inject hclog as the default logger in the main binary.	6 years ago
James Bardin	6ca477f042	move helper/logging to internal ... remove a dead code file too	6 years ago
Martin Atkins	30204ecded	command/cliconfig: Allow development overrides for providers ... For normal provider installation we want to associate each provider with a selected version number and find a suitable package for that version that conforms to the official hashes for that release. Those requirements are very onerous for a provider developer currently testing a not-yet-released build, though. To allow for that case this new CLI configuration feature allows overriding specific providers to refer to give local filesystem directories. Any provider overridden in this way is not subject to the usual restrictions about selected versions or checksum conformance, and activating an override won't cause any changes to the selections recorded in the lock file because it's intended to be a temporary setting for one developer only. This is, in a sense, a spiritual successor of an old capability we had to override specific plugins in the CLI configuration file. There were some vestiges of that left in the main package and CLI config package but nothing has actually been honoring them for several versions now and so this commit removes them to avoid confusion with the new mechanism.	6 years ago
Pam Selle	328baaad84	Fix un-saved error on chdir	6 years ago
Martin Atkins	efe78b2910	main: new global option -chdir ... This new option is intended to address the previous inconsistencies where some older subcommands supported partially changing the target directory (where Terraform would use the new directory inconsistently) where newer commands did not support that override at all. Instead, now Terraform will accept a -chdir command at the start of the command line (before the subcommand) and will interpret it as a request to direct all actions that would normally be taken in the current working directory into the target directory instead. This is similar to options offered by some other similar tools, such as the -C option in "make". The new option is only accepted at the start of the command line (before the subcommand) as a way to reflect that it is a global command (not specific to a particular subcommand) and that it takes effect _before_ executing the subcommand. This also means it'll be forced to appear before any other command-specific arguments that take file paths, which hopefully communicates that those other arguments are interpreted relative to the overridden path. As a measure of pragmatism for existing uses, the path.cwd object in the Terraform language will continue to return the _original_ working directory (ignoring -chdir), in case that is important in some exceptional workflows. The path.root object gives the root module directory, which will always match the overriden working directory unless the user simultaneously uses one of the legacy directory override arguments, which is not a pattern we intend to support in the long run. As a first step down the deprecation path, this commit adjusts the documentation to de-emphasize the inconsistent old command line arguments, including specific guidance on what to use instead for the main three workflow commands, but all of those options remain supported in the same way as they were before. In a later commit we'll make those arguments produce a visible deprecation warning in Terraform's output, and then in an even later commit we'll remove them entirely so that -chdir is the single supported way to run Terraform from a directory other than the one containing the root module configuration.	6 years ago
Alisdair McDiarmid	f1f24df7ff	main: Pass untyped nil for missing creds source ... If we are unable to create a credentials source for some reason, we can rely on the disco object to nil-check it before calling any of its methods. However to do this we must ensure that we pass untyped nil. This commit rearranges the initialization to ensure that this happens. The user-facing bug that triggered this work is that running init when the HOME environment variable is unset would result in a panic on macOS.	6 years ago
Paddy	5127f1ef8b	command: Unmanaged providers ... This adds supports for "unmanaged" providers, or providers with process lifecycles not controlled by Terraform. These providers are assumed to be started before Terraform is launched, and are assumed to shut themselves down after Terraform has finished running. To do this, we must update the go-plugin dependency to v1.3.0, which added support for the "test mode" plugin serving that powers all this. As a side-effect of not needing to manage the process lifecycle anymore, Terraform also no longer needs to worry about the provider's binary, as it won't be used for anything anymore. Because of this, we can disable the init behavior that concerns itself with downloading that provider's binary, checking its version, and otherwise managing the binary. This is all managed on a per-provider basis, so managed providers that Terraform downloads, starts, and stops can be used in the same commands as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable is added, and is a JSON encoding of the provider's address to the information we need to connect to it. This change enables two benefits: first, delve and other debuggers can now be attached to provider server processes, and Terraform can connect. This allows for attaching debuggers to provider processes, which before was difficult to impossible. Second, it allows the SDK test framework to host the provider in the same process as the test driver, while running a production Terraform binary against the provider. This allows for Go's built-in race detector and test coverage tooling to work as expected in provider tests. Unmanaged providers are expected to work in the exact same way as managed providers, with one caveat: Terraform kills provider processes and restarts them once per graph walk, meaning multiple times during most Terraform CLI commands. As unmanaged providers can't be killed by Terraform, and have no visibility into graph walks, unmanaged providers are likely to have differences in how their global mutable state behaves when compared to managed providers. Namely, unmanaged providers are likely to retain global state when managed providers would have reset it. Developers relying on global state should be aware of this.	6 years ago
Martin Atkins	94b87e056b	fixup main.go comment about providersource	6 years ago
Martin Atkins	5af1e6234a	main: Honor explicit provider_installation CLI config when present ... If the CLI configuration contains a provider_installation block then we'll use the source configuration it describes instead of the implied one we'd build otherwise.	6 years ago
Martin Atkins	8c928e8358	main: Consult local directories as potential mirrors of providers ... This restores some of the local search directories we used to include when searching for provider plugins in Terraform 0.12 and earlier. The directory structures we are expecting in these are different than before, so existing directory contents will not be compatible without restructuring, but we need to retain support for these local directories so that users can continue to sideload third-party provider plugins until the explicit, first-class provider mirrors configuration (in CLI config) is implemented, at which point users will be able to override these to whatever directories they want. This also includes some new search directories that are specific to the operating system where Terraform is running, following the documented layout conventions of that platform. In particular, this follows the XDG Base Directory specification on Unix systems, which has been a somewhat-common request to better support "sideloading" of packages via standard Linux distribution package managers and other similar mechanisms. While it isn't strictly necessary to add that now, it seems ideal to do all of the changes to our search directory layout at once so that our documentation about this can cleanly distinguish "0.12 and earlier" vs. "0.13 and later", rather than having to document a complex sequence of smaller changes. Because this behavior is a result of the integration of package main with package command, this behavior is verified using an e2etest rather than a unit test. That test, TestInitProvidersVendored, is also fixed here to create a suitable directory structure for the platform where the test is being run. This fixes TestInitProvidersVendored.	6 years ago
Martin Atkins	e9d0822b2a	command: Accept a "provider source" from the main package ... Following the same approach we use for other CLI-Config-able objects like the service discovery system, the main package is responsible for producing a suitable implementation of this interface which the command package can then use. When unit testing in the command package we can then substitute mocks as necessary, following the dependency inversion principle.	6 years ago
Pam Selle	78b1220558	Remove config.go and update things using its aliases	6 years ago
Radek Simko	32f9722d9d	Replace import paths & set UA string where necessary	7 years ago
Martin Atkins	cfc1c4900d	command/login: Use Cli.Ask to request confirmation ... This is more straightforward than using readline because it already works properly with panicwrap.	7 years ago
Martin Atkins	22a2580e93	main: Use the new cliconfig package credentials source ... This should not cause any change in behavior yet, but using this new implementation will allow the "terraform login" and "terraform logout" commands to store and forget credentials when they are implemented in subsequent commits.	7 years ago
Martin Atkins	e1590d0a70	command/cliconfig: Factor out CLI config handling ... This is just a wholesale move of the CLI configuration types and functions from the main package into its own package, leaving behind some type aliases and wrappers for now to keep existing callers working. This commit alone doesn't really achieve anything, but in future commits we'll expand the functionality in this package.	7 years ago
Matthew Sanabria	be04e70a0f	Adding documentation for TF_CLI_CONFIG_FILE environment variable (#20834) ... Fixes #15849	7 years ago
Sander van Harmelen	52a1b22f7a	Implement the remote enhanced backend ... This is a refactored version of the `remote` backend that was initially added to Terraform v0.11.8 which should now be compatible with v0.12.0.	8 years ago
Martin Atkins	a3403f2766	terraform: Ugly huge change to weave in new State and Plan types ... Due to how often the state and plan types are referenced throughout Terraform, there isn't a great way to switch them out gradually. As a consequence, this huge commit gets us from the old world to a _compilable_ new world, but still has a large number of known test failures due to key functionality being stubbed out. The stubs here are for anything that interacts with providers, since we now need to do the follow-up work to similarly replace the old terraform.ResourceProvider interface with its replacement in the new "providers" package. That work, along with work to fix the remaining failing tests, will follow in subsequent commits. The aim here was to replace all references to terraform.State and its downstream types with states.State, terraform.Plan with plans.Plan, state.State with statemgr.State, and switch to the new implementations of the state and plan file formats. However, due to the number of times those types are used, this also ended up affecting numerous other parts of core such as terraform.Hook, the backend.Backend interface, and most of the CLI commands. Just as with 5861dbf3fc before, I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.	8 years ago
Martin Atkins	bd10b84a8e	command/format: include source snippets in diagnostics ... If we get a diagnostic message that references a source range, and if the source code for the referenced file is available, we'll show a snippet of the source code with the source range highlighted. At the moment we have no cache of source code, so in practice this codepath can never be visited. Callers to format.Diagnostic will be gradually updated in subsequent commits.	8 years ago
Sander van Harmelen	7fb2d1b8de	Implement the Enterprise enhanced remote backend	8 years ago
Sander van Harmelen	179b32d426	Add a `CredentialsForHost` method to disco.Disco ... By adding this method you now only have to pass a `*disco.Disco` object around in order to do discovery and use any configured credentials for the discovered hosts. Of course you can also still pass around both a `*disco.Disco` and a `auth.CredentialsSource` object if there is a need or a reason for that!	8 years ago
Martin Atkins	275ab4a74e	main: don't print the CLI config into the logs ... Now that we're expecting "credentials" blocks in the config (with auth tokens for private module registries, etc) we should not print out the config contents into the log, or else people will probably end up accidentally disclosing their credentials when sharing debug output with us, or will be reluctant to share debug output.	9 years ago
Martin Atkins	11ba1d2a4c	main: factor out CLI config loading into its own function ... Previously we handled all of the config sources directly within the main function. We're going to make CLI config loading more complex shortly, so having this encapsulated in its own function will avoid creating even more clutter inside the main function. Along the way here we also switch from using native Go "error" to using tfdiags.Diagnostics, so that we can potentially issue warnings here too in future, and so that we can return multiple errors.	9 years ago
Martin Atkins	a2c59c6ecd	main: validate credentials blocks in CLI config ... We require that each "credentials" block has a valid hostname and that there be no more than one "credentials_helper" block. There are some more sophisticated validations we could do here, such as checking if the same host is declared more than once, but since this config handling will be rewritten to use HCL2 in the near future, and this sort of check is easier to do in the HCL2 API, we just check the basic stuff for now and plan to revisit later.	9 years ago
Martin Atkins	cb17a9a607	main: allow enabling plugin caching via config file or environment ... Either the environment variable TF_PLUGIN_CACHE_DIR or a setting in the CLI config file (~/.terraformrc on Unix) allow opting in to the plugin caching behavior. This is opt-in because for new users we don't want to pollute their system with extra directories they don't know about. By opting in to caching, the user is assuming the responsibility to occasionally prune the cache over time as older plugins become stale and unused.	9 years ago
Martin Atkins	3f401f0cd4	main: make configuration available when initializing commands ... This, in principle, allows us to make use of configuration information when we populate the Meta structure, though we won't actually make use of that until a subsequent commit.	9 years ago
Martin Atkins	9b5ae9143a	main: enable basic subcommand autocomplete ... The CLI package has automatic support for shell autocomplete (bash and zsh, at time of writing) for subcommands, so all we need to do here is just opt into it. Users can install this into their shells by running: terraform -install-autocomplete	9 years ago
Martin Atkins	8364383c35	Push plugin discovery down into command package ... Previously we did plugin discovery in the main package, but as we move towards versioned plugins we need more information available in order to resolve plugins, so we move this responsibility into the command package itself. For the moment this is just preserving the existing behavior as long as there are only internal and unversioned plugins present. This is the final state for provisioners in 0.10, since we don't want to support versioned provisioners yet. For providers this is just a checkpoint along the way, since further work is required to apply version constraints from configuration and support additional plugin search directories. The automatic plugin discovery behavior is not desirable for tests because we want to mock the plugins there, so we add a new backdoor for the tests to use to skip the plugin discovery and just provide their own mock implementations. Most of this diff is thus noisy rework of the tests to use this new mechanism.	9 years ago
Martin Atkins	5ac311e2a9	main: synchronize writes to VT100-faker on Windows ... We use a third-party library "colorable" to translate VT100 color sequences into Windows console attribute-setting calls when Terraform is running on Windows. colorable is not concurrency-safe for multiple writes to the same console, because it writes to the console one character at a time and so two concurrent writers get their characters interleaved, creating unreadable garble. Here we wrap around it a synchronization mechanism to ensure that there can be only one Write call outstanding across both stderr and stdout, mimicking the usual behavior we expect (when stderr/stdout are a normal file handle) of each Write being completed atomically.	9 years ago
Mitchell Hashimoto	3dfc46abd3	log the Go runtime version at TF startup ... This will help us debug issues which have been caused by Go bugs.	9 years ago
Mitchell Hashimoto	518ae5ef02	support nested subcommands with TF_CLI_ARGS	9 years ago
Mitchell Hashimoto	df93e5120c	allow targeted TF_CLI_ARGS_x	9 years ago
Mitchell Hashimoto	53796fcdb0	test for blank args for TF_CLI_ARGS	9 years ago
Mitchell Hashimoto	7f67b32169	main: add TF_CLI_ARGS to specify additional CLI args	9 years ago
Mitchell Hashimoto	e9c35eae32	Forward SIGTERM and handle that as an interrupt	9 years ago
Mitchell Hashimoto	557375f2a2	Update panicwrap to pass through all interrupt signals	9 years ago
Justin Nauman	bf48d0132c	Checkpoint signature fixes ... - Currently the disable_checkpoint setting from $HOME/.terraformrc is never respsected due to: -- The runCheckpoint go routine being fired off prior to loading configuration -- The config.Merge method not actually merging in the c2s settings	10 years ago
James Bardin	797a1b339d	DebugInfo and DebugGraph ... Implement debugInfo and the DebugGraph DebugInfo will be a global variable through which graph debug information can we written to a compressed archive. The DebugInfo methods are all safe for concurrent use, and noop with a nil receiver. The API outside of the terraform package will be to call SetDebugInfo to create the archive, and CloseDebugInfo() to properly close the file. Each write to the archive will be flushed and sync'ed individually, so in the event of a crash or a missing call to Close, the archive can still be recovered. The DebugGraph is a representation of a terraform Graph to be written to the debug archive, currently in dot format. The DebugGraph also contains an internal buffer with Printf and Write methods to add to this buffer. The buffer will be written to an accompanying file in the debug archive along with the graph. This also adds a GraphNodeDebugger interface. Any node implementing `NodeDebug() string` can output information to annotate the debug graph node, and add the data to the log. This interface may change or be removed to provide richer options for debugging graph nodes. The new graph builders all delegate the build to the BasicGraphBuilder. Having a Name field lets us differentiate the actual builder implementation in the debug graphs.	10 years ago
James Bardin	6b7a079bff	Add a way to avoid panicwrap when debugging ... When trying to use a debugger, you don't want the terraform to run in a new child process. Setting TF_FORK=0 will skip panicwrap and continue running the program in the current process. This will also give direct access to log output, and stdout.	10 years ago
Mitchell Hashimoto	b8036095d4	Logs contain CLI args	10 years ago
Chris Bednarski	3c774af9c2	Warn when an internal plugin is overridden ... Also added documentation explaining what happened and how to fix it	10 years ago
Chris Bednarski	e942a74def	Set a log prefix for each plugin and remove go-dynect global log prefix (#6336)	10 years ago