12 Commits (ccf19fbf5e2b61ba62efd1ac47a3a3fd3917d494)

Author	SHA1	Message	Date
James Bardin	f8893785f0	separate core and provider loggers ... Now that hclog can independently set levels on related loggers, we can separate the log levels for different subsystems in terraform. This adds the new environment variables, `TF_LOG_CORE` and `TF_LOG_PROVIDER`, which each take the same set of log level arguments, and only applies to logs from that subsystem. This means that setting `TF_LOG_CORE=level` will not show logs from providers, and `TF_LOG_PROVIDER=level` will not show logs from core. The behavior of `TF_LOG` alone does not change. While it is not necessarily needed since the default is to disable logs, there is also a new level argument of `off`, which reflects the associated level in hclog.	6 years ago
James Bardin	c2af5333e8	use a log sink to capture logs for panicwrap ... Use a separate log sink to always capture trace logs for the panicwrap handler to write out in a crash log. This requires creating a log file in the outer process and passing that path to the child process to log to.	6 years ago
James Bardin	abf6b9b378	get properly configured hcloggers ... make sure plugins get hcloggers configured to match core	6 years ago
Martin Atkins	30204ecded	command/cliconfig: Allow development overrides for providers ... For normal provider installation we want to associate each provider with a selected version number and find a suitable package for that version that conforms to the official hashes for that release. Those requirements are very onerous for a provider developer currently testing a not-yet-released build, though. To allow for that case this new CLI configuration feature allows overriding specific providers to refer to give local filesystem directories. Any provider overridden in this way is not subject to the usual restrictions about selected versions or checksum conformance, and activating an override won't cause any changes to the selections recorded in the lock file because it's intended to be a temporary setting for one developer only. This is, in a sense, a spiritual successor of an old capability we had to override specific plugins in the CLI configuration file. There were some vestiges of that left in the main package and CLI config package but nothing has actually been honoring them for several versions now and so this commit removes them to avoid confusion with the new mechanism.	6 years ago
Martin Atkins	e70ab09bf1	command: new cache directory .terraform/providers for providers ... Terraform v0.10 introduced .terraform/plugins as a cache directory for automatically-installed plugins, Terraform v0.13 later reorganized the directory structure inside but retained its purpose as a cache. The local cache used to also serve as a record of specifically which packages were selected in a particular working directory, with the intent that a second run of "terraform init" would always select the same packages again. That meant that in some sense it behaved a bit like a local filesystem mirror directory, even though that wasn't its intended purpose. Due to some unfortunate miscommunications, somewhere a long the line we published some documentation that _recommended_ using the cache directory as if it were a filesystem mirror directory when working with Terraform Cloud. That was really only working as an accident of implementation details, and Terraform v0.14 is now going to break that because the source of record for the currently-selected provider versions is now the public-facing dependency lock file rather than the contents of an existing local cache directory on disk. After some consideration of how to move forward here, this commit implements a compromise that tries to avoid silently doing anything surprising while still giving useful guidance to folks who were previously using the unsupported strategy. Specifically: - The local cache directory will now be .terraform/providers rather than .terraform/plugins, because .terraform/plugins is effectively "poisoned" by the incorrect usage that we can't reliably distinguish from prior version correct usage. - The .terraform/plugins directory is now the "legacy cache directory". It is intentionally _not_ now a filesystem mirror directory, because that would risk incorrectly interpreting providers automatically installed by Terraform v0.13 as if they were a local mirror, and thus upgrades and checksum fetches from the origin registry would be blocked. - Because of the previous two points, someone who _was_ trying to use the legacy cache directory as a filesystem mirror would see installation fail for any providers they manually added to the legacy directory. To avoid leaving that user stumped as to what went wrong, there's a heuristic for the case where a non-official provider fails installation and yet we can see it in the legacy cache directory. If that heuristic matches then we'll produce a warning message hinting to move the provider under the terraform.d/plugins directory, which is a _correct_ location for "bundled" provider plugins that belong only to a single configuration (as opposed to being installed globally on a system). This does unfortunately mean that anyone who was following the incorrectly-documented pattern will now encounter an error (and the aforementioned warning hint) after upgrading to Terraform v0.14. This seems like the safest compromise because Terraform can't automatically infer the intent of files it finds in .terraform/plugins in order to decide automatically how best to handle them. The internals of the .terraform directory are always considered implementation detail for a particular Terraform version and so switching to a new directory for the _actual_ cache directory fits within our usual set of guarantees, though it's definitely non-ideal in isolation but okay when taken in the broader context of this problem, where the alternative would be silent misbehavior when upgrading.	6 years ago
Martin Atkins	b3f5c7f1e6	command/init: Read, respect, and update provider dependency locks ... This changes the approach used by the provider installer to remember between runs which selections it has previously made, using the lock file format implemented in internal/depsfile. This means that version constraints in the configuration are considered only for providers we've not seen before or when -upgrade mode is active.	6 years ago
Alisdair McDiarmid	8e87ccb689	providercache: Lazily detect executable file ... Instead of searching the installed provider package directory for a binary as we install it, we can lazily detect the executable as it is required. Doing so allows us to separately report an invalid unpacked package, giving the user more actionable error messages.	6 years ago
Paddy	5127f1ef8b	command: Unmanaged providers ... This adds supports for "unmanaged" providers, or providers with process lifecycles not controlled by Terraform. These providers are assumed to be started before Terraform is launched, and are assumed to shut themselves down after Terraform has finished running. To do this, we must update the go-plugin dependency to v1.3.0, which added support for the "test mode" plugin serving that powers all this. As a side-effect of not needing to manage the process lifecycle anymore, Terraform also no longer needs to worry about the provider's binary, as it won't be used for anything anymore. Because of this, we can disable the init behavior that concerns itself with downloading that provider's binary, checking its version, and otherwise managing the binary. This is all managed on a per-provider basis, so managed providers that Terraform downloads, starts, and stops can be used in the same commands as unmanaged providers. The TF_REATTACH_PROVIDERS environment variable is added, and is a JSON encoding of the provider's address to the information we need to connect to it. This change enables two benefits: first, delve and other debuggers can now be attached to provider server processes, and Terraform can connect. This allows for attaching debuggers to provider processes, which before was difficult to impossible. Second, it allows the SDK test framework to host the provider in the same process as the test driver, while running a production Terraform binary against the provider. This allows for Go's built-in race detector and test coverage tooling to work as expected in provider tests. Unmanaged providers are expected to work in the exact same way as managed providers, with one caveat: Terraform kills provider processes and restarts them once per graph walk, meaning multiple times during most Terraform CLI commands. As unmanaged providers can't be killed by Terraform, and have no visibility into graph walks, unmanaged providers are likely to have differences in how their global mutable state behaves when compared to managed providers. Namely, unmanaged providers are likely to retain global state when managed providers would have reset it. Developers relying on global state should be aware of this.	6 years ago
Martin Atkins	958ea4f7d1	internal/providercache: Handle built-in providers ... Built-in providers are special providers that are distributed as part of Terraform CLI itself, rather than being installed separately. They always live in the terraform.io/builtin/... namespace so it's easier to see that they are special, and currently there is only one built-in provider named "terraform". Previous commits established the addressing scheme for built-in providers. This commit makes the installer aware of them to the extent that it knows not to try to install them the usual way and it's able to report an error if the user requests a built-in provider that doesn't exist or tries to impose a particular version constraint for a built-in provider. For the moment the tests for this are the ones in the "command" package because that's where the existing testing infrastructure for this functionality lives. A later commit should add some more focused unit tests here in the internal/providercache package, too.	6 years ago
Martin Atkins	c4fb22863c	command/init: Support -plugin-dir again ... This is a slightly different approach than we used to take for this option: rather than disabling the installer and causing all future commands to look elsewhere for plugins, we'll now leave the installer enabled by constrain it to only look at the given directories. This is overall simpler because it doesn't require any special tracking of the plugin directories for subsequent commands. Instead, the selections file generated by the installer will record the versions it selected from the specified directories, and we'll link them in to the local cache just as we would normally so that other commands don't need to do anything special to select the right plugins in either case.	6 years ago
Martin Atkins	549aede792	Remove terraform.ResourceProvider, use providercache.Installer instead ... Back when we first introduced provider versioning in Terraform 0.10, we did the provider version resolution in terraform.NewContext because we weren't sure yet how exactly our versioning model was going to play out (whether different versions could be selected per provider configuration, for example) and because we were building around the limitations of our existing filesystem-based plugin discovery model. However, the new installer codepath is new able to do all of the selections up front during installation, so we don't need such a heavy inversion of control abstraction to get this done: the command package can select the exact provider versions and pass their factories directly to terraform.NewContext as a simple static map. The result of this commit is that CLI commands other than "init" are now able to consume the local cache directory and selections produced by the installation process in "terraform init", passing all of the selected providers down to the terraform.NewContext function for use in implementing the main operations. This commit is just enough to get the providers passing into the terraform.Context. There's still plenty more to do here, including to repair all of the tests this change has additionally broken.	6 years ago
Martin Atkins	754b7ebb65	command: Expose providercache package objects for use elsewhere ... These new functions allow command implementations to get hold of the providercache objects and installation source object derived from the current CLI configuration.	6 years ago