173 Commits (ccf19fbf5e2b61ba62efd1ac47a3a3fd3917d494)

Author	SHA1	Message	Date
James Bardin	ccf19fbf5e	regenerate plugin proto with correct tooling ... The proto file is structurally unchanged, though it now requires a go import path option.	5 years ago
James Bardin	7eb491719f	normalize temp dir paths in tests ... The temporary directory on some systems (most notably MacOS) contains symlinks, which would not be recorded by the installer. In order to make these paths comparable in the tests we need to eval the symlinks in the paths before giving them to the installer.	5 years ago
James Bardin	d52e17e111	hide provider crashes from panicwrap when logging ... When logging is turned on, panicwrap will still see provider crashes and falsely report them as core crashes, hiding the formatted provider error. We can trick panicwrap by slightly obfuscating the error line.	6 years ago
Alisdair McDiarmid	fd43bc7847	Merge pull request #26783 from hashicorp/alisdair/normalize-version-constraints-before-deduplication ... getproviders: Normalize versions before dedupe	6 years ago
Alisdair McDiarmid	90c4f11b61	go get github.com/hashicorp/go-getter@v1.5.0 ... Includes fix for breaking upstream API change.	6 years ago
Alisdair McDiarmid	b1bc0e5d92	getproviders: Normalize versions before dedupe ... When rendering a set of version constraints to a string, we normalize partially-constrained versions. This means converting a version like 2.68.* to 2.68.0. Prior to this commit, this normalization was done after deduplication. This could result in a version constraints string with duplicate entries, if multiple partially-constrained versions are equivalent. This commit fixes this by normalizing before deduplicating and sorting.	6 years ago
Martin Atkins	e6e0b6ee46	providercache: verify locked hashes for local package dirs ... Previously we were only verifying locked hashes for local archive zip files, but if we have non-ziphash hashes available then we can and should also verify that a local directory matches at least one of them. This does mean that folks using filesystem mirrors but yet also running Terraform across multiple platforms will need to take some extra care to ensure the hashes pass on all relevant platforms, which could mean using "terraform providers lock" to pre-seed their lock files with hashes across all platforms, or could mean using the "packed" directory layout for the filesystem mirror so that Terraform will end up in the install-from-archive codepath instead of this install-from-directory codepath, and can thus verify ziphash too. (There's no additional documentation about the above here because there's already general information about this in the lock file documentation due to some similar -- though not identical -- situations with network mirrors.)	6 years ago
Martin Atkins	24d32e9ca2	providercache: More exhaustive testing of the main installer ... We previously had some tests for some happy paths and a few specific failures into an empty directory with no existing locks, but we didn't have tests for the installer respecting existing lock file entries. This is a start on a more exhaustive set of tests for the installer, aiming to visit as many of the possible codepaths as we can reasonably test using this mocking strategy. (Some other codepaths require different underlying source implementations, etc, so we'll have to visit those in other tests separately.)	6 years ago
Martin Atkins	fc5a41b5e5	depsfile: Allow loading locks from a byte array in memory ... This won't be a typical usage pattern for normal code, but will be useful for tests that need to work with locks as input so that they don't need to write out a temporary file on disk just to read it back in immediately.	6 years ago
Martin Atkins	430318e262	getproviders: Consistent ordering of terms in VersionConstraintsString ... An earlier commit made this remove duplicates, which set the precedent that this function is trying to canonically represent the _meaning_ of the version constraints rather than exactly how they were expressed in the configuration. Continuing in that vein, now we'll also apply a consistent (though perhaps often rather arbitrary) ordering to the terms, so that it doesn't change due to irrelevant details like declarations being written in a different order in the configuration. The ordering here is intended to be reasonably intuitive for simple cases, but constraint strings with many different constraints are hard to interpret no matter how we order them so the main goal is consistency, so those watching how the constraints change over time (e.g. in logs of Terraform output, or in the dependency log file) will see fewer noisy changes that don't actually mean anything.	6 years ago
Alisdair McDiarmid	8eed942482	Merge pull request #26678 from hashicorp/alisdair/suppress-duplicate-version-constraints ... internal: Suppress duplicate version constraints	6 years ago
James Bardin	3225d9ac11	record all plugin panics, and print on main exit ... Create a logger that will record any apparent crash output for later processing. If the cli command returns with a non-zero exit status, check for any recorded crashes and add those to the output.	6 years ago
James Bardin	f8893785f0	separate core and provider loggers ... Now that hclog can independently set levels on related loggers, we can separate the log levels for different subsystems in terraform. This adds the new environment variables, `TF_LOG_CORE` and `TF_LOG_PROVIDER`, which each take the same set of log level arguments, and only applies to logs from that subsystem. This means that setting `TF_LOG_CORE=level` will not show logs from providers, and `TF_LOG_PROVIDER=level` will not show logs from core. The behavior of `TF_LOG` alone does not change. While it is not necessarily needed since the default is to disable logs, there is also a new level argument of `off`, which reflects the associated level in hclog.	6 years ago
Alisdair McDiarmid	fe9a9fadfb	internal: Suppress duplicate version constraints ... A set of version constraints can contain duplicates. This can happen if multiple identical constraints are specified throughout a configuration. When rendering the set, it is confusing to display redundant constraints. This commit changes the string renderer to only show the first instance of a given constraint, and adds unit tests for this function to cover this change. This also fixes a bug with the locks file generation: previously, a configuration with redundant constraints would result in this error on second init: Error: Invalid provider version constraints on .terraform.lock.hcl line 6: (source code not available) The recorded version constraints for provider registry.terraform.io/hashicorp/random must be written in normalized form: "3.0.0".	6 years ago
James Bardin	bc1a841d65	Merge pull request #26665 from hashicorp/jbardin/logging ... Restore "crash.log" behavior and remove prefixedio	6 years ago
James Bardin	c2af5333e8	use a log sink to capture logs for panicwrap ... Use a separate log sink to always capture trace logs for the panicwrap handler to write out in a crash log. This requires creating a log file in the outer process and passing that path to the child process to log to.	6 years ago
James Bardin	b61488a8ba	write traceback to log crash log	6 years ago
James Bardin	1d9d82973b	move panicwrap handler to logging package	6 years ago
Alisdair McDiarmid	7a31e56cb7	Merge pull request #26637 from hashicorp/alisdair/fix-locksfile-unconstrained-versions ... internal: Fix lockfile constraint output for 1.2.*	6 years ago
Martin Atkins	1fe314b70b	getproviders: Don't log if EvalSymlinks changes nothing ... Previously this codepath was generating a confusing message in the absense of any symlinks, because filepath.EvalSymlinks returns a successful result if the target isn't a symlink. Now we'll emit the log line only if filepath.EvalSymlinks returns a result that's different in a way that isn't purely syntactic (which filepath.Clean would "fix"). The new message is a little more generic because technically we've not actually ensured that a difference here was caused by a symlink and so we shouldn't over-promise and generate something potentially misleading.	6 years ago
Martin Atkins	2332a7ab47	replacefile: don't create the temporary file in TMPDIR ... ioutil.TempFile has a special case where an empty string for its dir argument is interpreted as a request to automatically look up the system temporary directory, which is commonly /tmp . We don't want that behavior here because we're specifically trying to create the temporary file in the same directory as the file we're hoping to replace. If the file gets created in /tmp then it might be on a different device and thus the later atomic rename won't work. Instead, we'll add our own special case to explicitly use "." when the given filename is in the current working directory. That overrides the special automatic behavior of ioutil.TempFile and thus forces the behavior we need. This hadn't previously mattered for earlier callers of this code because they were creating files in subdirectories, but this codepath was failing for the dependency lock file due to it always being created directly in the current working directory. Unfortunately since this is a picky implementation detail I couldn't find a good way to write a unit test for it without considerable refactoring. Instead, I verified manually that the temporary filename wasn't in /tmp on my Linux system, and hope that the comment inline will explain this situation well enough to avoid an accidental regression in future maintenence.	6 years ago
Alisdair McDiarmid	9576a5b2d8	internal: Fix lockfile constraint output for 1.2.* ... If a configuration requires a partial provider version (with some parts unspecified), Terraform considers this as a constrained-to-zero version. For example, a version constraint of 1.2 will result in an attempt to install version 1.2.0, even if 1.2.1 is available. When writing the dependency locks file, we previously would write 1.2.*, as this is the in-memory representation of 1.2. This would then cause an error on re-reading the locks file, as this is not a valid constraint format. Instead, we now explicitly convert the constraint to its zero-filled representation before writing the locks file. This ensures that it correctly round-trips. Because this change is made in getproviders.VersionConstraintsString, it also affects the output of the providers sub-command.	6 years ago
James Bardin	a3a20e0396	remove LevelFilter ... We want to always be using the hclogger to filter whenever possible	6 years ago
James Bardin	0b31ffa587	use a single log writer ... Use a single log writer instance for all std library logging. Setup the std log writer in the logging package, and remove boilerplate from test packages.	6 years ago
James Bardin	abf6b9b378	get properly configured hcloggers ... make sure plugins get hcloggers configured to match core	6 years ago
James Bardin	211edf5d75	use hclog as the default logger ... Inject hclog as the default logger in the main binary.	6 years ago
James Bardin	6ca477f042	move helper/logging to internal ... remove a dead code file too	6 years ago
Martin Atkins	0009768c7f	internal/depsfile: Update the dependency lock file atomically ... In this case, "atomic" means that there will be no situation where the file contains only part of the newContent data, and therefore other software monitoring the file for changes (using a mechanism like inotify) won't encounter a truncated file. It does _not_ mean that there can't be existing filehandles open against the old version of the file. On Windows systems the write will fail in that case, but on Unix systems the write will typically succeed but leave the existing filehandles still pointing at the old version of the file. They'll need to reopen the file in order to see the new content.	6 years ago
Martin Atkins	55e6f64977	internal/depsfile: Factor out our atomic file replacement logic ... This originated in the cliconfig code to write out credentials files. The Windows implementation of this in particular was quite onerous to get right because it needs a very specific sequence of operations to avoid running into exclusive file locks, and so by factoring this out with only cosmetic modification we can avoid repeating all of that engineering effort for other atomic file writing use-cases.	6 years ago
Martin Atkins	0bbbb9c64b	configs: Experimental support for optional object type attributes ... This builds on an experimental feature in the underlying cty library which allows marking specific attribtues of an object type constraint as optional, which in turn modifies how the cty conversion package handles missing attributes in a source value: it will silently substitute a null value of the appropriate type rather than returning an error. In order to implement the experiment this commit temporarily forks the HCL typeexpr extension package into a local internal/typeexpr package, where I've extended the type constraint syntax to allow annotating object type attributes as being optional using the HCL function call syntax. If the experiment is successful -- both at the Terraform layer and in the underlying cty library -- we'll likely send these modifications to upstream HCL so that other HCL-based languages can potentially benefit from this new capability. Because it's experimental, the optional attribute modifier is allowed only with an explicit opt-in to the module_variable_optional_attrs experiment.	6 years ago
Martin Atkins	e1cf0ac801	internal/depsfile: Control how the "hashes" value is formatted ... Previously we were just letting hclwrite do its default formatting behavior here. The current behavior there isn't ideal anyway -- it puts big data structures all on one line -- but even ignoring that our goal for this file format is to keep things in a highly-normalized shape so that diffs against the file are clear and easy to read. With that in mind, here we directly control how we write that value into the file, which means that later changes to hclwrite's list/set presentation won't affect it, regardless of what form they take.	6 years ago
Martin Atkins	2611e08430	command/init: Mention using the lock file for provider selection ... This probably isn't the best UI we could do here, but it's a placeholder for now just to avoid making it seem like we're ignoring the lock file and checking for new versions anyway.	6 years ago
Martin Atkins	b3f5c7f1e6	command/init: Read, respect, and update provider dependency locks ... This changes the approach used by the provider installer to remember between runs which selections it has previously made, using the lock file format implemented in internal/depsfile. This means that version constraints in the configuration are considered only for providers we've not seen before or when -upgrade mode is active.	6 years ago
Martin Atkins	4a1b081afb	depsfile: Locks.Equal and Locks.Empty methods ... These are helper functions to give the installation UI some hints about whether the lock file has changed so that it can in turn give the user advice about it. The UI-layer callers of these will follow in a later commit.	6 years ago
Kristin Laemmert	04be220f5f	deprecate helper/copy ... helper/copy CopyDir was used heavily in tests. It differes from internal/copydir in a few ways, the main one being that it creates the dst directory while the internal version expected the dst to exist (there are other differences, which is why I did not just switch tests to using internal's CopyDir). I moved the CopyDir func from helper/copy into command_test.go; I could also have moved it into internal/copy and named it something like CreateDirAndCopy so if that seems like a better option please let me know. helper/copy/CopyFile was used in a couple of spots so I moved it into internal, at which point I thought it made more sense to rename the package copy (instead of copydir). There's also a `go mod tidy` included.	6 years ago
Kristin Laemmert	3933cbd491	remove LegacyProvider (#26433)	6 years ago
Martin Atkins	593cf7b4d5	didyoumean: move from "helper" to "internal" ... This new-ish package ended up under "helper" during the 0.12 cycle for want of some other place to put it, but in retrospect that was an odd choice because the "helper/" tree is otherwise a bunch of legacy code from when the SDK lived in this repository. Here we move it over into the "internal" directory just to distance it from the guidance of not using "helper/" packages in new projects; didyoumean is a package we actively use as part of error message hints.	6 years ago
Martin Atkins	59b116f7bf	command/init: Remove support for legacy provider addresses ... We no longer need to support 0.12-and-earlier-style provider addresses because users should've upgraded their existing configurations and states on Terraform 0.13 already. For now this is only checked in the "init" command, because various test shims are still relying on the idea of legacy providers the core layer. However, rejecting these during init is sufficient grounds to avoid supporting legacy provider addresses in the new dependency lock file format, and thus sets the stage for a more severe removal of legacy provider support in a later commit.	6 years ago
Martin Atkins	0b734a2803	command: Make provider installation interruptible ... In earlier commits we started to make the installation codepath context-aware so that it could be canceled in the event of a SIGINT, but we didn't complete wiring that through the API of the getproviders package. Here we make the getproviders.Source interface methods, along with some other functions that can make network requests, take a context.Context argument and act appropriately if that context is cancelled. The main providercache.Installer.EnsureProviderVersions method now also has some context-awareness so that it can abort its work early if its context reports any sort of error. That avoids waiting for the process to wind through all of the remaining iterations of the various loops, logging each request failure separately, and instead returns just a single aggregate "canceled" error. We can then set things up in the "terraform init" and "terraform providers mirror" commands so that the context will be cancelled if we get an interrupt signal, allowing provider installation to abort early while still atomically completing any local-side effects that may have started.	6 years ago
Martin Atkins	ef64df950c	getproviders: Prepare for having multiple valid hashes per package ... As we continue iterating towards saving valid hashes for a package in a depsfile lock file after installation and verifying them on future installation, this prepares getproviders for the possibility of having multiple valid hashes per package. This will arise in future commits for two reasons: - We will need to support both the legacy "zip hash" hashing scheme and the new-style content-based hashing scheme because currently the registry protocol is only able to produce the legacy scheme, but our other installation sources prefer the content-based scheme. Therefore packages will typically have a mixture of hashes of both types. - Installing from an upstream registry will save the hashes for the packages across all supported platforms, rather than just the current platform, and we'll consider all of those valid for future installation if we see both successful matching of the current platform checksum and a signature verification for the checksums file as a whole. This also includes some more preparation for the second case above in that signatureAuthentication now supports AcceptableHashes and returns all of the zip-based hashes it can find in the checksums file. This is a bit of an abstraction leak because previously that authenticator considered its "document" to just be opaque bytes, but we want to make sure that we can only end up trusting _all_ of the hashes if we've verified that the document is signed. Hopefully we'll make this better in a future commit with some refactoring, but that's deferred for now in order to minimize disruption to existing codepaths while we work towards a provider locking MVP.	6 years ago
Martin Atkins	6694cfaa0e	getproviders: Add a real type Hash for package hashes ... The logic for what constitutes a valid hash and how different hash schemes are represented was starting to get sprawled over many different files and packages. Consistently with other cases where we've used named types to gather the definition of a particular string into a single place and have the Go compiler help us use it properly, this introduces both getproviders.Hash representing a hash value and getproviders.HashScheme representing the idea of a particular hash scheme. Most of this changeset is updating existing uses of primitive strings to uses of getproviders.Hash. The new type definitions are in internal/getproviders/hash.go.	6 years ago
Martin Atkins	264a3cf031	depsfile: Flatten the "hashes" locks to a single set of strings ... Although origin registries return specific [filename, hash] pairs, our various different installation methods can't produce a structured mapping from platform to hash without breaking changes. Therefore, as a compromise, we'll continue to do platform-specific checks against upstream data in the cases where that's possible (installation from origin registry or network mirror) but we'll treat the lock file as just a flat set of equally-valid hashes, at least one of which must match after we've completed whatever checks we've made against the upstream-provided checksums/signatures. This includes only the minimal internal/getproviders updates required to make this compile. A subsequent commit will update that package to actually support the idea of verifying against multiple hashes.	6 years ago
Martin Atkins	b2c0ccdf96	internal/getproviders: Allow PackageMeta to carry acceptable hashes ... The "acceptable hashes" for a package is a set of hashes that the upstream source considers to be good hashes for checking whether future installs of the same provider version are considered to match this one. Because the acceptable hashes are a package authentication concern and they already need to be known (at least in part) to implement the authenticators, here we add AcceptableHashes as an optional extra method that an authenticator can implement. Because these are hashes chosen by the upstream system, the caller must make its own determination about their trustworthiness. The result of authentication is likely to be an input to that, for example by distrusting hashes produced by an authenticator that succeeds but doesn't report having validated anything.	6 years ago
Martin Atkins	e843097e52	internal/getproviders: Formalize the "ziphash" hashing scheme ... This is the pre-existing hashing scheme that was initially built for releases.hashicorp.com and then later reused for the provider registry protocol, which takes a SHA256 hash of the official distribution .zip file and formats it as lowercase hex. This is a non-ideal hash scheme because it works only for PackageLocalArchive locations, and so we can't verify package directories on local disk against such hashes. However, the registry protocol is now a compatibility constraint and so we're going to need to support this hashing scheme for the foreseeable future.	6 years ago
Martin Atkins	773dd56b42	internal/depsfile: Introduce the concept of "non-lockable" providers ... It doesn't make sense for a built-in provider to appear in a lock file because built-in providers have no version independent of the version of Terraform they are compiled into. We also exclude legacy providers here, because they were supported only as a transitional aid to enable the Terraform 0.13 upgrade process and are not intended for explicit selection. The provider installer will, once it's updated to understand dependency locking, use this concept to decide which subset of its selections to record in the dependency lock file for reference for future installation requests.	6 years ago
Martin Atkins	98e2e69abb	internal/depsfile: SaveLocksToFile implementation ... This is an initial implementation of writing locks back to a file on disk. This initial implementation is incomplete because it does not write the changes to the new file atomically. We'll revisit that in a later commit as we return to polish these codepaths, once we've proven out this package's design by integrating it with Terraform's provider installer.	6 years ago
Martin Atkins	92723661d0	internal/depsfile: Loading locks from HCL files on disk ... This is the initial implementation of the parser/decoder portion of the new dependency lock file handler. It's currently dead code because the caller isn't written yet. We'll continue to build out this functionality here until we have the basic level of both load and save functionality before introducing this into the provider installer codepath.	6 years ago
Martin Atkins	6993ecb0a6	internal/getproviders: VersionConstraintsString for "~> 2" input ... The version constraint parser allows "~> 2", but it behavior is identical to "~> 2.0". Due to a quirk of the constraint parser (caused by the fact that it supports both Ruby-style and npm/cargo-style constraints), it ends up returning "~> 2" with the minor version marked as "unconstrained" rather than as zero, but that means the same thing as zero in this context anyway and so we'll prefer to stringify as "~> 2.0" so that we can be clearer about how Terraform is understanding that version constraint.	6 years ago
Alisdair McDiarmid	5587509bcf	internal: Fix providercache test failures on macOS ... For reasons that are unclear, these two tests just started failing on macOS very recently. The failure looked like: PackageDir: strings.Join({ "/", + "private/", "var/folders/3h/foobar/T/terraform-test-p", "rovidercache655312854/registry.terraform.io/hashicorp/null/2.0.0", "/windows_amd64", }, Speculating that the macOS temporary directory moved into the /private directory, I added a couple of EvalSymlinks calls and the tests pass again. No other unit tests appear to be affected by this at the moment.	6 years ago
Alisdair McDiarmid	fc7e467d19	command: Add redirect support to 0.13upgrade ... If a provider changes namespace in the registry, we can detect this when running the 0.13upgrade command. As long as there is a version matching the user's constraints, we now use the provider's new source address. Otherwise, warn the user that the provider has moved and a version upgrade is necessary to move to it.	6 years ago