35 Commits (94de72c63834acea811c27a585c19eae29374a4b)

Author	SHA1	Message	Date
Eng Zer Jun	fedd315275	test: use `T.TempDir` to create temporary test directory (#30803) ... This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The directory created by `t.TempDir` is automatically removed when the test and all its subtests complete. Prior to this commit, temporary directory created using `ioutil.TempDir` needs to be removed manually by calling `os.RemoveAll`, which is omitted in some tests. The error handling boilerplate e.g. defer func() { if err := os.RemoveAll(dir); err != nil { t.Fatal(err) } } is also tedious, but `t.TempDir` handles this for us nicely. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	4 years ago
Martin Atkins	affe2c3295	addrs: Expose the registry address parser's error messages ... Previously we ended up losing all of the error message detail produced by the registry address parser, because we treated any registry address failure as cause to parse the address as a go-getter-style remote address instead. That led to terrible feedback in the situation where the user _was_ trying to write a module address but it was invalid in some way. Although we can't really tighten this up in the default case due to our compatibility promises, it's never been valid to use the "version" argument with anything other than a registry address and so as a compromise here we'll use the presence of "version" as a heuristic for user intent to parse the source address as a registry address, and thus we can return a registry-address-specific error message in that case and thus give more direct feedback about what was wrong. This unfortunately won't help someone trying to install from the registry _without_ a version constraint, but I didn't want to let perfect be the enemy of the good here, particularly since we recommend using version constraints with registry modules anyway; indeed, that's one of the main benefits of using a registry rather than a remote source directly.	4 years ago
kmoe	40ec62c139	command: make module installation interruptible ... Earlier work to make "terraform init" interruptible made the getproviders package context-aware in order to allow provider installation to be cancelled. Here we make a similar change for module installation, which is now also cancellable with SIGINT. This involves plumbing context through initwd and getmodules. Functions which can make network requests now include a context parameter whose cancellation cancels those requests. Since the module installation code is shared, "terraform get" is now also interruptible during module installation.	4 years ago
Martin Atkins	51b0aee36c	addrs: ModuleRegistryPackage for representing module registry packages ... Previously we had a separation between ModuleSourceRemote and ModulePackage as a way to represent within the type system that there's an important difference between a module source address and a package address, because module packages often contain multiple modules and so a ModuleSourceRemote combines a ModulePackage with a subdirectory to represent one specific module. This commit applies that same strategy to ModuleSourceRegistry, creating a new type ModuleRegistryPackage to represent the different sort of package that we use for registry modules. Again, the main goal here is to try to reflect the conceptual modelling more directly in the type system so that we can more easily verify that uses of these different address types are correct. To make use of that, I've also lightly reworked initwd's module installer to use addrs.ModuleRegistryPackage directly, instead of a string representation thereof. This was in response to some earlier commits where I found myself accidentally mixing up package addresses and source addresses in the installRegistryModule method; with this new organization those bugs would've been caught at compile time, rather than only at unit and integration testing time. While in the area anyway, I also took this opportunity to fix some historical confusing names of fields in initwd.ModuleInstaller, to be clearer that they are only for registry packages and not for all module source address types.	5 years ago
Martin Atkins	7b2a0284e0	initwd: Fix registry acceptance tests for upstream registry changes ... We have some tests in this package that install real modules from the real registry at registry.terraform.io. Those tests were written at an earlier time when the registry's behavior was to return the URL of a .tar.gz archive generated automatically by GitHub, which included an extra level of subdirectory that would then be reflected in the paths to the local copies of these modules. GitHub started rate limiting those tar archives in a way that Terraform's module installer couldn't authenticate to, and so the registry switched to returning direct git repository URLs instead, which don't have that extra subdirectory and so the local paths on disk now end up being a little different, because the actual module directories are at a different subdirectory of the package.	5 years ago
Martin Atkins	1a8da65314	Refactoring of module source addresses and module installation ... It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.	5 years ago
Martin Atkins	4e74a7a4f1	initwd: Error message for local paths escaping module packages ... Our module installer has a somewhat-informal idea of a "module package", which is some external thing we can go fetch in order to add one or more modules to the current configuration. Our documentation doesn't talk much about it because most users seem to have found the distinction between external and local modules pretty intuitive without us throwing a lot of funny terminology at them, but there are some situations where the distinction between a module and a module package are material to the end-user. One such situation is when using an absolute rather than relative filesystem path: we treat that as an external package in order to make the resulting working directory theoretically "portable" (although users can do various other things to defeat that), and so Terraform will copy the directory into .terraform/modules in the same way as it would download and extract a remote archive package or clone a git repository. A consequence of this, though, is that any relative paths called from inside a module loaded from an absolute path will fail if they try to traverse upward into the parent directory, because at runtime we're actually running from a copy of the directory that's been taking out of its original context. A similar sort of situation can occur in a truly remote module package if the author accidentally writes a "../" source path that traverses up out of the package root, and so this commit introduces a special error message for both situations that tries to be a bit clearer about there being a package boundary and use that to explain why installation failed. We would ideally have made escaping local references like that illegal in the first place, but sadly we did not and so when we rebuilt the module installer for Terraform v0.12 we ended up keeping the previous behavior of just trying it and letting it succeed if there happened to somehow be a matching directory at the given path, in order to remain compatible with situations that had worked by coincidence rather than intention. For that same reason, I've implemented this as a replacement error message we will return only if local module installation was going to fail anyway, and thus it only modifies the error message for some existing error situations rather than introducing new error situations. This also includes some light updates to the documentation to say a little more about how Terraform treats absolute paths, though aiming not to get too much into the weeds about module packages since it's something that most users can get away with never knowing.	5 years ago
Martin Atkins	31349a9c3a	Move configs/ to internal/configs/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	b9a93a0fe7	Move addrs/ to internal/addrs/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	05caff2ca3	Move tfdiags/ to internal/tfdiags/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	4c5b866dea	Move registry/ to internal/registry/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
James Bardin	0b31ffa587	use a single log writer ... Use a single log writer instance for all std library logging. Setup the std log writer in the logging package, and remove boilerplate from test packages.	6 years ago
James Bardin	6ca477f042	move helper/logging to internal ... remove a dead code file too	6 years ago
Kristin Laemmert	04be220f5f	deprecate helper/copy ... helper/copy CopyDir was used heavily in tests. It differes from internal/copydir in a few ways, the main one being that it creates the dst directory while the internal version expected the dst to exist (there are other differences, which is why I did not just switch tests to using internal's CopyDir). I moved the CopyDir func from helper/copy into command_test.go; I could also have moved it into internal/copy and named it something like CreateDirAndCopy so if that seems like a better option please let me know. helper/copy/CopyFile was used in a couple of spots so I moved it into internal, at which point I thought it made more sense to rename the package copy (instead of copydir). There's also a `go mod tidy` included.	6 years ago
Alan D. Salewski	f63c38d6a1	internal/initwd: allow tests to pass when $PWD contains symlinks (#26015) ... Fixes #26014	6 years ago
Kristin Laemmert	1b8f4566fa	internal/initwd: fix panics with relative submodules in DirFromModule (#25250) ... * internal/initwd: fix panics with relative submodules in DirFromModule There were two related issues here: 1. panic with any local module with submodules 1. panic with a relative directory that was above the workdir ("../") The first panic was caused by the local installer looking up the root module with the (nonexistant) key "root.", instead of "". The second panic was caused by the installer trying to determine the relative path from ".". This was fixed by detecting "." as the source path and using the absolute path for the call to filepath.Rel. Added test cases for both panics and updated the existing e2e tests with the correct install paths.	6 years ago
Alisdair McDiarmid	62d826e066	command/init: Use full config for provider reqs ... Relying on the early config for provider requirements was necessary in Terraform 0.12, to allow the 0.12upgrade command to run after init installs providers. However in 0.13, the same restrictions do not apply, and the detection of provider requirements has changed. As a result, the early config loader gives incorrect provider requirements in some circumstances, such as those in the new test in this commit. Therefore we are changing the init command to use the requirements found by the full configuration loader. This also means that we can remove the internal initwd CheckCoreVersionRequirements function.	6 years ago
Martin Atkins	072c6d9aed	internal/copydir: Factor out our recursive directory copy for reuse ... We've previously been copying this function around so it could remain unexported while being used in various packages. However, it's a non-trivial function with lots of specific assumptions built into it, so here we'll put it somewhere that other packages can depend on it _and_ document the assumptions it seems to be making for future reference. As a bonus, this now uses os.SameFile to detect when two paths point to the same physical file, instead of the slightly buggy local implementation we had before which only worked on Unix systems and did not correctly handle when the paths were on different physical devices. The copy of the function I extracted here is the one from internal/initwd, so this commit also includes the removal of that unexported version and updating the callers in that package to use at at this new location.	6 years ago
findkim	d0e13320d5	registry: configurable retry client	6 years ago
Kristin Laemmert	4045a6e5d0	initwd: cache registry responses for module versions and download URL (#23727) ... * initwd: cache registry responses for module versions and download URL Closes #23544	6 years ago
Martin Atkins	2b2ac1f6de	configs: use local set of go-getter detectors ... In an earlier change we switched to defining our own sets of detectors, getters, etc for go-getter in order to insulate us from upstream changes to those sets that might otherwise change the user-visible behavior of Terraform's module installer. However, we apparently neglected to actually refer to our local set of detectors, and continued to refer to the upstream set. Here we catch up with the latest detectors from upstream (taken from the version of go-getter we currently have vendored) and start using that fixed set. Currently we are maintaining these custom go-getter sets in two places due to the configload vs. initwd distinction. That was already true for goGetterGetters and goGetterDecompressors, and so I've preserved that for now just to keep this change relatively simple; in later change it would be nice to factor these "get with go getter" functions out into a shared location which we can call from both configload and initwd.	6 years ago
Kristin Laemmert	68dfc3046d	backend/remote-state: etcdv3, oss, and manta acc tests should fail ... faster The acceptance tests for etcdv3, oss and manta were not validating required env variablea, chosing to assume that if one was running acceptance tests they had already configured the credentials. It was not always clear if this was a bug in the tests or the provider, so I opted to make the tests fail faster when required attributes were unset (or "").	7 years ago
Kristin Laemmert	69b6791ef1	internal/initwd: fix issues with tests and symlinks ... filepath.EvalSymlinks is our friend! The code already does this, the tests needed to be updated to do the same.	7 years ago
Radek Simko	5b9f2fafc8	Standardise directory name for test data	7 years ago
Pam Selle	59c5cc4788	Merge pull request #21254 from davewongillies/gcs ... Add GCS source support for modules	7 years ago
Martin Atkins	c5285021fa	internal/initwd: Test that version is rejected for local modules ... This was already working, but since that codepath is separate from the go-getter install codepath it's helpful to have a separate test for it, in addition to the existing one for go-getter modules.	7 years ago
David Gillies	8b45443b21	Add GCS source support for modules	7 years ago
Matt Morrison	cbebb7cdf1	command/init: Don't panic if go-getter-fetched module has version constraint	7 years ago
Martin Atkins	f5a7f45ffb	internal/initwd: Fix module installation error ... The "err" variable in the MaybeRelativePathErr condition was masking the original err with nil in the "else" case of this branch, causing the error message to be incomplete. While here, also tweaked the wording to say "Could not download" rather than "Error attempting to download", both to say the same thing in fewer words and because the summary line above already starts with "Error:" when we print out this message, so it looks weird to have both lines start with the same word.	7 years ago
Kristin Laemmert	9327eedb04	internal/initwd: follow local module path symlink (#21063) ... * internal/initwd: follow local module path symlink Fixes #21060 While a previous commit fixed a problem when the local module directory contained a symlink, it did not account for the possibility that the entire directory was a symlink.	7 years ago
Kristin Laemmert	a15a4acf2f	configs/configupgrade: detect possible relative module sources (#20646) ... * configs/configupgrade: detect possible relative module sources If a module source appears to be a relative local path but does not have a preceding ./, print a #TODO message for the user. * internal/initwd: limit go-getter detectors to those supported by terraform * internal/initwd: move isMaybeRelativeLocalPath check into getWithGoGetter To avoid making two calls to getter.Detect, which potentially makes non-trivial API calls, the "isMaybeRelativeLocalPath" check was moved to a later step and a custom error type was added so user-friendly diagnostics could be displayed in the event that a possible relative local path was detected.	7 years ago
Kristin Laemmert	bf04503f04	internal/initwd: identify possible relative module sources ... Identify module sources that look like relative paths ("child" instead of "./child", for example) and surface a helpful error. Previously, such module sources would be passed to go-getter, which would fail because it was expecting an absolute, or properly relative, path. This commit moves the check for improper relative paths sooner so a user-friendly error can be displayed.	7 years ago
Kristin Laemmert	0c5fd835ce	copyDir: detect if the module install path is a symlink to a directory (#20603) ... configs/configload and internal/initwd both had a copyDir function that would fail if the source directory contained a symlinked directory, because the os.FileMode.IsDir() returns false for symlinks. This PR adds a check for a symlink and copies that symlink in the target directory. It handles symlinks for both files and directories (with included tests). Fixes #20539	7 years ago
Martin Atkins	86c02d5c35	command: "terraform init" can partially initialize for 0.12upgrade ... There are a few constructs from 0.11 and prior that cause 0.12 parsing to fail altogether, which previously created a chicken/egg problem because we need to install the providers in order to run "terraform 0.12upgrade" and thus fix the problem. This changes "terraform init" to use the new "early configuration" loader for module and provider installation. This is built on the more permissive parser in the terraform-config-inspect package, and so it allows us to read out the top-level blocks from the configuration while accepting legacy HCL syntax. In the long run this will let us do version compatibility detection before attempting a "real" config load, giving us better error messages for any future syntax additions, but in the short term the key thing is that it allows us to install the dependencies even if the configuration isn't fully valid. Because backend init still requires full configuration, this introduces a new mode of terraform init where it detects heuristically if it seems like we need to do a configuration upgrade and does a partial init if so, before finally directing the user to run "terraform 0.12upgrade" before running any other commands. The heuristic here is based on two assumptions: - If the "early" loader finds no errors but the normal loader does, the configuration is likely to be valid for Terraform 0.11 but not 0.12. - If there's already a version constraint in the configuration that excludes Terraform versions prior to v0.12 then the configuration is probably _already_ upgraded and so it's just a normal syntax error, even if the early loader didn't detect it. Once the upgrade process is removed in 0.13.0 (users will be required to go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can be simplified to remove that special mode, but the idea of doing the dependency version checks against the liberal parser will remain valuable to increase our chances of reporting version-based incompatibilities rather than syntax errors as we add new features in future.	7 years ago
Martin Atkins	0c0a437bcb	Move module install functionality over to internal/initwd	7 years ago