terraform

Commit Graph

Author	SHA1	Message	Date
Daniel Banck	fb60c15670	Update init to use new graph-based workflow The init command and the module installer are now using the new graph-based workflow to install modules instead of using the recursive BuildConfig.	1 month ago
Radek Simko	0fe906fa8c	make copyrightfix	2 months ago
Liam Cervante	698cb0d59d	init: detect and report invalid submodules (#35839 )	2 years ago
James Bardin	3a1a9408d9	fix Descendants spelling this has proliferated everywhere	2 years ago
Brandon Croft	3a5221269b	Revert "Add Warning for Deprecated Modules in `Init` (Second Revision)"	2 years ago
Mark DeCrane	64f385fbb1	Apply suggestions from @nfagerlund code review Co-authored-by: Nick Fagerlund <nick.fagerlund@gmail.com>	2 years ago
Mark DeCrane	7dd7360007	Use package addr instead of mod name in warning, promote failed registry call log message to WARN	2 years ago
Mark DeCrane	1368c93862	Add godoc comments related to portions pertaining to public API	2 years ago
Mark DeCrane	cacc5dffa2	DRY up the code	2 years ago
Mark DeCrane	1e7c29243e	Added Extra field for TFC diagnostic processing	2 years ago
Mark DeCrane	724adea97a	Surface deprecation warnings for module to be installed and already installed modules	2 years ago
Martin Atkins	84df5562f2	moduleaddrs: Terraform now owns module source address normalization We've been gradually chipping away at how much we use go-getter for source packages, because it's generally been a bit of a nightmare and sharing it with other codebases means that any time someone wants to change something we end up needing to find some way to prevent it breaking Terraform's compatibility promises. Here we make one further step: Terraform owns the "detectors" idea that deals with source address normalization, and now always produces fully-qualified addresses for go-getter to chew on only for the getting and decompressing steps. Retaining go-getter for the actual getting part is helpful because we can then benefit from security fixes upstream, but Terraform owning the first layer of parsing means that we can fix in place the definition of what "module source address" syntax means, and thus we can avoid having everything in this codebase indirectly depend on go-getter just because it wants to parse module source addresses. Now only the module installer actually depends indirectly on go-getter, which finally disconnects go-getter's subtree from all of the remote state backend dependency graphs.	2 years ago
Martin Atkins	5d87e32eb9	moduleaddrs: Split module source address parsing from package addrs Our package addrs ends up getting imported from just about ever other package in Terraform, because it contains the types we use to talk about various different kinds of objects. Therefore we typically try to keep its transitive dependency graph relatively small, because anything it depends on becomes an indirect dependency of nearly everything else. A while back we moved the module source address models into package addrs, which also brought with them the code for parsing strings to produce those addresses. Unfortunately, remote module source addresses are defined using the external dependency go-getter, which is pretty heavy itself and also brings with it numerous other external dependencies such as the AWS SDK, the Google Cloud Platform SDK, etc. Since only relatively few packages actually need to _parse_ source addresses -- with most either not caring about them at all or only consuming addresses that were already parsed by someone else -- we'll move the parser functions into their own package, while keeping the resulting address types in package addrs. This does still retain the package addrs dependency on external module github.com/hashicorp/terraform-registry-address, which is not ideal but that one at least has a relatively shallow dependency subgraph, so there's not so much urgency to tidy that one.	2 years ago
Liam Cervante	ee9aa07e10	terraform test: implement the source attribute for mock providers (#34192 ) * complete implementation * add tests * address comments	2 years ago
CJ Horton	0f2314fb62	initwd: require valid module name (#33745 ) We install remote modules prior to showing any validation errors during init so that we can show errors about the core version requirement before we do anything else. Unfortunately, this means that we don't validate module names until after remote modules have been installed, which may cause unexpected problems if we can't convert the module name into a valid path.	3 years ago
hashicorp-copywrite[bot]	53c34ff49c	Update copyright file headers to BUSL-1.1	3 years ago
James Bardin	f6768de218	handle install errs in installDescendentModules This will allow us to use the same logic for `get` too.	3 years ago
James Bardin	29c3f650cd	allow config errors when using -from-module When initializing the working directory from a module, some use cases may still want the module source even when the configuration cannot be loaded due to validation errors. Isolate the installation errors during the initialization sequence, so we can report them if there is a problem fetching the actual module source. Once we have the module, convert configuration errors to warnings so the cli initialization can proceed.	3 years ago
Liam Cervante	6882dd9530	testing framework: introduce test command optional flags (#33504 ) * testing framework: introduce test command optional flags * address consistency checks	3 years ago
Liam Cervante	5acc95dda7	[Testing Framework] Add module block to test run blocks (#33456 ) * [testing framework] prepare for beta phase of development * [Testing Framework] Add module block to test run blocks	3 years ago
hashicorp-copywrite[bot]	325d18262e	[COMPLIANCE] Add Copyright and License Headers	3 years ago
Alisdair McDiarmid	8df065a2fe	initwd: Switch from earlyconfig to configs This is a mostly mechanical refactor with a handful of changes which are necessary due to the semantic difference between earlyconfig and configs. When parsing root and descendant modules in the module installer, we now check the core version requirements inline. If the Terraform version is incompatible, we drop any other module loader diagnostics. This ensures that future language additions don't clutter the output and confuse the user. We also add two new checks during the module load process: * Don't try to load a module with a `nil` source address. This is a necessary change due to the move away from earlyconfig. * Don't try to load a module with a blank name (i.e. `module ""`). Because our module loading manifest uses the stringified module path as its map key, this causes a collision with the root module, and a later panic. This is the bug which triggered this refactor in the first place.	3 years ago
Liam Cervante	6af6540233	Use the apparentlymart/go-versions library to parse module constraints (#32377 ) * Use the apparentlymart/go-versions library to parse module constraints * goimports * Update comments, and parse versions carefully * add acceptance tests to verify behaviour of partial matches * goimports	3 years ago
James Bardin	553b8c6de5	expand module subdir globs	4 years ago
Radek Simko	7feef1c4aa	Use hashicorp/terraform-registry-address as a decoupled library (#28338 ) * refactor: Use tfaddr for provider address parsing * refactor: Use tfaddr for module address parsing * deps: introduce hashicorp/terraform-registry-address	4 years ago
Martin Atkins	affe2c3295	addrs: Expose the registry address parser's error messages Previously we ended up losing all of the error message detail produced by the registry address parser, because we treated any registry address failure as cause to parse the address as a go-getter-style remote address instead. That led to terrible feedback in the situation where the user _was_ trying to write a module address but it was invalid in some way. Although we can't really tighten this up in the default case due to our compatibility promises, it's never been valid to use the "version" argument with anything other than a registry address and so as a compromise here we'll use the presence of "version" as a heuristic for user intent to parse the source address as a registry address, and thus we can return a registry-address-specific error message in that case and thus give more direct feedback about what was wrong. This unfortunately won't help someone trying to install from the registry _without_ a version constraint, but I didn't want to let perfect be the enemy of the good here, particularly since we recommend using version constraints with registry modules anyway; indeed, that's one of the main benefits of using a registry rather than a remote source directly.	4 years ago
kmoe	40ec62c139	command: make module installation interruptible Earlier work to make "terraform init" interruptible made the getproviders package context-aware in order to allow provider installation to be cancelled. Here we make a similar change for module installation, which is now also cancellable with SIGINT. This involves plumbing context through initwd and getmodules. Functions which can make network requests now include a context parameter whose cancellation cancels those requests. Since the module installation code is shared, "terraform get" is now also interruptible during module installation.	4 years ago
Martin Atkins	51b0aee36c	addrs: ModuleRegistryPackage for representing module registry packages Previously we had a separation between ModuleSourceRemote and ModulePackage as a way to represent within the type system that there's an important difference between a module source address and a package address, because module packages often contain multiple modules and so a ModuleSourceRemote combines a ModulePackage with a subdirectory to represent one specific module. This commit applies that same strategy to ModuleSourceRegistry, creating a new type ModuleRegistryPackage to represent the different sort of package that we use for registry modules. Again, the main goal here is to try to reflect the conceptual modelling more directly in the type system so that we can more easily verify that uses of these different address types are correct. To make use of that, I've also lightly reworked initwd's module installer to use addrs.ModuleRegistryPackage directly, instead of a string representation thereof. This was in response to some earlier commits where I found myself accidentally mixing up package addresses and source addresses in the installRegistryModule method; with this new organization those bugs would've been caught at compile time, rather than only at unit and integration testing time. While in the area anyway, I also took this opportunity to fix some historical confusing names of fields in initwd.ModuleInstaller, to be clearer that they are only for registry packages and not for all module source address types.	5 years ago
Martin Atkins	1a8da65314	Refactoring of module source addresses and module installation It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.	5 years ago
Martin Atkins	4e74a7a4f1	initwd: Error message for local paths escaping module packages Our module installer has a somewhat-informal idea of a "module package", which is some external thing we can go fetch in order to add one or more modules to the current configuration. Our documentation doesn't talk much about it because most users seem to have found the distinction between external and local modules pretty intuitive without us throwing a lot of funny terminology at them, but there are some situations where the distinction between a module and a module package are material to the end-user. One such situation is when using an absolute rather than relative filesystem path: we treat that as an external package in order to make the resulting working directory theoretically "portable" (although users can do various other things to defeat that), and so Terraform will copy the directory into .terraform/modules in the same way as it would download and extract a remote archive package or clone a git repository. A consequence of this, though, is that any relative paths called from inside a module loaded from an absolute path will fail if they try to traverse upward into the parent directory, because at runtime we're actually running from a copy of the directory that's been taking out of its original context. A similar sort of situation can occur in a truly remote module package if the author accidentally writes a "../" source path that traverses up out of the package root, and so this commit introduces a special error message for both situations that tries to be a bit clearer about there being a package boundary and use that to explain why installation failed. We would ideally have made escaping local references like that illegal in the first place, but sadly we did not and so when we rebuilt the module installer for Terraform v0.12 we ended up keeping the previous behavior of just trying it and letting it succeed if there happened to somehow be a matching directory at the given path, in order to remain compatible with situations that had worked by coincidence rather than intention. For that same reason, I've implemented this as a replacement error message we will return only if local module installation was going to fail anyway, and thus it only modifies the error message for some existing error situations rather than introducing new error situations. This also includes some light updates to the documentation to say a little more about how Terraform treats absolute paths, though aiming not to get too much into the weeds about module packages since it's something that most users can get away with never knowing.	5 years ago
Martin Atkins	b9a93a0fe7	Move addrs/ to internal/addrs/ This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	05caff2ca3	Move tfdiags/ to internal/tfdiags/ This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	4c5b866dea	Move registry/ to internal/registry/ This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
findkim	d0e13320d5	registry: configurable retry client	6 years ago
Kristin Laemmert	4045a6e5d0	initwd: cache registry responses for module versions and download URL (#23727 ) * initwd: cache registry responses for module versions and download URL Closes #23544	6 years ago
Matt Morrison	cbebb7cdf1	command/init: Don't panic if go-getter-fetched module has version constraint	7 years ago
Martin Atkins	f5a7f45ffb	internal/initwd: Fix module installation error The "err" variable in the MaybeRelativePathErr condition was masking the original err with nil in the "else" case of this branch, causing the error message to be incomplete. While here, also tweaked the wording to say "Could not download" rather than "Error attempting to download", both to say the same thing in fewer words and because the summary line above already starts with "Error:" when we print out this message, so it looks weird to have both lines start with the same word.	7 years ago
Kristin Laemmert	9327eedb04	internal/initwd: follow local module path symlink (#21063 ) * internal/initwd: follow local module path symlink Fixes #21060 While a previous commit fixed a problem when the local module directory contained a symlink, it did not account for the possibility that the entire directory was a symlink.	7 years ago
Kristin Laemmert	a15a4acf2f	configs/configupgrade: detect possible relative module sources (#20646 ) * configs/configupgrade: detect possible relative module sources If a module source appears to be a relative local path but does not have a preceding ./, print a #TODO message for the user. * internal/initwd: limit go-getter detectors to those supported by terraform * internal/initwd: move isMaybeRelativeLocalPath check into getWithGoGetter To avoid making two calls to getter.Detect, which potentially makes non-trivial API calls, the "isMaybeRelativeLocalPath" check was moved to a later step and a custom error type was added so user-friendly diagnostics could be displayed in the event that a possible relative local path was detected.	7 years ago
Kristin Laemmert	bf04503f04	internal/initwd: identify possible relative module sources Identify module sources that look like relative paths ("child" instead of "./child", for example) and surface a helpful error. Previously, such module sources would be passed to go-getter, which would fail because it was expecting an absolute, or properly relative, path. This commit moves the check for improper relative paths sooner so a user-friendly error can be displayed.	7 years ago
Martin Atkins	86c02d5c35	command: "terraform init" can partially initialize for 0.12upgrade There are a few constructs from 0.11 and prior that cause 0.12 parsing to fail altogether, which previously created a chicken/egg problem because we need to install the providers in order to run "terraform 0.12upgrade" and thus fix the problem. This changes "terraform init" to use the new "early configuration" loader for module and provider installation. This is built on the more permissive parser in the terraform-config-inspect package, and so it allows us to read out the top-level blocks from the configuration while accepting legacy HCL syntax. In the long run this will let us do version compatibility detection before attempting a "real" config load, giving us better error messages for any future syntax additions, but in the short term the key thing is that it allows us to install the dependencies even if the configuration isn't fully valid. Because backend init still requires full configuration, this introduces a new mode of terraform init where it detects heuristically if it seems like we need to do a configuration upgrade and does a partial init if so, before finally directing the user to run "terraform 0.12upgrade" before running any other commands. The heuristic here is based on two assumptions: - If the "early" loader finds no errors but the normal loader does, the configuration is likely to be valid for Terraform 0.11 but not 0.12. - If there's already a version constraint in the configuration that excludes Terraform versions prior to v0.12 then the configuration is probably _already_ upgraded and so it's just a normal syntax error, even if the early loader didn't detect it. Once the upgrade process is removed in 0.13.0 (users will be required to go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can be simplified to remove that special mode, but the idea of doing the dependency version checks against the liberal parser will remain valuable to increase our chances of reporting version-based incompatibilities rather than syntax errors as we add new features in future.	7 years ago
Martin Atkins	0c0a437bcb	Move module install functionality over to internal/initwd	7 years ago

42 Commits (add-missing-variable-loading)