30 Commits (653a819cc3a2d2c7fb3f6a26b03f4e8a95e4d585)

Author	SHA1	Message	Date
Martin Atkins	2b2ac1f6de	configs: use local set of go-getter detectors ... In an earlier change we switched to defining our own sets of detectors, getters, etc for go-getter in order to insulate us from upstream changes to those sets that might otherwise change the user-visible behavior of Terraform's module installer. However, we apparently neglected to actually refer to our local set of detectors, and continued to refer to the upstream set. Here we catch up with the latest detectors from upstream (taken from the version of go-getter we currently have vendored) and start using that fixed set. Currently we are maintaining these custom go-getter sets in two places due to the configload vs. initwd distinction. That was already true for goGetterGetters and goGetterDecompressors, and so I've preserved that for now just to keep this change relatively simple; in later change it would be nice to factor these "get with go getter" functions out into a shared location which we can call from both configload and initwd.	6 years ago
Pam Selle	cca36025d6	Merge pull request #22946 from hashicorp/kmoe/copy_dir_dotfiles ... remove dotfile exclusion in copyDir	7 years ago
Radek Simko	32f9722d9d	Replace import paths & set UA string where necessary	7 years ago
Martin Atkins	39e609d5fd	vendor: switch to HCL 2.0 in the HCL repository ... Previously we were using the experimental HCL 2 repository, but now we'll shift over to the v2 import path within the main HCL repository as part of actually releasing HCL 2.0 as stable. This is a mechanical search/replace to the new import paths. It also switches to the v2.0.0 release of HCL, which includes some new code that Terraform didn't previously have but should not change any behavior that matters for Terraform's purposes. For the moment the experimental HCL2 repository is still an indirect dependency via terraform-config-inspect, so it remains in our go.sum and vendor directories for the moment. Because terraform-config-inspect uses a much smaller subset of the HCL2 functionality, this does still manage to prune the vendor directory a little. A subsequent release of terraform-config-inspect should allow us to completely remove that old repository in a future commit.	7 years ago
Katy Moe	2c44a7aacb	remove dotfile exclusion in copyDir ... copyDir is used in configload/getter.go to copy previously downloaded modules instead of using the go-getter client every time. The go-getter client downloads dotfiles, but copyDir did not copy dotfiles, leading to inconsistent behaviour when reusing the same module source.	7 years ago
Radek Simko	5b9f2fafc8	Standardise directory name for test data	7 years ago
Pam Selle	59c5cc4788	Merge pull request #21254 from davewongillies/gcs ... Add GCS source support for modules	7 years ago
Martin Atkins	e85093ce08	configs/configload: Don't panic when version constraint is added ... Previously, adding a version constraint to a module that was previously recorded without a version in the module manifest would cause a panic. Instead, we now use a slight variant of the "dependencies have changed" error that doesn't try to print out a specific version number.	7 years ago
David Gillies	8b45443b21	Add GCS source support for modules	7 years ago
Kristin Laemmert	0c5fd835ce	copyDir: detect if the module install path is a symlink to a directory (#20603) ... configs/configload and internal/initwd both had a copyDir function that would fail if the source directory contained a symlinked directory, because the os.FileMode.IsDir() returns false for symlinks. This PR adds a check for a symlink and copies that symlink in the target directory. It handles symlinks for both files and directories (with included tests). Fixes #20539	7 years ago
Martin Atkins	86c02d5c35	command: "terraform init" can partially initialize for 0.12upgrade ... There are a few constructs from 0.11 and prior that cause 0.12 parsing to fail altogether, which previously created a chicken/egg problem because we need to install the providers in order to run "terraform 0.12upgrade" and thus fix the problem. This changes "terraform init" to use the new "early configuration" loader for module and provider installation. This is built on the more permissive parser in the terraform-config-inspect package, and so it allows us to read out the top-level blocks from the configuration while accepting legacy HCL syntax. In the long run this will let us do version compatibility detection before attempting a "real" config load, giving us better error messages for any future syntax additions, but in the short term the key thing is that it allows us to install the dependencies even if the configuration isn't fully valid. Because backend init still requires full configuration, this introduces a new mode of terraform init where it detects heuristically if it seems like we need to do a configuration upgrade and does a partial init if so, before finally directing the user to run "terraform 0.12upgrade" before running any other commands. The heuristic here is based on two assumptions: - If the "early" loader finds no errors but the normal loader does, the configuration is likely to be valid for Terraform 0.11 but not 0.12. - If there's already a version constraint in the configuration that excludes Terraform versions prior to v0.12 then the configuration is probably _already_ upgraded and so it's just a normal syntax error, even if the early loader didn't detect it. Once the upgrade process is removed in 0.13.0 (users will be required to go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can be simplified to remove that special mode, but the idea of doing the dependency version checks against the liberal parser will remain valuable to increase our chances of reporting version-based incompatibilities rather than syntax errors as we add new features in future.	7 years ago
Martin Atkins	0c0a437bcb	Move module install functionality over to internal/initwd	7 years ago
Martin Atkins	0ce2add59f	internal/modsdir: Factor out module manifest models to separate package	7 years ago
Martin Atkins	f001cb8654	command: Fix terraform init -from-module with relative paths ... Since our new approach here works by installing with a synthetic module configuration block, we need to treat relative paths as a special case for two reasons: - Relative paths in module addresses are relative to the file containing the call rather than the working directory, but -from-module uses the working directory (and the call is in a synthetic "file" anyway) - We need to force Terraform to pass the path through to go-getter rather than just treating it as a relative reference, since we really do want a copy of the directory in this case, even if it is local. To address both of these things, we'll detect a relative path and turn it into an absolute path before beginning installation. This is a bit hacky, but this is consistent with the general philosophy of the -from-module implementation where it does hacky things so that the rest of the installer code can be spared of dealing with its special cases. This is covered by a couple of existing tests that run init -from-module, including TestInit_fromModule_dstInSrc which now passes.	8 years ago
Martin Atkins	741d334ee4	command: Even more fixes for "apply" command tests	8 years ago
Martin Atkins	7d0be84300	configload: Don't fail if one module is a child directory of another ... Given a module foo and a module foo/bar, the previous code might incorrectly treat "bar" as a file within "foo" rather than as a module directory in its own right.	8 years ago
Martin Atkins	a3403f2766	terraform: Ugly huge change to weave in new State and Plan types ... Due to how often the state and plan types are referenced throughout Terraform, there isn't a great way to switch them out gradually. As a consequence, this huge commit gets us from the old world to a _compilable_ new world, but still has a large number of known test failures due to key functionality being stubbed out. The stubs here are for anything that interacts with providers, since we now need to do the follow-up work to similarly replace the old terraform.ResourceProvider interface with its replacement in the new "providers" package. That work, along with work to fix the remaining failing tests, will follow in subsequent commits. The aim here was to replace all references to terraform.State and its downstream types with states.State, terraform.Plan with plans.Plan, state.State with statemgr.State, and switch to the new implementations of the state and plan file formats. However, due to the number of times those types are used, this also ended up affecting numerous other parts of core such as terraform.Hook, the backend.Backend interface, and most of the CLI commands. Just as with 5861dbf3fc before, I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.	8 years ago
Kristin Laemmert	d1f27ce8c2	registry: renaming module-specific registry functions	8 years ago
Martin Atkins	fc0e28b2b4	configload: Don't download the same module source multiple times ... It is common for the same module source package to be referenced multiple times in the same configuration, either because there are literally multiple instances of the same module source or because a single package (or repository) contains multiple modules in sub-directories and many of them are referenced. To optimize this, here we introduce a simple caching behavior where the module installer will detect if it's asked to install multiple times from the same source and produce the second and subsequent directories by copying the first, rather than by downloading again over the network. This optimization is applied once all of the go-getter detection has completed and sub-directory portions have been trimmed, so it is also able to normalize differently-specified source addresses that all ultimately detect to the same resolved address. When installing, we always extract the entire specified package (or repository) and then reference the specified sub-directory, so we can safely re-use existing directories when the base package is the same, even if the sub-directory is different. However, as a result we do not yet address the fact that the same package will be stored multiple times _on disk_, which may still be problematic when referencing large repositories multiple times in disk-storage-constrained environments. We could address this in a subsequent change by investigating the use of symlinks where possible. Since the Registry installer is implemented just as an extra resolution step in front of go-getter, this optimization applies to registry modules too. This does not apply to local relative references, which will continue to just resolve into the already-prepared directory of their parent module. The cache of previously installed paths lives only for the duration of one call to InstallModules, so we will never re-use directories that were created by previous runs of "terraform init" and there is no risk that older versions will pollute the cache when attempting an upgrade from a source address that doesn't explicitly specify a version. No additional tests are added here because the existing module installer tests (when TF_ACC=1) already cover the case of installing multiple modules from the same source.	8 years ago
Martin Atkins	eb7aaf2414	configload: Configuration snapshots ... Here we introduce a new idea of a "configuration snapshot", which is an in-memory copy of the source code of each of the files that make up the configuration. The primary intended purpose for this is as an intermediate step before writing the configuration files into a plan file, and then reading them out when that plan file is later applied. During earlier configs package development we expected to use an afero vfs implementation to read directly from the zip file, but that doesn't work in practice because we need to preserve module paths from the source file system that might include parent directory traversals (../) while retaining the original path for use in error messages. The result, for now, is a bit of an abstraction inversion: we implement a specialized afero vfs implementation that makes the sparse filesystem representation from a snapshot appear like a normal filesystem just well enough that the config loader and parser can work with it. In future we may wish to rework the internals here so that the main abstraction is at a similar level to the snapshot and then that API is mapped to the native filesystem in the normal case, removing afero. For now though, this approach avoids the need for a significant redesign of the parser/loader internals, at the expense of some trickiness in the case where we're reading from a snapshot. This commit does not yet include the reading and writing of snapshots into plan files. That will follow in a subsequent commit.	8 years ago
Martin Atkins	fa2a76fa23	configs: use addrs.Module for module path, rather than []string ... addrs.Module is itself internally just []string, but this better communicates our intent here and makes this integrate better with other code which is using this type for this purposes.	8 years ago
Martin Atkins	591aaf1e6a	configload: helper functions for tests ... These utility functions are intended to allow concisely loading a configuration from a fixture directory in a test, bailing out early if there are any unexpected errors.	8 years ago
Martin Atkins	6cf9346dfd	configs/configload: Helper for recognizing a config dir (or not) ... This is just a small wrapper around the parser's IsConfigDir method, for convenience and for law of demeter.	8 years ago
Sander van Harmelen	179b32d426	Add a `CredentialsForHost` method to disco.Disco ... By adding this method you now only have to pass a `*disco.Disco` object around in order to do discovery and use any configured credentials for the discovered hosts. Of course you can also still pass around both a `*disco.Disco` and a `auth.CredentialsSource` object if there is a need or a reason for that!	8 years ago
Martin Atkins	ea868026b8	configs/configload: installer tests inspect their result ... Previously we were just loading the module and asserting no diagnostics, but that is not really good enough since if we install modules incorrectly it's possible that we are still able to load an empty configuration successfully. Now we'll do some basic inspecetion of the module tree that results from loading what we installed, to ensure that all of the expected modules are present at the right locations in the tree.	8 years ago
Martin Atkins	d859bacbdd	configs/configload: InitDirFromModule ... This will provide the functionality of "terraform init -from-module=...", which uses the contents of a given module to populate the working directory. This mechanism is intended for installing e.g. examples from Terraform Registry or elsewhere. It's not fully-general since it can't reasonably install a module from a subdir that refers up to a parent directory, but that isn't an issue for all reasonable uses of this option.	8 years ago
Martin Atkins	74afcb4a7f	configs/configload: some loaders can't install modules ... Originally the hope was to use the afero filesystem abstraction for all loader operations, but since we install modules using go-getter we cannot (without a lot of refactoring) support vfs for installation. The vfs use-case is for reading configuration from plan zip files anyway, and so we have no real reason to support installation into a vfs. For now at least we will just add the possibility that a loader might not be install-capable. At the moment we have no non-install-capable loaders, but we'll add one later once we get to loading configuration from plan files.	8 years ago
Martin Atkins	59939cf320	configs/configload: installation from registry and go-getter ... Unlike the old installer in config/module, this uses new-style installation directories that include the static module path so that paths we show in diagnostics will be more meaningful to the user. As before, we retrieve the entire "package" associated with the given source string, rather than any given subdirectory directly, because the retrieved module may contain ../ references into parent directories which must be resolvable after extraction.	8 years ago
Martin Atkins	7feef98517	configs/configload: installation of local modules ... Enough of the InstallModules method to install local modules (those with relative paths). "Install" is actually a bit of an exaggeration for these since we actually just record them in our manifest after verifying that the source directory exists. This is a change of behavior relative to the old module installer since we no longer create a symlink to the module directory inside the .terraform/modules directory. Instead, we record the module's true location in our manifest so that the loader will find it later. The use of a symlink here predated the manifest file. Now that we have a manifest file the symlinks are redundant. Using the "natural" location of the module leads to more helpful error messages, since we'll refer to the module path as the user expects it, rather than to an internal alias.	8 years ago
Martin Atkins	72ad927c4d	configs/configload: package for loading configurations ... Previously the behavior for loading and installing modules was included in the same package as the representation of the module tree (in the config/module package). In our new world, the model of a module tree (now called a "Config") is included in "configs" along with the Module and File structs. This new package replaces the loading and installation functionality previously in config/module with new equivalents that work with the model objects in "configs". As of this commit, only the loading functionality is implemented. The installation functionality will follow in subsequent commits.	8 years ago