53 Commits (eac87798708e714fd3a7e2ae496bddc021a32fa0)

Author	SHA1	Message	Date
Martin Atkins	f77e7a61b0	various: helpers for collecting necessary provider types ... Since schemas are required to interpret provider, resource, and provisioner attributes in configs, states, and plans, these helpers intend to make it easier to gather up the the necessary provider types in order to preload all of the needed schemas before beginning further processing. Config.ProviderTypes returns directly the list of provider types, since at this level further detail is not useful: we've not yet run the provider allocation algorithm, and so the only thing we can reliably extract here is provider types themselves. State.ProviderAddrs and Plan.ProviderAddrs each return a list of absolute provider addresses, which can then be turned into a list of provider types using the new helper providers.AddressedTypesAbs. Since we're already using configs.Config throughout core, this also updates the terraform.LoadSchemas helper to use Config.ProviderTypes to find the necessary providers, rather than implementing its own discovery logic. states.State is not yet plumbed in, so we cannot yet use State.ProviderAddrs to deal with the state but there's a TODO comment to remind us to update that in a later commit when we swap out terraform.State for states.State. A later commit will probably refactor this further so that we can easily obtain schema for the providers needed to interpret a plan too, but that is deferred here because further work is required to make core work with the new plan types first. At that point, terraform.LoadSchemas may become providers.LoadSchemas with a different interface that just accepts lists of provider and provisioner names that have been gathered by the caller using these new helpers.	8 years ago
Martin Atkins	ffe5f7c4e6	command: 0.12upgrade command ... This is the frontend to the work-in-progress codepath for upgrading the source code for a module written for Terraform v0.11 or earlier to use the new syntax and idiom of v0.12. The underlying upgrade code is not yet complete as of this commit, and so the command is not yet very useful. We will continue to iterate on the upgrade code in subsequent commits.	8 years ago
Martin Atkins	95b7b883a3	configupgrade: Basic expression formatting ... This covers all of the expression node types in HIL's AST, and also includes initial support for some of our top-level blocks so that we can easily test that. The initial implementations of the "variable" and "output" blocks are pretty redundant and messy, so we can hopefully improve on these in a later pass.	8 years ago
Martin Atkins	a345533573	configupgrade: Beginnings of Upgrade function ... This function is the main functionality of this package. So far it just deals with detecting and renaming JSON files that are mislabeled as native syntax files. Other functionality will follow in later commits.	8 years ago
Martin Atkins	1132898fbc	configupgrade: Load source code for a module and detect already upgraded ... This package will do all of its work in-memory so that it can avoid making partial updates and then failing, so we need to be able to load the sources files from a particular directory into memory. The upgrade process isn't idempotent, so we also attempt to detect heuristically whether an upgrade has already been performed (can parse with the new parser and has a version constraint that prevents versions earlier than 0.12) so that the CLI tool that will eventually wrap this will be able to produce a warning and prompt for confirmation in that case.	8 years ago
Martin Atkins	2f85b47586	configupgrade: new package for upgrading configs for 0.12 ... Although the new HCL implementation and configuration loader is broadly compatible with the prior implementation, it has a number of new idiomatic forms and it also cannot parse some more extreme non-idiomatic usages that were possible under the old parser. To help users migrate to the new implementation, this package will rewrite configuration to comply with the new idiom and fix as many cases as possible where the legacy parser was too liberal or exposed implementation details.	8 years ago
Martin Atkins	fc0e28b2b4	configload: Don't download the same module source multiple times ... It is common for the same module source package to be referenced multiple times in the same configuration, either because there are literally multiple instances of the same module source or because a single package (or repository) contains multiple modules in sub-directories and many of them are referenced. To optimize this, here we introduce a simple caching behavior where the module installer will detect if it's asked to install multiple times from the same source and produce the second and subsequent directories by copying the first, rather than by downloading again over the network. This optimization is applied once all of the go-getter detection has completed and sub-directory portions have been trimmed, so it is also able to normalize differently-specified source addresses that all ultimately detect to the same resolved address. When installing, we always extract the entire specified package (or repository) and then reference the specified sub-directory, so we can safely re-use existing directories when the base package is the same, even if the sub-directory is different. However, as a result we do not yet address the fact that the same package will be stored multiple times _on disk_, which may still be problematic when referencing large repositories multiple times in disk-storage-constrained environments. We could address this in a subsequent change by investigating the use of symlinks where possible. Since the Registry installer is implemented just as an extra resolution step in front of go-getter, this optimization applies to registry modules too. This does not apply to local relative references, which will continue to just resolve into the already-prepared directory of their parent module. The cache of previously installed paths lives only for the duration of one call to InstallModules, so we will never re-use directories that were created by previous runs of "terraform init" and there is no risk that older versions will pollute the cache when attempting an upgrade from a source address that doesn't explicitly specify a version. No additional tests are added here because the existing module installer tests (when TF_ACC=1) already cover the case of installing multiple modules from the same source.	8 years ago
Martin Atkins	eb7aaf2414	configload: Configuration snapshots ... Here we introduce a new idea of a "configuration snapshot", which is an in-memory copy of the source code of each of the files that make up the configuration. The primary intended purpose for this is as an intermediate step before writing the configuration files into a plan file, and then reading them out when that plan file is later applied. During earlier configs package development we expected to use an afero vfs implementation to read directly from the zip file, but that doesn't work in practice because we need to preserve module paths from the source file system that might include parent directory traversals (../) while retaining the original path for use in error messages. The result, for now, is a bit of an abstraction inversion: we implement a specialized afero vfs implementation that makes the sparse filesystem representation from a snapshot appear like a normal filesystem just well enough that the config loader and parser can work with it. In future we may wish to rework the internals here so that the main abstraction is at a similar level to the snapshot and then that API is mapped to the native filesystem in the normal case, removing afero. For now though, this approach avoids the need for a significant redesign of the parser/loader internals, at the expense of some trickiness in the case where we're reading from a snapshot. This commit does not yet include the reading and writing of snapshots into plan files. That will follow in a subsequent commit.	8 years ago
James Bardin	e0e177374f	don't hanlde "type" when parsing connection blocks ... There's no provisioner schema yet, and we need to handle the type dynamically during evaluation.	8 years ago
Martin Atkins	1f859ba8d4	configs: Handle object constructor keys when shimming traversals ... This is important in particular for shimming the "providers" map in module blocks: providers = { "aws" = "aws.foo" } We call this shim for both the key and the value here, and the value would previously have worked. However, the key is wrapped up by the parser in an ObjectConsKeyExpr container, which deals with the fact that in normal use an object constructor key that is just a bare identifier is actually interpreted as a string. We don't care about that interpretation for our shimming purposes, and so we can just unwrap it here.	8 years ago
Martin Atkins	c4fac74371	configs: Fix crasher in provider configuration reference parsing ... Due to a logic error, this would panic if given an empty traversal.	8 years ago
Martin Atkins	cd584309b9	configs: Fix typo in the deprecation warning about ignore_changes = ["*"]	8 years ago
Martin Atkins	5cf791861f	configs: Allow looking up resources by resource addresses. ... Throughout the main "terraform" package we identify resources using the address types, and so this helper is useful to make concise transitions between the address types and the configuration types. As part of this, we use the address types to produce the keys used in our resource maps. This has no visible change in behavior since the prior implementation produced an equal result, but this change ensures that ResourceByAddr cannot be broken by hypothetical future changes to the key serialization.	8 years ago
Martin Atkins	70f1635416	configs: record the source directory for modules ... We can only do this when modules are loaded with Parser.LoadConfigDir, but in practice this is the common case anyway. This is important to support the path.module and path.root expressions in configuration.	8 years ago
Martin Atkins	fa2a76fa23	configs: use addrs.Module for module path, rather than []string ... addrs.Module is itself internally just []string, but this better communicates our intent here and makes this integrate better with other code which is using this type for this purposes.	8 years ago
Martin Atkins	24dce0c624	configs: Helper methods to integrate with "addrs" package ... Our new "addrs" package gives us some nice representations of various kinds of "address" within Terraform. To talk to APIs that use these, it's convenient to be able to easily derive such addresses from the configuration objects. These new methods, along with a recasting of the existing Resource.ProviderConfigKey method to Resource.ProviderConfigAddr, give us some key integration points to support the configuration graph transforms in the main "terraform" package.	8 years ago
Martin Atkins	b6fdd0446e	configs: parse the "providers" map for module calls ... This was accidentally missed on the first pass of module call decoding. As before, this is a map from child provider config address to parent provider config address, allowing the set of providers to be projected in arbitrary ways into a child module.	8 years ago
Martin Atkins	dd62cd97c9	configs: DisabledModuleWalker ... This is a built-in implementation of ModuleWalker that just returns an error any time it's asked for a module. This is intended for simple unit tests where no child modules are needed anyway.	8 years ago
Martin Atkins	2eba023537	configs: NewEmptyConfig function ... This is useful for creating a valid placeholder configuration, but not much else. Most callers should use BuildConfig to build a configuration that actually has something in it.	8 years ago
Martin Atkins	c07b0a7806	configs: Re-unify the ManagedResource and DataResource types ... Initially the intent here was to tease these apart a little more since they don't really share much behavior in common in core, but in practice it'll take a lot of refactoring to tease apart these assumptions in core right now and so we'll keep these things unified at the configuration layer in the interests of minimizing disruption at the core layer. The two types are still kept in separate maps to help reinforce the fact that they are separate concepts with some behaviors in common, rather than the same concept.	8 years ago
Martin Atkins	cd51864d84	configs: Start using the new "addrs" package types for modules ... We initially just mimicked our old practice of using []string for module paths here, but the addrs package now gives us a pair of types that better capture the two different kinds of module addresses we are dealing with: static addresses (nodes in the configuration tree) and dynamic/instance addresses (which can represent the situation where multiple instances are created from a single module call). This distinction still remains rather artificial since we don't yet have support for count or for_each on module calls, but this is intended to lay the foundations for that to be added later, and in the mean time just gives us some handy helper functions for parsing and formatting these address types.	8 years ago
Martin Atkins	4ed06a9227	terraform: HCL2-flavored module dependency resolver ... For the moment this is just a lightly-adapted copy of ModuleTreeDependencies named ConfigTreeDependencies, with the goal that the two can live concurrently for the moment while not all callers are yet updated and then we can drop ModuleTreeDependencies and its helper functions altogether in a later commit. This can then be used to make "terraform init" and "terraform providers" work properly with the HCL2-powered configuration loader.	8 years ago
Martin Atkins	ebafa51723	command: Various updates for the new backend package API ... This is a rather-messy, complex change to get the "command" package building again against the new backend API that was updated for the new configuration loader. A lot of this is mechanical rewriting to the new API, but meta_config.go and meta_backend.go in particular saw some major changes to interface with the new loader APIs and to deal with the change in order of steps in the backend API.	8 years ago
Martin Atkins	591aaf1e6a	configload: helper functions for tests ... These utility functions are intended to allow concisely loading a configuration from a fixture directory in a test, bailing out early if there are any unexpected errors.	8 years ago
Martin Atkins	6cf9346dfd	configs/configload: Helper for recognizing a config dir (or not) ... This is just a small wrapper around the parser's IsConfigDir method, for convenience and for law of demeter.	8 years ago
Martin Atkins	5dd6b839d0	configs: Export MergeBodies and new SynthBody function ... We have a few special use-cases in Terraform where an object is constructed from a mixture of different sources, such as a configuration file, command line arguments, and environment variables. To represent this within the HCL model, we introduce a new "synthetic" HCL body type that just represents a map of values that are interpreted as attributes. We then export the previously-private MergeBodies function to allow the synthetic body to be used as an override for a "real" body, which then allows us to combine these various sources together while still retaining the proper source location information for each individual attribute. Since a synthetic body doesn't actually exist in configuration, it does not produce source locations that can be turned into source snippets but we can still use placeholder strings to help the user to understand which of the many different sources a particular value came from.	8 years ago
Sander van Harmelen	179b32d426	Add a `CredentialsForHost` method to disco.Disco ... By adding this method you now only have to pass a `*disco.Disco` object around in order to do discovery and use any configured credentials for the discovered hosts. Of course you can also still pass around both a `*disco.Disco` and a `auth.CredentialsSource` object if there is a need or a reason for that!	8 years ago
Martin Atkins	5661ab5991	configs: allow full type constraints for variables ... Previously we just ported over the simple "string", "list", and "map" type hint keywords from the old loader, which exist primarily as hints to the CLI for whether to treat -var=... arguments and environment variables as literal strings or as HCL expressions. However, we've been requested before to allow more specific constraints here because it's generally better UX for a type error to be detected within an expression in a calling "module" block rather than at some point deep inside a third-party module. To allow for more specific constraints, here we use the type constraint expression syntax defined as an extension within HCL, which uses the variable and function call syntaxes to represent types rather than values, like this: - string - number - bool - list(string) - list(any) - list(map(string)) - object({id=string,name=string}) In native HCL syntax this looks like: variable "foo" { type = map(string) } In JSON, this looks like: { "variable": { "foo": { "type": "map(string)" } } } The selection of literal processing or HCL parsing of CLI-set values is now explicit in the model and separate from the type, though it's still derived from the type constraint and thus not directly controllable in configuration. Since this syntax is more complex than the keywords that replaced it, for now the simpler keywords are still supported and "list" and "map" are interpreted as list(any) and map(any) respectively, mimicking how they were interpreted by Terraform 0.11 and earlier. For the time being our documentation should continue to recommend these shorthand versions until we gain more experience with the more-specific type constraints; most users should just make use of the additional primitive type constraints this enables: bool and number. As a result of these more-complete type constraints, we can now type-check the default value at config load time, which has the nice side-effect of allowing us to produce a tailored error message if an override file produces an invalid situation; previously the result was rather confusing because the error message referred to the original definition of the variable and not the overridden parts.	8 years ago
Martin Atkins	c05a1050fc	configs: Disable deprecation warning for quoted keywords/references ... Although we do still consider these deprecated for 0.12, we'll defer actually generating warnings for them until a later minor release so that module authors can retain their quoted identifiers for a period after 0.12 release for backward-compatibility with Terraform 0.11.	8 years ago
Martin Atkins	a26ff56f01	configs: highlight resource name in diags when invalid ... Previously we were erroneously highlighting the resource type name instead.	8 years ago
Martin Atkins	7549ce15f0	configs: update values file invalid syntax test for new HCL behavior ... The error-handling behavior of the HCL parser was improved, which causes the number of diagnostics and the diagnostics messages to be different in cases where a block-like introduction is given but without any following body.	8 years ago
Martin Atkins	4fa8c16ead	configs: support ignore_changes wildcards ... The initial pass of implementation here missed the special case where ignore_changes can, in the old parser, be set to ["*"] to ignore changes to all attributes. Since that syntax is awkward and non-obvious, our new decoder will instead expect ignore_changes = all, using HCL2's capability to interpret an expression as a literal keyword. For compatibility with old configurations we will still accept the ["*"] form but emit a deprecation warning to encourage moving to the new form.	8 years ago
Martin Atkins	c5f5340b15	configs: Additional guidance in doc.go ... There's quite a lot in this package, so hopefully this additional paragraph will help readers get oriented.	8 years ago
Martin Atkins	36fb5b52e7	configs: quoted keywords/references are warnings, not errors ... In our new loader we are changing certain values in configuration to be naked keywords or references rather than quoted strings as before. Since many of these have been shown in books, tutorials, and our own documentation we will make the old forms generate deprecation warnings rather than errors so that newcomers starting from older documentation can be eased into the new syntax, rather than getting blocked. This will also avoid creating a hard compatibility wall for reusable modules that are already published, allowing them to still be used in spite of these warnings and then fixed when the maintainer is able.	8 years ago
Martin Atkins	ea868026b8	configs/configload: installer tests inspect their result ... Previously we were just loading the module and asserting no diagnostics, but that is not really good enough since if we install modules incorrectly it's possible that we are still able to load an empty configuration successfully. Now we'll do some basic inspecetion of the module tree that results from loading what we installed, to ensure that all of the expected modules are present at the right locations in the tree.	8 years ago
Martin Atkins	d859bacbdd	configs/configload: InitDirFromModule ... This will provide the functionality of "terraform init -from-module=...", which uses the contents of a given module to populate the working directory. This mechanism is intended for installing e.g. examples from Terraform Registry or elsewhere. It's not fully-general since it can't reasonably install a module from a subdir that refers up to a parent directory, but that isn't an issue for all reasonable uses of this option.	8 years ago
Martin Atkins	74afcb4a7f	configs/configload: some loaders can't install modules ... Originally the hope was to use the afero filesystem abstraction for all loader operations, but since we install modules using go-getter we cannot (without a lot of refactoring) support vfs for installation. The vfs use-case is for reading configuration from plan zip files anyway, and so we have no real reason to support installation into a vfs. For now at least we will just add the possibility that a loader might not be install-capable. At the moment we have no non-install-capable loaders, but we'll add one later once we get to loading configuration from plan files.	8 years ago
Martin Atkins	59939cf320	configs/configload: installation from registry and go-getter ... Unlike the old installer in config/module, this uses new-style installation directories that include the static module path so that paths we show in diagnostics will be more meaningful to the user. As before, we retrieve the entire "package" associated with the given source string, rather than any given subdirectory directly, because the retrieved module may contain ../ references into parent directories which must be resolvable after extraction.	8 years ago
Martin Atkins	3d551e25e0	configs: BuildConfig sorts child modules by name ... This is not strictly necessary, but since this is not a performance-critical codepath we'll do this because it makes life easier for callers that want to print out user-facing logs about build process, or who are logging actions taken as part of a unit test.	8 years ago
Martin Atkins	7feef98517	configs/configload: installation of local modules ... Enough of the InstallModules method to install local modules (those with relative paths). "Install" is actually a bit of an exaggeration for these since we actually just record them in our manifest after verifying that the source directory exists. This is a change of behavior relative to the old module installer since we no longer create a symlink to the module directory inside the .terraform/modules directory. Instead, we record the module's true location in our manifest so that the loader will find it later. The use of a symlink here predated the manifest file. Now that we have a manifest file the symlinks are redundant. Using the "natural" location of the module leads to more helpful error messages, since we'll refer to the module path as the user expects it, rather than to an internal alias.	8 years ago
Martin Atkins	72ad927c4d	configs/configload: package for loading configurations ... Previously the behavior for loading and installing modules was included in the same package as the representation of the module tree (in the config/module package). In our new world, the model of a module tree (now called a "Config") is included in "configs" along with the Module and File structs. This new package replaces the loading and installation functionality previously in config/module with new equivalents that work with the model objects in "configs". As of this commit, only the loading functionality is implemented. The installation functionality will follow in subsequent commits.	8 years ago
Martin Atkins	9153bb448e	configs: include the module call source range in our module tree	8 years ago
Martin Atkins	8929eca405	configs: BuildConfig function ... BuildConfig creates a module tree by recursively walking through module calls in the root module and any descendent modules. This is intended to be used both for the simple case of loading already-installed modules and the more complex case of installing modules inside "terraform init", both of which will be dealt with in a separate package.	8 years ago
Martin Atkins	cc38e91612	configs: Implementation of mergeBody ... mergeBody is a hcl.Body implementation that deals with our override file merging behavior for the portions of the configuration that are not processed until full eval time. Mimicking the behavior of our old config merge implementation from the "config" package, the rules here are: - Attributes in the override body hide attributes of the same name in the base body. - Any block in the override body hides all blocks with the same type name that appear in the base body. This is tested by a new test for the overriding of module arguments, which asserts the correct behavior of the merged body as part of its work.	8 years ago
Martin Atkins	7c8efe103e	configs: allow overrides files to omit args that primary files can't ... Some of the fields in our config structs are either mandatory in primary files or there is a default value that we apply if absent. Unfortunately override files impose the additional constraint that we be allowed to omit required fields (which have presumably already been set in the primary files) and that we are able to distinguish between a default value and omitting a value entirely. Since most of our fields were already acceptable for override files, here we just add some new fields to deal with the few cases where special handling is required and a helper function to disable the "Required" flag on attributes in a given schema.	8 years ago
Martin Atkins	4e5efa498a	configs: Parser.LoadConfigDir ... This method wraps LoadConfigFile to load all of the .tf and .tf.json files in a given directory and then bundle them together into a Module object. This function also deals with the distinction between primary and override files, first appending together the primary files in lexicographic order by filename, and then merging in override files in the same order. The merging behavior is not fully implemented as of this commit, and so will be expanded in future commits.	8 years ago
Martin Atkins	9be399d49c	configs: Another simple test for _invalid_ config files ... Much like TestParserLoadConfigFileSuccess, this is intended to be an easy-to-maintain collection of bad examples to test different permutations of our error handling. As with TestParserLoadConfigFileSuccess, we should also have more specific tests alongside this that check that the error outcome is what was expected, since this test just accepts any error and may thus not be testing what we think it is.	8 years ago
Martin Atkins	e524d1eb95	configs: Simple test of loading valid configuration files ... This test is intended to be an easy-to-maintain catalog of good examples that we can use to catch certain parsing or decoding regressions easily. It's not a fully-comprehensive test since it doesn't check the result of decoding, instead just accepting any decode that completes without errors. However, an easy-to-maintain test like this is a good complement to some more specialized tests since we can easily collect good examples over time and just add them in here.	8 years ago
Martin Atkins	e15ec486bf	configs: Parser.LoadConfigFile ... This is a first pass of decoding of the main Terraform configuration file format. It hasn't yet been tested with any real-world configurations, so it will need to be revised further as we test it more thoroughly.	8 years ago
Martin Atkins	13fa73c63e	configs: stub out main configuration structs ... These types represent the individual elements within configuration, the modules a configuration is made of, and the configuration (static module tree) itself.	8 years ago