63 Commits (3f8a97384636479b7aafdb4ee4eefa946caee08a)

Author	SHA1	Message	Date
Martin Atkins	44bc7519a6	terraform: More wiring in of new provider types ... This doesn't actually work yet, but it builds and then panics in a pretty satisfying way.	8 years ago
Martin Atkins	549544f201	configschema: Handle nested blocks containing dynamic-typed attributes ... We need to make the collection itself be a tuple or object rather than list or map in this case, since otherwise all of the elements of the collection are constrained to be of the same type and that isn't the intent of a provider indicating that it accepts any type.	8 years ago
Martin Atkins	0317da9911	plans/objchange: logic for merging prior state with config ... This produces a "proposed new state", which already has prior computed values propagated into it (since that behavior is standard for all resource types) but could be customized further by the provider to make the "_planned_ new state". In the process of implementing this it became clear that our configschema DecoderSpec behavior is incorrect, since it's producing list values for NestingList and map values for NestingMap. While that seems like it should be right, we should actually be using tuple and object types respectively to allow each block to have a different runtime type in situations where an attribute is given the type cty.DynamicPseudoType. That's not fixed here, and so without a further fix list and map blocks will panic here. The DecoderSpec implementation will be fixed in a subsequent commit.	8 years ago
Martin Atkins	a3403f2766	terraform: Ugly huge change to weave in new State and Plan types ... Due to how often the state and plan types are referenced throughout Terraform, there isn't a great way to switch them out gradually. As a consequence, this huge commit gets us from the old world to a _compilable_ new world, but still has a large number of known test failures due to key functionality being stubbed out. The stubs here are for anything that interacts with providers, since we now need to do the follow-up work to similarly replace the old terraform.ResourceProvider interface with its replacement in the new "providers" package. That work, along with work to fix the remaining failing tests, will follow in subsequent commits. The aim here was to replace all references to terraform.State and its downstream types with states.State, terraform.Plan with plans.Plan, state.State with statemgr.State, and switch to the new implementations of the state and plan file formats. However, due to the number of times those types are used, this also ended up affecting numerous other parts of core such as terraform.Hook, the backend.Backend interface, and most of the CLI commands. Just as with 5861dbf3fc before, I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.	8 years ago
Kristin Laemmert	d1f27ce8c2	registry: renaming module-specific registry functions	8 years ago
James Bardin	8f295fcb22	CoerceValue should insert Null for unset attrs ... This matches the decoder spec, where a value unset in the configuration is always Null.	8 years ago
James Bardin	50e099ad10	don't set Unknown for computed values ... Any value that is not set in the configuration should decode as a Null value. Unknown should still be returned if a computed value expression is unknown.	8 years ago
Martin Atkins	d8bf3cc4e0	configschema: Fix ConfigSchema bugs with nested blocks ... We were iterating over the wrong value to recursively coerce content for nested blocks, and also incorrectly constructing the cty.Path used in errors.	8 years ago
Martin Atkins	479c6b2466	move "configschema" from "config" to "configs" ... The "config" package is no longer used and will be removed as part of the 0.12 release cleanup. Since configschema is part of the "new world" of configuration modelling, it makes more sense for it to live as a subdirectory of the newer "configs" package.	8 years ago
Martin Atkins	adb88eaa16	configupgrade: Analysis of input configuration ... In order to properly migrate the contents of resource, data, provider and provisioner blocks we will need the provider's schema in order to understand what is expected, so we can resolve some ambiguities inherent in the legacy HCL AST. This includes an initial prototype of migrating the content of resource blocks just to verify that the information is being gathered correctly. As with the rest of the upgrade_native.go file, this will be reorganized significantly once the basic end-to-end flow is established and we can see how to organize this code better.	8 years ago
Martin Atkins	f77e7a61b0	various: helpers for collecting necessary provider types ... Since schemas are required to interpret provider, resource, and provisioner attributes in configs, states, and plans, these helpers intend to make it easier to gather up the the necessary provider types in order to preload all of the needed schemas before beginning further processing. Config.ProviderTypes returns directly the list of provider types, since at this level further detail is not useful: we've not yet run the provider allocation algorithm, and so the only thing we can reliably extract here is provider types themselves. State.ProviderAddrs and Plan.ProviderAddrs each return a list of absolute provider addresses, which can then be turned into a list of provider types using the new helper providers.AddressedTypesAbs. Since we're already using configs.Config throughout core, this also updates the terraform.LoadSchemas helper to use Config.ProviderTypes to find the necessary providers, rather than implementing its own discovery logic. states.State is not yet plumbed in, so we cannot yet use State.ProviderAddrs to deal with the state but there's a TODO comment to remind us to update that in a later commit when we swap out terraform.State for states.State. A later commit will probably refactor this further so that we can easily obtain schema for the providers needed to interpret a plan too, but that is deferred here because further work is required to make core work with the new plan types first. At that point, terraform.LoadSchemas may become providers.LoadSchemas with a different interface that just accepts lists of provider and provisioner names that have been gathered by the caller using these new helpers.	8 years ago
Martin Atkins	ffe5f7c4e6	command: 0.12upgrade command ... This is the frontend to the work-in-progress codepath for upgrading the source code for a module written for Terraform v0.11 or earlier to use the new syntax and idiom of v0.12. The underlying upgrade code is not yet complete as of this commit, and so the command is not yet very useful. We will continue to iterate on the upgrade code in subsequent commits.	8 years ago
Martin Atkins	95b7b883a3	configupgrade: Basic expression formatting ... This covers all of the expression node types in HIL's AST, and also includes initial support for some of our top-level blocks so that we can easily test that. The initial implementations of the "variable" and "output" blocks are pretty redundant and messy, so we can hopefully improve on these in a later pass.	8 years ago
Martin Atkins	a345533573	configupgrade: Beginnings of Upgrade function ... This function is the main functionality of this package. So far it just deals with detecting and renaming JSON files that are mislabeled as native syntax files. Other functionality will follow in later commits.	8 years ago
Martin Atkins	1132898fbc	configupgrade: Load source code for a module and detect already upgraded ... This package will do all of its work in-memory so that it can avoid making partial updates and then failing, so we need to be able to load the sources files from a particular directory into memory. The upgrade process isn't idempotent, so we also attempt to detect heuristically whether an upgrade has already been performed (can parse with the new parser and has a version constraint that prevents versions earlier than 0.12) so that the CLI tool that will eventually wrap this will be able to produce a warning and prompt for confirmation in that case.	8 years ago
Martin Atkins	2f85b47586	configupgrade: new package for upgrading configs for 0.12 ... Although the new HCL implementation and configuration loader is broadly compatible with the prior implementation, it has a number of new idiomatic forms and it also cannot parse some more extreme non-idiomatic usages that were possible under the old parser. To help users migrate to the new implementation, this package will rewrite configuration to comply with the new idiom and fix as many cases as possible where the legacy parser was too liberal or exposed implementation details.	8 years ago
Martin Atkins	fc0e28b2b4	configload: Don't download the same module source multiple times ... It is common for the same module source package to be referenced multiple times in the same configuration, either because there are literally multiple instances of the same module source or because a single package (or repository) contains multiple modules in sub-directories and many of them are referenced. To optimize this, here we introduce a simple caching behavior where the module installer will detect if it's asked to install multiple times from the same source and produce the second and subsequent directories by copying the first, rather than by downloading again over the network. This optimization is applied once all of the go-getter detection has completed and sub-directory portions have been trimmed, so it is also able to normalize differently-specified source addresses that all ultimately detect to the same resolved address. When installing, we always extract the entire specified package (or repository) and then reference the specified sub-directory, so we can safely re-use existing directories when the base package is the same, even if the sub-directory is different. However, as a result we do not yet address the fact that the same package will be stored multiple times _on disk_, which may still be problematic when referencing large repositories multiple times in disk-storage-constrained environments. We could address this in a subsequent change by investigating the use of symlinks where possible. Since the Registry installer is implemented just as an extra resolution step in front of go-getter, this optimization applies to registry modules too. This does not apply to local relative references, which will continue to just resolve into the already-prepared directory of their parent module. The cache of previously installed paths lives only for the duration of one call to InstallModules, so we will never re-use directories that were created by previous runs of "terraform init" and there is no risk that older versions will pollute the cache when attempting an upgrade from a source address that doesn't explicitly specify a version. No additional tests are added here because the existing module installer tests (when TF_ACC=1) already cover the case of installing multiple modules from the same source.	8 years ago
Martin Atkins	eb7aaf2414	configload: Configuration snapshots ... Here we introduce a new idea of a "configuration snapshot", which is an in-memory copy of the source code of each of the files that make up the configuration. The primary intended purpose for this is as an intermediate step before writing the configuration files into a plan file, and then reading them out when that plan file is later applied. During earlier configs package development we expected to use an afero vfs implementation to read directly from the zip file, but that doesn't work in practice because we need to preserve module paths from the source file system that might include parent directory traversals (../) while retaining the original path for use in error messages. The result, for now, is a bit of an abstraction inversion: we implement a specialized afero vfs implementation that makes the sparse filesystem representation from a snapshot appear like a normal filesystem just well enough that the config loader and parser can work with it. In future we may wish to rework the internals here so that the main abstraction is at a similar level to the snapshot and then that API is mapped to the native filesystem in the normal case, removing afero. For now though, this approach avoids the need for a significant redesign of the parser/loader internals, at the expense of some trickiness in the case where we're reading from a snapshot. This commit does not yet include the reading and writing of snapshots into plan files. That will follow in a subsequent commit.	8 years ago
James Bardin	e0e177374f	don't hanlde "type" when parsing connection blocks ... There's no provisioner schema yet, and we need to handle the type dynamically during evaluation.	8 years ago
Martin Atkins	1f859ba8d4	configs: Handle object constructor keys when shimming traversals ... This is important in particular for shimming the "providers" map in module blocks: providers = { "aws" = "aws.foo" } We call this shim for both the key and the value here, and the value would previously have worked. However, the key is wrapped up by the parser in an ObjectConsKeyExpr container, which deals with the fact that in normal use an object constructor key that is just a bare identifier is actually interpreted as a string. We don't care about that interpretation for our shimming purposes, and so we can just unwrap it here.	8 years ago
Martin Atkins	c4fac74371	configs: Fix crasher in provider configuration reference parsing ... Due to a logic error, this would panic if given an empty traversal.	8 years ago
Martin Atkins	cd584309b9	configs: Fix typo in the deprecation warning about ignore_changes = ["*"]	8 years ago
Martin Atkins	5cf791861f	configs: Allow looking up resources by resource addresses. ... Throughout the main "terraform" package we identify resources using the address types, and so this helper is useful to make concise transitions between the address types and the configuration types. As part of this, we use the address types to produce the keys used in our resource maps. This has no visible change in behavior since the prior implementation produced an equal result, but this change ensures that ResourceByAddr cannot be broken by hypothetical future changes to the key serialization.	8 years ago
Martin Atkins	70f1635416	configs: record the source directory for modules ... We can only do this when modules are loaded with Parser.LoadConfigDir, but in practice this is the common case anyway. This is important to support the path.module and path.root expressions in configuration.	8 years ago
Martin Atkins	fa2a76fa23	configs: use addrs.Module for module path, rather than []string ... addrs.Module is itself internally just []string, but this better communicates our intent here and makes this integrate better with other code which is using this type for this purposes.	8 years ago
Martin Atkins	24dce0c624	configs: Helper methods to integrate with "addrs" package ... Our new "addrs" package gives us some nice representations of various kinds of "address" within Terraform. To talk to APIs that use these, it's convenient to be able to easily derive such addresses from the configuration objects. These new methods, along with a recasting of the existing Resource.ProviderConfigKey method to Resource.ProviderConfigAddr, give us some key integration points to support the configuration graph transforms in the main "terraform" package.	8 years ago
Martin Atkins	b6fdd0446e	configs: parse the "providers" map for module calls ... This was accidentally missed on the first pass of module call decoding. As before, this is a map from child provider config address to parent provider config address, allowing the set of providers to be projected in arbitrary ways into a child module.	8 years ago
Martin Atkins	dd62cd97c9	configs: DisabledModuleWalker ... This is a built-in implementation of ModuleWalker that just returns an error any time it's asked for a module. This is intended for simple unit tests where no child modules are needed anyway.	8 years ago
Martin Atkins	2eba023537	configs: NewEmptyConfig function ... This is useful for creating a valid placeholder configuration, but not much else. Most callers should use BuildConfig to build a configuration that actually has something in it.	8 years ago
Martin Atkins	c07b0a7806	configs: Re-unify the ManagedResource and DataResource types ... Initially the intent here was to tease these apart a little more since they don't really share much behavior in common in core, but in practice it'll take a lot of refactoring to tease apart these assumptions in core right now and so we'll keep these things unified at the configuration layer in the interests of minimizing disruption at the core layer. The two types are still kept in separate maps to help reinforce the fact that they are separate concepts with some behaviors in common, rather than the same concept.	8 years ago
Martin Atkins	cd51864d84	configs: Start using the new "addrs" package types for modules ... We initially just mimicked our old practice of using []string for module paths here, but the addrs package now gives us a pair of types that better capture the two different kinds of module addresses we are dealing with: static addresses (nodes in the configuration tree) and dynamic/instance addresses (which can represent the situation where multiple instances are created from a single module call). This distinction still remains rather artificial since we don't yet have support for count or for_each on module calls, but this is intended to lay the foundations for that to be added later, and in the mean time just gives us some handy helper functions for parsing and formatting these address types.	8 years ago
Martin Atkins	4ed06a9227	terraform: HCL2-flavored module dependency resolver ... For the moment this is just a lightly-adapted copy of ModuleTreeDependencies named ConfigTreeDependencies, with the goal that the two can live concurrently for the moment while not all callers are yet updated and then we can drop ModuleTreeDependencies and its helper functions altogether in a later commit. This can then be used to make "terraform init" and "terraform providers" work properly with the HCL2-powered configuration loader.	8 years ago
Martin Atkins	ebafa51723	command: Various updates for the new backend package API ... This is a rather-messy, complex change to get the "command" package building again against the new backend API that was updated for the new configuration loader. A lot of this is mechanical rewriting to the new API, but meta_config.go and meta_backend.go in particular saw some major changes to interface with the new loader APIs and to deal with the change in order of steps in the backend API.	8 years ago
Martin Atkins	591aaf1e6a	configload: helper functions for tests ... These utility functions are intended to allow concisely loading a configuration from a fixture directory in a test, bailing out early if there are any unexpected errors.	8 years ago
Martin Atkins	6cf9346dfd	configs/configload: Helper for recognizing a config dir (or not) ... This is just a small wrapper around the parser's IsConfigDir method, for convenience and for law of demeter.	8 years ago
Martin Atkins	5dd6b839d0	configs: Export MergeBodies and new SynthBody function ... We have a few special use-cases in Terraform where an object is constructed from a mixture of different sources, such as a configuration file, command line arguments, and environment variables. To represent this within the HCL model, we introduce a new "synthetic" HCL body type that just represents a map of values that are interpreted as attributes. We then export the previously-private MergeBodies function to allow the synthetic body to be used as an override for a "real" body, which then allows us to combine these various sources together while still retaining the proper source location information for each individual attribute. Since a synthetic body doesn't actually exist in configuration, it does not produce source locations that can be turned into source snippets but we can still use placeholder strings to help the user to understand which of the many different sources a particular value came from.	8 years ago
Sander van Harmelen	179b32d426	Add a `CredentialsForHost` method to disco.Disco ... By adding this method you now only have to pass a `*disco.Disco` object around in order to do discovery and use any configured credentials for the discovered hosts. Of course you can also still pass around both a `*disco.Disco` and a `auth.CredentialsSource` object if there is a need or a reason for that!	8 years ago
Martin Atkins	5661ab5991	configs: allow full type constraints for variables ... Previously we just ported over the simple "string", "list", and "map" type hint keywords from the old loader, which exist primarily as hints to the CLI for whether to treat -var=... arguments and environment variables as literal strings or as HCL expressions. However, we've been requested before to allow more specific constraints here because it's generally better UX for a type error to be detected within an expression in a calling "module" block rather than at some point deep inside a third-party module. To allow for more specific constraints, here we use the type constraint expression syntax defined as an extension within HCL, which uses the variable and function call syntaxes to represent types rather than values, like this: - string - number - bool - list(string) - list(any) - list(map(string)) - object({id=string,name=string}) In native HCL syntax this looks like: variable "foo" { type = map(string) } In JSON, this looks like: { "variable": { "foo": { "type": "map(string)" } } } The selection of literal processing or HCL parsing of CLI-set values is now explicit in the model and separate from the type, though it's still derived from the type constraint and thus not directly controllable in configuration. Since this syntax is more complex than the keywords that replaced it, for now the simpler keywords are still supported and "list" and "map" are interpreted as list(any) and map(any) respectively, mimicking how they were interpreted by Terraform 0.11 and earlier. For the time being our documentation should continue to recommend these shorthand versions until we gain more experience with the more-specific type constraints; most users should just make use of the additional primitive type constraints this enables: bool and number. As a result of these more-complete type constraints, we can now type-check the default value at config load time, which has the nice side-effect of allowing us to produce a tailored error message if an override file produces an invalid situation; previously the result was rather confusing because the error message referred to the original definition of the variable and not the overridden parts.	8 years ago
Martin Atkins	c05a1050fc	configs: Disable deprecation warning for quoted keywords/references ... Although we do still consider these deprecated for 0.12, we'll defer actually generating warnings for them until a later minor release so that module authors can retain their quoted identifiers for a period after 0.12 release for backward-compatibility with Terraform 0.11.	8 years ago
Martin Atkins	a26ff56f01	configs: highlight resource name in diags when invalid ... Previously we were erroneously highlighting the resource type name instead.	8 years ago
Martin Atkins	7549ce15f0	configs: update values file invalid syntax test for new HCL behavior ... The error-handling behavior of the HCL parser was improved, which causes the number of diagnostics and the diagnostics messages to be different in cases where a block-like introduction is given but without any following body.	8 years ago
Martin Atkins	4fa8c16ead	configs: support ignore_changes wildcards ... The initial pass of implementation here missed the special case where ignore_changes can, in the old parser, be set to ["*"] to ignore changes to all attributes. Since that syntax is awkward and non-obvious, our new decoder will instead expect ignore_changes = all, using HCL2's capability to interpret an expression as a literal keyword. For compatibility with old configurations we will still accept the ["*"] form but emit a deprecation warning to encourage moving to the new form.	8 years ago
Martin Atkins	c5f5340b15	configs: Additional guidance in doc.go ... There's quite a lot in this package, so hopefully this additional paragraph will help readers get oriented.	8 years ago
Martin Atkins	36fb5b52e7	configs: quoted keywords/references are warnings, not errors ... In our new loader we are changing certain values in configuration to be naked keywords or references rather than quoted strings as before. Since many of these have been shown in books, tutorials, and our own documentation we will make the old forms generate deprecation warnings rather than errors so that newcomers starting from older documentation can be eased into the new syntax, rather than getting blocked. This will also avoid creating a hard compatibility wall for reusable modules that are already published, allowing them to still be used in spite of these warnings and then fixed when the maintainer is able.	8 years ago
Martin Atkins	ea868026b8	configs/configload: installer tests inspect their result ... Previously we were just loading the module and asserting no diagnostics, but that is not really good enough since if we install modules incorrectly it's possible that we are still able to load an empty configuration successfully. Now we'll do some basic inspecetion of the module tree that results from loading what we installed, to ensure that all of the expected modules are present at the right locations in the tree.	8 years ago
Martin Atkins	d859bacbdd	configs/configload: InitDirFromModule ... This will provide the functionality of "terraform init -from-module=...", which uses the contents of a given module to populate the working directory. This mechanism is intended for installing e.g. examples from Terraform Registry or elsewhere. It's not fully-general since it can't reasonably install a module from a subdir that refers up to a parent directory, but that isn't an issue for all reasonable uses of this option.	8 years ago
Martin Atkins	74afcb4a7f	configs/configload: some loaders can't install modules ... Originally the hope was to use the afero filesystem abstraction for all loader operations, but since we install modules using go-getter we cannot (without a lot of refactoring) support vfs for installation. The vfs use-case is for reading configuration from plan zip files anyway, and so we have no real reason to support installation into a vfs. For now at least we will just add the possibility that a loader might not be install-capable. At the moment we have no non-install-capable loaders, but we'll add one later once we get to loading configuration from plan files.	8 years ago
Martin Atkins	59939cf320	configs/configload: installation from registry and go-getter ... Unlike the old installer in config/module, this uses new-style installation directories that include the static module path so that paths we show in diagnostics will be more meaningful to the user. As before, we retrieve the entire "package" associated with the given source string, rather than any given subdirectory directly, because the retrieved module may contain ../ references into parent directories which must be resolvable after extraction.	8 years ago
Martin Atkins	3d551e25e0	configs: BuildConfig sorts child modules by name ... This is not strictly necessary, but since this is not a performance-critical codepath we'll do this because it makes life easier for callers that want to print out user-facing logs about build process, or who are logging actions taken as part of a unit test.	8 years ago
Martin Atkins	7feef98517	configs/configload: installation of local modules ... Enough of the InstallModules method to install local modules (those with relative paths). "Install" is actually a bit of an exaggeration for these since we actually just record them in our manifest after verifying that the source directory exists. This is a change of behavior relative to the old module installer since we no longer create a symlink to the module directory inside the .terraform/modules directory. Instead, we record the module's true location in our manifest so that the loader will find it later. The use of a symlink here predated the manifest file. Now that we have a manifest file the symlinks are redundant. Using the "natural" location of the module leads to more helpful error messages, since we'll refer to the module path as the user expects it, rather than to an internal alias.	8 years ago