22162 Commits (eac87798708e714fd3a7e2ae496bddc021a32fa0)

 

All Branches   

Search

Author	SHA1	Message	Date
Martin Atkins	eac8779870	command: validate must set values for root variables ... Since the intent of the validate command is to check config validity regardless of context (input variables, state, etc), we use unknown values of the requested type here, which will then allow us to complete type checking against the specified types of the variables without assuming any particular values.	7 years ago
Martin Atkins	f77e7a61b0	various: helpers for collecting necessary provider types ... Since schemas are required to interpret provider, resource, and provisioner attributes in configs, states, and plans, these helpers intend to make it easier to gather up the the necessary provider types in order to preload all of the needed schemas before beginning further processing. Config.ProviderTypes returns directly the list of provider types, since at this level further detail is not useful: we've not yet run the provider allocation algorithm, and so the only thing we can reliably extract here is provider types themselves. State.ProviderAddrs and Plan.ProviderAddrs each return a list of absolute provider addresses, which can then be turned into a list of provider types using the new helper providers.AddressedTypesAbs. Since we're already using configs.Config throughout core, this also updates the terraform.LoadSchemas helper to use Config.ProviderTypes to find the necessary providers, rather than implementing its own discovery logic. states.State is not yet plumbed in, so we cannot yet use State.ProviderAddrs to deal with the state but there's a TODO comment to remind us to update that in a later commit when we swap out terraform.State for states.State. A later commit will probably refactor this further so that we can easily obtain schema for the providers needed to interpret a plan too, but that is deferred here because further work is required to make core work with the new plan types first. At that point, terraform.LoadSchemas may become providers.LoadSchemas with a different interface that just accepts lists of provider and provisioner names that have been gathered by the caller using these new helpers.	7 years ago
Martin Atkins	51aeba2cf2	plans: Record provider config address in each resource change ... This allows us to record the resource-to-provider associations we made during the plan phase and ensure that we use the same config during apply. This isn't technically necessary, since we can in principle just repeat the same matching algorithm against the config, but that algorithm is relatively complicated and so if we execute it just once during plan we remove the risk of bugs causing different decisions to be made during the apply phase. This also includes updates to the plan file format to include the provider addresses. Since we've not actually shipped any version of Terraform using our protobuf schema yet, we renumber here all of the fields after the new one to keep them incrementing consecutively. This example should not be followed after we have a released version of Terraform using this, as a courtesy to anyone trying to parse these files (even though we're not yet guaranteeing compatibility between releases).	7 years ago
Martin Atkins	ffe5f7c4e6	command: 0.12upgrade command ... This is the frontend to the work-in-progress codepath for upgrading the source code for a module written for Terraform v0.11 or earlier to use the new syntax and idiom of v0.12. The underlying upgrade code is not yet complete as of this commit, and so the command is not yet very useful. We will continue to iterate on the upgrade code in subsequent commits.	7 years ago
Martin Atkins	95b7b883a3	configupgrade: Basic expression formatting ... This covers all of the expression node types in HIL's AST, and also includes initial support for some of our top-level blocks so that we can easily test that. The initial implementations of the "variable" and "output" blocks are pretty redundant and messy, so we can hopefully improve on these in a later pass.	7 years ago
Martin Atkins	a345533573	configupgrade: Beginnings of Upgrade function ... This function is the main functionality of this package. So far it just deals with detecting and renaming JSON files that are mislabeled as native syntax files. Other functionality will follow in later commits.	7 years ago
Martin Atkins	1132898fbc	configupgrade: Load source code for a module and detect already upgraded ... This package will do all of its work in-memory so that it can avoid making partial updates and then failing, so we need to be able to load the sources files from a particular directory into memory. The upgrade process isn't idempotent, so we also attempt to detect heuristically whether an upgrade has already been performed (can parse with the new parser and has a version constraint that prevents versions earlier than 0.12) so that the CLI tool that will eventually wrap this will be able to produce a warning and prompt for confirmation in that case.	7 years ago
Martin Atkins	2f85b47586	configupgrade: new package for upgrading configs for 0.12 ... Although the new HCL implementation and configuration loader is broadly compatible with the prior implementation, it has a number of new idiomatic forms and it also cannot parse some more extreme non-idiomatic usages that were possible under the old parser. To help users migrate to the new implementation, this package will rewrite configuration to comply with the new idiom and fix as many cases as possible where the legacy parser was too liberal or exposed implementation details.	7 years ago
Martin Atkins	fc0e28b2b4	configload: Don't download the same module source multiple times ... It is common for the same module source package to be referenced multiple times in the same configuration, either because there are literally multiple instances of the same module source or because a single package (or repository) contains multiple modules in sub-directories and many of them are referenced. To optimize this, here we introduce a simple caching behavior where the module installer will detect if it's asked to install multiple times from the same source and produce the second and subsequent directories by copying the first, rather than by downloading again over the network. This optimization is applied once all of the go-getter detection has completed and sub-directory portions have been trimmed, so it is also able to normalize differently-specified source addresses that all ultimately detect to the same resolved address. When installing, we always extract the entire specified package (or repository) and then reference the specified sub-directory, so we can safely re-use existing directories when the base package is the same, even if the sub-directory is different. However, as a result we do not yet address the fact that the same package will be stored multiple times _on disk_, which may still be problematic when referencing large repositories multiple times in disk-storage-constrained environments. We could address this in a subsequent change by investigating the use of symlinks where possible. Since the Registry installer is implemented just as an extra resolution step in front of go-getter, this optimization applies to registry modules too. This does not apply to local relative references, which will continue to just resolve into the already-prepared directory of their parent module. The cache of previously installed paths lives only for the duration of one call to InstallModules, so we will never re-use directories that were created by previous runs of "terraform init" and there is no risk that older versions will pollute the cache when attempting an upgrade from a source address that doesn't explicitly specify a version. No additional tests are added here because the existing module installer tests (when TF_ACC=1) already cover the case of installing multiple modules from the same source.	7 years ago
James Bardin	8c3c0418d4	add GetSchema to provisioner interface	7 years ago
James Bardin	12521366e1	Add ImportedResource type ... The ImportResourceStateResponse command can return mutliple types. ImportedResource maps the state to the appropriate type name, and also allows providers to return Private data for each resource.	7 years ago
James Bardin	dc78269a3c	use cty.Value for connection info ... For now this will be an encoded map, but it allows for a more complex schema in the future.	7 years ago
James Bardin	960553e18b	rename UpgradedState	7 years ago
James Bardin	6a1b57a1e4	split PriorStateRaw into JSON and Flatmap ... This way the provider doesn't need to be concerned with determining the state encoding.	7 years ago
James Bardin	91b5bbbde0	add versions to the provider schemas ... We need to know the schema version for all resources. This is stored in order to allow providers to upgrade the state from a known previous version.	7 years ago
James Bardin	1473d09c50	rename provider and provisioner packages ... Using plural names to avoid collisions in existing code.	7 years ago
Martin Atkins	0742e756e5	tfdiags: Sort order for diagnostics ... Because we gather together diagnostics from many different parts of the codebase, the list often ends up being in a non-ideal order. Here we define a partial ordering for diagnostics that should hopefully make them easier to scan when many are present, by grouping together diagnostics that are of the same severity and belong to the same file. We use sort.Stable here because we have a partial order and so we need to make sure that diagnostics that do not have a relative ordering will remain in their original order. This sorting is applied just in time before rendering the diagnostics in command.Meta.showDiagnostics.	7 years ago
James Bardin	27d90c7550	Add provider plugin interface and types ... This adds the new interface for the resource providers. The methods here closely follow the grpc service methods, using the types expected in terraform core	7 years ago
James Bardin	fdfbc5c1a7	Add provisioner interface and request types ... This adds the interface for the new resource provisioners. The methods closely follow the grpc service methods, using the type expected within terraform core.	7 years ago
Martin Atkins	a2eb462f5d	plans/planfile: Reading and writing the new plan format ... The new format is radically different in than the old in physical structure, but still has the same logical parts: the plan itself, a snapshot of the input configuration, and a snapshot of the state as it existed when the plan was created. Rather than creating plan-specific serializations of state and config, the new format instead leans on the existing file formats implemented elsewhere, wrapping the result up in a zip archive with some internal file naming conventions. The plan portion of the file is serialized with protobuf, consistent with our general strategy of replacing all use of encoding/gob with protobuf moving forward.	7 years ago
Martin Atkins	7357e7f734	plans: New package for in-memory plan models ... The types in this package are intended to replace plan- and diff-related types from the "terraform" package, although those older types must remain for now so that they can be used to implement shims for older codepaths. type "Changes" is approximately equivalent to terraform.Diff, but renamed since it now describes whole objects before and after rather than an attribute-level diff as before. The term "diff" is now reserved for the visual rendition of the changes we'll display to the user, although rendering of this new Changes model is not yet implemented.	7 years ago
Martin Atkins	eb7aaf2414	configload: Configuration snapshots ... Here we introduce a new idea of a "configuration snapshot", which is an in-memory copy of the source code of each of the files that make up the configuration. The primary intended purpose for this is as an intermediate step before writing the configuration files into a plan file, and then reading them out when that plan file is later applied. During earlier configs package development we expected to use an afero vfs implementation to read directly from the zip file, but that doesn't work in practice because we need to preserve module paths from the source file system that might include parent directory traversals (../) while retaining the original path for use in error messages. The result, for now, is a bit of an abstraction inversion: we implement a specialized afero vfs implementation that makes the sparse filesystem representation from a snapshot appear like a normal filesystem just well enough that the config loader and parser can work with it. In future we may wish to rework the internals here so that the main abstraction is at a similar level to the snapshot and then that API is mapped to the native filesystem in the normal case, removing afero. For now though, this approach avoids the need for a significant redesign of the parser/loader internals, at the expense of some trickiness in the case where we're reading from a snapshot. This commit does not yet include the reading and writing of snapshots into plan files. That will follow in a subsequent commit.	7 years ago
Martin Atkins	93cda6dbd2	govendor fetch github.com/zclconf/go-cty/cty/... ... This allows automatic conversions between different object types as long as the target type is a subset of the given type.	7 years ago
Martin Atkins	b1247bf7af	govendor fetch github.com/zclconf/go-cty/cty/... ... The existing cty packages were already at the latest version, but we were not yet vendoring the msgpack package. This also imports some dependencies from: github.com/vmihailenco/msgpack	7 years ago
Martin Atkins	da17afaa85	govendor fetch github.com/golang/protobuf/proto/...	7 years ago
Martin Atkins	a33f941778	states: New SyncState type ... This is a wrapper around State that is able to perform higher-level manipulations (at the granularity of the entire state) in a concurrency-safe manner, using the lower-level APIs exposed by State and all of the types it contains. The granularity of a SyncState operation roughly matches the granularity off a state-related EvalNode in the "terraform" package, performing a sequence of more primitive operations while guaranteeing atomicity of the entire change. As a compromise for convenience of usage, it's still possible to access the individual state data objects via this API, but they are always copied before returning to ensure that two distinct callers cannot have data races. Callers should access the most granular object possible for their operation.	7 years ago
Martin Atkins	53cafc542b	statemgr: New package for state managers ... This idea of a "state manager" was previously modelled via the confusingly-named state.State interface, which we've been calling a "state manager" only in some local variable names in situations where there were also *terraform.State variables. As part of reworking our state models to make room for the new type system, we also need to change what was previously the state.StateReader interface. Since we've found the previous organization confusing anyway, here we just copy all of those interfaces over into statemgr where we can make the relationship to states.State hopefully a little clearer. This is not yet a complete move of the functionality from "state", since we're not yet ready to break existing callers. In a future commit we'll turn the interfaces in the old "state" package into aliases of the interfaces in this package, and update all the implementers of what will by then be statemgr.Reader to use *states.State instead of *terraform.State. This also includes an adaptation of what was previously state.LocalState into statemgr.FileSystem, using the new state serialization functionality from package statefile instead of the old terraform.ReadState and terraform.WriteState.	7 years ago
Martin Atkins	5c1c6e9d9c	statefile: New package for loading and saving state files ... Whereas the parent directory "states" contains the models that represent state in memory, this package's responsibility is in serializing a subset of that data to a JSON-based file format and then reloading that data back into memory later. For reading, this package supports state file formats going back to version 1, using lightly-adapted versions of the migration code previously used in the "terraform" package. State data is upgraded to the latest version step by step and then transformed into the in-memory state representation, which is distinct from any of the file format structs in this package to enable these to evolve separately. For writing, only the latest version (4) is supported, which is a new format that is a slightly-flattened version of the new in-memory state models introduced in the prior commit. This format retains the outputs from only the root module and it flattens out the module and instance parts of the hierarchy by including the identifiers for these inside the child object. The loader then reconstructs the multi-layer structure we use for more convenient access in memory. For now, the only testing in this package is of round-tripping different versions of state through a read and a write, ensuring the output is as desired. This exercises all of the reading, upgrading, and writing functions but should be augmented in later commits to improve coverage and introduce more focused tests for specific parts of the functionality.	7 years ago
Martin Atkins	b975ada8db	states: New package with modern models for Terraform state ... Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.	7 years ago
Martin Atkins	ba894ee05c	addrs: More string parsing helpers for addresses ... Our main "parse" methods in this package work with hcl.Traversals, but we're gradually adding helpers to parse these directly froms strings since the visual noise of doing the traversal parse first is inconvenient in situations where addresses are coming from non-config locations where no source information is available anyway.	7 years ago
Martin Atkins	00199cd2ed	addrs: "Less" comparison method for resource and module instances ... This can be used to sort lists of resource instance and module instance addresses, such as in a rendered plan.	7 years ago
Kristin Laemmert	c23a971ed1	minor fixes	7 years ago
Kristin Laemmert	0dbecc54c0	functions: ValuesFunc - cleanup return type	7 years ago
Kristin Laemmert	a213c4a648	functions: add tests and support for unknown values	7 years ago
Kristin Laemmert	d802d5c624	functions: pr feedback fixes	7 years ago
Kristin Laemmert	ff4b3d763b	functions: fix lookup()'s handling of numberical defaults	7 years ago
Kristin Laemmert	4f5c03339a	functions: ZipmapFunc	7 years ago
Martin Atkins	2ef56b3e05	Fix up some missed "go fmt" ... Because of the size of some of these files, automatic format-on-save was implicitly disabled in my editor, which I didn't notice before committing.	7 years ago
James Bardin	82d2c5b484	core: use absolute address in CloserProvider ... Otherwsie relative addresses may collide and close the incorrect provider.	7 years ago
James Bardin	a2c76a28f5	core: run ProvisionerTransformer during plan ... This makes sure the graph is complete for validation	7 years ago
Martin Atkins	0b710792a8	website: "functions" layout "Terraform Language" is back ... In the heirarchy, both "Terraform Language" and "Functions" are "up" from the individual function reference pages, so we'll class them as such to use the back-facing arrow instead of the forward-facing arrow.	7 years ago
Martin Atkins	1360948a41	website: document the functions "keys", "lookup", and "values" ... I missed these on the first pass because in the legacy function table they are, for some reason, added in a different place than the others.	7 years ago
Martin Atkins	3c10a3b213	helper/schema: Tolerate incorrectly-specified collection elems ... We historically tolerated this, so we need to tolerate it here too in order to work correctly with existing provider code.	7 years ago
Martin Atkins	91def7b6f6	core: Better failure output for TestContext2Apply_moduleDestroyOrder	7 years ago
Martin Atkins	bab9f7ce09	core: update Stringer implementations for GraphType and walkOperation ... These are no longer correct due to the input walk being removed in an earlier commit.	7 years ago
Martin Atkins	38f37af7e9	core: remove TestContext2Apply_outputInvalid ... This was testing that returning a number as an output would be reported as an error, but that is intentionally allowed now.	7 years ago
Martin Atkins	b4d7882e2f	core: Fix TestContextImport_providerVarConfig ... I updated the "Variables" map incorrectly in earlier commit 10fe50bbdb while making bulk updates to get the tests compiling again with the changed underlying APIs. The original value here was "bar", incorrectly changed to "foo" in that commit. Here we return it back to "bar".	7 years ago
James Bardin	6eb897e293	core: attach provisioner schemas in subgraphs ... DynamicExpand also needs to add the provisioner schemas to make sure all references are found in the subgraph.	7 years ago
Martin Atkins	059de66fcc	core: Don't save provider input for non-root module ... We only support provider input for the root module. This is already checked in ProviderInput, but was not checked in SetProviderInput. We can't actually do anything particularly clever with an invalid call here, but we will at least generate a WARN log to help with debugging. Also need to update TestBuiltinEvalContextProviderInput to expect this new behavior of ignoring input for non-root modules.	7 years ago
Martin Atkins	b99b31ebea	core: Correct schema for TestContext2Apply_issue5254 ... This test is now failing due to the fact that WritePlan is currently disabled pending a rewrite. This will be addressed in a subsequent commit.	7 years ago