21937 Commits (ca4875128e49463be1d66a445e80ff5a67863876)

 

All Branches   

Search

Author	SHA1	Message	Date
Martin Atkins	ca4875128e	configupgrade: Basic expression formatting ... This covers all of the expression node types in HIL's AST, and also includes initial support for some of our top-level blocks so that we can easily test that. The initial implementations of the "variable" and "output" blocks are pretty redundant and messy, so we can hopefully improve on these in a later pass.	8 years ago
Martin Atkins	1c5d05b1a0	configupgrade: Beginnings of Upgrade function ... This function is the main functionality of this package. So far it just deals with detecting and renaming JSON files that are mislabeled as native syntax files. Other functionality will follow in later commits.	8 years ago
Martin Atkins	79b361160a	configupgrade: Load source code for a module and detect already upgraded ... This package will do all of its work in-memory so that it can avoid making partial updates and then failing, so we need to be able to load the sources files from a particular directory into memory. The upgrade process isn't idempotent, so we also attempt to detect heuristically whether an upgrade has already been performed (can parse with the new parser and has a version constraint that prevents versions earlier than 0.12) so that the CLI tool that will eventually wrap this will be able to produce a warning and prompt for confirmation in that case.	8 years ago
Martin Atkins	34b0a20ee5	configupgrade: new package for upgrading configs for 0.12 ... Although the new HCL implementation and configuration loader is broadly compatible with the prior implementation, it has a number of new idiomatic forms and it also cannot parse some more extreme non-idiomatic usages that were possible under the old parser. To help users migrate to the new implementation, this package will rewrite configuration to comply with the new idiom and fix as many cases as possible where the legacy parser was too liberal or exposed implementation details.	8 years ago
Martin Atkins	2b124a1d00	configload: Don't download the same module source multiple times ... It is common for the same module source package to be referenced multiple times in the same configuration, either because there are literally multiple instances of the same module source or because a single package (or repository) contains multiple modules in sub-directories and many of them are referenced. To optimize this, here we introduce a simple caching behavior where the module installer will detect if it's asked to install multiple times from the same source and produce the second and subsequent directories by copying the first, rather than by downloading again over the network. This optimization is applied once all of the go-getter detection has completed and sub-directory portions have been trimmed, so it is also able to normalize differently-specified source addresses that all ultimately detect to the same resolved address. When installing, we always extract the entire specified package (or repository) and then reference the specified sub-directory, so we can safely re-use existing directories when the base package is the same, even if the sub-directory is different. However, as a result we do not yet address the fact that the same package will be stored multiple times _on disk_, which may still be problematic when referencing large repositories multiple times in disk-storage-constrained environments. We could address this in a subsequent change by investigating the use of symlinks where possible. Since the Registry installer is implemented just as an extra resolution step in front of go-getter, this optimization applies to registry modules too. This does not apply to local relative references, which will continue to just resolve into the already-prepared directory of their parent module. The cache of previously installed paths lives only for the duration of one call to InstallModules, so we will never re-use directories that were created by previous runs of "terraform init" and there is no risk that older versions will pollute the cache when attempting an upgrade from a source address that doesn't explicitly specify a version. No additional tests are added here because the existing module installer tests (when TF_ACC=1) already cover the case of installing multiple modules from the same source.	8 years ago
James Bardin	119665ec6a	add GetSchema to provisioner interface	8 years ago
James Bardin	661848d30d	Add ImportedResource type ... The ImportResourceStateResponse command can return mutliple types. ImportedResource maps the state to the appropriate type name, and also allows providers to return Private data for each resource.	8 years ago
James Bardin	b385c86269	use cty.Value for connection info ... For now this will be an encoded map, but it allows for a more complex schema in the future.	8 years ago
James Bardin	483490f8c2	rename UpgradedState	8 years ago
James Bardin	d694076fb2	split PriorStateRaw into JSON and Flatmap ... This way the provider doesn't need to be concerned with determining the state encoding.	8 years ago
James Bardin	39650828c0	add versions to the provider schemas ... We need to know the schema version for all resources. This is stored in order to allow providers to upgrade the state from a known previous version.	8 years ago
James Bardin	b942eb7f7f	rename provider and provisioner packages ... Using plural names to avoid collisions in existing code.	8 years ago
Martin Atkins	3d304efd91	tfdiags: Sort order for diagnostics ... Because we gather together diagnostics from many different parts of the codebase, the list often ends up being in a non-ideal order. Here we define a partial ordering for diagnostics that should hopefully make them easier to scan when many are present, by grouping together diagnostics that are of the same severity and belong to the same file. We use sort.Stable here because we have a partial order and so we need to make sure that diagnostics that do not have a relative ordering will remain in their original order. This sorting is applied just in time before rendering the diagnostics in command.Meta.showDiagnostics.	8 years ago
James Bardin	2b5ae042b8	Add provider plugin interface and types ... This adds the new interface for the resource providers. The methods here closely follow the grpc service methods, using the types expected in terraform core	8 years ago
James Bardin	56a9af829f	Add provisioner interface and request types ... This adds the interface for the new resource provisioners. The methods closely follow the grpc service methods, using the type expected within terraform core.	8 years ago
Martin Atkins	4466754bb9	plans/planfile: Reading and writing the new plan format ... The new format is radically different in than the old in physical structure, but still has the same logical parts: the plan itself, a snapshot of the input configuration, and a snapshot of the state as it existed when the plan was created. Rather than creating plan-specific serializations of state and config, the new format instead leans on the existing file formats implemented elsewhere, wrapping the result up in a zip archive with some internal file naming conventions. The plan portion of the file is serialized with protobuf, consistent with our general strategy of replacing all use of encoding/gob with protobuf moving forward.	8 years ago
Martin Atkins	ab3dccff1f	plans: New package for in-memory plan models ... The types in this package are intended to replace plan- and diff-related types from the "terraform" package, although those older types must remain for now so that they can be used to implement shims for older codepaths. type "Changes" is approximately equivalent to terraform.Diff, but renamed since it now describes whole objects before and after rather than an attribute-level diff as before. The term "diff" is now reserved for the visual rendition of the changes we'll display to the user, although rendering of this new Changes model is not yet implemented.	8 years ago
Martin Atkins	628a846b24	configload: Configuration snapshots ... Here we introduce a new idea of a "configuration snapshot", which is an in-memory copy of the source code of each of the files that make up the configuration. The primary intended purpose for this is as an intermediate step before writing the configuration files into a plan file, and then reading them out when that plan file is later applied. During earlier configs package development we expected to use an afero vfs implementation to read directly from the zip file, but that doesn't work in practice because we need to preserve module paths from the source file system that might include parent directory traversals (../) while retaining the original path for use in error messages. The result, for now, is a bit of an abstraction inversion: we implement a specialized afero vfs implementation that makes the sparse filesystem representation from a snapshot appear like a normal filesystem just well enough that the config loader and parser can work with it. In future we may wish to rework the internals here so that the main abstraction is at a similar level to the snapshot and then that API is mapped to the native filesystem in the normal case, removing afero. For now though, this approach avoids the need for a significant redesign of the parser/loader internals, at the expense of some trickiness in the case where we're reading from a snapshot. This commit does not yet include the reading and writing of snapshots into plan files. That will follow in a subsequent commit.	8 years ago
Martin Atkins	f98ca58f75	govendor fetch github.com/zclconf/go-cty/cty/... ... This allows automatic conversions between different object types as long as the target type is a subset of the given type.	8 years ago
Martin Atkins	2a5d6348e8	govendor fetch github.com/zclconf/go-cty/cty/... ... The existing cty packages were already at the latest version, but we were not yet vendoring the msgpack package. This also imports some dependencies from: github.com/vmihailenco/msgpack	8 years ago
Martin Atkins	40816dcda2	govendor fetch github.com/golang/protobuf/proto/...	8 years ago
Martin Atkins	8f45d020a6	states: New SyncState type ... This is a wrapper around State that is able to perform higher-level manipulations (at the granularity of the entire state) in a concurrency-safe manner, using the lower-level APIs exposed by State and all of the types it contains. The granularity of a SyncState operation roughly matches the granularity off a state-related EvalNode in the "terraform" package, performing a sequence of more primitive operations while guaranteeing atomicity of the entire change. As a compromise for convenience of usage, it's still possible to access the individual state data objects via this API, but they are always copied before returning to ensure that two distinct callers cannot have data races. Callers should access the most granular object possible for their operation.	8 years ago
Martin Atkins	1235b69f48	statemgr: New package for state managers ... This idea of a "state manager" was previously modelled via the confusingly-named state.State interface, which we've been calling a "state manager" only in some local variable names in situations where there were also *terraform.State variables. As part of reworking our state models to make room for the new type system, we also need to change what was previously the state.StateReader interface. Since we've found the previous organization confusing anyway, here we just copy all of those interfaces over into statemgr where we can make the relationship to states.State hopefully a little clearer. This is not yet a complete move of the functionality from "state", since we're not yet ready to break existing callers. In a future commit we'll turn the interfaces in the old "state" package into aliases of the interfaces in this package, and update all the implementers of what will by then be statemgr.Reader to use *states.State instead of *terraform.State. This also includes an adaptation of what was previously state.LocalState into statemgr.FileSystem, using the new state serialization functionality from package statefile instead of the old terraform.ReadState and terraform.WriteState.	8 years ago
Martin Atkins	cb8225d2ee	statefile: New package for loading and saving state files ... Whereas the parent directory "states" contains the models that represent state in memory, this package's responsibility is in serializing a subset of that data to a JSON-based file format and then reloading that data back into memory later. For reading, this package supports state file formats going back to version 1, using lightly-adapted versions of the migration code previously used in the "terraform" package. State data is upgraded to the latest version step by step and then transformed into the in-memory state representation, which is distinct from any of the file format structs in this package to enable these to evolve separately. For writing, only the latest version (4) is supported, which is a new format that is a slightly-flattened version of the new in-memory state models introduced in the prior commit. This format retains the outputs from only the root module and it flattens out the module and instance parts of the hierarchy by including the identifiers for these inside the child object. The loader then reconstructs the multi-layer structure we use for more convenient access in memory. For now, the only testing in this package is of round-tripping different versions of state through a read and a write, ensuring the output is as desired. This exercises all of the reading, upgrading, and writing functions but should be augmented in later commits to improve coverage and introduce more focused tests for specific parts of the functionality.	8 years ago
Martin Atkins	60ac0956f7	states: New package with modern models for Terraform state ... Our previous state models in the "terraform" package had a few limitations that are addressed here: - Instance attributes were stored as map[string]string with dot-separated keys representing traversals through a data structure. Now that we have a full type system, it's preferable to store it as a real data structure. - The existing state structures skipped over the "resource" concept and went straight to resource instance, requiring heuristics to decide whether a particular resource should appear as a single object or as a list of objects when used in configuration expressions. - Related to the previous point, the state models also used incorrect terminology where "ResourceState" was really a resource instance state and "InstanceState" was really the state of a particular remote object associated with an instance. These new models use the correct names for each of these, introducing the idea of a "ResourceInstanceObject" as the local record of a remote object associated with an instance. This is a first pass at fleshing out a new model for state. Undoubtedly there will be further iterations of this as we work on integrating these new models into the "terraform" package. These new model types no longer serve double-duty as a description of the JSON state file format, since they are for in-memory use only. A subsequent commit will introduce a separate package that deals with persisting state to files and reloading those files later.	8 years ago
Martin Atkins	92dca16302	addrs: More string parsing helpers for addresses ... Our main "parse" methods in this package work with hcl.Traversals, but we're gradually adding helpers to parse these directly froms strings since the visual noise of doing the traversal parse first is inconvenient in situations where addresses are coming from non-config locations where no source information is available anyway.	8 years ago
Martin Atkins	05d295eb61	addrs: "Less" comparison method for resource and module instances ... This can be used to sort lists of resource instance and module instance addresses, such as in a rendered plan.	8 years ago
Kristin Laemmert	013d51463d	minor fixes	8 years ago
Kristin Laemmert	2a28480c8c	functions: ValuesFunc - cleanup return type	8 years ago
Kristin Laemmert	99c6aff05b	functions: add tests and support for unknown values	8 years ago
Kristin Laemmert	825cd8822d	functions: pr feedback fixes	8 years ago
Kristin Laemmert	2a62b1a51a	functions: fix lookup()'s handling of numberical defaults	8 years ago
Kristin Laemmert	c4e42da7d5	functions: ZipmapFunc	8 years ago
Martin Atkins	7f205645a1	Fix up some missed "go fmt" ... Because of the size of some of these files, automatic format-on-save was implicitly disabled in my editor, which I didn't notice before committing.	8 years ago
James Bardin	63ca238c72	core: use absolute address in CloserProvider ... Otherwsie relative addresses may collide and close the incorrect provider.	8 years ago
James Bardin	bd12441030	core: run ProvisionerTransformer during plan ... This makes sure the graph is complete for validation	8 years ago
Martin Atkins	d65dfcddcc	website: "functions" layout "Terraform Language" is back ... In the heirarchy, both "Terraform Language" and "Functions" are "up" from the individual function reference pages, so we'll class them as such to use the back-facing arrow instead of the forward-facing arrow.	8 years ago
Martin Atkins	e405b58c79	website: document the functions "keys", "lookup", and "values" ... I missed these on the first pass because in the legacy function table they are, for some reason, added in a different place than the others.	8 years ago
Martin Atkins	fa9d7f4d20	helper/schema: Tolerate incorrectly-specified collection elems ... We historically tolerated this, so we need to tolerate it here too in order to work correctly with existing provider code.	8 years ago
Martin Atkins	a607951007	core: Better failure output for TestContext2Apply_moduleDestroyOrder	8 years ago
Martin Atkins	c9bfbf58c1	core: update Stringer implementations for GraphType and walkOperation ... These are no longer correct due to the input walk being removed in an earlier commit.	8 years ago
Martin Atkins	368a76ea47	core: remove TestContext2Apply_outputInvalid ... This was testing that returning a number as an output would be reported as an error, but that is intentionally allowed now.	8 years ago
Martin Atkins	44c88dc9d9	core: Fix TestContextImport_providerVarConfig ... I updated the "Variables" map incorrectly in earlier commit 10fe50bbdb while making bulk updates to get the tests compiling again with the changed underlying APIs. The original value here was "bar", incorrectly changed to "foo" in that commit. Here we return it back to "bar".	8 years ago
James Bardin	5b66703cb2	core: attach provisioner schemas in subgraphs ... DynamicExpand also needs to add the provisioner schemas to make sure all references are found in the subgraph.	8 years ago
Martin Atkins	edb56a7e16	core: Don't save provider input for non-root module ... We only support provider input for the root module. This is already checked in ProviderInput, but was not checked in SetProviderInput. We can't actually do anything particularly clever with an invalid call here, but we will at least generate a WARN log to help with debugging. Also need to update TestBuiltinEvalContextProviderInput to expect this new behavior of ignoring input for non-root modules.	8 years ago
Martin Atkins	d216b2aacd	core: Correct schema for TestContext2Apply_issue5254 ... This test is now failing due to the fact that WritePlan is currently disabled pending a rewrite. This will be addressed in a subsequent commit.	8 years ago
Martin Atkins	3895b23ffa	core: Fix TestContext2Refresh_dataState ... Now that we fetch schemas during NewContext, we need to configure the mock GetSchema method before constructing the context.	8 years ago
James Bardin	acd5e2240e	core: TestContext2Apply_dataDependsOn	8 years ago
James Bardin	56be68f6bc	core: TestContext2Validate_interpolateMap	8 years ago
Martin Atkins	824efdde62	core: Don't create indirect provider dependencies for references ... The prior commit changed the schema-access model so that all schemas are fetched up front during context creation and are then readily available for use throughout graph building and evaluation. As a result, we no longer need to create dependency edges to a provider when one of its resources is referenced by another node, and so the ProviderTransformer needs only to worry about direct ownership dependencies. This also avoids the need for us to run AttachSchemaTransformer twice, since ProviderTransformer no longer needs schema and we can therefore defer attaching until just before ReferenceTransformer, when all of the referencable and referencing nodes are already present in the graph.	8 years ago