414 Commits (dcf2d3c1efa2165cfe953794bcfcb8d074d2ed9b)

Author	SHA1	Message	Date
James Bardin	8ed9a270e5	Merge pull request #29559 from hashicorp/jbardin/optional-attrs ... Prevent `ObjectWithOptionalAttrs` from escaping	5 years ago
James Bardin	7f26531d4f	variable types should always be populated	5 years ago
James Bardin	53a73a8ab6	configs: add ConstraintType to config.Variable ... In order to handle optional attributes, the Variable type needs to keep track of the type constraint for decoding and conversion, as well as the concrete type for creating values and type comparison. Since the Type field is referenced throughout the codebase, and for future refactoring if the handling of optional attributes changes significantly, the constraint is now loaded into an entirely new field called ConstraintType. This prevents types containing ObjectWithOptionalAttrs from escaping the decode/conversion codepaths into the rest of the codebase.	5 years ago
James Bardin	43123d284e	optional attrs should not be in ProposedNew	5 years ago
James Bardin	af4f4540a9	configschema: do not expose optional attributes ... Objects with optional attributes are only used for the decoding of HCL, and those types should never be exposed elsewhere within terraform. Separate the external ImpliedType method from the cty.Type generated internally for the decoder spec. This unfortunately causes our ImpliedType method to return a different type than the hcldec.ImpliedType function, but the former is only used within terraform for concrete values, while the latter is used to decode HCL. Renaming the ImpliedType methods could be done to further differentiate them, but that does cause fairly large diff in the codebase that does not seem worth the effort at this time.	5 years ago
Martin Atkins	65e0c448a0	workdir: Start of a new package for working directory state management ... Thus far our various interactions with the bits of state we keep associated with a working directory have all been implemented directly inside the "command" package -- often in the huge command.Meta type -- and not managed collectively via a single component. There's too many little codepaths reading and writing from the working directory and data directory to refactor it all in one step, but this is an attempt at a first step towards a future where everything that reads and writes from the current working directory would do so via an object that encapsulates the implementation details and offers a high-level API to read and write all of these session-persistent settings. The design here continues our gradual path towards using a dependency injection style where "package main" is solely responsible for directly interacting with the OS command line, the OS environment, the OS working directory, the stdio streams, and the CLI configuration, and then communicating the resulting information to the rest of Terraform by wiring together objects. It seems likely that eventually we'll have enough wiring code in package main to justify a more explicit organization of that code, but for this commit the new "workdir.Dir" object is just wired directly in place of its predecessors, without any significant change of code organization at that top layer. This first commit focuses on the main files and directories we use to find provider plugins, because a subsequent commit will lightly reorganize the separation of concerns for plugin launching with a similar goal of collecting all of the relevant logic together into one spot.	5 years ago
Martin Atkins	343279110a	core: Graph walk loads plugin schemas opportunistically ... Previously our graph walker expected to recieve a data structure containing schemas for all of the provider and provisioner plugins used in the configuration and state. That made sense back when terraform.NewContext was responsible for loading all of the schemas before taking any other action, but it no longer has that responsiblity. Instead, we'll now make sure that the "contextPlugins" object reaches all of the locations where we need schema -- many of which already had access to that object anyway -- and then load the needed schemas just in time. The contextPlugins object memoizes schema lookups, so we can safely call it many times with the same provider address or provisioner type name and know that it'll still only load each distinct plugin once per Context object. As of this commit, the Context.Schemas method is now a public interface only and not used by logic in the "terraform" package at all. However, that does leave us in a rather tenuous situation of relying on the fact that all practical users of terraform.Context end up calling "Schemas" at some point in order to verify that we have all of the expected versions of plugins. That's a non-obvious implicit dependency, and so in subsequent commits we'll gradually move all responsibility for verifying plugin versions into the caller of terraform.NewContext, which'll heal a long-standing architectural wart whereby the caller is responsible for installing and locating the plugin executables but not for verifying that what's installed is conforming to the current configuration and dependency lock file.	5 years ago
Martin Atkins	38ec730b0e	core: Opportunistic schema loading during graph construction ... Previously the graph builders all expected to be given a full manifest of all of the plugin component schemas that they could need during their analysis work. That made sense when terraform.NewContext would always proactively load all of the schemas before doing any other work, but we now have a load-as-needed strategy for schemas. We'll now have the graph builders use the contextPlugins object they each already hold to retrieve individual schemas when needed. This avoids the need to prepare a redundant data structure to pass alongside the contextPlugins object, and leans on the memoization behavior inside contextPlugins to preserve the old behavior of loading each provider's schema only once.	5 years ago
Martin Atkins	a59c2fe1b9	core: EvalContextBuiltin no longer has a "Schemas" ... By tolerating ProviderSchema and ProvisionerSchema potentially returning errors, we can slightly simplify EvalContextBuiltin by having it retrieve individual schemas when needed directly from the "Plugins" object. EvalContextBuiltin already needs to be holding a contextPlugins instance for other reasons anyway, so this allows us to get the same result with fewer moving parts.	5 years ago
Martin Atkins	2bf1de1f5d	core: Context.Schemas in terms of contextPlugins methods ... The responsibility for actually instantiating a single plugin and reading out its schema now belongs to the contextPlugins type, which memoizes the results by each plugin's unique identifier so that we can avoid retrieving the same schemas multiple times when working with the same context. This doesn't change the API of Context.Schemas but it does restore the spirit of an earlier version of terraform.Context which did all of the schema loading proactively inside terraform.NewContext. In an earlier commit we reduced the scope of terraform.NewContext, making schema loading a separate step, but in the process of doing that removed the effective memoization of the schema results that terraform.NewContext was providing. The memoization here will play out in a different way than before, because we'll be treating each plugin call as separate rather than proactively loading them all up front, but this is effectively the same because all of our operation methods on Context call Context.Schemas early in their work and thus end up forcing all of the necessary schemas to load up front nonetheless.	5 years ago
Martin Atkins	80b3fcf93e	core: Replace contextComponentFactory with contextPlugins ... In the v0.12 timeframe we made contextComponentFactory an interface with the expectation that we'd write mocks of it for tests, but in practice we ended up just always using the same "basicComponentFactory" implementation throughout. In the interests of simplification then, here we replace that interface and its sole implementation with a new concrete struct type contextPlugins. Along with the general benefit that this removes an unneeded indirection, this also means that we can add additional methods to the struct type without the usual restriction that interface types prefer to be small. In particular, in a future commit I'm planning to add methods for loading provider and provisioner schemas, working with the currently-unused new fields this commit has included in contextPlugins, as compared to its predecessor basicComponentFactory.	5 years ago
Martin Atkins	dcfa077adf	core: contextComponentFactory doesn't need to enumerate components ... In earlier Terraform versions we used the set of all available plugins of each type to make graph-building decisions, but in modern Terraform we make those decisions based entirely on the configuration. Consequently, we no longer need the methods which can enumerate all of the known plugin components of a given type. Instead, we just try to instantiate each of the plugins that the configuration refers to and then handle the error when that fails, which typically means that the user needs to run "terraform init" to install some new plugins.	5 years ago
Martin Atkins	d51921f085	core: Provider transformers don't use the set of all available providers ... In earlier incarnations of these transformers we used the set of all available providers for tasks such as generating implied provider configuration nodes. However, in modern Terraform we can extract all of the information we need from the configuration itself, and so these transformers weren't actually using this set of provider addresses. These also ended up getting left behind as sets of string rather than sets of addrs.Provider in our earlier refactoring work, which didn't really matter because the result wasn't used anywhere anyway. Rather than updating these to use addrs.Provider instead, I've just removed the unused arguments entirely in the hope of making it easier to see what inputs these transformers use to make their decisions.	5 years ago
Martin Atkins	feb219a9c4	core: Un-export the LoadSchemas function ... The public interface for loading schemas is Context.Schemas, which can take into account the context's records of which plugin versions and checksums we're expecting. loadSchemas is an implementation detail of that, representing the part we run only after we've verified all of the plugins.	5 years ago
James Bardin	782132da33	remove incorrect computed check ... The config is already validated, and does not need to be checked in AssertPlanValid. Add some more coverage for plan validation.	5 years ago
Alisdair McDiarmid	cd29c3e5fd	Revert "json-output: Release format version 1.0"	5 years ago
Laura Pacilio	a819d7db3a	Merge pull request #29502 from hashicorp/alisdair/json-format-version ... json-output: Release format version 1.0	5 years ago
Alisdair McDiarmid	7f3a29b46e	Merge pull request #29523 from hashicorp/alisdair/moved-ui ... command: Render "moved" annotations in plan UI	5 years ago
James Bardin	1ad0421e15	Merge pull request #29522 from hashicorp/jbardin/json-blocktoattr ... allow json output to marshal ConfigModeAttr blocks	5 years ago
Alisdair McDiarmid	29999c1d6f	command: Render "moved" annotations in plan UI ... For resources which are planned to move, render the previous run address as additional information in the plan UI. For the case of a move-only resource (which otherwise is unchanged), we also render that as a planned change, but without any corresponding action symbol. If all changes in the plan are moves without changes, the plan is no longer considered "empty". In this case, we skip rendering the action symbols in the UI.	5 years ago
James Bardin	ac2a870ea0	allow json output to marshal ConfigModeAttr blocks ... In order to marshal config blocks using ConfigModeAttr, we need to insert the fixup body to map hcl blocks to the attribute in the schema.	5 years ago
Alisdair McDiarmid	06956773dc	configs: Disable experiment warnings at link time ... This package level variable can be overridden at link time to allow temporarily disabling the UI warning when experimental features are enabled. This makes it easier to understand how UI will render when the feature is no longer experimental. This change is only for those developing Terraform.	5 years ago
Alisdair McDiarmid	b60f201eed	json-output: Release format version 1.0	5 years ago
James Bardin	863963e7a6	de-linting	5 years ago
James Bardin	7c02ef6220	Merge pull request #29482 from hashicorp/jbardin/computed-obj-attrs ... Computed values within nested object types in `AssertPlanValid`	5 years ago
James Bardin	f195ce7fd4	remove temp test	5 years ago
Martin Atkins	7803f69d42	core: Enable TestContext2Plan_movedResourceBasic ... This is the first test exercising the basic functionality of config-driven move. We previously had it skipped because Terraform's previous design of treating all three of the state artifacts as mutable attributes of terraform.Context meant that it was too late during planning to deal with the move operations, and thus this test was failing. Thanks to the previous commit, which changes the terraform.Context API such that we can defer creating the three state artifacts until we're already doing planning, this test now works and shows Terraform correctly handling a resource that was formerly called "a" and is now called "b", with a "moved" block recording that renaming.	5 years ago
Martin Atkins	89b05050ec	core: Functional-style API for terraform.Context ... Previously terraform.Context was built in an unfortunate way where all of the data was provided up front in terraform.NewContext and then mutated directly by subsequent operations. That made the data flow hard to follow, commonly leading to bugs, and also meant that we were forced to take various actions too early in terraform.NewContext, rather than waiting until a more appropriate time during an operation. This (enormous) commit changes terraform.Context so that its fields are broadly just unchanging data about the execution context (current workspace name, available plugins, etc) whereas the main data Terraform works with arrives via individual method arguments and is returned in return values. Specifically, this means that terraform.Context no longer "has-a" config, state, and "planned changes", instead holding on to those only temporarily during an operation. The caller is responsible for propagating the outcome of one step into the next step so that the data flow between operations is actually visible. However, since that's a change to the main entry points in the "terraform" package, this commit also touches every file in the codebase which interacted with those APIs. Most of the noise here is in updating tests to take the same actions using the new API style, but this also affects the main-code callers in the backends and in the command package. My goal here was to refactor without changing observable behavior, but in practice there are a couple externally-visible behavior variations here that seemed okay in service of the broader goal: - The "terraform graph" command is no longer hooked directly into the core graph builders, because that's no longer part of the public API. However, I did include a couple new Context functions whose contract is to produce a UI-oriented graph, and _for now_ those continue to return the physical graph we use for those operations. There's no exported API for generating the "validate" and "eval" graphs, because neither is particularly interesting in its own right, and so "terraform graph" no longer supports those graph types. - terraform.NewContext no longer has the responsibility for collecting all of the provider schemas up front. Instead, we wait until we need them. However, that means that some of our error messages now have a slightly different shape due to unwinding through a differently-shaped call stack. As of this commit we also end up reloading the schemas multiple times in some cases, which is functionally acceptable but likely represents a performance regression. I intend to rework this to use caching, but I'm saving that for a later commit because this one is big enough already. The proximal reason for this change is to resolve the chicken/egg problem whereby there was previously no single point where we could apply "moved" statements to the previous run state before creating a plan. With this change in place, we can now do that as part of Context.Plan, prior to forking the input state into the three separate state artifacts we use during planning. However, this is at least the third project in a row where the previous API design led to piling more functionality into terraform.NewContext and then working around the incorrect order of operations that produces, so I intend that by paying the cost/risk of this large diff now we can in turn reduce the cost/risk of future projects that relate to our main workflow actions.	5 years ago
Martin Atkins	4faac6ee43	core: Record move result information in the plan ... Here we wire through the "move results" into the graph walk data structures so that all of the the nodes which produce plans.ResourceInstanceChange values can capture the "PrevRunAddr" for each resource instance. This doesn't actually quite work yet, because the logic in Context.Plan isn't actually correct and so the updated state from refactoring.ApplyMoves isn't actually visible as the "previous run state". For that reason, the context test in this commit is currently skipped, with the intent of re-enabling it once the updated state is properly propagating into the plan graph walk and thus we can actually react to the result of the move while choosing actions for those addresses.	5 years ago
Martin Atkins	22b36d1f4c	Field for the previous address of each resource instance in the plan ... In order to expose the effect of any relevant "moved" statements we dealt with prior to creating the plan, we'll record with each ResourceInstanceChange both is current address and the address it was tracked at for the previous run. To save consumers of these objects from having to special-case the situation where there _was_ no previous run (e.g. because this is a Create change), we'll just pretend the previous run address was the same as the current address in that case, the same as for an update without any renaming in effect. This includes a breaking change to the plan file format, but one that doesn't require a version number increment because there is no ambiguity between the two formats and so mismatched parsers will already fail with an error message. As of this commit we've just added the new field but not yet populated it with any useful information: it always just matches Addr. A future commit will wire this up to the result of applying the moves so that we can populate it correctly. We also don't yet expose this new information anywhere in the UI layer.	5 years ago
James Bardin	ea68d79ea2	nested object values can be computed ... While blocks were not allowed to be computed by the provider, nested objects can be. Remove the errors regarding blocks and verify unknown values are valid.	5 years ago
James Bardin	903797084a	objects vs maps in nested object types ... When using NestedMap objects, unify the codepath for both maps and objects as they may be interchangeable.	5 years ago
James Bardin	7ca6be8285	correctly verify planned nested object values ... The validation for nested object types with computed attributes was using the incorrect function call.	5 years ago
Martin Atkins	ce96d82de0	build: Centralize our protobuf compilation steps ... We have a few different .proto files in this repository that all need to get recompiled into .pb.go files each time we change them, but we were previously handling that with some scripts that just assumed that protoc and the relevant plugins were already installed on the system somewhere, at the right versions. In practice we've been constantly flopping between different versions of these tools due to folks having different versions installed in their development environments. In particular, the state of the .pb.go files in the prior commit wasn't reproducible by any single version of the tools because they've all slightly diverged from one another. In the interests of being more consistent here and avoiding accidental inconsistencies, we'll now centralize the protocol buffer compile steps all into a single tool that knows how to fetch and install the expected versions of the various tools we need and then run those tools with the right options to get a stable result. If we want to upgrade to either a newer protoc or a newer protoc-gen-go in future then we'll do that in a central location and update all of the .pb.go files at the same time, so that we're always consistently tracking the same version of protocol buffers everywhere. While doing this I attempted to keep as close as possible to the toolchain we'd most recently used, but since they were not consistent with each other they've now all changed which version numbers they record at minimum, and the planproto stub in particular now also has a slightly different descriptor serialization but is otherwise offering the same API.	5 years ago
James Bardin	249e05f827	Merge pull request #29437 from hashicorp/jbardin/move ... refactoring: complete CanChainFrom and NestedWithin	5 years ago
Alisdair McDiarmid	40dd4292b8	Merge pull request #29438 from hashicorp/alisdair/init-force-copy-multiple-workspaces ... command: Suppress prompt for init -force-copy	5 years ago
James Bardin	30afb492ab	fix ApplyMove test with nested modules working ... The addrs package can now correctly handle the combinations of nested module endpoints, which fixes these 2 tests.	5 years ago
James Bardin	bc60f7aae4	Extend CanChainFrom to handle relative modules ... CanChainFrom needs to be able to handle move statements from different relative modules, re-implementing with addrs.anyKey Add the anyKey InstanceKey value to the addrs package to simplify module path comparison. This allows all combinations of module path representation to be normalized into a ModuleInstance which can be compared directly, rather than dealing with multiple levels of different prefix types.	5 years ago
Alisdair McDiarmid	2762a940c0	command: Suppress prompt for init -force-copy ... The -force-copy flag to init should automatically migrate state. Previously this was not applied to one case: when migrating from a backend with multiple workspaces to another backend supporting multiple workspaces. I believe this was an oversight so this commit fixes that.	5 years ago
Alisdair McDiarmid	70a4f7a6b6	command: Fix stale lock after running add	5 years ago
James Bardin	11561b22cd	Merge pull request #29330 from hashicorp/jbardin/move ... refactoring: CanChainFrom and NestedWithin	5 years ago
Martin Atkins	43d753e727	command: "terraform add" is experimental ... We're aware of several quirks of this command's current design, which result from some existing architectural limitations that we can't address immediately. However, we do still want to make this command available in its current capacity as an incremental improvement, so as a compromise we'll document it as experimental. Our intent here is to exclude it from the Terraform 1.0 Compatibility Promises so that we can have the space to continue to improve the design as other parts of the overall Terraform system gain new capabilities. We don't currently have any concrete plan for this command to be stabilized and subject to compatibility promises. That decision will follow from ongoing discussions with other teams whose systems may need to change in order to support the final design of "terraform add".	5 years ago
James Bardin	553d6525d2	more move tests	5 years ago
James Bardin	2dff0481c8	missed relMatch for AbsModuleCall in SelectsModule	5 years ago
James Bardin	58aabb54ca	Merge pull request #29410 from hashicorp/jbardin/format-empty-nested-attrs ... handle null and unknown values in attr diffs	5 years ago
James Bardin	68ed50616e	handle null and unknown values in attr diffs ... The code adopted from block diffs was not set to handle null and unknown values, as those are not allowed for blocks. We also revert the change to formatting nested object types as single attributes, because the attribute formatter cannot handle sensitive values from the schema. This presents some awkward syntax for diffs for now, but should suffice until the entire formatter can be refactored to better handle these new nested types.	5 years ago
James Bardin	da007517b0	handle null NestingSingle values ... Null NestingSingle attributes were not being handled in ProposedNew	5 years ago
Martin Atkins	c23a7fce4e	lang/funcs: Preserve IP address leading zero behavior from Go 1.16 ... Go 1.17 includes a breaking change to both net.ParseIP and net.ParseCIDR functions to reject IPv4 address octets written with leading zeros. Our use of these functions as part of the various CIDR functions in the Terraform language doesn't have the same security concerns that the Go team had in evaluating this change to the standard library, and so we can't justify an exception to our v1.0 compatibility promises on the same sort of security grounds that the Go team used to justify their compatibility exception. For that reason, we'll now use our own fork of the Go library functions which has the new check disabled in order to preserve the prior behavior. We're taking this path, rather than pre-normalizing the IP address before calling into the standard library, because an additional normalization layer would be entirely new code and additional complexity, whereas this fork is relatively minor in terms of code size and avoids any significant changes to our own calls to these functions. Thanks to the Kubernetes team for their prior work on carving out a subset of the "net" package for their similar backward-compatibility concern. Our "ipaddr" package here is a lightly-modified fork of their fork, with only the comments changed to talk about Terraform instead of Kubernetes. This fork is not intended for use in any other future feature implementations, because they wouldn't be subject to the same compatibility constraints as our existing functions. We will use these forked implementations for new callers only if consistency with the behavior of the existing functions is a key requirement.	5 years ago
Martin Atkins	383bbdeebc	Upgrade to Go 1.17 ... This includes the addition of the new "//go:build" comment form in addition to the legacy "// +build" notation, as produced by gofmt to ensure consistent behavior between Go versions. The new directives are all equivalent to what was present before, so there's no change in behavior. Go 1.17 continues to use the Unicode 13 tables as in Go 1.16, so this upgrade does not require also upgrading our Unicode-related dependencies. This upgrade includes the following breaking changes which will also appear as breaking changes for Terraform users, but that are consistent with the Terraform v1.0 compatibility promises. - On MacOS, Terraform now requires macOS 10.13 High Sierra or later. This upgrade also includes the following breaking changes which will appear as breaking changes for Terraform users that are inconsistent with our compatibility promises, but have justified exceptions as follows: - cidrsubnet, cidrhost, and cidrnetmask will now reject IPv4 CIDR addresses whose decimal components have leading zeros, where previously they would just silently ignore those leading zeros. This is a security-motivated exception to our compatibility promises, because some external systems interpret zero-prefixed octets as octal numbers rather than decimal, and thus the previous lenient parsing could lead to a different interpretation of the address between systems, and thus potentially allow bypassing policy when configuring firewall rules etc. This upgrade also includes the following breaking changes which could _potentially_ appear as breaking changes for Terraform users, but that do not in practice for the reasons given: - The Go net/url package no longer allows query strings with pairs separated by semicolons instead of ampersands. This primarily affects HTTP servers written in Go, and Terraform includes a special temporary HTTP server as part of its implementation of OAuth for "terraform login", but that server only needs to accept URLs created by Terraform itself and Terraform does not generate any URLs that would be rejected.	5 years ago
James Bardin	a94155d0ca	Merge pull request #29397 from hashicorp/jbardin/format-id-name-marks ... unmark object ID or Name for formatting	5 years ago