138 Commits (148b19b0d7e36b6f2b8a169d2cb7b6069ab5d00a)

Author	SHA1	Message	Date
Daniel Schmidt	f3764ead4c	Merge pull request #35118 from hashicorp/consistency-provider-deferred ... stacks: error if provider emits deferred action when it's not supported	2 years ago
Daniel Schmidt	f684df82f6	stack: improve wording ... Co-authored-by: Brian Flad <bflad417@gmail.com>	2 years ago
Daniel Schmidt	08917d1628	stacks: error if provider emits deferred action when it's not supported	2 years ago
James Bardin	2562420d16	prevent panics with null objects in nested attrs ... When descending into nested structural attributes, don't try to extract attributes from null objects. Unlike with blocks, nested attributes allow the possibility of assigning null values. While these technically aren't allowed to be altered, we need to accept these for compatibility.	2 years ago
Martin Atkins	30e2fd6525	Handle marks a little more consistently ... In the very first implementation of "sensitive values" we were unfortunately not disciplined about separating the idea of "marked value" from the idea of "sensitive value" (where the latter is a subset of the former). The first implementation just assumed that any marking whatsoever meant "sensitive". We later improved that by adding the marks package and the marks.Sensitive value to standardize on the representation of "sensitive value" as being a value marked with _that specific mark_. However, we did not perform a thorough review of all of the mark-handling codepaths to make sure they all agreed on that definition. In particular, the state and plan models were both designed as if they supported arbitrary marks but then in practice marks other than marks.Sensitive would be handled in various inconsistent ways: dropped entirely, or interpreted as if marks.Sensitive, and possibly do so inconsistently when a value is used only in memory vs. round-tripped through a wire/file format. The goal of this commit is to resolve those oddities so that there are now two possible situations: - General mark handling: some codepaths genuinely handle marks generically, by transporting them from input value to output value in a way consistent with how cty itself deals with marks. This is the ideal case because it means we can add new marks in future and assume these codepaths will handle them correctly without any further modifications. - Sensitive-only mark preservation: the codepaths that interact with our wire protocols and file formats typically have only specialized support for sensitive values in particular, and lack support for any other marks. Those codepaths are now subject to a new rule where they must return an error if asked to deal with any other mark, so that if we introduce new marks in future we'll be forced either to define how we'll avoid those markings reaching the file/wire formats or extend the file/wire formats to support the new marks. Some new helper functions in package marks are intended to standardize how we deal with the "sensitive values only" situations, in the hope that this will make it easier to keep things consistent as the codebase evolves in future. In practice the modules runtime only ever uses marks.Sensitive as a mark today, so all of these checks are effectively covering "should never happen" cases. The only other mark Terraform uses is an implementation detail of "terraform console" and does not interact with any of the codepaths that only support sensitive values in particular.	2 years ago
Martin Atkins	ea60ca7d70	plans/planfile: Don't use deprecated "io/ioutil" package ... Everything from ioutil has been moved into other locations in modern Go. This particular package only used ioutil.ReadAll and ioutil.ReadFile, which have been replaced by io.ReadAll and os.ReadFile respectively.	2 years ago
Martin Atkins	6be3ddf9e7	planfile: Reject value marks that we cannot serialize ... The plan file format can only preserve the "sensitive" mark, but previously it was just silently treating any other mark as if it were the sensitive mark. Now we'll reject any other marks at serialization time. There are not currently any other marks used by the modules runtime and so in practice this cannot fail yet, but this is here to guard against misbehavior if we introduce new marks in future without considering whether and how they are to be serialized. (For any mark we choose not to serialize, it'll be the caller's responsibility to replace the values using unsupported marks with suitable unmarked placeholders.)	2 years ago
Daniel Schmidt	57018c8d57	stacks: add test case for ReportDataSourceInstanceDeferred	2 years ago
Daniel Schmidt	7e3fb0850c	stacks: add ReportDataSourceInstanceDeferred method	2 years ago
Daniel Schmidt	4db9ab8536	stacks: check if resource instance is deferred ... before marking it as deferred	2 years ago
Daniel Schmidt	5b8f77d2a4	stacks: add deferred actions to plugin protocol ... We are moving the deferred reason from plans/deferring to providers to have a single representation of deferred reason to be used from providers to the plan. Using the plan's representation causes circular dependencies. Co-authored-by: Matej Risek <matej.risek@hashicorp.com>	2 years ago
Liam Cervante	cbee7469b0	Complete functionality for the unknown instances processing path for resources (#34945) ... * Expand functionality of the unknown count/for_each path when processing resources * remove reference to experiment in tests * fix race tests * update tests * address comments	2 years ago
Liam Cervante	28797e7938	Add the deferred changes into the plan (#34946) ... * Add the deferred changes into the plan * add copywrite headers	2 years ago
Nick Fagerlund	d3cdb2b26d	Add DeferralAllowed fields to PlanOpts, graphWalkOpts, and Deferred ... The concept behind deferred actions is that various objects (resources, data sources, modules, stack components) can end up being *deferred*, meaning that we're unable to produce an actionable plan for them and intend to run the *rest* of the plan to completion and apply it without taking action on those objects, which will have to wait for another future plan. For the most part, we only *find out* about the kinds of absences that force deferral at the time of *graph walk.* That's because deferral is caused by unknown values in count/for_each, absent dependencies reported by the provider, or dependency on something that got deferred, none of which can really be know at graph construction. Also, we don't always want deferral to be an available option -- for the time being, we want to preserve existing behavior in the modules runtime ("normal terraform") while enabling deferral in Stacks. So, *things that happen during graph walk need to know whether they can defer.* (And if not, they'll throw an error if they hit such an absence.) That means the top-level signal that deferral is enabled (which should arrive as part of the PlanOpts) needs to make it all the way down into the graph walk methods. The graph walk methods generally have access to an EvalContext, and EvalContext conveniently gives you access to a Deferred tracker, the pointer to which is shared across all related EvalContexts. So, the Deferred will be what tells you whether you can defer.	2 years ago
Martin Atkins	d3b8a55781	go.mod: go get google.golang.org/protobuf@v1.33.0 ... Many of the changes between these two versions seem to be in parts of the library that Terraform doesn't use, but there are a few sensitive parts: - Slight changes to details of how the library implements the JSON encoding of protobuf. Terraform CLI/Core don't use this, but some of the backends might use it indirectly when talking to their underlying APIs. - Some details of the "protoreflect" package for working with protobuf descriptors dynamically (rather than using generated stubs) have changed, but it seems pretty marginal and unlikely to cause problems.	2 years ago
Martin Atkins	5f2eab6944	providerreqs: Provider requirement/version types in their own package ... As we previously found with "package addrs", any package that contains types used as identifiers or addresses tends to get imported from all over, which means that if we mix real logic with those types in the same package then a lot more packages end up depending indirectly on our external dependencies. In this case, the getproviders package depends on various things we need only during provider installation, such as the library we use for GPG signature verification. Lots of different packages in Terraform need to talk about provider requirements, but only the CLI layer actually needs to do provider installation, and so this commit splits the types we use to talk about requirements into a new package "providerreqs", which package getproviders then depends on. To keep the size of this diff relatively small to start I've left forwarding aliases in package getproviders, which means that not all of the callers needed to be updated all at once. This commit does update all of the packages that the remote state backends depend on though, because that then allows us to reduce the set of indirect dependencies for each of those backends. The nested module go.mod files show that this has disconnected the indirect dependencies on github.com/ProtonMail/go-crypto and on github.com/cloudflare/circl , both of which Terraform CLI uses only for provider installation.	2 years ago
Martin Atkins	d86eee6c1f	go.mod: Eliminate github.com/apparentlymart/go-dump ... I originally wrote this library as a stopgap measure because davecgh/go-spew produced a hard-to-read representation of cty.Value due to exposing the raw internals, whereas my "go-dump" instead just pretty-prints whatever fmt.Sprintf("%#v", ...) would return for a given value. However, in the meantime I wrote ctydebug as a more cty-specialized library for this purpose, and we switched to using go-cmp as our primary library for deep-comparing data structures, and so our few remaining uses of apparentlymart/go-dump were vestigial in tests that haven't been updated recently enough to have been updated to our new standards. This commit therefore swaps out all use of that library with combinations of go-cmp and ctydebug, matching our current standards for how to do these sorts of things in tests. This removes our dependency on "go-dump" altogether. At least one of our dependencies is still using go-dump as part of its own test suite, and so the Go toolchain still includes its declared version in our go.sum file. However, this library is no longer linked in to Terraform either in real builds or in test binaries, and so it gets dropped entirely from our go.mod file.	2 years ago
CJ Horton	5a750561e6	round-trip check results through stack plans ... Components expect that check results will be round-tripped through the plan in order to update unknown check results during the apply. We hadn't wired this up for stack plans, resulting in panics when the apply tries to update a check status that doesn't exist.	2 years ago
Martin Atkins	db07b905e9	deferring: A new package for tracking deferred actions ... When we defer planning actions that have direct side-effects -- that is, resource instance actions -- we need to remember that we did so because we then need to force deferring planning for any other resources downstream to preserve the required order of operations. That means we need to keep track of what we've already deferred so we can check whether a newly-visited resource instance must also be deferred. We also need to track deferrals for the purpose of reporting them as part of the plan presented to the end-user. This change _starts_ to implement that, but for now the public API is limited only to generating a boolean signal for whether there are any deferrals at all, because that's the minimum possible reporting that allows for correct behavior (so the caller can treat the resulting plan slight differently in the workflow). In future we'll also want to be able to return placeholder values for not-yet-expanded resource instances, but that's not fully built out here yet since for an initial implementation we can cheat and use cty.DynamicVal as a universal "we know nothing at all" placeholder. This API is likely to grow and change in forthcoming commits as we continue wiring these new behaviors into the Terraform modules runtime.	2 years ago
Martin Atkins	884e1fb2a4	terraform: Plans can be "complete" and "applyable" ... These ideas are both already implied by some logic elsewhere in the system, but until now we didn't have the decision logic centralized in a single place that could therefore evolve over time without necessarily always updating every caller together. We'll now have the modules runtime produce its own boolean ruling about each characteristic, which callers can rely on for the mechanical decision-making of whether to offer the user an "approve" prompt, and whether to remind the user after apply that it was an incomplete plan that will probably therefore need at least one more plan/apply round to converge. The "Applyable" flag directly replaces the previous method Plan.CanApply, with equivalent logic. Making this a field instead of a method means that we can freeze it as part of a saved plan, rather than recalculating it when we reload the plan, and we can export the field value in our export formats like JSON while ensuring it'll always be consistent with what Terraform is using internally. Callers can (and should) still use other context in the plan to return more tailored messages for specific situations they already know about that might be useful to users, but with these flags as a baseline callers can now just fall back to a generic presentation when encountering a situation they don't yet understand, rather than making the wrong decision and causing something strange to happen. That is: a lack of awareness of a new rule will now cause just a generic message in the UI, rather than incorrect behavior. This commit mostly just deals with populating the flags, and then all of the direct consequences of that on our various tests. Further changes to actually make use of these flags elsewhere in the system will follow in later commits, both in this repository and in other repositories.	2 years ago
James Bardin	bc945f22eb	Merge pull request #34515 from hashicorp/jbardin/provider-function-validation ... Add validation for provider function results	2 years ago
James Bardin	a3aa517589	store and verify provider functions calls in plan ... Store the hashed function calls within the plan file, so that Terraform can validate the results during apply as well. The error won't be as good, since we don't have access to the prior results to help debugging, but it is very helpful to indicate when a function returned an incorrect result rather than show that the resource attempted to apply a different value from what was originally planned.	2 years ago
Alisdair McDiarmid	fc75657113	terraform: Support sensitive input variable values ... The stacks runtime interacts directly with the modules runtime's planning operations, rather than through the usual CLI paths. As a result, root module input variable values can be marked as sensitive upon entry. This commit adds support for such marked values to the modules runtime and the `plans.Plan` type. This is sufficient for stacks, which does not use the planfile serialization, but we may in future choose to serialize these decoded marks also.	2 years ago
Martin Atkins	a4d61733f6	stackplan: Include extra info in raw component instance plan ... We'll now track the planned action, the component addresses we depend on, and the planned output values.	2 years ago
kmoe	6ac20050a5	planproto: parse forget actions (#34433)	2 years ago
kmoe	cc3e3bd513	planproto: add Forget, CreateThenForget actions (#34430)	2 years ago
CJ Horton	89aea31bbd	stackstate: sensitive values are preserved when deserializing ... Without this, sensitive values cause a permadiff, since their sensitivity is lost on every round trip through state.	2 years ago
Martin Atkins	caeecbebcb	planfile: Move some of the proto convert funcs into package planproto ... Previously we had much of the mechanical conversion logic for going from in-memory structures to protobuf messages as unexported functions in the planfile package. We're now using planproto for parts of the representation of stack plans and states too, and so having those helpers exported is helpful. In some of our other protobuf packages we've adopted the convention of putting simple conversion functions in the same package as the protobuf stubs themselves, following the convention used upstream such as the additional encoding/decoding helper functions in the "anypb" package. This therefore follows that lead by adding some helpful constructor functions to package planproto and then making the equivalent functions back in planfile just thinly wrap them for now, to avoid churning the callers too much and thus making backports potentially harder.	2 years ago
kmoe	a718f70f85	Add removed block (#34251) ... * terraform: remove redundant code NodeDestroyResourceInstance is never instantiated with a DeposedKey of anything other than states.NotDeposed, so the deleted code is never run. Deposed objects get a NodeDestroyDeposedResourceInstanceObject instead. * tfdiags: add helper func * configs: introduce removed block type * terraform: add forget action * renderer: render forget actions * terraform: deposed objects can be forgotten Deposed objects encountered during planning spawn NodePlanDeposedResourceInstanceObject, which previously generated a destroy change. Now it will generate a forget change if the deposed object is a forget target, and a destroy change otherwise. The apply graph gains a new node type, NodeForgetDeposedResourceInstanceObject, whose execution simply removes the object from the state. * configs: add RemoveTarget address type * terraform: modules can be forgotten * terraform: error if removed obj still in config * tests: better error on restore state fail * Update CHANGELOG.md	2 years ago
Martin Atkins	4e34e6ebfa	stacks: Track prior state for all resource instance objects ... Previously our raw plan data structure only included information about resource instances that have planned actions, which meant we were missing those which only get their state updated during refresh and don't actually need any real work done. Now we'll track everything, accepting that some "planned changes" for resource instance objects will not actually include a planned change in the typical sense, and instead only represent a state update.	2 years ago
Martin Atkins	1f7dbaf3a8	plans: ResourceInstanceChange.ObjectAddr ... We've now many times made the mistake of thinking that the resource instance address is a suitable unique identifier for a resource instance change in a plan -- the name certainly seems to suggest it -- but really these represent changes to resource instance _objects_ and so it's important to also include the DeposedKey field as part of the unique identifier for a change. Now we'll expose ObjectAddr as a new method that returns the two together for convenient use as a single address value, along with adding some commentary to the fields to make it clearer that Addr is not sufficient. Ideally we'd rename Addr to make it less of an attractive nuisance, but that was too invasive a change for today so will need to wait for another time.	2 years ago
Martin Atkins	13e26b60dd	addrs: Adopt DeposedKey as an address type; remove states.Generation ... Previously we had states.DeposedKey as our representation of the unique keys used to differentiate between multiple deposed objects on the same resource instance. That type is useful in various places, but its presence in the "states" package made it troublesome to import for some callers. Since its purpose is as an identifier, this type is more at home in the "addrs" package. The states package retains a small number of aliases to preserve compatibility with a few existing callers for now, until we have the stomach for a more invasive change. This also removes the largely-redundant states.Generation interface type, which initially was intended as a way to distinguish between "current" objects and deposed objects, but didn't really add anything because we already have addrs.NotDeposed to represent that situation. Instead, we just ended up with various bits of boilerplate adapter code converting between the two representations. Everything now uses addrs.DeposedKey and uses addrs.NotDeposed to identify non-deposed objects.	2 years ago
Martin Atkins	dd04b2caef	rpcapi: Track action for changes to stack component instances	2 years ago
Martin Atkins	c81b9e926f	planfile: Expose some protobuf decoding helpers ... Our internal format for stack plans will share some of the message types from the main plan file format but use a different overall container, so now we'll expose a few helpers for the sub-parts of the structure that can appear in both a stack plan and a normal plan.	2 years ago
Martin Atkins	9942deded1	plans: Move "planproto" up to toplevel ... The Go package containing the generated stubs for the plan file format's protobuf messages was previously nestled as an internal package under "plans", making it importable only for source files under "plans". To avoid duplicating various message types into the new "stack plan" serialization, we'll now make this package internal to the whole of this codebase rather than just to "plans", and then have the stack data protocol buffers schema import it to reuse the same message types.	2 years ago
Liam Cervante	4ce385a19b	testing framework: implement overrides in terraform graph (#34169) ... * testing framework: implement overrides in terraform graph * fix bug for modules and module instances * add test for ModuleInstance.ContainingModule()	3 years ago
CJ Horton	98b81d3c50	add missing copyright headers	3 years ago
hashicorp-copywrite[bot]	53c34ff49c	Update copyright file headers to BUSL-1.1	3 years ago
Nick Fagerlund	f98f920b67	Add error wrapper types to highlight bad plan/state data ... This commit uses Go's error wrapping features to transparently add some optional info to certain planfile/state read errors. Specifically, we wrap errors when we think we've identified the file type but are somehow unable to use it. Callers that aren't interested in what we think about our input can just ignore the wrapping; callers that ARE interested can use `errors.As()`.	3 years ago
Nick Fagerlund	0df3c143bb	New plans.Quality type for display-relevant facts about a plan ... This commit replaces the existing jsonformat.PlanRendererOpt type with a new type with identical semantics, located in the plans package. We needed to be able to exchange the facts represented by `jsonformat.PlanRendererOpt` across some package boundaries, but the jsonformat package is implicated in too many dependency chains to be safe for that purpose! So, we had to make a new one. The plans package seems safe to import from all the places that must emit or accept this info, and already contains plans.Mode, which is effectively a sibling of this type.	3 years ago
Nick Fagerlund	f9d937a4dd	Apply a confirmable run when given a saved cloud plan (#33270) ... It displays a run header with link to web UI, like starting a new plan does, then confirms the run and streams the apply logs. If you can't apply the run (it's from a different workspace, is in an unconfirmable state, etc. etc.), it displays an error instead. Notable points along the way: * Implement `WrappedPlanFile` sum type, and update planfile consumers to use it instead of a plain `planfile.Reader`. * Enable applying a saved cloud plan * Update TFC mocks — add org name to workspace, and minimal support for includes on MockRuns.ReadWithOptions.	3 years ago
Martin Atkins	a7807dac16	main: Optionally configure an OpenTelemetry OTLP exporter ... Terraform CLI is sometimes used as part of a larger distributed system, in which case it would be helpful to be able to gather telemetry from it as part of the larger request it's being run in response to. We'll now support optionally enabling an OTLP exporter by setting the environment variable OTEL_TRACES_EXPORTER=otlp (a standard OpenTelemetry convention). As of this commit there isn't actually anything emitting traces to the specified collector, but we'll gradually add tracing instrumentation to parts of Terraform CLI and Core in later commits.	3 years ago
Liam Cervante	c9bc7e8479	Add input validation into the 'checks' outputs and tracking (#33481)	3 years ago
Liam Cervante	3732bffe13	[Testing Framework] Adds TestContext for evaluating test assertions (#33326) ... * Add test structure to views package for rendering test output * Add test file HCL configuration and parser functionality * Adds a TestContext structure for evaluating assertions against the state and plan	3 years ago
Martin Atkins	d49e991c3c	plans/objchange: Support nested unknowns in our unrefinedValue shim ... Several parts of the objchange logic incorrectly use cty.Value.RawEquals for value comparison, instead of more appropriate comparison methods like cty.Value.Equals or c.Value.Range().Includes. That makes them incorrectly consider two unknown values with the same type but different refinements as always non-equal, rather than evaluating based on the overlap between the refinements (if any). As a short-term fix for that we previously added this unrefinedValue shim that just strips away the refinements for comparison, thus allowing callers to continue using RawEquals as long as they've already taken care of all of the other things that can make that go wrong, such as value marks. Unfortunately the shim was too simplistic and only supported direct unknown values. Unknown values with refinements can also appear nested inside known container values such as collections, so the shim needs to recursively un-refine the entire data structure in that case. This is still intended only as a temporary fix until we have time to revisit all of the callers and make them use cty's own logic for comparison. Using cty's own logic will make the results more precise, because e.g. it can notice if two unknown strings have different known prefixes and therefore cannot possibly be equal despite not being fully known. For now this shim will accept any pair of unknown values of the same type as equal, regardless of refinement.	3 years ago
James Bardin	f6c536e436	validate planned set sizes using refinements	3 years ago
James Bardin	3c8a163583	set length is unknown with partially known elems ... If a set contains partially known values the length is unknown which causes assertPlannedObjectValid to fail valid plans. Revert to the old method if using LengthInt for the set lengths, which returns the maximum number of possible elements, with a guard for entirely unknown set values.	3 years ago
Martin Atkins	4c439b099f	plans/objchange: Don't consider refinements when validating plans ... Providers that existed prior to refinements (all of them, at the time of writing) cannot preserve refinements sent in unknown values in the configuration, and even if one day providers _are_ aware of refinements there we might add new ones that existing providers don't know how to handle. For that reason we'll absolve providers of the responsibility of preserving refinements from config into plan by fixing some cases where we were incorrectly using RawEquals to compare values; that function isn't appropriate for comparing values that might be unknown. However, to avoid a disruptive change right now this initial fix just strips off the refinements before comparing. Ideally this should be using Value.Equals and handling unknown values more explicitly, but we'll save that for a possible later improvement. This does not include a similar exception for validating whether a final value conforms to a plan because the plan value and the final value are both produced by the same provider and so providers ought to be able to be consistent with their _own_ treatment of refinements, if any. Configuration is special because Terraform itself generates that, and so it can potentially contain refinements that a particular provider has no awareness of.	3 years ago
Martin Atkins	dfe5e1ddc4	plans/objchange: Providers must honor their unknown value refinements ... If the original value was unknown but its range was refined then the provider must return a value that is within the refined range, because otherwise downstream planning decisions could be invalidated. This relies on cty's definition of whether a value is in a refined range, which has pretty good coverage for the "false" case and so should give a pretty good signal, but it'll probably improve over time and so providers must not rely on any loopholes in the current implementation and must keep their promises even if Terraform can't currently check them.	3 years ago
CJ Horton	258bdbe89f	Merge pull request #33238 from hashicorp/radditude/import-plan-plumbing ... plannable import: correct plumbing when using a plan output file	3 years ago