29 Commits (8564e934aa57cc59459581ee06479879ebf41bf0)

Author	SHA1	Message	Date
Liam Cervante	07f6621091	stacks: include resources in state when calculating required providers (#34645) ... * stacks: include resources in state when calculating required providers * also support apply time * add copywrite headers	2 years ago
Martin Atkins	884e1fb2a4	terraform: Plans can be "complete" and "applyable" ... These ideas are both already implied by some logic elsewhere in the system, but until now we didn't have the decision logic centralized in a single place that could therefore evolve over time without necessarily always updating every caller together. We'll now have the modules runtime produce its own boolean ruling about each characteristic, which callers can rely on for the mechanical decision-making of whether to offer the user an "approve" prompt, and whether to remind the user after apply that it was an incomplete plan that will probably therefore need at least one more plan/apply round to converge. The "Applyable" flag directly replaces the previous method Plan.CanApply, with equivalent logic. Making this a field instead of a method means that we can freeze it as part of a saved plan, rather than recalculating it when we reload the plan, and we can export the field value in our export formats like JSON while ensuring it'll always be consistent with what Terraform is using internally. Callers can (and should) still use other context in the plan to return more tailored messages for specific situations they already know about that might be useful to users, but with these flags as a baseline callers can now just fall back to a generic presentation when encountering a situation they don't yet understand, rather than making the wrong decision and causing something strange to happen. That is: a lack of awareness of a new rule will now cause just a generic message in the UI, rather than incorrect behavior. This commit mostly just deals with populating the flags, and then all of the direct consequences of that on our various tests. Further changes to actually make use of these flags elsewhere in the system will follow in later commits, both in this repository and in other repositories.	2 years ago
Alisdair McDiarmid	1d3f863f2b	stackruntime: Support sensitive component inputs ... Components can emit sensitive values as outputs, which can be consumed as inputs to other components. This commit ensures that such values are correctly processed in order to pass their sensitivity to the modules runtime.	2 years ago
Martin Atkins	a4d61733f6	stackplan: Include extra info in raw component instance plan ... We'll now track the planned action, the component addresses we depend on, and the planned output values.	2 years ago
Martin Atkins	25a514f846	stackeval: Populate planned component instance more thoroughly ... We'll now track a more realistic Action, the components that the component instance depends on, and the planned output values.	2 years ago
Martin Atkins	bcccd67e32	tfstackdata1: A local DynamicValue message type ... Previously we were directly using the planproto.DynamicValue message type, but that's symmetrical with plans.DynamicValue in that it encodes only the value itself and not associated metadata such as the paths that have sensitive value marks. This new tfstackdata1.DynamicValue therefore echoes terraform1.DynamicValue by carrying both the value and the metadata together. This implies a slight change to one of the existing message types that was encoding a map of planned input values for each component instance, but in practice we're not currently making use of that data anyway -- it's there to enable a hypothetical extra correctness check in the stacks runtime so that we can detect inconsistency problems closer to their source -- and so this change doesn't affect anything in practice. Before we make use of it we'll probably want to change the internal API a little so that we can preserve the sensitive value metadata on those values, but that's beyond the scope of this PR. The main point of adding this is to track the output values for a component instance as part of the raw state, and so the new field for that is also added here although nothing will read or write it yet. Actual use of that new field will follow in future commits.	2 years ago
Matej Risek	8ac8f18636	Feature/add additional fields to resource proto (#34364) ... Add additional fields to the AppliedChange#ResourceInstance We're adding Resource Mode, Resource Type and Provider Address to the AppliedChange's ResourceInstance	2 years ago
Liam Cervante	afc83356ca	stacks: actions for a no-op change are a concrete list (#34260)	2 years ago
Martin Atkins	0952457778	stackplan: Remove stale TODO about output value changes ... This recorded that we needed to encode the type information for the output values. In the end we didn't need to do anything special for that because we're borrowing the pre-serialized data originally designed for the traditional Terraform CLI saved plan format, and that encodes the type information as part of the MessagePack encoding in-band.	2 years ago
Martin Atkins	972f1c0678	stackplan: Track provider config for every resource instance change ... Previously we were still partially relying on the record of this in the embedded planproto object, but since we're now allowing "planned changes" that don't actually include a change in that traditional sense we need to track the provider configuration address separately as a top-level field. As with some of the addresses before, this means we're now storing the provider config address redundantly in two places when there _is_ a planproto change. Hopefully we'll improve that someday, but we don't need to sweat it too much right now because this only affects the internal raw plan format that is not subject to any compatibility promises between releases, so we can freely change it in any future version.	2 years ago
Martin Atkins	4e34e6ebfa	stacks: Track prior state for all resource instance objects ... Previously our raw plan data structure only included information about resource instances that have planned actions, which meant we were missing those which only get their state updated during refresh and don't actually need any real work done. Now we'll track everything, accepting that some "planned changes" for resource instance objects will not actually include a planned change in the typical sense, and instead only represent a state update.	2 years ago
Martin Atkins	3d8e95d153	rpcapi(Stacks): Support multiple descriptions in a "PlannedChange" ... Previously we had an asymmetry between PlannedChange and AppliedChange, where the latter supported any number of "description" objects whereas the former only supported zero or one. This change then is primarily motivated by making these two APIs consistent, but also has some other benefits: - The generated RPC stub packages will now have a specific type to use to represent _just_ a change description for a plan, so callers can pass around and preserve just those objects in isolation, rather than always having to use the entire PlannedChange message for that purpose. - This gives us a little more flexibility if later we find we need to return more than one description associated with a single logical change in future. For example, if we end up needing to make a breaking change to how we describe a particular kind of change, we might send an object containing both the old form and the new form together, and instruct clients that support the new form to treat it as taking priority over the old form when both are present. We don't yet know of any specific reason to do this, but this protocol is likely to be with us for a long time and so we need to prioritize leaving room for future additions we haven't thought of yet. This is a breaking change to the API from a client's perspective, but that's acceptable at the current stage of development where nothing has been committed in a stable release yet.	2 years ago
Martin Atkins	8fd29e9439	stacks: Preserve prior state from plan to apply ... We need to retain the prior state (in a form that Terraform Core's modules runtime can accept) between the plan and apply phases because Terraform uses it to verify that the provider's "final plan" is consistent with the initial plan. We previously tried to do this by reusing the "old" value from the planned change, but that's not really possible in practice because of the historical implementation detail that Terraform wants state information in a JSON format and plan information in MessagePack. This also contains the beginnings of handling the "discarding" of unrecognized keys from the state data structures, although we'll probably need to do more in that area later since this is just the bare minimum.	2 years ago
Martin Atkins	5372d5eb98	stacks+rpcapi(stacks): Model deposed objects for resource instances ... Previously we incorrectly assumed that resource instances only have one "current" object each. However, resource instances can also have deposed objects which we must also keep track of. This is a breaking change to the rpcapi since we're now using a new message type for a resource instance _object_ address in several places. That breakage is intentional here because at the time of writing this commit the rpcapi is not yet in any stable release and we want to force updating our existing internal client to properly handle deposed objects too.	2 years ago
Martin Atkins	69aa51b9ff	stackruntime: Fix some quirks of data sent from plan to apply ... We previously took a few shortcuts just to more quickly produce a proof of concept. This gets us considerably closer to a properly-working handoff from plan to apply, at least for create, update, and replace actions. Destroying stuff that's been removed from the configuration will take some further work since currently everything we do is driven by objects in the configuration. That will follow in later commits.	2 years ago
Martin Atkins	c67920ce4b	stackplan: Add HashiCorp copyright comments	2 years ago
Martin Atkins	74215a4451	stacks: Include resource type schema in PlannedChange and AppliedChange ... Unfortunately for historical reasons the "terraform" package produces state and plan artifacts that have dynamic data elements pre-encoded in the formats that Terraform CLI's traditional plan and state formats use. Since Stacks uses different plan and state models, and exclusively uses MessagePack encoding for dynamic values in both, we'll sometimes need to transcode from one format to another when generating and decoding the stacks-oriented representations. Transcoding between these serialization formats requires access to the relevant schema because the formats themselves each have an infoset that is only a subset of Terraform's type system. Therefore we'll now annotate the PlannedChange and AppliedChange objects that include provider-specific data types with the schema required to transcode them. This commit does not yet actually arrange to make use of those schemas. That will follow in later commits.	2 years ago
Martin Atkins	48518b8233	stacks: A very basic stub of applied resource instance change announcements ... We'd like to start developing some clients for this part of the RPC API concurrently with remaining work here in Terraform Core, and so this is a bare-minimum implementation of emitting "applied change" events for resource instances just so there's something for client developers to test against. There are various things wrong with this, including but not limited to: - It always reports that the new state for a resource instance is an empty object, rather than including the correct updated object. - It only supports "current" resource instance objects, and ignores deposed ones. - It just uses the resource instance address alone as the key for the state description map, which is fine for now when we only have one kind of description object anyway but will not be sufficient for a real implementation that needs to emit various different kinds of object. We'll need to rework most of this in future commits, but this is hopefully sufficient to start implementing and testing API clients, at least to some extend, despite the crudity of the results.	2 years ago
Martin Atkins	d99985720f	stacks: Carry input variable values from plan to apply ... We need to preserve the input variable values provided during the plan phase so we can reuse them during the apply phase to recalculate derived results based on new information learned during the apply process. We also need to recalculate the input values for individual components during the apply phase, so we can react to previously-unknown values that come from other upstream components.	2 years ago
Martin Atkins	b8923a5f94	stackruntime: Persist a component's plantimestamp in the plan ... We need to preserve the module runtime's chosen plan timestamp through to the apply phase so that plantimestamp function calls during the apply phase will return the same result as during the planning phase. This also means that we will now no longer emit a "component instance" planned change object if the planning operation fails so much that there is no plans.Plan object returned, but that's at least consistent with the fact that we also don't report any resource instance changes in the same situation. Event emission is always best-effort when the result has errors.	2 years ago
Martin Atkins	f34633577f	stackplan: LoadFromProto must allocate its component map ... Previously this was panicking later on in the function when trying to insert into a nil map.	2 years ago
Martin Atkins	3c5a2ea7e5	stackplan: Use comparable version strings for plan version checking ... Previously we were comparing version.String() with version.SemVer.String(), which doesn't match because the latter excludes any prerelease version suffix.	2 years ago
Martin Atkins	fafa36e73a	stackeval: ApplyPlan function and initial supporting internals ... This is a first pass at actually driving the apply phase through to completion, using the ChangeExec function to schedule the real change operations from the plan and then, just like for planning, a visit to each participating object to ask it to check itself and report the results of any changes it has made. Many of our objects don't have any external side-effects of their own and so just need to check their results are still valid after the apply phase has replaced unknown values with known values. For those we can mostly just share the same logic between plan and apply aside from asking for ApplyPhase instead of PlanPhase. ComponentInstance and OutputValue are the two objects whose treatment differs the most between plan and apply, because both of those need to emit externally-visible "applied change" objects to explain their side-effects to the caller. The apply-walk driver has a lot of behavior in common with the existing plan-walk driver. For this commit we're just accepting that and having two very similar pieces of code that differ only in some leaf details. In a future commit we might try to generalize that so we can share more logic between the two, but only if we can do that without making the code significantly harder to read.	2 years ago
Martin Atkins	e66f4aacdb	stackplan: Include schema correlation info in resource change descriptions ... In order to find the appropriate schema to use when decoding the dynamic values for a particular resource instance, callers will need to know which provider's schema to consult, and which resource type and mode to consult within that provider's schema. We don't want callers to try to implement their own parsers for Terraform's resource address syntax, and the provider information isn't available any other way anyway, so we'll expose the three required lookup keys as new fields in the events that report that a resource instance change has been planned.	2 years ago
Alisdair McDiarmid	54062a52cb	stacks: More extensive in-progress plan events ... Expand the existing hooks to emit events throughout the planning process, providing enough information for the Terraform Cloud UI to render a live-updating representation of the plan. We also sketch out the equivalent hooks for the apply operation.	2 years ago
Martin Atkins	d3e93e4a31	stackruntime: Initial stubbing of Plan ... This is just enough implementation of "Plan" for embedded stacks with their own input variables and output values, and exposing the final top-level output values as "planned changes". This still doesn't really do anything end-user-useful but it's helpful to have the expression evaluation and plan walk driver machinery in place before moving on to the main useful language element: component blocks. Support for components will follow in subsequent commits.	2 years ago
Martin Atkins	a4ca5f687c	stackplan: Expose the applyable flag as external description ... Previously we retained the applyable flag only in the raw plan sequence, but callers of PlanStackChange also need to know whether the plan they've generated is applyable so that they can e.g. skip showing the plan approval UI. To support that we'll also expose the applyable flag as an externally-consumable PlannedChange.Description variant, just encoding the boolean result directly since there is no other metadata to represent in this case.	2 years ago
Martin Atkins	ba8af34679	stackplan: Stub of runtime models for representing plans ... This package includes both some model types to represent plans (and parts thereof) in memory, and logic for translating between the in-memory representation and our protocol-buffers-based storage serialization. Unlike the traditional saved plan file format, the stacks model for planning is slightly asymmetrical: while constructing the plan the system will emit chunks of the overall plan piecemeal as PlannedChange values, streamed to the rpcapi client. When reloading that plan to apply it, the client must provide all of the raw plan blobs in sequence which we can then decode one by one to construct a single Plan object covering the entire effect of the plan. Another intentional asymmetry is that PlannedChange includes enough information to produce both the raw internal plan representation and the public-facing plan description, whereas Plan only includes the subset of information actually required for Terraform Core to apply the plan, from the raw internal representation exclusively.	2 years ago
Martin Atkins	d857c75e33	stackplan: Initial models for a "meta-plan" over an entire stack	2 years ago