354 Commits (da74b59ea1d3aa4d7706579c95f63c778f2e515e)

Author	SHA1	Message	Date
Martin Atkins	c57f709b1d	stackruntime: Pass the apply-time variable values to stackeval ... This seems to have got lost while resolving some conflicts in 88eec94896. Without this, apply-time input variable values just get ignored entirely and if any are required the plan is not applyable at all. Now we'll pass through what the caller provided so stackeval can validate and use it properly.	2 years ago
Liam Cervante	b646dff26a	stacks: apply nested default values to inputs (#35349) ... * stacks: apply nested default values to inputs * group similar tests	2 years ago
Liam Cervante	284ce63947	stacks: applyable status should be dependent on applyable components (#35350)	2 years ago
Martin Atkins	1714729f87	stackeval: terraform.applying symbol ... This symbol produces an ephemeral boolean value that's true when the stack evaluator is configured for applying and false otherwise. The primary purpose of this is to allow specifying a more privileged auth role in a provider configuration during the apply phase, while allowing the plan phase to use a read-only or otherwise-less-privileged role.	2 years ago
Martin Atkins	e74896bd24	stackruntime: Handle apply-time-specified input variables ... When the topmost stack configuration declares an ephemeral input variable, its values must be provided separately for each of the plan and apply phases. Therefore here we extend the API to allow specifying input variable values during the apply phase, and add rules to check whether all of the apply-time-required input variables have been specified and whether any non-ephemeral variables are either unspecified or re-specified with equal values during the apply phase. This also extends the FindStackConfigurationComponents response to include more metadata about the input variables and output values so that a caller can know which ones are ephemeral. The name of that RPC function had already become a little too specific with the inclusion of embedded stack information and is even moreso now; we might choose to rename it to something more generic like "AnalyzeStackConfiguration" in future, but that'd be a breaking change and therefore requires more coordination.	2 years ago
Martin Atkins	79f78b268f	stackeval: Ephemeral output values ... Now we'll mark the result of an ephemeral output value as ephemeral, and reject attempts to assign any ephemeral values to a non-ephemeral output value.	2 years ago
Martin Atkins	71749d9682	stackeval: Ephemeral input variables ... We'll now handle input variables that are declared as ephemeral = true, and reject attempts to assign ephemeral values to any variables that are not so declared.	2 years ago
Liam Cervante	517f9944f7	stacks: support deferred imports in stacks events (#35334) ... * stacks: support deferred imports in stacks events * fix indentation * fix tests	2 years ago
Liam Cervante	108bd85053	Refactor deferred variable name to defer in terraform1 API (#35343) ... * Refactor deferred variable name to defer in terraform1 API * fix tests	2 years ago
Mutahhir Hayat	d398fa50c6	handle overlooked error in test	2 years ago
Mutahhir Hayat	cb28bf3379	remove unused test	2 years ago
Mutahhir Hayat	e20891ac1d	add tests for local values through data sources	2 years ago
Liam Cervante	9b95898f63	stacks: emit events for deferred actions (#35325) ... * stacks: emit events for deferred actions * deferral allowed is always on * Update internal/rpcapi/stacks.go Co-authored-by: Daniel Schmidt <danielmschmidt92@gmail.com> --------- Co-authored-by: Daniel Schmidt <danielmschmidt92@gmail.com>	2 years ago
Liam Cervante	1bb78c14c2	deferred actions: always enable deferred actions in stacks (#35321)	2 years ago
Liam Cervante	fc390038f3	stacks: add tests for and refactor steps when a provider can't be configured (#35294)	2 years ago
Liam Cervante	eeb2c53385	stacks: remove wildcard key from computed instances (#35284) ... * stacks: remove wildcard key from computed instances * fix compliation error	2 years ago
Liam Cervante	dc06717f69	stacks: add support for unknown for_each expressions in providers (#35267) ... * stacks: add support for unknown for_each expressions in providers * return more precise values for the unknown provider	2 years ago
Liam Cervante	1e9a186059	stacks: return dynamic types when referencing partial components (#35254)	2 years ago
Liam Cervante	7096ec85cc	stacks: Enable resource deferrals within stack tests (#35168)	2 years ago
Daniel Schmidt	59ead5356f	stacks: update RPC APIs with deferral information (#35125) ... * stacks: add deferred resource instance planned change to protobuf * stacks: add deferred resource instance to stack plan sequence * stacks: add planned change for deferred actions * stacks: refactor planned change resource instance planned moving the components out of the main function definition so that we can reuse the implementation for deferred resource instances which wraps the message used for PlannedChangeResourceInstancePlanned * stacks: track deferred changes in stackplan * add simple tests * fix tests * address comments --------- Co-authored-by: Liam Cervante <liam.cervante@hashicorp.com>	2 years ago
Alisdair McDiarmid	c8eec89c3d	Merge pull request #35247 from hashicorp/alisdair/rpcapi-stop ... rpcapi: Allow stopping long-running operations	2 years ago
Liam Cervante	b5d6b13b3b	stacks: convert sourcebundle source into directory for config SourceDir field (#35245)	2 years ago
Martin Atkins	5003af8335	stackeval: If context is cancelled, ask provider instances to stop ... Co-authored-by: Alisdair McDiarmid <alisdair@users.noreply.github.com>	2 years ago
Alisdair McDiarmid	39373cc0d6	stackeval: Stop operations on context cancellation ... When our context is cancelled, we instruct the modules runtime to stop. This aborts the graph traversal after any current node operations are completed.	2 years ago
Martin Atkins	9448845e91	stackeval: Improve coverage of namedPromiseReporter impls ... The "namedPromiseReporter" interface is a helper we use to collect user-friendly names for promises when we need to report a promise self-reference error to the user. We collect names for promises reactively on error, rather than proactively in the normal course of work, because proactively maintaining a suitable data structure for this cross-cutting concern would be both expensive and complicated due to it being a cross-cutting concern. This compromise therefore allows us to pay the cost of walking over all of the promise-owning objects only in the exceptional case where we need to report a self-reference error, but it does come at the expense of it being possible for us to forget to implement this interface for certain types and thus be unable to name a subset of the affected promises. This commit implements reportNamedPromises for all promise-owning types that are reachable from a stackeval.Main, although there is a notable gap here where we can't actually report promises from the individual instances of a multi-instance object (e.g. a component) because the promising API doesn't let us peek into a promising.Once to see if it's resolved without blocking until it becomes resolved. There are FIXME comments for that limitation in all of the places where it's relevant, and so hopefully in a future commit we'll devise a suitable way to perform a fallible non-blocking peek of a promise so that we can report promises nested in its result only if they are already present.	2 years ago
Martin Atkins	a4d19a8d13	stackeval: Stack.ProviderByLocalAddr must memoize its results ... A Provider object owns promises for related work, so we need to make sure we treat these as singletons to ensure that the associated work gets done only once and all callers end up sharing the same result. We've got away with this so far because the "related work" for providers is all deterministic based on the configuration anyway, and so without this we're just inefficient (doing the same work multiple times) rather than incorrect (producing inconsistent results). However, we should preserve the typical contract that objects containing promises are always treated as singletons, because otherwise this inconsistency will be a maintenance hazard for anyone who (reasonably) expects objects of this type to be treated the same as objects of other promise-owning types.	2 years ago
Liam Cervante	7a8ffff66e	stacks: Fix unknown provider values crashing validation phase (#35155) ... * test to reproduce * stacks: Fix unknown provider values crashing validation phase * remove extra type information --------- Co-authored-by: CJ Horton <cjhorton@hashicorp.com>	2 years ago
Daniel Schmidt	2c65affe05	stacks: add comments to tests	2 years ago
Daniel Schmidt	b1e50d0475	chore: fix comment ... Co-authored-by: Nick Fagerlund <nick.fagerlund@gmail.com>	2 years ago
Daniel Schmidt	664bb21738	stacks: add comment ... Co-authored-by: Nick Fagerlund <nick.fagerlund@gmail.com>	2 years ago
Mutahhir Hayat	6bbc8cdc76	Add more tests for local values	2 years ago
Mutahhir Hayat	2667a0f2c3	cleanup comments	2 years ago
Mutahhir Hayat	5e12bdf048	Stack evaluation should also visit local values when walking the stack	2 years ago
Mutahhir Hayat	be317d4d3a	Make Stack and Stack config aware of local values	2 years ago
Mutahhir Hayat	6554ce6bfd	Add planning test to test for locals usage	2 years ago
Mutahhir Hayat	129d58fed3	Create local value structs, including config and tests	2 years ago
Mutahhir Hayat	d695f8197c	Add local values test	2 years ago
Daniel Schmidt	3267718d3d	stacks: support unknown for_each on embedded stacks	2 years ago
kmoe	b74715e5fb	stacks: fix invalid filename (#35111)	2 years ago
Daniel Schmidt	35f44b01f0	stacks: fix comment and add assertion for diagnostics	2 years ago
Daniel Schmidt	aeb5b47723	stacks: remove unused test case	2 years ago
Daniel Schmidt	f847360a40	stacks: component.CheckInstances now reports on for each value diagnostics ... So there is no need for checkValid to do this during th walkDynamicObjects phase	2 years ago
Daniel Schmidt	c370e5e71d	stacks: use more precise name for variable	2 years ago
Daniel Schmidt	1afc9789ef	stacks: use UnknownForEachRepetitionData for instance map repetition data in for_each	2 years ago
Daniel Schmidt	91a18a6b30	stacks: fix typo ... Co-authored-by: Liam Cervante <liam.cervante@hashicorp.com>	2 years ago
Daniel Schmidt	7d0c4cfc4d	stacks: early exit on invalid for_each values for components	2 years ago
Daniel Schmidt	d1168ffa62	stacks: return instance value for wildcard component instances	2 years ago
Daniel Schmidt	6f95225a91	stacks: handle unknown dynamic values	2 years ago
Daniel Schmidt	3cbfd39721	stacks: validate unknown component for_each type	2 years ago
Daniel Schmidt	3a7150b27e	stacks: handle unknown values in component for_each	2 years ago
Martin Atkins	30e2fd6525	Handle marks a little more consistently ... In the very first implementation of "sensitive values" we were unfortunately not disciplined about separating the idea of "marked value" from the idea of "sensitive value" (where the latter is a subset of the former). The first implementation just assumed that any marking whatsoever meant "sensitive". We later improved that by adding the marks package and the marks.Sensitive value to standardize on the representation of "sensitive value" as being a value marked with _that specific mark_. However, we did not perform a thorough review of all of the mark-handling codepaths to make sure they all agreed on that definition. In particular, the state and plan models were both designed as if they supported arbitrary marks but then in practice marks other than marks.Sensitive would be handled in various inconsistent ways: dropped entirely, or interpreted as if marks.Sensitive, and possibly do so inconsistently when a value is used only in memory vs. round-tripped through a wire/file format. The goal of this commit is to resolve those oddities so that there are now two possible situations: - General mark handling: some codepaths genuinely handle marks generically, by transporting them from input value to output value in a way consistent with how cty itself deals with marks. This is the ideal case because it means we can add new marks in future and assume these codepaths will handle them correctly without any further modifications. - Sensitive-only mark preservation: the codepaths that interact with our wire protocols and file formats typically have only specialized support for sensitive values in particular, and lack support for any other marks. Those codepaths are now subject to a new rule where they must return an error if asked to deal with any other mark, so that if we introduce new marks in future we'll be forced either to define how we'll avoid those markings reaching the file/wire formats or extend the file/wire formats to support the new marks. Some new helper functions in package marks are intended to standardize how we deal with the "sensitive values only" situations, in the hope that this will make it easier to keep things consistent as the codebase evolves in future. In practice the modules runtime only ever uses marks.Sensitive as a mark today, so all of these checks are effectively covering "should never happen" cases. The only other mark Terraform uses is an implementation detail of "terraform console" and does not interact with any of the codepaths that only support sensitive values in particular.	2 years ago
Nick Fagerlund	4d46cc81e3	Fix several deferred actions tests by setting DeferralAllowed	2 years ago
Nick Fagerlund	2b4147e0ba	Add DeferralAllowed to stackeval.PlanOpts (temporary) ... Ultimately, we want deferred actions to always be enabled for Stacks. However, it's not ready to turn on yet, as it currently regresses other features. So, adding a top-level option that flows through to the main terraform.PlanOpts field will allow us to enable it selectively in tests while we sort out various behaviors.	2 years ago
Nick Fagerlund	d5be799cb9	End unknown_instances language experiment, in favor of whole-plan option ... We're still iterating on the ability to defer unplannable changes to a later plan. However, instead of enabling it via a "contagious" experimental feature switch that can bubble up from any module, it's now behind a per-plan option in the PlanOpts struct. That option travels a winding path into the internals (see previous commit), and node execution logic can then access it during the plan walk via the `Deferred.DeferralAllowed()` method. This commit updates the spot where we switch between the deferral yea/nay code paths to respect this top-level plan option rather than the experiment flag.	2 years ago
Alisdair McDiarmid	f409639934	Merge pull request #34765 from hashicorp/alisdair/stack-root-input-variable-defaults ... stackruntime: Apply defaults to root variables	2 years ago
Alisdair McDiarmid	f3ec86b17b	stackruntime: Treat unset and null equally ... When handling root input variable values, we now consider unset and null values to be equivalent to each other. This is consistent with how we handle variables in embedded stacks, and very similar to how we handle variable in the modules runtime with `nullable = false`. One difference from the modules runtime case is that we do not prevent a null default value for stack variables.	2 years ago
Alisdair McDiarmid	0fe26468cd	stackruntime: Apply defaults to root variables ... When evaluating a stack's root input variables, supplied by the caller, we must apply any default values specified in the variable configuration for variables with no specified value. This commit adds this default fallback case, using NilVal as a marker indicating the lack of a specified value. If no default value exists for a variable, it is therefore required to be supplied by the caller. This commit also reports a diagnostic error in this case.	2 years ago
CJ Horton	5a750561e6	round-trip check results through stack plans ... Components expect that check results will be round-tripped through the plan in order to update unknown check results during the apply. We hadn't wired this up for stack plans, resulting in panics when the apply tries to update a check status that doesn't exist.	2 years ago
Daniel Schmidt	49ad27f1c9	Merge pull request #34737 from hashicorp/stacks-for-each ... stacks: make component for_each more robust	2 years ago
Nick Fagerlund	c9c24706aa	Merge pull request #34741 from hashicorp/nf/feb24-provider-addrs-in-statuses ... Stacks, Hooks: Include full provider addresses	2 years ago
Daniel Schmidt	2a0962328d	stacks: make error message more precise	2 years ago
Daniel Schmidt	8da61e2d20	stacks: reword for_each error message	2 years ago
Daniel Schmidt	217879e510	stacks: make component for_each more robust	2 years ago
Liam Cervante	cec4b7b74c	stacks: validate providers based on types instead of local names (#34735) ... * stacks: validate providers based on types instead of local names * fix tests	2 years ago
Nick Fagerlund	f58c71e77c	Stacks: Update expectations in TestTerraformHook	2 years ago
Nick Fagerlund	7703a5aead	Stacks: Add ProviderAddr field to ResourceInstanceStatusHookData ... This updates the stackruntime `hooks.ResourceInstanceStatusHookData` struct to include the provider address, and updates everything that instantiates that struct to pass along the valid provider address it received from its own caller. In other words, this commit is a bridge between the terraform.Hook interface methods (which already have access to the provider address) and the stacks hook callbacks that result in RPC messages being sent to the agent.	2 years ago
Nick Fagerlund	94ea534e40	Refactor terraform.Hook to use a resource-identifying wrapper struct ... The terraform.Hook interface lets other areas of code perform streaming reactions to various events, generally in the service of some UI somewhere. Nearly all of the methods on this interface take an `addrs.AbsResourceInstance` as their first argument, to identify the resource that's being operated on. However, that addrs struct doesn't necessarily contain everything you might want in order to uniquely and usefully identify a resource. It has the module instance and resource instance addresses, but it lacks the provider source address, which can affect how the consuming UI should display the resource's events. (For example, Terraform Cloud wants reliable info about who maintains a given provider, what cloud provider it operates on, and where to find its documentation.) Instead of polluting `addrs.AbsResourceInstance` with extra information that isn't relevant to other call sites, let's change the first argument of each Hook method to be a wrapper struct defined in the package that owns the Hook interface, and add the provider address to that wrapper as a sibling of the resource address. This causes a big noisy commit today, but should streamline future updates to the UI-facing "identity" of a resource; existing callers can ignore any new fields they're uninterested in, or exploit new info as needed. Other than making new information available for future edits to Hook implementing types, this commit should have no effect on existing behavior.	2 years ago
Liam Cervante	31a7fa88d5	stacks: validate provider configurations during static analysis (#34730)	2 years ago
Alisdair McDiarmid	1459825e53	Merge pull request #34725 from hashicorp/alisdair/configs-sourcebundle-parser ... Add and use a source bundle aware configuration parser for stacks	2 years ago
Alisdair McDiarmid	2c22c213e9	stackeval: More robust absolute source addrs ... Instead of relying on the module call's source being unique, we now use the entire path as the key for looking up the parent's absolute source address.	2 years ago
Alisdair McDiarmid	7a043fe54a	stackeval: Another module source test case ... If multiple submodules of a given component share the same relative source, they should result in distint source addresses in the diagnostics. This commit introduces an example with the following structure: - component.remote_invalid_grandchildren - module.first_child - module.child (source = "./child") - module.second_child - module.child (source = "./child") Both of these use the same invalid module as the rest of the examples in this test, so we should (and do) see equivalent diagnostics differing only in filename.	2 years ago
Alisdair McDiarmid	7d956bbc11	stackeval: Fix module loader source addr building ... The source bundle aware module loader requires "absolute" source addresses, which are fully qualified rather than relative. We generate these during the module loading process. The previous implementation assumed that any local module source address should be parented by the root module source, but this is incorrect when a descendant module targets a remote or registry source. This commit addresses this by tracking each module request's generated absolute source address, and using it as the base for any descendant local module requests.	2 years ago
Martin Atkins	a45d4467f1	stackeval: Don't fail when data resource is removed ... Due to an oversight in our handling of resource instance objects that are neither in configuration nor plan -- which is true for data resources that have since been removed from the configuration -- we were generating plan change objects that were lacking a provider configuration address, which made them syntactically invalid and thus not reloadable using the raw plan parser. This is a bit of a strange situation since we don't technically _need_ a provider configuration address for these; all we're going to do is just unceremoniously delete them from the state during apply anyway. However, we always have the provider configuration address available anyway, so adding this in here is overall simpler than changing the parser, the models it populates, and all of the downstream users of those models to treat this field as optional. This commit is more test case than it is fix, since the fix was relatively straightforward once I had a test case to reproduce the problem it's fixing.	2 years ago
Liam Cervante	b2cc7dbadf	stacks: validate the types of input variables during validation (#34722)	2 years ago
Liam Cervante	68fd15dfdd	stacks: expand reference scope of static validations (#34721)	2 years ago
Liam Cervante	831630fabe	stacks: the plan function should stop on validation errors (#34720)	2 years ago
Alisdair McDiarmid	97a220fb82	stacks: Use source bundle aware config parser	2 years ago
Liam Cervante	b3abff5750	stacks: check providers blocks in components during validatation (#34707)	2 years ago
Liam Cervante	bced645a4d	stacks: call terraform.Validate on stack component configs (#34705)	2 years ago
Liam Cervante	2472f7c773	stacks: fix flaky tests (#34708)	2 years ago
Alisdair McDiarmid	48148eda5f	Merge pull request #34687 from hashicorp/alisdair/stackeval-hook-replace-action-counts ... stackeval: Add support for replace actions to hook	2 years ago
Liam Cervante	64b311d53a	stacks: apply sensitive marks to outputs from components (#34684) ... * stacks: apply sensitive marks to outputs from components * gen headers * ensure consistent test ordering	2 years ago
Liam Cervante	f0b61090a2	Move the testing MockProvider into a dedicated shared package (#34681) ... * move the testing MockProvider into a dedicated shared package * address comments * fix new changes	2 years ago
Alisdair McDiarmid	06723d3d4b	stackeval: Add support for replace actions to hook ... The terraform.Hook implementation in stackeval is used to track the operations performed during stack runtime operations, for later reporting to the caller. This hook did not correctly support the replace actions (create-then-destroy, destroy-then-create), resulting in a loss of data between plan and apply. This exhibited as a plan with a replace reporting 1 add and 1 remove operation, which when applied would report only 1 add operation. Previously, we stored the action performed for each apply operation on a given resource instance in the PreApply hook, then allowed access to it via the ResourceInstanceObjectAppliedAction method. Here we extend the PostApply hook to look for a previous action performed on this instance, and use that to reconstruct the planned replace action. This method is called in the stackruntime package, where replace operations are counted for both add and remove. No changes are needed at the call site to fix the bug.	2 years ago
Martin Atkins	fde6277565	stackruntime: Pass ExperimentsAllowed into stackeval for apply ... This was originally part of 7dad938fdb but unfortunately seems to have got lost during some rebasing, or some other similar sort of annoying reason. This now allows the "experiments allowed" flag to propagate into the stackeval package when we're running the apply phase, for consistency with all of the other phases. Without this, it's possible to plan a configuration that's participating in experiments, but then it fails in a strange way during the apply step due to Terraform suddenly thinking it's a stable release where experiments are disabled.	2 years ago
Liam Cervante	07f6621091	stacks: include resources in state when calculating required providers (#34645) ... * stacks: include resources in state when calculating required providers * also support apply time * add copywrite headers	2 years ago
Martin Atkins	e6665c4f43	stackeval: Basic support for deferred-change propagation ... This is the bare minimum functionality to ensure that we defer all actions in any component that depends on a component that already had deferred actions. We will also eventually need to propagate out a signal to the caller for whether the stack plan as a whole is complete or incomplete, but we'll save that for later commits, since the stack orchestration in Terraform Cloud will do the right thing regardless, aside from the cosmetic concern that it won't yet know to show a message to the user saying that there are deferred changes.	2 years ago
Martin Atkins	67d8a5137b	rpcapi+stacks: Stacks runtime can see whether experiments are allowed ... We allow experiments only in alpha builds, and so we propagate the flag for whether that's allowed in from "package main". We previously had that plumbed in only as far as the rpcapi startup. This plumbs the flag all the way into package stackeval so that we can in turn propagate it to Terraform's module config loader, which is ultimately the one responsible for ensuring that language experiments can be enabled only when the flag is set. Therefore it will now be possible to opt in to language experiments in modules that are used in stack components.	2 years ago
Liam Cervante	3475d22741	Fix nil pointer exception when planning stacks with undeclared variables (#34638) ... * fix nil pointer exception when planning with undeclared variables * another function does the same	2 years ago
Alisdair McDiarmid	55675b990a	stackeval: Unmark provider config for plugin RPC ... When validating or applying a provider configuration, we must unmark the value before attempting to use plugin RPC, as we cannot serialize marks.	2 years ago
Martin Atkins	884e1fb2a4	terraform: Plans can be "complete" and "applyable" ... These ideas are both already implied by some logic elsewhere in the system, but until now we didn't have the decision logic centralized in a single place that could therefore evolve over time without necessarily always updating every caller together. We'll now have the modules runtime produce its own boolean ruling about each characteristic, which callers can rely on for the mechanical decision-making of whether to offer the user an "approve" prompt, and whether to remind the user after apply that it was an incomplete plan that will probably therefore need at least one more plan/apply round to converge. The "Applyable" flag directly replaces the previous method Plan.CanApply, with equivalent logic. Making this a field instead of a method means that we can freeze it as part of a saved plan, rather than recalculating it when we reload the plan, and we can export the field value in our export formats like JSON while ensuring it'll always be consistent with what Terraform is using internally. Callers can (and should) still use other context in the plan to return more tailored messages for specific situations they already know about that might be useful to users, but with these flags as a baseline callers can now just fall back to a generic presentation when encountering a situation they don't yet understand, rather than making the wrong decision and causing something strange to happen. That is: a lack of awareness of a new rule will now cause just a generic message in the UI, rather than incorrect behavior. This commit mostly just deals with populating the flags, and then all of the direct consequences of that on our various tests. Further changes to actually make use of these flags elsewhere in the system will follow in later commits, both in this repository and in other repositories.	2 years ago
Martin Atkins	0994e6fbf9	stacks: The terraform.workspace attr is not available in Stacks ... The terraform.workspace attribute is a rare example of a CLI- and Cloud- specific concern bleeding into the Terraform language, and it can only really have meaning when used in the traditional Terraform workflow because otherwise there's no workspace to return the name of. In Stacks any variations between instances of a module must be created through input variables. Within Terraform Cloud in particular it's also possible to use stack-level input variables that are assigned different values from different stack deployments, and thus an author can recreate the effect of terraform.workspace using a stack-level input variable that has a different value for each deployment. This is one of the few cases where the Terraform module language differs in stacks compared to traditional Terraform. Any module that makes use of terraform.workspace will need to be generalized to use input variables instead before it can be used within a stack component. Prior to this change, references to terraform.workspace from a module used in a stack component would just panic altogether, because the stacks runtime doesn't provide the object that the workspace name would be taken from. Now we'll return a user-oriented error instead.	2 years ago
Martin Atkins	7d2d4dec5e	stackeval: When being destroyed, component instance result comes from plan ... Because we treat dependency edges as reversed when a component instance is being destroyed, the final result (an object representing output values) for a component instance being destroyed must not depend on anything else in the evaluation graph, or else we'd cause a promise self-reference as the downstream component tries to configure itself based on our outputs. As a special case then, for a component instance being destroyed we take the planned output values directly from the plan, relying on the fact that the plan phase sets them to the prior state output values in that case, and therefore the result for such a component is available immediately without blocking on any other expression evaluation during the apply phase. This combines with several previous commits to create a first pass at handling ordering correctly when planning and applying a full destroy. This commit also incorporates some fixes and improvements to stackeval's apply-time testing helpers, which had some quirks and bugs when first added in a recent commit. One of those problems also revealed that the raw state loader was not resilient to a buggy caller setting a state entry to nil instead of removing it altogether, and that mistake seems relatively easy to make (as I did here in the test helper) so we'll tolerate it to make it possible to recover if such a bug does end up occurring in real code too.	2 years ago
Martin Atkins	dcb176a557	stackeval: A Main for applying knows the plan it's applying	2 years ago
Martin Atkins	c06ad051a1	stackeval: Respect inter-component dependencies during apply ... Before we start any apply-time actions for a particular component, we'll block until other components' apply are complete. In the normal case a component instance waits until its dependencies have been applied. However, if component instances are being destroyed then they instead wait for their _dependents_ to complete being applied, since a component must outlive all other components that depend on it.	2 years ago
Martin Atkins	c345cde93c	stackeval: Extended utilities for plan and apply testing ... These helpers make it easier to write our tests because the plan and apply phases both work in a callback-based fashion where they gradually emit events, and that's not a very convenient API for test code to interact with. This exposes some more details about the planning results, and also adds a new similar helper for the apply phase. We'll make use of both of these in future commits.	2 years ago
Liam Cervante	9246b3a994	stacks: on module load convert relative paths to absolute (#34523)	2 years ago
Liam Cervante	7d143388f3	Add support for the cross resource type move operation in the proto schema (#34480) ... * Update proto schema and provider interfaces with support for moved across resource type RPCs * address comments * remove unused functions * remove support for flatmap format	2 years ago
Martin Atkins	3c14eeb945	stackeval: Make some provisioners available to stack components ... This makes the built-in "remote-exec" and "file" provisioners available for use in the modules that implement stack components. These are both relatively easy and low-risk to include because they are builtins and don't require anything from outside of Terraform itself. For now this intentionally excludes local-exec because we'll want to think about what constraints we want to put on it, if any, to help ensure we can meet the goal of stack configurations being portable between different execution environments without significant modification, and our current stacks execution environment doesn't guarantee the availability of any external software _at all_. The motivation for adding this now is just to give some better feedback when someone uses a module using one of these provisioners, since otherwise they'll see just a confusing generic error message from the modules runtime about the provisioners not being available. I expect we'll revisit this later and consider expanding it to at least include local-exec, and _maybe_ external provisioner plugins, although that's more questionable because the provisioner plugin mechanism is incredibly legacy and doesn't have any way to work outside of local Terraform CLI usage today. There are no tests here yet because these provisioners are not mockable and would depend on having an SSH or WinRM server to connect to. Later we should ponder how to make this more testable, which might mean making another part of the system responsible for actually providing the provisioner factories and thus our tests here can use fakes. The goal here is just to get this done in a relatively lightweight way for better feedback during preview though, so we're not yet ready to make significant time investments here.	2 years ago
Alisdair McDiarmid	1d3f863f2b	stackruntime: Support sensitive component inputs ... Components can emit sensitive values as outputs, which can be consumed as inputs to other components. This commit ensures that such values are correctly processed in order to pass their sensitivity to the modules runtime.	2 years ago
Alisdair McDiarmid	e4b319401e	stacksruntime: Add sensitive outputs tests	2 years ago
Martin Atkins	3961c18420	stackeval: Return an error if a module contains a provider config ... When using stacks the provider configurations belong in the stack configuration rather than inline in the individual modules. Shared modules with inline provider configurations has been a deprecated legacy practice for many years now, but traditional Terraform continued to support it for backward-compatibility with older modules despite the significant downsides of doing so. Stacks now finally removes that capability, since it isn't straightforward to continue supporting it once we've made the stacks runtime be responsible for instantiating and configuring providers.	2 years ago
Martin Atkins	a51c034cc3	stackeval: Include component configurations in our static walks ... This means both that the validate walk can now describe static problems in the component's module tree, and that we'll catch such problems earlier in the planning phase and thus avoid reporting them repeatedly in cases where a component block uses for_each to declare multiple instances. This includes a fix to a bug in StackConfig.Components, which was incorrectly using the input variable declarations as the source for its result, instead of the component declarations.	2 years ago
Martin Atkins	a3cd4a11ce	stackeval: Use relative, slashy paths in module-related diagnostics ... Until we've updated the module config loader to be sourcebundle-aware and thus return proper source addresses, this will at least make the paths we show in diagnostics a little less verbose, and more consistent across platforms.	2 years ago
Martin Atkins	a4d61733f6	stackplan: Include extra info in raw component instance plan ... We'll now track the planned action, the component addresses we depend on, and the planned output values.	2 years ago
Martin Atkins	25a514f846	stackeval: Populate planned component instance more thoroughly ... We'll now track a more realistic Action, the components that the component instance depends on, and the planned output values.	2 years ago
Martin Atkins	524da4f278	stackeval: Applyable objects have a set of required components ... To figure out the required order in which we'll apply the plans for all components we need to know how data flows between them. We'll use what's added here during the plan phase to record in the plan which components depend on which other components, and then the apply phase will use that coarse information to schedule the individual component apply operations into a suitable order. None of that is implemented here though, and will follow in future commits. This commit includes new documentation about the apply scheduling approach which includes some mechanisms that are not actually yet included in this commit, but will follow shortly after. This is just a pragmatic compromise to avoid spending time authoring short-lived partial documentation drafts for the intermediate steps while still keeping these increments small enough for practical review.	2 years ago
Martin Atkins	00761a9f70	stackeval: ApplyChecker is now Applyable ... We chose this name originally to communicate that implementations are only responsible for checking that an apply was successful, but in future commits this interface will also gain the additional responsibility of helping to gather the information required to understand the dependency relationships between components that will then drive the scheduling of the apply actions during the apply phase.	2 years ago
Martin Atkins	11bb10ba72	stackeval: Find references in arbitrary expressions and bodies ... Along with full expression/body evaluation, we'll also need to be able to determine what an expression or body refers to without evaluating it, so that we can use references to infer dependency relationships between significant side-effects in the apply phase, such as the relative ordering of applying the changes for different components.	2 years ago
Martin Atkins	306b9008c4	stackeval: Clean some debug junk in ComponentInstance ... I apparently missed this when I was preparing my previous round of commits in this area. Mea culpa.	2 years ago
James Bardin	2574b899e7	Merge pull request #34385 from hashicorp/jbardin/provider-functions-impl ... client-side implementation of provider functions	2 years ago
Matej Risek	039cced8ae	Add component address to ComponentInstance under AppliedChanges (#34418)	2 years ago
James Bardin	2923ba6ac3	add CallFunction to stub provider	2 years ago
Martin Atkins	be33a7e9e0	stackeval: Use prior state outputs when planning component destroy ... When a component is planned for full destroy, we should assume that its apply phase will be delayed until everything that depends on it has been destroyed, since otherwise we'd be violating the dependency relationship. That means that the expression value for a component pending destruction ought to be based on the output values from the prior state (which ought to have been refreshed as part of planning destroy) rather than the desired state (in which case all of the output values would be unset). This commit only deals with the _planning_ part of this problem, and only deals with the situation where we're planning a "full destroy" of the entire stack configuration and therefore all component instances in the configuration are going to be destroyed. In future commits we'll need at least the following additional work to complete this: - The apply phase needs to learn to schedule component instance destroy actions differently than other component instance actions, so that destroy cannot begin until everything that depends on the component has already been destroyed. - We also need to deal with the situation where only a particular component instance has been removed from the configuration, in which case only that one component instance is planned for destroy and everything else is planned as normal. In that case it would be invalid for anything to still be referring to that component instance in the configuration, and so the work in this commit would actually be irrelevant, but it'll take some other separate machinery to make sure that the removed component instances still get planned properly, which they don't today. This commit is just an incremental step toward the final correct behavior, which should allow the planning of a full destroy to succeed, although actually applying that plan is still likely to fail because the apply phase will not respect the required order of operations yet.	2 years ago
Martin Atkins	a47dc447c1	stackeval: Additional testing utilities ... In earlier commits we stubbed out some testing utilities as part of adding some unit tests, but we didn't end up using all of them in the initial round and it turned out that some of them were incorrect or incomplete. Now we'll improve the test utilities, this time with a focus mainly on integration-type tests, because we'll start adding those in the near future. Specifically: - utilities for rendering diagnostics, mainly in situations where they are unexpected and thus we just want to see the full information as part of the test results - helper functions for concisely asserting that there should be no diagnostics at all, or that there should be no error diagnostics - the existing helper for faking up a prior state based on hard-coded raw state messages was incorrect in that it was trying to use statekeys.Key values as map keys, and those values are not typically comparable. Instead, we should use the string representations that are designed to be used as unique map keys. - the testPlan function, and the lower-level testPlanOutput it's built with, help to encapsulate the asynchronous event-emitting behavior of Main.PlanAll into a "normal-looking" function that just blocks until it's done and then returns all of its results. This automatically round-trips the planned changes through serialization and deserialization as we would between a normal plan and apply phase, and so the calling test can make its assertions against the applyable plan object instead of the raw component parts it's made from.	2 years ago
Martin Atkins	2aba28eb1a	stacks: Track component instance existence in the state ... Although we can often use the presence of resource instances in a component instance as an implication of the component instance's existence, it's possible (albeit rare) for a component instance to have no resources in it at all, and thus it would be ambiguous whether it exists or not. Now we'll track the existence of the component instances themselves as extra objects in both the raw state and the state description. Along with their existence we'll also track a snapshot of the output values as they were at the most recent apply, which for now is primarily for external consumption but we'll also include them in the raw state in case it ends up being useful for something down the road.	2 years ago
Martin Atkins	b0b8d4aa6f	states: Only track root module output values ... For a very long time we've had an annoying discrepancy between the in-memory state model and our state snapshot format where the in-memory format stores output values for all modules whereas the snapshot format only tracks the root module output values because those are all we actually need to preserve between runs. That design wart was a result of us using the state both as an internal and an external artifact, due to having nowhere else to store the transient values of non-root module output values while Terraform Core does its work. We now have namedvals.State to internally track all of the throwaway results from named values that don't need to persist between runs, so now we'll use that for our internal work instead and reserve the states.State model only for the data that we will preserve between runs in state snapshots. The namedvals internal model isn't really designed to support enumerating all of the output values for a particular module call, but our expression evaluator currently depends on being able to do that and so we have a temporary inefficient implementation of that which just scans the entire table of values as a stopgap just to avoid this commit growing even larger than it already is. In a future commit we'll rework the evaluator to support the PartialEval mode and at the same time move the responsiblity for enumerating all of the output values into the evaluator itself, since it should be able to determine what it's expecting by analyzing the configuration rather than just by trusting that earlier evaluation has completed correctly. Because our legacy state string serialization previously included output values for all modules, some of our context tests were accidentally depending on the implementation detail of how those got stored internally. Those tests are updated here to test only the data that is a real part of Terraform Core's result, by ensuring that the relevant data appears somewhere either in a root output value or in a resource attribute.	2 years ago
Martin Atkins	b64953dba3	stacks: Summary docs giving an overview and some details about stackeval	2 years ago
Martin Atkins	3f569b55c5	stacks: Some code consistency cleanups in preparation for first merge	2 years ago
Martin Atkins	8a99c3c0e5	stacks+rpcapi: Add some missing copyright comments	2 years ago
Martin Atkins	3678815bb3	stackeval: Tests for provider instance configuration evaluation	2 years ago
Martin Atkins	1872c9d9be	stackeval: inPromisingTask cancels context when test completes ... This is just some extra insurance to reduce the risk of us leaving things dangling after a test terminates. Although tests should ideally not rely on this and should terminate background work properly as part of the test, this should catch some things that might get left behind if a test ends in an ungraceful way.	2 years ago
Martin Atkins	94ca9bc613	stackeval: testEvaluator arranges for its result to be cleaned up ... Some operations against an "evaluator" (type Main) register cleanup actions that need to be run once the evaluation has completed. To avoid leaking resources while we're running tests we'll arrange for any test that uses the "testEvaluator" helper to automatically run the cleanup actions on the returned evaluator once the test is complete. At the time of this commit the only possible cleanup action is terminating any provider pluginc client processes that might be running, and none of our tests are actually doing that so this doesn't really change anything, but it's here to avoid problems if we add additional cleanup actions in future, or if any tests start actually using child plugin processes.	2 years ago
Martin Atkins	d7461ce202	stackeval: Initial tests for type "Provider" ... These tests focus on the behaviors related to deciding the set of dynamic instances for a particular provider and producing the values that act as references to providers. These tests also revealed a small but important bug in ExprReferenceValue in the for_each case, which is now fixed as part of this commit.	2 years ago
Martin Atkins	d3e0b89e02	stackeval: Some initial tests for the Component type ... This includes Component.CheckInstances and Component.CheckResultValue.	2 years ago
Martin Atkins	91b8ea3d76	stackstate: A helper for loading state during tests ... The main entry point here assumes the caller is providing serialized state objects wrapped in anypb.Any messages. That's an inconvenient representation for hand-writing in tests, so the new function LoadFromDirectProto allows skipping the deserialization steps and instead has the caller write the wanted values as if they had already been parsed/unmarshaled.	2 years ago
Martin Atkins	95ff474ee9	stackeval: Tests for StackCall.ResultValue	2 years ago
Martin Atkins	333c502990	stackeval: Support each.key, each.value, count.index and self references ... For all contexts where these have a defined value, they can now be used in expressions and evaluate to whatever they ought to evaluate to.	2 years ago
Martin Atkins	75602fa9eb	stackeval: Tests for StackCall.Instances	2 years ago
Martin Atkins	48d2d15a40	stackeval: Some basic tests for OutputValue.CheckResultValue	2 years ago
Martin Atkins	13edd30479	stackeval: Test-only globals ... Because many stacks language features are dependent on others to do useful work, it's tempting to focus only on integration testing of combinations of features used together. However, we known from our experience with the modules runtime (the "terraform" package) that over time this becomes a huge maintenance burden, because any non-trivial change tends to invalidate hundreds or thousands of integration tests, and because of their broad scope its often hard in retrospect to figure out what exactly a particular test was aiming to test vs. what it was just relying on as a side-effect. To try to minimize these cross-dependencies and thus enable something closer to unit testing, here we introduce a special kind of symbol to the stacks language which is available only to unit tests in this package. "Test-only globals" -- an intentionally-clunky name to avoid squatting on useful names -- can be set as part of the Main object and, when defined, are available for use in all situations where we perform expression evaluation against a stack. The "globals" in the name represents that, unlike just about everything else, they are defined once but available in all stacks in the configuration tree. This design is a tradeoff: it introduces a bunch of extra code that is here entirely to support testing, but hopefully this code is segregated enough from everything else that it's unlikely to change significantly under future maintenance, thereby hopefully minimizing the need for future cross-cutting test maintenance too.	2 years ago
Martin Atkins	bb0e833785	stackeval: Remove stale comment about output value changes ... This comment was originally assuming we'd need a separate field to track the type information for the serialized output values, but we actually don't need that because the type information is encoded in-band in the MessagePack representation of the values.	2 years ago
Martin Atkins	0281170645	stackeval: Remove stale todo about prior state in plans ... This TODO was noting that an earlier version of this code was not including any prior state information in the "modules runtime" plan. At the time I wrote the comment I was expecting to handle that here as a separate step, but I later refactored this to include the prior state as part of the returned plan, since that's where the modules runtime expects to find it during the apply phase anyway. This comment is therefore now redundant, because no extra work is required in this location to get the prior state populated.	2 years ago
Martin Atkins	992f6d4115	stackeval: Fix input variable evaluation in embedded stacks ... Two bugs were making this not work quite right: - The Stack type wasn't treating its child stack objects as singletons, instead creating a new one on each request and therefore preventing child objects from being treated as singletons too. - StackCallInstance.CheckInputVariableValues wasn't actually using the value from its expression evaluation, and was instead just always returning an empty object. This commit also includes the tests that helped find these bugs.	2 years ago
Martin Atkins	9a943af56d	stackruntime: Arbitrary expression evaluation ... Here we introduce a new evaluation phase for the stacks runtime called "inspect", which aims to provide a read-only view of a configuration and an associated state snapshot. The main idea of this is to use it as a more convenient vehicle for unit testing despite it not really being viable to decompose the main evaluation logic into smaller parts: the language features are all directly interdependent on one another, but we can at least minimize our interactions with the Terraform modules runtime when we're testing anything other than actually planning and applying components. However, this also gives an opportunity to expose evaluation of arbitrary expressions in a similar manner as "terraform console", which is for now just another debugging aid but will probably someday appear as a Stacks-flavored version of the "terraform console" command.	2 years ago
Martin Atkins	d689237a48	stackeval: Tests for the "perEvalPhase" helper	2 years ago
Martin Atkins	5aaed3111e	stackeval: Reusable and testable for_each handling ... Previously we had some similar code duplicated into each of the three kinds of object that support for_each: components, embedded stack calls, and provider configurations. Instead, we'll hoist that code out into a function that all three can call. We can then actually test that function in isolation (independently of the three callers) to get more confidence that future maintenance won't inadvertently break the slightly-unusual contract of treating a nil map differently from a non-nil empty one, which is unfortunately likely to defy the expectations of a typical Go developer since those two values are normally considered equivalent.	2 years ago
Martin Atkins	1180096850	stackeval: Unit tests for EvalExpr and EvalBody	2 years ago
Martin Atkins	3c6c5effd5	stackeval: Applying must use apply-time provider clients ... Previously we were incorrectly trying to use the plan-phase provider clients during the apply phase, which causes the provider to get configured as it would during the plan phase and, in particular, crashes if the provider configuration refers to input variables because the plan-phase input variables are not available during the apply phase.	2 years ago
Martin Atkins	6e7eff9b4b	stackruntime: Report deletion of resource instance objects ... Previously our strategy for reporting the "applied change" results for resource instance objects was to iterate over the final state and emit an object for each resource instance we found in there. That's not a sufficient solution though, because if any objects are deleted as part of applying the plan then they won't be in the final state and so we won't know to tell the caller that they should drop the relevant objects from the stack-level state. Instead, we'll keep track of all of the resource instance objects that could potentially be affected by applying a plan and then use that set to decide which "applied change" objects to emit. If one of the affected objects is not present in the final state then we'll assume it was deleted and so command the caller to delete its records of that object from both the raw state and the state description. With this in place, it's now possible to create a destroy-mode plan and then apply it to end up with no resource instances at all. The stack runtime itself is still missing handling of destroying its own objects like components and embedded stacks, so this isn't a complete solution but it does at least allow properly cleaning up the records of objects that exist in remote systems, so it's now easier to clean up a development environment with real infrastructure backing it.	2 years ago
Martin Atkins	c6894cc617	stackeval: ComponentInstance: all output values during planning ... If planning fails partway through then we might have an incomplete set of planned changes for our root output values. Previously any output values we hadn't visited yet would get omitted altogether from the value representing a component instance, which is likely to cause confusing downstream errors if any other expression is referring to those missing output values. Instead, we'll use the configuration to find all of the statically-configured output values and fill out any that don't have a planned change with a placeholder cty.DynamicVal. This will make references to those output values return an unknown value, instead of raising an error.	2 years ago
Martin Atkins	972f1c0678	stackplan: Track provider config for every resource instance change ... Previously we were still partially relying on the record of this in the embedded planproto object, but since we're now allowing "planned changes" that don't actually include a change in that traditional sense we need to track the provider configuration address separately as a top-level field. As with some of the addresses before, this means we're now storing the provider config address redundantly in two places when there _is_ a planproto change. Hopefully we'll improve that someday, but we don't need to sweat it too much right now because this only affects the internal raw plan format that is not subject to any compatibility promises between releases, so we can freely change it in any future version.	2 years ago
Martin Atkins	4e34e6ebfa	stacks: Track prior state for all resource instance objects ... Previously our raw plan data structure only included information about resource instances that have planned actions, which meant we were missing those which only get their state updated during refresh and don't actually need any real work done. Now we'll track everything, accepting that some "planned changes" for resource instance objects will not actually include a planned change in the typical sense, and instead only represent a state update.	2 years ago
Martin Atkins	61ccc09681	stacks+rpcapi: Wire through the "plan mode" to the stacks runtime ... Previously we were accepting it at the API layer but then silently discarding it and always creating a "normal" plan. Now we'll pass the caller's selected planning mode all the way through to the modules runtime in component plans. This isn't actually quite sufficient to properly support destroy-mode plans since the stacks runtime itself doesn't yet really understand how to deal with things being destroyed, but this is intended to be just enough for those working on callers of this API to see _some_ effect from setting the planning mode, even though it's far from the full effect expected. More work on destroying will follow in subsequent commits.	2 years ago
Martin Atkins	af0c4764ef	stackeval: A FIXME about emitting prior state snapshots during apply ... Currently we're not explicitly writing the results from the refresh phase to the raw state or state description, which means that a refresh-only plan wouldn't do anything and even in a normal plan there are some edge-cases where we'll end up not saving the refreshed state. This is just a note to remind us of that in later work.	2 years ago
Martin Atkins	8fd29e9439	stacks: Preserve prior state from plan to apply ... We need to retain the prior state (in a form that Terraform Core's modules runtime can accept) between the plan and apply phases because Terraform uses it to verify that the provider's "final plan" is consistent with the initial plan. We previously tried to do this by reusing the "old" value from the planned change, but that's not really possible in practice because of the historical implementation detail that Terraform wants state information in a JSON format and plan information in MessagePack. This also contains the beginnings of handling the "discarding" of unrecognized keys from the state data structures, although we'll probably need to do more in that area later since this is just the bare minimum.	2 years ago
Martin Atkins	5372d5eb98	stacks+rpcapi(stacks): Model deposed objects for resource instances ... Previously we incorrectly assumed that resource instances only have one "current" object each. However, resource instances can also have deposed objects which we must also keep track of. This is a breaking change to the rpcapi since we're now using a new message type for a resource instance _object_ address in several places. That breakage is intentional here because at the time of writing this commit the rpcapi is not yet in any stable release and we want to force updating our existing internal client to properly handle deposed objects too.	2 years ago
Martin Atkins	2559151d0a	stackruntime: Count actions successfully applied ... Callers expect to be given an updated set of counts for how many of each action were completed, which we'll achieve by having the terraform.Hook implementation keep track of those and then announce them based on what it collected. We can't use the plan for this directly because terraform.Context.Plan seems to modify the plan it's given during its work, and on success the plan ends up being totally empty.	2 years ago
Martin Atkins	5d9cac85bc	stackruntime: Emit "pending" events for component instance plan/apply ... Previously we did have an event for planning but it wasn't in quite the right place, and so could potentially appear "late" and mess up the event sequence. Now we'll always emit the pending event immediately before the planning or applying event, which is perhaps a little redundant but having these separated means that we might be able to create some space between them later if we find that we need to do some additional work after we know about a component instance but before we're reading to say that we're planning or applying it.	2 years ago
Martin Atkins	947c483c49	rpcapi: Stack plan and apply use the same progress message type ... Previously we had the progress messages directly inlined as events of the PlanStackChanges operation, but that means that there's no common interface type for progress events across both the plan and apply phases, making it hard for Go implementations to share marshaling code between the two phases. Now we'll use a new StackChangeProgress message type to contain all of the progress message situations. This makes constructing and using the progress messages a little more verbose -- an extra layer of message -- but means that we can write code that works generically with the StackChangeProgress generated struct and thus share it between the two phases, leading to less code overall. As of this commit we don't yet have the apply phase generating any progress messages, but we'll extend it in a subsequent commit now that it's possible to share more of that event-generating code between the two phases.	2 years ago
Martin Atkins	1943ed2395	stackruntime: Emit some more progress hooks during the apply phase ... This is not quite comprehensive yet, but echoes the main component start/end events we emit during the planning phase, and some of the resource-instance-related events. The rpcapi layer is not yet consuming all of these events and so most of this will just be no-op right now; we'll add the rpcapi translation layer for these in subsequent commits.	2 years ago
Martin Atkins	c4d7ee93fb	stackeval: Use the right EvalPhase in various places ... In an earlier commit I generalized some code that was originally written for the plan phase to be shared between the plan and apply phases, but missed a few spots where it was still hard-coded for the plan phase. Hopefully this is all of them!	2 years ago
Martin Atkins	69aa51b9ff	stackruntime: Fix some quirks of data sent from plan to apply ... We previously took a few shortcuts just to more quickly produce a proof of concept. This gets us considerably closer to a properly-working handoff from plan to apply, at least for create, update, and replace actions. Destroying stuff that's been removed from the configuration will take some further work since currently everything we do is driven by objects in the configuration. That will follow in later commits.	2 years ago
Martin Atkins	569f0f553b	stackruntime: ComponentInstance considers previous state ... Now that we have access to a previous state snapshot during the planning phase, we can pass the relevant parts to the modules runtime when asking it to plan changes for a component instance. This is not yet complete enough for full support of consecutive plan and apply runs that preserve state, but it gets the needed data plumbed in and we'll continue making this more complete in subsequent commits.	2 years ago
Martin Atkins	aaba750095	rpcapi: Stacks: pass previous state to stack runtime during planning ... As of this commit it doesn't yet do anything with it, but we'll plumb that in more deeply in subsequent commits.	2 years ago
Martin Atkins	49511fe9a8	stackstate: Fuller round-tripping of resource instance object state ... This gets us ever closer to being able to preserve resource instance objects from one run to the next. In subsequent commits we'll make use of the "LoadFromProto" option to load the prior state during the planning phase and take it into account when we're deciding what actions to propose.	2 years ago
Martin Atkins	13e26b60dd	addrs: Adopt DeposedKey as an address type; remove states.Generation ... Previously we had states.DeposedKey as our representation of the unique keys used to differentiate between multiple deposed objects on the same resource instance. That type is useful in various places, but its presence in the "states" package made it troublesome to import for some callers. Since its purpose is as an identifier, this type is more at home in the "addrs" package. The states package retains a small number of aliases to preserve compatibility with a few existing callers for now, until we have the stomach for a more invasive change. This also removes the largely-redundant states.Generation interface type, which initially was intended as a way to distinguish between "current" objects and deposed objects, but didn't really add anything because we already have addrs.NotDeposed to represent that situation. Instead, we just ended up with various bits of boilerplate adapter code converting between the two representations. Everything now uses addrs.DeposedKey and uses addrs.NotDeposed to identify non-deposed objects.	2 years ago
Martin Atkins	d1b0632e82	stackruntime: Add HashiCorp copyright comments	2 years ago
Martin Atkins	74215a4451	stacks: Include resource type schema in PlannedChange and AppliedChange ... Unfortunately for historical reasons the "terraform" package produces state and plan artifacts that have dynamic data elements pre-encoded in the formats that Terraform CLI's traditional plan and state formats use. Since Stacks uses different plan and state models, and exclusively uses MessagePack encoding for dynamic values in both, we'll sometimes need to transcode from one format to another when generating and decoding the stacks-oriented representations. Transcoding between these serialization formats requires access to the relevant schema because the formats themselves each have an infoset that is only a subset of Terraform's type system. Therefore we'll now annotate the PlannedChange and AppliedChange objects that include provider-specific data types with the schema required to transcode them. This commit does not yet actually arrange to make use of those schemas. That will follow in later commits.	2 years ago
Martin Atkins	48518b8233	stacks: A very basic stub of applied resource instance change announcements ... We'd like to start developing some clients for this part of the RPC API concurrently with remaining work here in Terraform Core, and so this is a bare-minimum implementation of emitting "applied change" events for resource instances just so there's something for client developers to test against. There are various things wrong with this, including but not limited to: - It always reports that the new state for a resource instance is an empty object, rather than including the correct updated object. - It only supports "current" resource instance objects, and ignores deposed ones. - It just uses the resource instance address alone as the key for the state description map, which is fine for now when we only have one kind of description object anyway but will not be sufficient for a real implementation that needs to emit various different kinds of object. We'll need to rework most of this in future commits, but this is hopefully sufficient to start implementing and testing API clients, at least to some extend, despite the crudity of the results.	2 years ago
Martin Atkins	d99985720f	stacks: Carry input variable values from plan to apply ... We need to preserve the input variable values provided during the plan phase so we can reuse them during the apply phase to recalculate derived results based on new information learned during the apply process. We also need to recalculate the input values for individual components during the apply phase, so we can react to previously-unknown values that come from other upstream components.	2 years ago
Martin Atkins	b887d29894	stackeval: NewForApplying must save the ChangeExecResults object ... This was just an oversight in some earlier refactoring. An ApplyPhase instance of Main must store a pointer to the ChangeExecResults object that will capture all of the results of the main side-effects of applying the plan, since we'll need to retrieve those results to evaluate expressions and other similar downstream results based on those outcomes.	2 years ago
Martin Atkins	de170a9a30	stackeval: InputVariable should respect EvalPhase when deciding value ... Previously it was still hard-coding PlanPhase, which was a cheat we did in earlier work when we were just exploring the overall design of this package. It will now pass ApplyPhase when running in the apply phase, although Main.RootVariableValue doesn't yet have any implementation for retrieving variable values frozen during planning so root input variables will always be unknown values during the apply phase for now. We'll fix that in a future commit.	2 years ago
Martin Atkins	373f39d6ac	stackeval: ApplyPlan's ChangeExec must be in main task ... The ChangeExec call creates new promises, one of which the caller is responsible for resolving by calling the "begin" callback. To correctly follow the rules of the "promising" package we must resolve that promise in the same task that created it or explicitly delegate it to another task. In this case it's fine to just resolve it, but for that to work correctly we need to call "begin" from the same task as ChangeExec was called from.	2 years ago
Martin Atkins	b8923a5f94	stackruntime: Persist a component's plantimestamp in the plan ... We need to preserve the module runtime's chosen plan timestamp through to the apply phase so that plantimestamp function calls during the apply phase will return the same result as during the planning phase. This also means that we will now no longer emit a "component instance" planned change object if the planning operation fails so much that there is no plans.Plan object returned, but that's at least consistent with the fact that we also don't report any resource instance changes in the same situation. Event emission is always best-effort when the result has errors.	2 years ago
Martin Atkins	8f00a7bf5a	stackruntime: Make sure PlanResponse and ApplyResponse channels get closed ... The closure of these channels is how a caller knows that the operation has completed, so we must always close them before we return or the caller is likely to get wedged.	2 years ago
Alisdair McDiarmid	927350eca8	rpcapi: Add component expansion and change summary ... We want to report the result of component expansion to the client as early as possible, so that it can prepare to receive any following information for each instance. This message is critical when a component's expansion expression evaluates to an empty set: without this message's confirmation of this case, it will be unable to determine that the component will receive no plans for any instances until the plan completes. We also want to report a roll-up of the resource changes upon completion of each component's plan and apply operations, so that a client can summarize the operation changes without performing the roll-up operation on all received change events.	2 years ago
Martin Atkins	f1ceee5e69	rpcapi: Stacks.ApplyStackChanges initial implementation ... This wires up the stackruntime.ApplyPlan function to the RPC API with a similar strategy as we used for Stacks.PlanStackChanges wrapping stackruntime.Plan. This is not well tested at all and so probably doesn't quite work right. I'm currently optimizing for getting a reasonable stub of everything into place so that we can have multiple people work together on getting the details sorted afterwards.	2 years ago
Martin Atkins	2a0d407eb6	stackruntime: ApplyPlan function ... This is the exported entry-point for applying a plan that was created by an earlier call to stackruntime.Plan.	2 years ago
Martin Atkins	8719f13b58	stackeval: Stubby implementation of applying changes to components ... This implementation is likely naive and will probably need some refinements in future commits; the goal here is just to stub out the general shape of this so that we can potentially have multiple contributors work concurrently on the subsequent improvements.	2 years ago
Martin Atkins	f8d2fef129	stackeval: Factor out the two "walk drivers" ... With validate, plan, and apply now all at least stubbed out we can see that they all share some similar logic for visiting everything that's relevant to an evaluation phase and gathering up diagnostics and other external reports about each object. Since it's important that the phases all visit the same objects so that we can produce consistent results between phases, we'll accept a little inversion-of-control complexity here in return for now having only two "walk-driver" implementations: one for visiting the "static" objects and one for visiting the "dynamic" objects. - The validate phase only visits static objects. - The plan phase visits static objects first, and then dynamic objects only if the static walk doesn't produce any errors. - The apply phase only visits the dynamic objects. This also required some retroactive changes to some of my earlier work on the validation phase since we can no longer assume that the validation walk is the only one which visits the "static" objects. Their methods now take EvalPhase arguments similar to those for the dynamic objects, and we differentiate between the results in each phase so that we can potentially add slight differences between the phases in future if needed. As of this commit, though, the validate phase and the static portion of the plan phase should produce identical results.	2 years ago
Martin Atkins	fafa36e73a	stackeval: ApplyPlan function and initial supporting internals ... This is a first pass at actually driving the apply phase through to completion, using the ChangeExec function to schedule the real change operations from the plan and then, just like for planning, a visit to each participating object to ask it to check itself and report the results of any changes it has made. Many of our objects don't have any external side-effects of their own and so just need to check their results are still valid after the apply phase has replaced unknown values with known values. For those we can mostly just share the same logic between plan and apply aside from asking for ApplyPhase instead of PlanPhase. ComponentInstance and OutputValue are the two objects whose treatment differs the most between plan and apply, because both of those need to emit externally-visible "applied change" objects to explain their side-effects to the caller. The apply-walk driver has a lot of behavior in common with the existing plan-walk driver. For this commit we're just accepting that and having two very similar pieces of code that differ only in some leaf details. In a future commit we might try to generalize that so we can share more logic between the two, but only if we can do that without making the code significantly harder to read.	2 years ago
Martin Atkins	985b110afa	stackeval: ChangeExec returns states.State from component instance change ... Previously we were returning just a value representing all of the component's output values, but that's the wrong level of detail. In the plan phase we have the corresponding function of ComponentInstance return a plans.Plan object and then derive the object value representing outputs from it only on request. Having the apply result be a *states.State is the analogous level of detail, and then again we'll construct the cty object representing all of the output values only on request during expression evaluation.	2 years ago
Martin Atkins	959b02daa3	stackeval: ChangeExec allows component changes to return diagnostics ... It's annoying to have to complicate our utility function in this way, but it's the most pragmatic way to ensure that the result value and any relevant diagnostics stay associated with one another so we can ensure that we can always handle them both consistently.	2 years ago
Martin Atkins	0ec4acb4ce	stackeval: ChangeExec for orchestrating apply-time changes ... Unlike the plan phase where we just visit everything in no particular order and just let things be scheduled by the Go runtime wherever they happen to get scheduled, we'd like a little more control over the parts of the apply phase that might affect external infrastructure. In particular, we'd like to see some top-level code that is clearly making sure that all of the component instances from the plan are getting handled, rather than leaving that implicit as part of the overall evaluation process. This new ChangeExec function (and associated supporting types) model an explicit setup phase where we register all of the change actions we intend to take, and then an execution phase where those actions can run concurrently with other stackeval work while mediating through the stackeval.Main type as usual. This does require a small amount of tricky finagling to properly sequence the steps so that we can freeze the set of change tasks before we start executing but still give all of the tasks access to the appropriate stackeval.Main object. As usual with this package, the idea is to keep the weirdo promise-based scheduling stuff segregated from the main code so that for everyday maintenence we can hopefully just focus on the straight-through code that represents the main execution rules and worry less about implementation details such as promises and tasks.	2 years ago
Alisdair McDiarmid	54062a52cb	stacks: More extensive in-progress plan events ... Expand the existing hooks to emit events throughout the planning process, providing enough information for the Terraform Cloud UI to render a live-updating representation of the plan. We also sketch out the equivalent hooks for the apply operation.	2 years ago
Martin Atkins	6a45cdf7b0	stacks/stackruntime: Report if provider schema fetching fails ... We were previously just ignoring the errors on the assumption that some other codepath would propagate them, as is our assumption for all inter-dependencies between objects in stackeval. However, provider types are a bit of an oddity right now in that they are really _only_ accessed by implication, and so we don't directly visit them to get an opportunity to gather any errors. We'll hopefully improve on that later, but for now we'll just emit the error once for each component instance to ensure that it gets emitted _somewhere_. We must also stop before trying to call terraform.NewContext if schema loading failed, because otherwise terraform.NewContext itself will fail trying to interact with a provider whose schema it wasn't given.	2 years ago
Martin Atkins	2095a7d000	stacks/stackruntime: Don't panic if component modules are invalid ... ComponentConfig.RequiredProviderInstances was not considering the possibility that ComponentConfig.ModuleTree will return nil if the component's module tree is too invalid to load anything. As usual in stackeval, we need to react to these situations by returning safely to allow another goroutine to return the diagnostic messages that caused the nil result.	2 years ago
Martin Atkins	afd2282aa6	stacks/stackruntime: fix TestPlanWithSingleResource ... We've now upgraded the go-slug dependency to a newer version that fixed the bug that this FIXME comment was referring to, so we can now write the source address correctly and remove the note.	2 years ago
Martin Atkins	c1e7abafc3	stackruntime: Configure and assign provider instances ... Instead of the temporary hack of hard-coding the built-in "terraform" provider, it's now possible to declare configurations for arbitrary providers and assign them to the provider configuration slots of the root module of each component.	2 years ago
Martin Atkins	3dc3781904	stackruntime: Starting up and configuring provider plugins ... Each declared provider configuration is now started up and configured during planning. We don't actually do anything with them yet other than shut them down again once the plan is complete, but we'll improve on that in subsequent commits.	2 years ago
Martin Atkins	373889ae25	stackeval: Actually use the component block "inputs" argument ... Due to some carelessness in earlier refactoring, the components block evaluation was evaluating the "inputs" expression but not actually saving its result, causing all components to get planned with all their input variables set to default values (or failing, if there were any required).	2 years ago
Martin Atkins	7c1dbc9061	stackeval: Don't share objects between AsyncTask closures ... This is, of course, the classic Go for loop hazard where a closure in the loop captures the symbol itself rather than the value of the symbol, and the symbol gets modified directly on each iteration so all of the closures end up referring to whichever value came last. This is just an ugly tactical fix for that intentionally not changing the existing code too much. Hopefully in a future commit we'll rearrange this a little so that it's harder to accidentally reintroduce this bug later, and we should also add some more complex test cases with multiple embedded stack calls and multiple component calls.	2 years ago
Martin Atkins	d1eaf199c9	stackeval: Don't panic if we don't know the input type for a component ... If one of the modules under a component is invalid then we won't be able to determine the expected object type for specifying the input variable values, in which case we should just answer "I don't know" rather than crashing when we're asked to evaluate the input variables object.	2 years ago
Martin Atkins	fbc34005ba	stackeval: Emit PlannedChange for each component instance ... We need to explicitly announce each component instance that we find declared in the stack configuration before we announce any resource instances that belong to it, because callers will rely on this message to create a link between the component instance address on each resource instance and the unexpanded component addresses previously announced by the "find stack components" static analysis call.	2 years ago
Martin Atkins	2d2baee72d	staticcheck: Broadly ignore unused stuff in all of "stacks" ... We're currently actively developing stuff under the "internal/stacks" directory such that there are some loose ends not yet completed in order to make our PRs at least _slightly_ more digestible. For that reason we'll broadly turn off the "unused symbol" checks from staticcheck under that entire directory for now, instead of making many single-file exceptions that will be harder to clean up later. Once the work in this directory settles we should remove this exception so that the stacks-related packages are subject to the same rules as everything else. Unfortunately staticcheck can only work with configuration files if there are no explicit checks on the command line, so this also makes the top-level checks be a configuration file rather than just a command line option in our staticcheck.sh script. The effect for the rest of the codebase is the same, but having the top-level rules in a configuration file means that subdirectories (like "internal/stacks") can inherit from it to minimize the divergence from the typical rules.	2 years ago
Martin Atkins	ea3b70abf0	stackeval: Component calls are now referenceable ... This means it's valid to write references like "component.foo" in expressions in a stack configuration, where the result is built from the planned output values of the root module of each component instance. For embedded stacks we intentionally designed the language to require explicit types on all output values so that we'd still be able to do type checking even when we can't calculate the output values yet. Unfortunately we can't do the same trick for components because the main Terraform module language treats output values as dynamically-typed and so an output value we can't evaluate yet could have literally any type. Therefore we concede and just return cty.DynamicVal in most failure cases here; perhaps a future edition of the main Terraform language can improve this by requiring explicitly-typed output values there too.	2 years ago
Martin Atkins	4a179ed969	stackeval: Very very basic component planning ... This is the absolute bare-minimum integration with the main Terraform language runtime to plan changes for a component. It's intentionally not yet very robust and limited only to the built-in Terraform provider because we're just trying to get the bones in place for seeing stack planning working, rather than trying to build something complete enough to ship. This currently only deals with the resource instance changes that get planned and doesn't expose the root module's output values for use elsewhere in the stack configuration. Output value handling will follow in subsequent commits.	2 years ago
Martin Atkins	d435cd5a1b	stackconfig: Bring the source bundle with the config ... When we load stackconfig we don't immediately load any of the modules that component blocks refer to, so we need to bring the source bundle along with the configuration so that a downstream piece of code (the stack language runtime, currently) can load those from the same source bundle.	2 years ago
Martin Atkins	d3e93e4a31	stackruntime: Initial stubbing of Plan ... This is just enough implementation of "Plan" for embedded stacks with their own input variables and output values, and exposing the final top-level output values as "planned changes". This still doesn't really do anything end-user-useful but it's helpful to have the expression evaluation and plan walk driver machinery in place before moving on to the main useful language element: component blocks. Support for components will follow in subsequent commits.	2 years ago
Martin Atkins	e76e203c77	stackeval: walkState with custom diagnostics handling ... In an earlier commit we added walkState as something we could concisely pass between all of the functions participating in a config/state walk such as what we do in main_validating.go for validation, dealing with the two cross-cutting concerns of firstly detecting when all of the asynchronous operations have completed and secondly aggregating together diagnostics produced by the various objects into a single set. The diagnostics part of this problem was previously solved by just always appending diagnostics into a hidden syncDiagnostics object. That works okay for validation where we just wait until all the work is complete and return all of the diagnostics together as a normal return value, but for planning we expect to be able to stream out both planned changes and diagnostics gradually as the (potentially-slow) planning process proceeds, so the UI can give gradual feedback and thus feel more responsive than just showing an indeterminite progress indicator for the entire operation. To preserve the idea of just passing a single object through the walk functions while giving a little more flexibility, here we introduce an extra hooking capability where the plan phase can treat diagnostics differently while still preserving the same external walkState API. In future commits the plan walk driver functions will all share the same walkState object and the entry point will populate it with the custom callbacks required to create a bridge to the channel-based API for emitting streaming diagnostics. The old simple API is still available for the validate walk to use. Instead of having the internal syncDiagnostics object directly inside the walkState object, it now lives in the local scope created by calling newWalkState, captured into the closures of the two callback functions. The effect is equivalent.	2 years ago
Martin Atkins	355bae32da	stackruntime: Allow hooking the start and end of the overall plan operation ... This broad pair of hooks allows the caller (i.e. rpcapi) to generate a parent tracing span around the entire plan operation, which all other traced operations will descend from. This is also in part just a way to show that the hook API is workable in practice, since these are the first actual hooks.	2 years ago
Martin Atkins	863a91359a	stackruntime: Plan calls into stackeval ... This nudges the placeholder "this is not a real plan" warning down one more level in the stack, with stackruntime.Plan now being equipped to propagate the individual planned changes and diagnostics up to the rpcapi caller, which will in turn propagate them to its client.	2 years ago
Martin Atkins	0111408ca8	stackruntime: Stub of "Plan" function ... This establishes the overall API shape but implementation-wise just moves the warning that the implementation is fake one level deeper, with stackruntime now being the one to return that warning instead of the rpcapi package directly. Package rpcapi now monitors the two output channels from stackruntime.Plan and translates what it sees into protobuf events to transmit to the client. Of course given the above that's currently always just exactly one diagnostic and no planned changes, but we'll start to make the stackruntime.Plan function real in subsequent commits.	2 years ago
Martin Atkins	8af46fd38b	stackruntime: An extensible "hooks" API ... Following a similar model that we've previously used in other parts of Terraform to expose UI-oriented progress events, here we define stackruntime.Hooks as a collection of optional callbacks a caller can use to subscribe to events, and the supporting utility code to help us concisely and correctly call those hook functions from the main evaluation code without adding too much extra noise to the code. Although the primary focus of this API is driving realtime UI updates, the API design also aims to accommodate distributed tracing use-cases by allowing callers to propagate arbitrary values between related event notifications and to inject additional values into the context.Context used for downstream operations that might not be under the direct supervision of stackruntime/stackeval.	2 years ago
Martin Atkins	2b7fad5bee	stackruntime: Our fake telemetry span must be passed by pointer ... Most of our Span actions don't mutate the object because we just immediately emit messages to the test log, but SetName directly modifies the span name so we need to pass by pointer so we can honor that rename.	2 years ago
Martin Atkins	03c981d9d0	promising,rpcapi,stackruntime: Quiet some staticcheck errors ... This code is still under development and so we have some stuff that's there to support further work despite not actually being used yet. We should reconsider these changes later on once the work here has settled.	2 years ago
Martin Atkins	791a5fb8a6	stackruntime: Top-level "Validate" function ... This is the public entry point for the overall validation of an entire stack configuration. So far it supports validation only of input variables, embedded stack calls, and output values. We'll grow this to support other language constructs in later commits. This is a relatively large commit due to introducing various supporting infrastructure to help make evaluation possible at all. Subsequent commits will hopefully be more focused due to being able to depend on the foundations introduced here.	2 years ago
Martin Atkins	8c7acd5378	stackruntime: Helper for viewing traces in tests ... Since our stackeval package is instrumented for OpenTelemetry tracing, it's helpful to be able to see the trace events as part of the test log when debugging. tracesToTestLog arranges for any spans to be announced into the main test log for the remainder of the current test, and then restores the original trace provider afterwards as a test cleanup action.	2 years ago
Martin Atkins	4754889950	stackruntime: Beginnings of Validate function ... This stubs out the basic structure of stackeval's equivalent of a graph walk, which is really just a recursive tree walk starting async tasks that might depend on each other via the abstractions in package "promising". This initial work is sufficient to validate the inputs to an embedded stack call when referring to input variables in the calling stack, showing that the expression evaluation model seems to be working and the use of promises to deal with dependency ordering seems viable.	2 years ago