2447 Commits (eecbfbf2cccedfb581469ee63599c4b4abc5c60f)

Author	SHA1	Message	Date
Martin Atkins	0994e6fbf9	stacks: The terraform.workspace attr is not available in Stacks ... The terraform.workspace attribute is a rare example of a CLI- and Cloud- specific concern bleeding into the Terraform language, and it can only really have meaning when used in the traditional Terraform workflow because otherwise there's no workspace to return the name of. In Stacks any variations between instances of a module must be created through input variables. Within Terraform Cloud in particular it's also possible to use stack-level input variables that are assigned different values from different stack deployments, and thus an author can recreate the effect of terraform.workspace using a stack-level input variable that has a different value for each deployment. This is one of the few cases where the Terraform module language differs in stacks compared to traditional Terraform. Any module that makes use of terraform.workspace will need to be generalized to use input variables instead before it can be used within a stack component. Prior to this change, references to terraform.workspace from a module used in a stack component would just panic altogether, because the stacks runtime doesn't provide the object that the workspace name would be taken from. Now we'll return a user-oriented error instead.	2 years ago
Martin Atkins	9851cd456e	terraform: decide module instances using new API ... We've been gradually improving the design of how we keep track of "named values" such as output values, but the expression evaluator for output values was still doing some things in more manual/bespoke way, rather than making use of the new abstractions. We'll now use a combination of instances.Expander and namedvalues.State as the source of truth for which instances exist for a given module call and what output values each of those instances have. The configuration remains the decider for which output value names are declared, but the final result associated with each of the declared output values is tracked by namedvalues.State. This also removes the temporary namedvals GetOutputValuesForModuleCall method we added as a stopgap to allow introducing the namedvals.State API without such a disruptive rewrite of our output value handling. We have no further need for that helper because instances.Expander already knows which instances should exist for each module call, and the configuration knows which output values should exist in the result, and so namedvals only needs to be able to return one final output value result at a time, via namedvals.State.GetOutputValue. With all of the tricky inline logic factored out of this part of the evaluator, the TestEvaluatorGetModule test becomes considerably simpler since it's no longer the evaluator's responsibility to choose between various different sources for the module output values depending on the situation, and instead we just need to set the evaluator and the instance expander up in a realistic way, with the correct handling of both of those utilities already being covered by this package's main integration tests.	2 years ago
Martin Atkins	0c3a581ff0	instances: Expander.ExpandAbsModuleCall ... This is a similar idea to Expander.ExpandModule but for situations where we already know exactly which containing module we're evaluating inside, and thus we can just pluck out the leaf instance keys of that specific dynamic call, rather than the full recursive expansion of every containing module. Expander.ExpandModule is useful when implementing DynamicExpand for a graph node because main graph nodes always represent static configuration objects, but this new Expander.ExpandAbsModuleCall is more appropriate for use during expression evaluation of references to a module call because in that case we'll know which specific module instance address we're using as the evaluation scope. A future commit will actually use this new method from the expression evaluator in the modules runtime.	2 years ago
Martin Atkins	d37436a39a	addrs: ModuleInstance.AbsCall ... This is just a variation of the pre-existing ModuleInstance.Call that returns its result as a single value of type AbsModuleCall. This should really have been the signature of ModuleInstance.Call in the first place, but it took us some time to realize that we would benefit from an absolute address type for module calls themselves as opposed to the instances they imply once expanded, and so AbsModuleCall didn't exist at the time we originally wrote ModuleInstance.Call. Perhaps one day we'll update all existing callers of ModuleInstance.Call to use ModuleInstance.AbsCall instead, but for now we'll be pragmatic and keep both to avoid causing unnecessary churn.	2 years ago
Martin Atkins	f3c48314ab	addrs: AbsModuleCall.StaticModule ... This new method calculates the static Module address corresponding to the receiver, effectively discarding any instance keys present in the containing module instance path.	2 years ago
Brian Flad	3b4964270f	internal/command/jsonprovider: Remove root jsonfunction object from function output (#34536) ... Previously, the `providers schema -json` output would include the root object from `metadata functions -json`. This object had its own `format_version` property, which would be confusing with the root `format_version` property already present. This change still uses the `jsonfunction` package for consistency between cty and provider function JSON handling, but removes that extra object, instead making `functions` directly a mapping of names to signatures/definitions. This also adds a code comment to hint maintainers that jsonprovider format versioning is tied to jsonfunction format versioning. Example output prior to change: ```jsonc { "format_version": "1.0", "provider_schemas": { "registry.terraform.io/bflad/framework": { // ... "functions": { "format_version": "1.0", "function_signatures": { "example": { "description": "Echoes given argument as result", "summary": "Example function", "return_type": "string", "parameters": [ { "name": "input", "description": "String to echo", "type": "string" } ] } } } } } } ``` Example output after change: ```jsonc { "format_version": "1.0", "provider_schemas": { "registry.terraform.io/bflad/framework": { // ... "functions": { "example": { "description": "Echoes given argument as result", "summary": "Example function", "return_type": "string", "parameters": [ { "name": "input", "description": "String to echo", "type": "string" } ] } } } } } ```	2 years ago
Lauren	6d9b3b2cd7	call Colorize() on CLI output	2 years ago
Martin Atkins	7d2d4dec5e	stackeval: When being destroyed, component instance result comes from plan ... Because we treat dependency edges as reversed when a component instance is being destroyed, the final result (an object representing output values) for a component instance being destroyed must not depend on anything else in the evaluation graph, or else we'd cause a promise self-reference as the downstream component tries to configure itself based on our outputs. As a special case then, for a component instance being destroyed we take the planned output values directly from the plan, relying on the fact that the plan phase sets them to the prior state output values in that case, and therefore the result for such a component is available immediately without blocking on any other expression evaluation during the apply phase. This combines with several previous commits to create a first pass at handling ordering correctly when planning and applying a full destroy. This commit also incorporates some fixes and improvements to stackeval's apply-time testing helpers, which had some quirks and bugs when first added in a recent commit. One of those problems also revealed that the raw state loader was not resilient to a buggy caller setting a state entry to nil instead of removing it altogether, and that mistake seems relatively easy to make (as I did here in the test helper) so we'll tolerate it to make it possible to recover if such a bug does end up occurring in real code too.	2 years ago
Martin Atkins	dcb176a557	stackeval: A Main for applying knows the plan it's applying	2 years ago
Martin Atkins	c06ad051a1	stackeval: Respect inter-component dependencies during apply ... Before we start any apply-time actions for a particular component, we'll block until other components' apply are complete. In the normal case a component instance waits until its dependencies have been applied. However, if component instances are being destroyed then they instead wait for their _dependents_ to complete being applied, since a component must outlive all other components that depend on it.	2 years ago
Martin Atkins	c345cde93c	stackeval: Extended utilities for plan and apply testing ... These helpers make it easier to write our tests because the plan and apply phases both work in a callback-based fashion where they gradually emit events, and that's not a very convenient API for test code to interact with. This exposes some more details about the planning results, and also adds a new similar helper for the apply phase. We'll make use of both of these in future commits.	2 years ago
James Bardin	4d1499b559	Merge pull request #34525 from hashicorp/jbardin/unknown-errors ... Add error diagnostics when receiving an unknown value from `ReadResource` or `UpgradeResourceState`	2 years ago
James Bardin	d8a335f520	add unknown val error for MoveResourceState	2 years ago
Liam Cervante	9246b3a994	stacks: on module load convert relative paths to absolute (#34523)	2 years ago
Martin Atkins	4a0d06c13e	terraform test: Report skipped runs in JUnit XML output ... Previously we were just ignoring skipped runs completely, but JUnit XML has some elements/attributes for describing those and so we'll populate those too now.	2 years ago
Martin Atkins	b43ddd8ff6	terraform test: Experimental JUnit XML reporting ... This adds an experimental new option -junit-xml=FILENAME for the "terraform test" command. When specified, it writes a JUnit XML report to the specified filename once the test run is complete, while continuing to report test progress in the UI in the usual way. This is only experimental for now because it remains to be seen if this particular mapping to the JUnit XML schema is actually useful in real software -- this format is woefully underdocumented and implemented slightly differently by each consumer -- and so we might change this significantly before stabilizing it, or remove it altogether if it turns out that there's no useful mapping to JUnit XML here. Hopefully those who are interested in JUnit XML reports will try this experiment against their favorite JUnit XML-consuming software and report back whether the report is presented in a helpful way. It's a de-facto convention for JUnit XML to be reported separately to a file, rather than replacing the normal test run output, since tools that consume this format tend to present its results in a separate and less prominent place than the output of the command itself. This option is designed to follow that convention for consistency with various other software that produces this format. The implementation here is intentionally pretty minimal and simplistic just as a starting point for gathering feedback. The main priority is that it be easy to evolve this based on feedback and to remove it altogether if we decide not to stabilize this at all. If this does become stabilized, it might deserve being factored out into a separate package so that we can minimize the amount of logic embedded directly inside the views package, and it will certainly need some unit tests to represent what we've committed to supporting in future versions.	2 years ago
James Bardin	105b2044d5	remove leftover debug output	2 years ago
James Bardin	ef44fd604e	error when UpgradeResourceState return unknown value	2 years ago
James Bardin	0b4e468de6	error when ReadResource returns unknown value	2 years ago
Brian Flad	1298fcd412	Add missing function deprecation_message to JSON output and website documentation (#34520) ... Reference: https://github.com/hashicorp/terraform/pull/34450	2 years ago
James Bardin	bc945f22eb	Merge pull request #34515 from hashicorp/jbardin/provider-function-validation ... Add validation for provider function results	2 years ago
James Bardin	5c7b6614f5	add Validate to providerFunctionImpurePlan test ... In normal operation a Validate always precedes Plan, so add a check for that phase in the Plan test.	2 years ago
James Bardin	a5337b3b20	remove global results table state ... In the case of now possible long-running terraform processes, we don't want the cache to grow indefinitely. Move the handling of the results into the terraform package, and inject the shared table during the walk into the `BuildFunction` wrapper.	2 years ago
Liam Cervante	0ca03ecf08	Extend refactoring package to support cross-provider moves (#34501) ... * Extend refactoring package to support cross-provider moves * implement provider cache * make generate	2 years ago
Liam Cervante	7d143388f3	Add support for the cross resource type move operation in the proto schema (#34480) ... * Update proto schema and provider interfaces with support for moved across resource type RPCs * address comments * remove unused functions * remove support for flatmap format	2 years ago
James Bardin	94d64364a4	test provider function consistency	2 years ago
James Bardin	a3aa517589	store and verify provider functions calls in plan ... Store the hashed function calls within the plan file, so that Terraform can validate the results during apply as well. The error won't be as good, since we don't have access to the prior results to help debugging, but it is very helpful to indicate when a function returned an incorrect result rather than show that the resource attempted to apply a different value from what was originally planned.	2 years ago
James Bardin	b3c5590f01	Merge pull request #34450 from hashicorp/jbardin/provider-functions-json-schema ... Add functions definitions to the `providers schema -json` output	2 years ago
James Bardin	2e74fec583	add optional Summary field to function json	2 years ago
Graham Davison	57ec131e80	Ignores `AccessDenied` errors when listing bucket with default prefix	2 years ago
Graham Davison	40e593ddc1	Adds constant for default workspace prefix	2 years ago
Martin Atkins	3c14eeb945	stackeval: Make some provisioners available to stack components ... This makes the built-in "remote-exec" and "file" provisioners available for use in the modules that implement stack components. These are both relatively easy and low-risk to include because they are builtins and don't require anything from outside of Terraform itself. For now this intentionally excludes local-exec because we'll want to think about what constraints we want to put on it, if any, to help ensure we can meet the goal of stack configurations being portable between different execution environments without significant modification, and our current stacks execution environment doesn't guarantee the availability of any external software _at all_. The motivation for adding this now is just to give some better feedback when someone uses a module using one of these provisioners, since otherwise they'll see just a confusing generic error message from the modules runtime about the provisioners not being available. I expect we'll revisit this later and consider expanding it to at least include local-exec, and _maybe_ external provisioner plugins, although that's more questionable because the provisioner plugin mechanism is incredibly legacy and doesn't have any way to work outside of local Terraform CLI usage today. There are no tests here yet because these provisioners are not mockable and would depend on having an SSH or WinRM server to connect to. Later we should ponder how to make this more testable, which might mean making another part of the system responsible for actually providing the provisioner factories and thus our tests here can use fakes. The goal here is just to get this done in a relatively lightweight way for better feedback during preview though, so we're not yet ready to make significant time investments here.	2 years ago
Martin Atkins	6071e06f1a	tfstackdata1: DynamicValueFromTFStackData1 decodes sensitive marks	2 years ago
Alisdair McDiarmid	1d3f863f2b	stackruntime: Support sensitive component inputs ... Components can emit sensitive values as outputs, which can be consumed as inputs to other components. This commit ensures that such values are correctly processed in order to pass their sensitivity to the modules runtime.	2 years ago
Alisdair McDiarmid	fc75657113	terraform: Support sensitive input variable values ... The stacks runtime interacts directly with the modules runtime's planning operations, rather than through the usual CLI paths. As a result, root module input variable values can be marked as sensitive upon entry. This commit adds support for such marked values to the modules runtime and the `plans.Plan` type. This is sufficient for stacks, which does not use the planfile serialization, but we may in future choose to serialize these decoded marks also.	2 years ago
Alisdair McDiarmid	e4b319401e	stacksruntime: Add sensitive outputs tests	2 years ago
Martin Atkins	3961c18420	stackeval: Return an error if a module contains a provider config ... When using stacks the provider configurations belong in the stack configuration rather than inline in the individual modules. Shared modules with inline provider configurations has been a deprecated legacy practice for many years now, but traditional Terraform continued to support it for backward-compatibility with older modules despite the significant downsides of doing so. Stacks now finally removes that capability, since it isn't straightforward to continue supporting it once we've made the stacks runtime be responsible for instantiating and configuring providers.	2 years ago
Martin Atkins	a51c034cc3	stackeval: Include component configurations in our static walks ... This means both that the validate walk can now describe static problems in the component's module tree, and that we'll catch such problems earlier in the planning phase and thus avoid reporting them repeatedly in cases where a component block uses for_each to declare multiple instances. This includes a fix to a bug in StackConfig.Components, which was incorrectly using the input variable declarations as the source for its result, instead of the component declarations.	2 years ago
Martin Atkins	a3cd4a11ce	stackeval: Use relative, slashy paths in module-related diagnostics ... Until we've updated the module config loader to be sourcebundle-aware and thus return proper source addresses, this will at least make the paths we show in diagnostics a little less verbose, and more consistent across platforms.	2 years ago
Martin Atkins	a4d61733f6	stackplan: Include extra info in raw component instance plan ... We'll now track the planned action, the component addresses we depend on, and the planned output values.	2 years ago
Martin Atkins	25a514f846	stackeval: Populate planned component instance more thoroughly ... We'll now track a more realistic Action, the components that the component instance depends on, and the planned output values.	2 years ago
Martin Atkins	524da4f278	stackeval: Applyable objects have a set of required components ... To figure out the required order in which we'll apply the plans for all components we need to know how data flows between them. We'll use what's added here during the plan phase to record in the plan which components depend on which other components, and then the apply phase will use that coarse information to schedule the individual component apply operations into a suitable order. None of that is implemented here though, and will follow in future commits. This commit includes new documentation about the apply scheduling approach which includes some mechanisms that are not actually yet included in this commit, but will follow shortly after. This is just a pragmatic compromise to avoid spending time authoring short-lived partial documentation drafts for the intermediate steps while still keeping these increments small enough for practical review.	2 years ago
Martin Atkins	5bfdb12098	stackaddrs: Absolute references and referenceables ... This allows us to talk about references between expressions in absolute terms, taking into account the stack where a particular expression would be resolved. This isn't an appropriate representation for references like each.key, since those require a more specific scope than just a stack, but that's okay because those contextual reference types can't depend on anything other than what they are embedded inside anyway, and so we never need to consider them when we're doing global reference analysis.	2 years ago
Martin Atkins	00761a9f70	stackeval: ApplyChecker is now Applyable ... We chose this name originally to communicate that implementations are only responsible for checking that an apply was successful, but in future commits this interface will also gain the additional responsibility of helping to gather the information required to understand the dependency relationships between components that will then drive the scheduling of the apply actions during the apply phase.	2 years ago
Martin Atkins	11bb10ba72	stackeval: Find references in arbitrary expressions and bodies ... Along with full expression/body evaluation, we'll also need to be able to determine what an expression or body refers to without evaluating it, so that we can use references to infer dependency relationships between significant side-effects in the apply phase, such as the relative ordering of applying the changes for different components.	2 years ago
Martin Atkins	8cb51f415b	collections: go-cmp helpers for the collection types ... A downside of using custom collection types is that go-cmp doesn't support them automatically, and so we need to help it out a little by providing additional options. An additional awkwardness is that go-cmp does all of its work at runtime using reflection, at which point the generic types have all been erased and lowered to many separate concrete types. Therefore we need some helper functions to produce a dynamically representation using interface types, cautiously exposing some internal details of these types only for go-cmp's use. The dynamic adapter here unfortunately means that the compiler likely won't be able to optimize away CmpOptions in non-test builds, but we'll accept that for now since the cost of that inert data and code is relatively small. If it turns out to be a problem then we'll need to find a different strategy, but hopefully not.	2 years ago
Martin Atkins	306b9008c4	stackeval: Clean some debug junk in ComponentInstance ... I apparently missed this when I was preparing my previous round of commits in this area. Mea culpa.	2 years ago
Liam Cervante	d2502265dc	removed: child modules should be removed if they are within a removed module (#34487)	2 years ago
Nick Fagerlund	9d7737b2fe	Merge pull request #34447 from hashicorp/nf/dec23-grpc-metadata-cloudplugin ... Cloudplugin: Pass configuration via gRPC metadata	2 years ago
Nick Fagerlund	ff62192819	Adjust CloudPluginConfig doc comments	2 years ago
James Bardin	d94a61735e	Merge pull request #34472 from hashicorp/jbardin/schema-filter-attrs ... `Block.Filter` can panic with a nested object which has no attributes	2 years ago
Liam Cervante	7e18b1183d	Add test verifying issue #34476 has been fixed (#34478)	2 years ago
Liam Cervante	bd40c411f7	Override provider schemas for mocked providers (#34481) ... * Override provider schemas for mocked providers * address comments	2 years ago
Nick Fagerlund	1a492b9dbb	Add test for (CloudPluginConfig).ToMetadata()	2 years ago
Nick Fagerlund	c398d0340f	Catch a missing assign-after-append ... I believe this is a slice append under the hood, which would mean `.Append()` consumes the original receiver and you shouldn't touch it afterwards.	2 years ago
Nick Fagerlund	7a1c49c61d	Append cloud plugin config metadata in plugin client requests ... - The cloud plugin wants a go-tfe client to get stuff from TFC. - Terraform knows how to find all the info you need in order to configure a go-tfe client, but it's sometimes scattered across configs and env vars and credentials helpers, and trying to re-implement the resolution logic identically in another codebase would be error-prone. - Therefore, it'd be best if Terraform did that resolution and just passed the plugin all the config info it needs. There are two options for that handoff: 1. Adding it as an additional argument to the primary Execute RPC interface. 2. Passing it as gRPC metadata, which is a pile of arbitrary key => list-of-strings pairs transmitted via HTTP/2 headers. The crux of the decision here is whether a proliferation of plugin protocol versions whenever we need to add more config data is better or worse than a fuzzy and typeless key/value bag. This PR is an implementation of option 2, for discussion and evaluation. (And after all, this is considered an experimental feature anyway.) It assembles the necessary information, translates it to the required metadata format, and attaches it to the request at the proper moment to get everything over to the plugin. As previous commits foreshadowed, we're leaning on the existing logic in the Cloud backend for pretty much all of this.	2 years ago
Nick Fagerlund	e1f0165693	Cloud command: Use cloud backend to find plugin config and service URLs ... The plugin needs some config info in order to actually build a go-tfe client and _do_ anything, and the Cloud backend is the one place that already knows how to look up and reconcile all the possible sources of that info. So, we'll just find a Cloud backend and pick its pockets. This also replaces our reimplementations of hostname lookup and service discovery, using work the Cloud backend already did.	2 years ago
Nick Fagerlund	c02533c889	Cloud backend: Cache service discovery in public ServicesHost field ... Most of the time, the Cloud backend is the only thing that needs to read the results of this discovery, and the only thing it cares about is the URL to the TFE v2 API service. However: - Other things might need to know about services that your TFC instance provides, including services other than the TFE v2 API. - There's a lot of complexity involved in determining which TFC instance you should be talking to (as well as the credentials for contacting it, the default org/workspace/project you're interested in, etc.), and it makes sense to leave the Cloud backend as the owner of that complexity rather than re-implement it in other potential consumers! So, since other TFC consumers will likely be configuring a Cloud backend to get their information anyway, might as well cache the known services so we don't have to repeat the discovery request. As part of this commit, note that I removed some code that swallowed a particular error during discovery. This was vestigial behavior that came over from the `remote` backend, but we were no longer performing the second part of the process (i.e. producing a more descriptive error in place of the one we swallowed). See https://github.com/hashicorp/terraform/pull/19659 for discussion about the original intent.	2 years ago
Nick Fagerlund	0248b6cf90	Give cloud TestRunner its own service discovery function ... I'm about to refactor the shared one away into a form it can't re-use. This is the only other callsite, so it can just own this logic.	2 years ago
Nick Fagerlund	054fdf93cf	Publicize Cloud's token and organization fields ... Sorry... I need them.	2 years ago
James Bardin	1a52436a88	validate provider function consistency ... Provider functions must be pure, and return consistent results for the same arguments. Store the results from all provider function calls in a global cache, and ensure future calls always match prior results. In the future we will store the data in the plan file, which will allow us to verify the results again during apply.	2 years ago
James Bardin	7ccb21d864	Block.Filter can panic with no attrs ... Make sure that we only dereference an attribute is it was set in the filter. While not extremely useful, a provider could have a nested object with no attributes.	2 years ago
Liam Cervante	3a75362cce	terraform test: run block names should be valid HCL identifiers (#34469) ... * terraform test: run block names should be valid HCL identifiers * fix test	2 years ago
hc-github-team-es-release-engineering	6114ec4390	Merge pull request #34367 from hashicorp/RELPLAT-955-EOY-license-updates ... [DO NOT MERGE UNTIL EOY] EOY license updates	2 years ago
CJ Horton	fcdeab5da9	Merge pull request #34423 from komisan19/fix/delete-min-function ... refactor: removed min function and adopted go standard min	2 years ago
James Bardin	6553e55dfa	marshal provider functions into the json schemas ... Add provider functions to the json schema output for providers. We leverage the jsonfunction package here to ensure that the serialization is kept identical, so that the consumers of the json function definitions can work equally with both.	2 years ago
James Bardin	a74c847656	test MarshalProviderFunctions ... Provider functions should marshal identically to an equivalent cty functions.	2 years ago
James Bardin	8c41a1e036	MarshalProviderFunctions	2 years ago
James Bardin	8125fe56a4	cannot statically validate an import target moved ... The validation for checking if an import target is the source of a moved block is incorrect. Besides the noted FIXME of the check being incorrect for moved modules (and also resources scoped within modules); an import can only be statically identified at the granularity of a resource block, but moved block can have individual instances in the `from` field. Since we already validate that a resource block must _not_ exist in the config for `moved`, we can rely on that causing `import` to fail with a nonexistant resource block during evaluation if they have the same target.	2 years ago
James Bardin	26188550c4	remove []byte literal from test	2 years ago
James Bardin	1644c2f63b	check function descriptions directly ... Now that we dynamically add scoped versions to the global functions we can't check the static Descriptions list. This should be OK since the function descriptions are taken directly from Descriptions list, and if any other scoped functions were added in the future, they may not be coming from the current global set of functions anyway.	2 years ago
James Bardin	d122b2cddd	add plan context provider func test	2 years ago
Martin Atkins	8b76d5052f	core: Minimal working implementation of provider-contributed functions ... This establishes all of the wiring necessary for a provider's declared functions to appear in the hcl.EvalContext when we evaluate expressions inside a module that depends on a provider that contributes functions. This does not yet make any attempts to guarantee that the provider-contributed functions correctly honor the contracts such as behaving as pure functions. Properly checking this is important because if a function doesn't uphold Terraform's expectations then it will cause confusing errors reported downstream, incorrectly blaming other components for the inconsistency.	2 years ago
James Bardin	b07f66a942	add a function to the simple-v6 test provider	2 years ago
James Bardin	33be99db1f	cache providers used for functions ... We need to be able to reuse providers which are used for function calls, so cache them in the EvalContext. This does require a way to close the provider however, which we can do via the normal root module close node since it already does the same for provisioners. Update the `CloseProvisioners` method to be a more general `ClosePlugins` method, and use that to call Close on the cached providers too.	2 years ago
Martin Atkins	17d7f7ff24	lang: Scope accepts "External Functions" ... This is intended as a bridge to functions defined outside of the core language, which will live in other namespaces in order to avoid ambiguity with any built-in functions. For now we only accept provider-contributed external functions. The lang package doesn't really know anything special about providers, so its only responsibility here is to enforce the naming scheme that any provider-contributed functions always live under the namespace prefix "provider::NAME::", where NAME is a name decided by whatever codepath instantiated the state as a unique identifier for a particular provider. This commit also includes updates to the core package to make it possible to pass in ExternalFuncs in principle, although in practice we don't yet have any codepaths which do so, and so this is effectively a no-op commit.	2 years ago
James Bardin	d7a1ebb1b7	add special cases for jsonfunction ... The jsonfunction package has special cases for some internal functions, and wee need to add the namespaced equivalents to those lists.	2 years ago
Martin Atkins	d1df979115	lang: Expose built-in functions under core:: namespace ... This essentially doubles up the registrations for all of the built-in functions to have both non-namespaced and core::-namespaced versions of each function. This is in preparation for later commits introducing other namespaces, such as a provider:: namespace which could hold functions that were contributed by providers that are currently in scope.	2 years ago
James Bardin	2574b899e7	Merge pull request #34385 from hashicorp/jbardin/provider-functions-impl ... client-side implementation of provider functions	2 years ago
kmoe	6ac20050a5	planproto: parse forget actions (#34433)	2 years ago
kmoe	cc3e3bd513	planproto: add Forget, CreateThenForget actions (#34430)	2 years ago
Jamie Finnigan	05b0bdac99	adopt hashicorp/cli fork of mitchellh/cli (#34429)	2 years ago
komisan19	e3a9dce998	fix: delete min function	2 years ago
Matej Risek	039cced8ae	Add component address to ComponentInstance under AppliedChanges (#34418)	2 years ago
Brandon Croft	2cf2943626	Update go-tfe mocks	2 years ago
James Bardin	78b30b71ab	Merge pull request #34383 from hashicorp/jbardin/provider-functions-proto ... Add protobuf for provider-defined functions	2 years ago
Graham Davison	ebb8b503bc	Merge pull request #34386 from hashicorp/s3/b-configurable-endpoints ... backend/s3: Fix upstream error with endpoints in shared config files	2 years ago
kmoe	8f3aa0e099	jsonformat: render forget-only attr changes as no-op (#34403) ... * jsonplan: document forget actions * jsonformat: format forget changes as no-op Previous to this commit, forget-only actions (i.e. "forget", not "create then forget") would be rendered using the forget action symbol for the top-level resource, and the delete action symbol for each resource attribute, with a new value of "null". This attribute rendering is identical to that for resource deletion, which might suggest to some users that Terraform plans to delete the resource, not just remove it from state. This commit tweaks the renderer so forget-only changes render as no-ops but with the forget action symbol and resource change comment.	2 years ago
James Bardin	e536ae9e07	round trip func decl tests	2 years ago
CJ Horton	89aea31bbd	stackstate: sensitive values are preserved when deserializing ... Without this, sensitive values cause a permadiff, since their sensitivity is lost on every round trip through state.	2 years ago
Graham Davison	a8028da56f	Adds test for standard envvar overriding legacy envvar	2 years ago
Graham Davison	c9eae74ba7	Adds tests to handle legacy config and envvars	2 years ago
Graham Davison	c701ad1e2a	Adds tests for STS endpoint override	2 years ago
James Bardin	b0baae5d3e	add file headers	2 years ago
James Bardin	2923ba6ac3	add CallFunction to stub provider	2 years ago
James Bardin	f9b93d13a2	update grpcwrap to handle provider functions	2 years ago
James Bardin	9abaf2f3b0	update provider code to match new rpc methods ... Add the types, methods, and basic framework to use provider defined functions.	2 years ago
James Bardin	c85859f26d	add provider funcs to internal interfaces	2 years ago
James Bardin	5c8fd0b147	add grpcwrap function methods to make it compile	2 years ago
James Bardin	9e164b5488	add mocks for new plugin protocol	2 years ago
James Bardin	c439b72599	proto: add provider-defined functions	2 years ago
James Bardin	e26ec58c63	Merge pull request #34380 from hashicorp/jbardin/unmanaged-plugin-schema-cache ... ensure unmanaged plugins have an address	2 years ago
Martin Atkins	be33a7e9e0	stackeval: Use prior state outputs when planning component destroy ... When a component is planned for full destroy, we should assume that its apply phase will be delayed until everything that depends on it has been destroyed, since otherwise we'd be violating the dependency relationship. That means that the expression value for a component pending destruction ought to be based on the output values from the prior state (which ought to have been refreshed as part of planning destroy) rather than the desired state (in which case all of the output values would be unset). This commit only deals with the _planning_ part of this problem, and only deals with the situation where we're planning a "full destroy" of the entire stack configuration and therefore all component instances in the configuration are going to be destroyed. In future commits we'll need at least the following additional work to complete this: - The apply phase needs to learn to schedule component instance destroy actions differently than other component instance actions, so that destroy cannot begin until everything that depends on the component has already been destroyed. - We also need to deal with the situation where only a particular component instance has been removed from the configuration, in which case only that one component instance is planned for destroy and everything else is planned as normal. In that case it would be invalid for anything to still be referring to that component instance in the configuration, and so the work in this commit would actually be irrelevant, but it'll take some other separate machinery to make sure that the removed component instances still get planned properly, which they don't today. This commit is just an incremental step toward the final correct behavior, which should allow the planning of a full destroy to succeed, although actually applying that plan is still likely to fail because the apply phase will not respect the required order of operations yet.	2 years ago
Martin Atkins	a47dc447c1	stackeval: Additional testing utilities ... In earlier commits we stubbed out some testing utilities as part of adding some unit tests, but we didn't end up using all of them in the initial round and it turned out that some of them were incorrect or incomplete. Now we'll improve the test utilities, this time with a focus mainly on integration-type tests, because we'll start adding those in the near future. Specifically: - utilities for rendering diagnostics, mainly in situations where they are unexpected and thus we just want to see the full information as part of the test results - helper functions for concisely asserting that there should be no diagnostics at all, or that there should be no error diagnostics - the existing helper for faking up a prior state based on hard-coded raw state messages was incorrect in that it was trying to use statekeys.Key values as map keys, and those values are not typically comparable. Instead, we should use the string representations that are designed to be used as unique map keys. - the testPlan function, and the lower-level testPlanOutput it's built with, help to encapsulate the asynchronous event-emitting behavior of Main.PlanAll into a "normal-looking" function that just blocks until it's done and then returns all of its results. This automatically round-trips the planned changes through serialization and deserialization as we would between a normal plan and apply phase, and so the calling test can make its assertions against the applyable plan object instead of the raw component parts it's made from.	2 years ago
Martin Atkins	2aba28eb1a	stacks: Track component instance existence in the state ... Although we can often use the presence of resource instances in a component instance as an implication of the component instance's existence, it's possible (albeit rare) for a component instance to have no resources in it at all, and thus it would be ambiguous whether it exists or not. Now we'll track the existence of the component instances themselves as extra objects in both the raw state and the state description. Along with their existence we'll also track a snapshot of the output values as they were at the most recent apply, which for now is primarily for external consumption but we'll also include them in the raw state in case it ends up being useful for something down the road.	2 years ago
Martin Atkins	bcccd67e32	tfstackdata1: A local DynamicValue message type ... Previously we were directly using the planproto.DynamicValue message type, but that's symmetrical with plans.DynamicValue in that it encodes only the value itself and not associated metadata such as the paths that have sensitive value marks. This new tfstackdata1.DynamicValue therefore echoes terraform1.DynamicValue by carrying both the value and the metadata together. This implies a slight change to one of the existing message types that was encoding a map of planned input values for each component instance, but in practice we're not currently making use of that data anyway -- it's there to enable a hypothetical extra correctness check in the stacks runtime so that we can detect inconsistency problems closer to their source -- and so this change doesn't affect anything in practice. Before we make use of it we'll probably want to change the internal API a little so that we can preserve the sensitive value metadata on those values, but that's beyond the scope of this PR. The main point of adding this is to track the output values for a component instance as part of the raw state, and so the new field for that is also added here although nothing will read or write it yet. Actual use of that new field will follow in future commits.	2 years ago
Martin Atkins	caeecbebcb	planfile: Move some of the proto convert funcs into package planproto ... Previously we had much of the mechanical conversion logic for going from in-memory structures to protobuf messages as unexported functions in the planfile package. We're now using planproto for parts of the representation of stack plans and states too, and so having those helpers exported is helpful. In some of our other protobuf packages we've adopted the convention of putting simple conversion functions in the same package as the protobuf stubs themselves, following the convention used upstream such as the additional encoding/decoding helper functions in the "anypb" package. This therefore follows that lead by adding some helpful constructor functions to package planproto and then making the equivalent functions back in planfile just thinly wrap them for now, to avoid churning the callers too much and thus making backports potentially harder.	2 years ago
James Bardin	a9cc3baf1a	ensure unmanaged plugins have an address ... Unmanaged plugins use a slightly different factory function which was missed when adding the address to the plugin struct. This made little difference in most cases, but now with the new global schema cache it would prevent the plugin's schema from being cached due to lack of a key. The end result was acceptance tests for very large providers would take considerably longer due to the overhead of repeatedly generating the schema. There is no good way to check the internals of an unmanaged provider in a unit test, but this PR puts the setup functionality in a single helper function so that unmanaged providers use the same setup codepath as the standard set of plugins.	2 years ago
Matej Risek	8ac8f18636	Feature/add additional fields to resource proto (#34364) ... Add additional fields to the AppliedChange#ResourceInstance We're adding Resource Mode, Resource Type and Provider Address to the AppliedChange's ResourceInstance	2 years ago
Martin Atkins	17f420102f	states: Local values no longer live in state ... Back when we added local values (a long time ago now!) we put their results in state mainly just because it was the only suitable shared data structure to keep them in. They are a bit ideosyncratic there because we intentionally discard them when serializing state to a snapshot, and that's just fine because they never need to be retained between runs anyway. We now have namedvals.State for all of our named value result storage needs, so we can remove the local-value-related fields of states.Module and use the relevant map inside the local value state instead. As of this commit, states.State now tracks only the data that we actually persist between runs in state snapshots, which will hopefully avoid future bugs resulting from the former difference in fidelity between a freshly-created in-memory state vs. one loaded from a snapshot.	2 years ago
Martin Atkins	b0b8d4aa6f	states: Only track root module output values ... For a very long time we've had an annoying discrepancy between the in-memory state model and our state snapshot format where the in-memory format stores output values for all modules whereas the snapshot format only tracks the root module output values because those are all we actually need to preserve between runs. That design wart was a result of us using the state both as an internal and an external artifact, due to having nowhere else to store the transient values of non-root module output values while Terraform Core does its work. We now have namedvals.State to internally track all of the throwaway results from named values that don't need to persist between runs, so now we'll use that for our internal work instead and reserve the states.State model only for the data that we will preserve between runs in state snapshots. The namedvals internal model isn't really designed to support enumerating all of the output values for a particular module call, but our expression evaluator currently depends on being able to do that and so we have a temporary inefficient implementation of that which just scans the entire table of values as a stopgap just to avoid this commit growing even larger than it already is. In a future commit we'll rework the evaluator to support the PartialEval mode and at the same time move the responsiblity for enumerating all of the output values into the evaluator itself, since it should be able to determine what it's expecting by analyzing the configuration rather than just by trusting that earlier evaluation has completed correctly. Because our legacy state string serialization previously included output values for all modules, some of our context tests were accidentally depending on the implementation detail of how those got stored internally. Those tests are updated here to test only the data that is a real part of Terraform Core's result, by ensuring that the relevant data appears somewhere either in a root output value or in a resource attribute.	2 years ago
Martin Atkins	1bea2155ec	core: Remove variable-specific methods from EvalContext ... Now that we have the separate namedvals.State type to encapsulate all of the named-value tracking we can simplify the EvalContext API to just return that object directly. This removes the slightly odd evolved API for setting and retrieving input variable values, instead now just calling directly into the relevant namedvals.State methods. It also slightly simplifies some of our test code because there's no longer any need to mock accesses to what is just a temporary in-memory data store anyway.	2 years ago
Martin Atkins	55c86a1919	core: use namedvals.State to track input variable evaluation ... This replaces the direct manipulation of a map shared between three different components, encapsulating that manipulation now inside a single wrapping API that itself ensures safe concurrent access. In future commits we'll do the same for local values and output values, but for now those part of namedvals.State remain unused.	2 years ago
hc-github-team-es-release-engineering	a90ba0bfe4	update year in license files	2 years ago
Martin Atkins	48a2c00f09	core: Use a separate node type for externally-configured providers ... Previously we tried to handle externally-configured providers by treating them mostly like normal provider configurations except quietly skipping the configuration step when visiting them. However, that wasn't sufficient because the runtime would still try to decode a placeholder empty configuration to send into the InitProvider method, which failed in cases where the provider had at least one required configuration argument. Instead then, we'll use a separate graph node type to represent externally-configured providers, which allows us to use a much simpler implementation for them and also, as a bonus, makes it considerably easier to confirm that they are being treated as expected when reading the trace log output, because we include the concrete node types in the graph dumps and because we'll generate different trace messages when visiting those nodes. Altogether this avoids misleading error messages being returned when using a provider with at least one required configuration argument through the stacks runtime. Previously this external providers concept had been tested exclusively through the stacks runtime integration tests and not directly inside "package terraform", and so this also introduces an explicit test for this that's closer to the implementation, so we can inspect the behavior a little more closely.	2 years ago
Liam Cervante	5cbf7c5f96	Add docs for automatic var files in test directories (#34349)	2 years ago
Nick Fagerlund	bf9dd097ba	Merge pull request #34352 from hashicorp/nf/dec23-no-saved-plans-on-vcs ... Disallow saved cloud plans in VCS workspaces	2 years ago
Nick Fagerlund	9aa18a4107	Merge pull request #34353 from hashicorp/nf/nov23-cloudplugin-dev-overrides ... Add TF_CLOUD_PLUGIN_DEV_OVERRIDE to enable cloudplugin dev	2 years ago
Jared Baker	4729af3ccf	Merge pull request #34337 from hashicorp/s3/use_legacy_workflow-deprecation ... backend/s3: default `use_legacy_workflow` to false, deprecate	2 years ago
kmoe	ebd31f5022	core: remove import,forget targets from PlanOpts (#34355) ... PlanOpts should contain only static options which are known before planning. Previous to this commit, ctx.plan was modifying the opts object, which, since it is passed via pointer, could be shared by other ctx.plan calls happening concurrently, producing weird behaviour. Such weird behaviour was only noticeable during tests, in which it is common to pass a pointer to DefaultPlanOpts. When one test that modified opts.forgetResources was run at the same time as another test, DefaultPlanOpts.forgetResources would be populated when it shouldn't be. The real Terraform binary does not do this and ctx.plan is only called once per run. One fix for the race condition would be to have ctx.plan take a deep copy of the entire PlanOpts object which it can modify without touching the caller's object. Due to the multiple nested data structures in PlanOpts this would be a large amount of code, with a lot of DeepCopy methods it would be easy to miss when adding struct fields. Instead we just remove ImportTargets, forgetResources, and forgetModules from PlanOpts, since they did not belong there in the first place. These slices must be populated dynamically during planning once the entire config tree is available and passed to the graph builder.	2 years ago
Nick Fagerlund	d2e07f6f03	Add `TF_CLOUD_PLUGIN_DEV_OVERRIDE` to enable cloudplugin dev ... Currently, Terraform will only run properly signed versions of the experimental cloud plugin that were downloaded from a TFC instance that provides the appropriate service. That obstructs development on new cloud plugin features! Our internal teams will need a "dev override" capability, like what we offer provider authors. However, unlike with providers, we don't have to integrate this into a heterogeneous mix of sources for mirroring and caching a wide range of binaries. There's only one cloud plugin, HashiCorp controls it, and end users should never need to override the location of the binary for non-development reasons. Thus, we have the luxury of being quite a bit stupider in how we handle the override signal. Instead of adding it to the CLI config file schema, we'll just use a single environment variable whose value is the path to an alternate binary. Enter: `TF_CLOUD_PLUGIN_DEV_OVERRIDE`.	2 years ago
Nick Fagerlund	4ec407af08	Disallow saved cloud plans in VCS workspaces ... By convention, Terraform Cloud workspaces connected to a VCS repo should should only apply configs that come from the designated branch, to ensure that version control remains the source of truth for that infrastructure. The cloud version of `terraform apply` already includes a guardrail for that, but we forgot to include an equivalent one for `terraform plan` when performing saved plans (which can be applied later, unlike a purely speculative plan). We suspect that TFC actually needs some additional formal access controls around assigning configuration versions to workspaces, as everyone tends to be surprised to learn that the API doesn't actually prevent this. But in the meantime, what's good for the goose (apply command) is good for the gander (plan command), so I'm copying over the check to voluntarily enforce the VCS workflow for plans that can be applied.	2 years ago
Nick Fagerlund	9a7fae5ff1	Typo fix	2 years ago
Liam Cervante	dee12de5e0	stacks: Add ValidateStackConfiguration RPC to stacks API service (#34347)	2 years ago
Kubo	c98e355ca1	terraform test: Collect variables from default var file within testing directory (#34341) ... * Add logic to collect variables for terrafrom test * Add tests for test variable collection * Update the test variable collection implementation * Update internal/backend/local/test.go Co-authored-by: Liam Cervante <liam.cervante@hashicorp.com> * Update internal/backend/local/test.go Co-authored-by: Liam Cervante <liam.cervante@hashicorp.com> * Move test variables into var file * resolve diff from cross-branch switch * go fmt --------- Co-authored-by: Liam Cervante <liam.cervante@hashicorp.com>	2 years ago
Martin Atkins	7055650270	terraform console: Option to evaluate in a planned state ... Previously "terraform console" always evaluated in a kinda strange context where resource instance data comes from the prior state, but other derived of values end up being calculated dynamically based on the current configuration, which is okay for simple cases but can be confusing if the configuration has changed significantly since the most recent apply, or if there haven't yet been any applied changes. Now we'll allow an optional new mode where Terraform runs the normal plan phase (as if running "terraform plan") and then uses the resulting _planned state_ as the basis for evaluation, allowing evaluation against a partial approximation of what the world ought to look like if these changes were applied, without having to actually apply them first. As with the previous use of the eval walk, it's possible that an erroneous situation will still produce a partial evaluation scope, and so the console still allows evaluation against that scope but with a caveat that it might produce unexpected results. In practice this can be useful for debugging situations like when unknown values block planning of an object, to allow inspection of the values that are contributing to that blocked planning operation even though the plan is not yet complete.	2 years ago
Martin Atkins	0f845adfdb	moduletest: EvalContext lives outside of the "terraform" package ... The initial implementation of the module testing language included some elements deeply embedded inside "package terraform", which is primarily concerned with the traditional Terraform language, which we might now call the "module language" to distinguish it from the test language and the stacks language. That was a reasonable design decision due to the modules runtime needing to evaluate some expressions in a similar way to how the modules language does it, but unfortunately it caused the test language runtime to depend a little too closely on some current implementation details of how the modules runtime deals with expression evaluation, and those details are about to change as we teach Terraform to be able to deal with unknown values in new places, as part of resolving this issue: https://github.com/hashicorp/terraform/issues/30937 This commit therefore refactors the test runtime a little so that it doesn't intrude so deeply into the module runtime's business. Instead of directly extending the module runtime's expression evaluator, we'll instead call into it in a similar way to how the "terraform console" command does, by wrapping the evaluation data produced by the module runtime to extend it with the only two special things that the test runtime needs: references to the output values of previous runs, and references to test-only variables that aren't declared as part of the module under test. This effectively means that the test runtime can now evaluate a superset of the expressions that would be valid in the global scope of the module under test using only the public API of the modules runtime, with no testing-specific evaluation logic embedded inside the modules runtime. This new strategy is also somewhat similar to how the stacks runtime calls into the modules runtime to evaluate components, but the stacks runtime only actually cares about output values and so doesn't need a full evaluation scope. This means the result here is a little more symmetrical with the approach we took in stacks, which will hopefully allow experience working on one to translate better to the other.	2 years ago
Martin Atkins	a2ca1208a4	lang: EvalCheckErrorMessage is language-agnostic ... We have a design goal of reusing the same design patterns for all of our "condition-like" configuration constructs across all of Terraform's languages, including custom conditions, validation rules, and check blocks. Now that we have the stacks language and the test language in addition to what I guess we'll now call the "modules language", we'll eventually want to hoist much of the handling of such blocks into a language-agnostic place where all three languages can use it. This is a very meagre start towards that goal, just factoring out the one function for evaluating and validating the "error_message" argument associated with a condition. I've factored it out into package "lang" because that is currently the closest thing we have to a language-agnostic place for this. It's not _quite_ right because lang.Scope is quite specific to the modules language, but we're already importing things from this package in both the test runtime and the stacks runtime and so it's a good enough place until in future we find a more complete design to represent the common elements across all three languages.	2 years ago
Martin Atkins	1249666f9f	core: Plan and apply can optionally return an expression scope ... This is a similar idea to terraform.Context.Eval, but rather than using the special eval-only walk (which just reads data from the latest state snapshot) we can now produce an evaluation scope based on the result of either a plan or an apply operation, evaluating against either the planned state or the new state respectively. This will allow (in future commits) some code reuse for the test harness and also the option to use "terraform console" to evaluate expressions against planned state rather than only against prior state. However, as of this commit there are not yet any direct callers of these new methods except for the two new test cases.	2 years ago
Martin Atkins	8924fbc91b	states: SyncState becomes read-only after close ... Previously a SyncState became completely invalid after a call to Close, but that's a little too strict because SyncState objects live on as part of any returned lang.Scope objects that a caller might want to use for (read-only) expression evaluation. As a pragmatic compromise then, we'll instead continue to allow read access through the SyncState object after it's closed, which is safe as long as the caller does not try to perform those reads concurrently with its own modifications to the underlying state.	2 years ago
Jared Baker	470a851e45	backend/s3: default use_legacy_workflow to false, deprecate ... In Terraform 1.7 we'll begin phasing out the legacy credential chain evaluation order by defaulting use_legacy_workflow to false and deprecating the argument. This will bring the default behavior into alignment with the AWS SDKs and CLI. Practitioners can still preserve the legacy behavior by setting this argument to `true`. In Terraform 1.8 this argument will be removed, and the S3 backend will always use the default credential chain evaluation order from the AWS SDK for Go.	2 years ago
Liam Cervante	9ea1aa0a86	terraform test: skip destroy step for empty run blocks (#34331)	2 years ago
Liam Cervante	e703eb7b17	stacks: add the packages service to the rpc api (#34319) ... * stacks: add the packages service to the rpc api * go generate	2 years ago
Martin Atkins	94b3242123	core: Include some rejected changes in the plan ... Some time ago we ruled that Terraform Core would make a best effort to return a partial plan describing the subset of actions that were successfully proposed before encountering an error, which Terraform CLI and Terraform Cloud then rely on to present some additional context to support the associated error messages. This change aims to improve that effort by making a distinction between the failure of the planning operation itself vs. something else in the configuration ruling that the successfully-created plan is unacceptable for some separate reason. In that case, it's helpful to still include that proposed change in the plan, because the planning step itself succeeded and these problems are in a sense "between" the planning operations, blocking any downstream work from starting. In particular, this change arranges for a failed prevent_destroy check or a failed postcondition check to still include the planning result that they were checked against, which then allows the UI the option of displaying that planned action alongside the error describing why it was unacceptable. This doesn't include any Terraform CLI UI changes, but the UI layer is already built to show partial plans returned alongside errors and so as of this change the additional planned changes are already included. It'll be up to future maintainers of Terraform CLI to decide whether and how to refine that output, but the existing behavior is sufficient for now.	2 years ago
Martin Atkins	3c87c0a643	core: NodeAbstraceResourceInstance.checkPreventDestroy returns diagnostics ... This method was returning diagnostics wrapped inside an error, and all of its callers were then immediately unwrapping it and using the diagnostics inside anyway. To make that clearer, we'll just return diagnostics directly. (It's likely that this was originally written as a function that returned an error before we even _had_ the concept of diagnostics, and over time we've gradually migrated all of the callers to be ready to accept diagnostics but neglected to actually update the function itself.)	2 years ago
Martin Atkins	d905271f27	core: Connect resource instance nodes with their expanders during apply ... This was inadvertently trying to compare the string representation of an addrs.AbsResource with the string representation of an addrs.ConfigResource, making the two not match whenever the containing module has count or for_each set. The effects of this were pretty subtle: - During normal apply the whole-resource state updates were not necessarily happening before the instances got evaluated, but we got away with it most of the time because the provider configuration for a resource rarely changes and because we were redundantly re-calculating the for_each/count for every instance and thus not relying on the expander's idea of the instance expansion. - During destroy-time apply the pruning transformer was pruning the expansion nodes out altogether due to them having no dependents, and so we were not updating the state and expander at all in that case. This now correctly ensures that the expander node runs first and so gets a chance to update the resource's selected provider configuration and the expander's sense of the resource expansion before we visit any instance-specific code that relies on those. This also switches to using `addrs.Map` instead of a direct Go map because that allows the Go compiler to to type-check that we're using the address types consistently, thereby reducing the risk of regressing this again in the future. There are unfortunately no unit tests here because (as usual) our graph transformation process resists unit testing as a result of all of the transformers being subtly interdependent. However, this didn't regress any existing tests and I manually inspected the graphs of the configurations that caused me to spot this in the first place to verify that they are now being constructed correctly.	2 years ago
Liam Cervante	392e2a6e21	Fix the compile error in command/test_test.go (#34324)	2 years ago
Martin Atkins	8fe469e3d4	instances: "skipUnregistered" instead of "skipUnknown" internally ... Recently we've introduced the idea of the expansion of a particular module or resource not yet being known, which has inadvertently introduced some ambiguity about what "unknown" means in this package: "unknown" previously represented "not registered yet", which is different than having explicitly registered that we don't know. To remove that ambiguity, we'll now use the term "unregistered" to talk about the "not registered yet" situation, and reserve "unknown" for when we know that we definitely don't know. This is only an internal naming change within the package and doesn't affect any exported API.	2 years ago
Martin Atkins	f41f47563e	namedvals: Can ask if a particular value has been registered ... We shouldn't need this in normal code because the graph walk should ensure that values are always registered before we read them, but this will be useful in test code to avoid panicking in situations where the graph walk doesn't behave correctly and we _do_ end up reading and writing in the wrong order.	2 years ago
Martin Atkins	869750da61	namedvals: A new package to encapsulate the tracking of named values ... Currently we have a slightly different treatment for each of the different kinds of named values, which makes it hard to generalize the implementations of their very similar behaviors. This package should over time take ownership of the problem of tracking all three kinds of named value during a graph walk, with result data structures that use this information (such as state and plan objects) copying the relevant values out of here only once the graph walk is complete. Nothing is using this yet, though. Some retrofitting of this into Terraform Core will follow in subsequent commits.	2 years ago
Martin Atkins	908c5fa44f	addrs: PartialExpandedModule.String correct for entirely-unexpanded ... This logic was previously incorrect for any case where the number of expanded steps was zero, causing the string representation to begin with a dot when it ought not to.	2 years ago
Liam Cervante	6670ab43f7	terraform test: rearrange the order of destroy operations (#34293)	2 years ago
Liam Cervante	08eafe0f2e	Fix flaky test in command/test_test.go (#34320)	2 years ago
kmoe	a718f70f85	Add removed block (#34251) ... * terraform: remove redundant code NodeDestroyResourceInstance is never instantiated with a DeposedKey of anything other than states.NotDeposed, so the deleted code is never run. Deposed objects get a NodeDestroyDeposedResourceInstanceObject instead. * tfdiags: add helper func * configs: introduce removed block type * terraform: add forget action * renderer: render forget actions * terraform: deposed objects can be forgotten Deposed objects encountered during planning spawn NodePlanDeposedResourceInstanceObject, which previously generated a destroy change. Now it will generate a forget change if the deposed object is a forget target, and a destroy change otherwise. The apply graph gains a new node type, NodeForgetDeposedResourceInstanceObject, whose execution simply removes the object from the state. * configs: add RemoveTarget address type * terraform: modules can be forgotten * terraform: error if removed obj still in config * tests: better error on restore state fail * Update CHANGELOG.md	2 years ago
Graham Davison	bfd0c3029c	Merge pull request #34313 from hashicorp/s3/update-aws-sdk-go-base-v2-beta.43 ... Updates `aws-sdk-go-base`	2 years ago
Graham Davison	39b7def2bd	Updates `aws-sdk-go-base`	2 years ago
Martin Atkins	844b1616e8	command/graph: Simpler resource-only graph by default ... Unless a user specifically requests a real operation graph using the -type option, we'll by default present a simplified graph which only represents the relationships between resources, since resources are the main side-effects and so the ordering of these is more interesting than the ordering of Terraform's internal implementation details.	2 years ago
Martin Atkins	135f14275b	core: Graph.ResourceGraph ... This helper method derives from an execution graph a typically-smaller graph which describes only the relationships between resources. This could then be used for analyses that only care about the relationships between resources and don't need to consider all of the other objects that might indirectly glue the resources together.	2 years ago
Martin Atkins	ce267925e6	addrs: Graph of addresses ... This is another contribution to our slowly-growing library of data structures of address types that implement UniqueKeyer. This is currently just a wrapper around *dag.AcyclicGraph which avoids the need for the caller to type-assert the results and which has hopefully less confusing method names for analyzing the edges. However, that's just an implementation detail and we should not expose that in the public API of this type.	2 years ago
Martin Atkins	b64953dba3	stacks: Summary docs giving an overview and some details about stackeval	2 years ago
Martin Atkins	4b763be987	rpcapi: A README summarizing the role and behavior of this package	2 years ago
Martin Atkins	d46f0ed015	promising: A readme giving an overview of how this package works	2 years ago
Martin Atkins	d9a4f9a06b	instances: Methods for retrieving the instance keys of modules/resources ... These new methods both take into account the possibility of there being unknown keys, which will be useful for the expression evaluator to decide whether it can return a specific type or not.	2 years ago
Martin Atkins	ea0fd23407	instances: RepetitionData placeholders to use for unknown expansion ... Now that Expander can support unknown expansions for modules and resource instances, we'll need some placeholders to use when evaluating the configurations of those objects to predict as much as possible what the final configurations of all of their instances will look like. This is a similar problem to checking whether a not-yet-expanded object has valid configuration during the "validate" phase: we're essentially asking what configuration all instances of the object will have in common, and leaving unknown anything that might vary between instances. Nothing is using these yet, but uses will follow in later commits.	2 years ago
Martin Atkins	ad53ca6020	instances: Expander.UnknownResourceInstances ... This is the resource instance equivalent of UnknownModuleInstances added in an earlier commit, returning a result that represents a set of possible resource instance addresses that _might_ be declared by the currently-unknown count or for_each argument.	2 years ago
Martin Atkins	9e80dfde2a	instances: Expander.UnknownModuleInstances ... This new method effectively returns a set of all of the instances of a particular module whose addresses can't be fully determined yet, but does so in the form of a set of patterns that match all of those as-yet-unknown instance addresses. So far this isn't used anywhere but in a future commit Terraform Core can use this to generate placeholder unknown values when walking inside unexpanded modules once we make it legal to have an unknown value for the repetition argument of a module call.	2 years ago
Martin Atkins	4d0087e3ef	addrs: PartialExpandedModule and PartialExpandedResource ... In preparation for making it not an error to have unknown values in count or for_each arguments, we will need a way to talk about the as-yet-undetermined set of module or resource instance addresses that will result once we _do_ learn the instance keys in a later step. Unlike many of our address types these have unexported internals so that we can do some semi-uncool things with our address types that external callers are not normally supposed to do. This means they therefore need a bunch more methods than our address types typically need so that it's possible to access the relevant parts of the internal representation.	2 years ago
Martin Atkins	a58ee3eceb	instances: Allow registering expansion as "unknown" ... In preparation for supporting partial planning when count or for_each are not known enough to finalize instance keys, this allows the core graph walk to register when it encounters an object whose repetition argument isn't yet "known enough" to actually expand, which then allows downstreams to still expand the subset of instances whose addresses are fully known but (for now) quietly ignores any subtrees under objects that cannot be expanded yet. In subsequent commits we'll add additional API for finding the partial addresses of objects that _cannot_ fully expand yet, so that the graph walk can still keep track of those and report them to the user.	2 years ago
Liam Cervante	e38ca6ea60	Fix race conditions in mock provider and mock tfc client (#34306)	2 years ago
Martin Atkins	f1d68e89c0	addrs: AbsLocalValue is a UniqueKeyer	2 years ago
CJ Horton	d60a3db261	stackstate: unmark values before serializing apply change descriptions ... When emitting apply descriptions for changes that contain sensitive values, we need to unmark the value before re-serializing it with MsgPack.	2 years ago
Liam Cervante	afc83356ca	stacks: actions for a no-op change are a concrete list (#34260)	2 years ago
James Bardin	f4a5b11dc1	Merge pull request #34240 from hashicorp/jbardin/generate-config-no-instances ... import: don't panic if expanded import returns no instances with config generation	2 years ago
Martin Atkins	72543dc4ed	rpcapi: Some code consistently cleanups in preparation for first merge	2 years ago
Martin Atkins	3f569b55c5	stacks: Some code consistency cleanups in preparation for first merge	2 years ago
Martin Atkins	8a99c3c0e5	stacks+rpcapi: Add some missing copyright comments	2 years ago
Martin Atkins	1c734048af	addrs: Add a missing copyright comment	2 years ago
Martin Atkins	3678815bb3	stackeval: Tests for provider instance configuration evaluation	2 years ago
Martin Atkins	1872c9d9be	stackeval: inPromisingTask cancels context when test completes ... This is just some extra insurance to reduce the risk of us leaving things dangling after a test terminates. Although tests should ideally not rely on this and should terminate background work properly as part of the test, this should catch some things that might get left behind if a test ends in an ungraceful way.	2 years ago
Martin Atkins	94ca9bc613	stackeval: testEvaluator arranges for its result to be cleaned up ... Some operations against an "evaluator" (type Main) register cleanup actions that need to be run once the evaluation has completed. To avoid leaking resources while we're running tests we'll arrange for any test that uses the "testEvaluator" helper to automatically run the cleanup actions on the returned evaluator once the test is complete. At the time of this commit the only possible cleanup action is terminating any provider pluginc client processes that might be running, and none of our tests are actually doing that so this doesn't really change anything, but it's here to avoid problems if we add additional cleanup actions in future, or if any tests start actually using child plugin processes.	2 years ago
Martin Atkins	d7461ce202	stackeval: Initial tests for type "Provider" ... These tests focus on the behaviors related to deciding the set of dynamic instances for a particular provider and producing the values that act as references to providers. These tests also revealed a small but important bug in ExprReferenceValue in the for_each case, which is now fixed as part of this commit.	2 years ago
Martin Atkins	d3e0b89e02	stackeval: Some initial tests for the Component type ... This includes Component.CheckInstances and Component.CheckResultValue.	2 years ago
Martin Atkins	91b8ea3d76	stackstate: A helper for loading state during tests ... The main entry point here assumes the caller is providing serialized state objects wrapped in anypb.Any messages. That's an inconvenient representation for hand-writing in tests, so the new function LoadFromDirectProto allows skipping the deserialization steps and instead has the caller write the wanted values as if they had already been parsed/unmarshaled.	2 years ago
Martin Atkins	95ff474ee9	stackeval: Tests for StackCall.ResultValue	2 years ago
Martin Atkins	333c502990	stackeval: Support each.key, each.value, count.index and self references ... For all contexts where these have a defined value, they can now be used in expressions and evaluate to whatever they ought to evaluate to.	2 years ago
Martin Atkins	75602fa9eb	stackeval: Tests for StackCall.Instances	2 years ago
Martin Atkins	48d2d15a40	stackeval: Some basic tests for OutputValue.CheckResultValue	2 years ago
Martin Atkins	13edd30479	stackeval: Test-only globals ... Because many stacks language features are dependent on others to do useful work, it's tempting to focus only on integration testing of combinations of features used together. However, we known from our experience with the modules runtime (the "terraform" package) that over time this becomes a huge maintenance burden, because any non-trivial change tends to invalidate hundreds or thousands of integration tests, and because of their broad scope its often hard in retrospect to figure out what exactly a particular test was aiming to test vs. what it was just relying on as a side-effect. To try to minimize these cross-dependencies and thus enable something closer to unit testing, here we introduce a special kind of symbol to the stacks language which is available only to unit tests in this package. "Test-only globals" -- an intentionally-clunky name to avoid squatting on useful names -- can be set as part of the Main object and, when defined, are available for use in all situations where we perform expression evaluation against a stack. The "globals" in the name represents that, unlike just about everything else, they are defined once but available in all stacks in the configuration tree. This design is a tradeoff: it introduces a bunch of extra code that is here entirely to support testing, but hopefully this code is segregated enough from everything else that it's unlikely to change significantly under future maintenance, thereby hopefully minimizing the need for future cross-cutting test maintenance too.	2 years ago
Martin Atkins	bb0e833785	stackeval: Remove stale comment about output value changes ... This comment was originally assuming we'd need a separate field to track the type information for the serialized output values, but we actually don't need that because the type information is encoded in-band in the MessagePack representation of the values.	2 years ago
Martin Atkins	0281170645	stackeval: Remove stale todo about prior state in plans ... This TODO was noting that an earlier version of this code was not including any prior state information in the "modules runtime" plan. At the time I wrote the comment I was expecting to handle that here as a separate step, but I later refactored this to include the prior state as part of the returned plan, since that's where the modules runtime expects to find it during the apply phase anyway. This comment is therefore now redundant, because no extra work is required in this location to get the prior state populated.	2 years ago
Martin Atkins	9c698a1de6	stackstate: Remove stale comment in AppliedChangeResourceInstanceObject ... This comment was noting that an earlier stub of this function was only returning an external facing "state description" object and not including the raw state for use when creating future plans. That was subsequently fixed -- this now returns both description and raw forms -- but the comment stuck around.	2 years ago
Martin Atkins	0952457778	stackplan: Remove stale TODO about output value changes ... This recorded that we needed to encode the type information for the output values. In the end we didn't need to do anything special for that because we're borrowing the pre-serialized data originally designed for the traditional Terraform CLI saved plan format, and that encodes the type information as part of the MessagePack encoding in-band.	2 years ago
Martin Atkins	b54a5be8e9	rpcapi: Remove Stacks.FindStackConfigurationProviders function ... This was unimplemented because we didn't end up needing it. We might find a use for something like it someday, but we'll wait and see what the requirements are before we properly define it as part of the API. (The current way we achieve "finding providers" is to use a dependency lock file which is assumed to already contain exact provider selections. The most likely reason for something like FindStackConfigurationProviders to return is to allow building a tool for generating those dependency lock files, which will require analyzing the configuration to find which providers it actually depends on, but it remains to be seen exactly what API would make sense for that operation.)	2 years ago
Martin Atkins	992f6d4115	stackeval: Fix input variable evaluation in embedded stacks ... Two bugs were making this not work quite right: - The Stack type wasn't treating its child stack objects as singletons, instead creating a new one on each request and therefore preventing child objects from being treated as singletons too. - StackCallInstance.CheckInputVariableValues wasn't actually using the value from its expression evaluation, and was instead just always returning an empty object. This commit also includes the tests that helped find these bugs.	2 years ago
Martin Atkins	9a943af56d	stackruntime: Arbitrary expression evaluation ... Here we introduce a new evaluation phase for the stacks runtime called "inspect", which aims to provide a read-only view of a configuration and an associated state snapshot. The main idea of this is to use it as a more convenient vehicle for unit testing despite it not really being viable to decompose the main evaluation logic into smaller parts: the language features are all directly interdependent on one another, but we can at least minimize our interactions with the Terraform modules runtime when we're testing anything other than actually planning and applying components. However, this also gives an opportunity to expose evaluation of arbitrary expressions in a similar manner as "terraform console", which is for now just another debugging aid but will probably someday appear as a Stacks-flavored version of the "terraform console" command.	2 years ago
Martin Atkins	d689237a48	stackeval: Tests for the "perEvalPhase" helper	2 years ago
Martin Atkins	5aaed3111e	stackeval: Reusable and testable for_each handling ... Previously we had some similar code duplicated into each of the three kinds of object that support for_each: components, embedded stack calls, and provider configurations. Instead, we'll hoist that code out into a function that all three can call. We can then actually test that function in isolation (independently of the three callers) to get more confidence that future maintenance won't inadvertently break the slightly-unusual contract of treating a nil map differently from a non-nil empty one, which is unfortunately likely to defy the expectations of a typical Go developer since those two values are normally considered equivalent.	2 years ago
Martin Atkins	1180096850	stackeval: Unit tests for EvalExpr and EvalBody	2 years ago
Martin Atkins	3c6c5effd5	stackeval: Applying must use apply-time provider clients ... Previously we were incorrectly trying to use the plan-phase provider clients during the apply phase, which causes the provider to get configured as it would during the plan phase and, in particular, crashes if the provider configuration refers to input variables because the plan-phase input variables are not available during the apply phase.	2 years ago
Martin Atkins	6e7eff9b4b	stackruntime: Report deletion of resource instance objects ... Previously our strategy for reporting the "applied change" results for resource instance objects was to iterate over the final state and emit an object for each resource instance we found in there. That's not a sufficient solution though, because if any objects are deleted as part of applying the plan then they won't be in the final state and so we won't know to tell the caller that they should drop the relevant objects from the stack-level state. Instead, we'll keep track of all of the resource instance objects that could potentially be affected by applying a plan and then use that set to decide which "applied change" objects to emit. If one of the affected objects is not present in the final state then we'll assume it was deleted and so command the caller to delete its records of that object from both the raw state and the state description. With this in place, it's now possible to create a destroy-mode plan and then apply it to end up with no resource instances at all. The stack runtime itself is still missing handling of destroying its own objects like components and embedded stacks, so this isn't a complete solution but it does at least allow properly cleaning up the records of objects that exist in remote systems, so it's now easier to clean up a development environment with real infrastructure backing it.	2 years ago
Martin Atkins	c6894cc617	stackeval: ComponentInstance: all output values during planning ... If planning fails partway through then we might have an incomplete set of planned changes for our root output values. Previously any output values we hadn't visited yet would get omitted altogether from the value representing a component instance, which is likely to cause confusing downstream errors if any other expression is referring to those missing output values. Instead, we'll use the configuration to find all of the statically-configured output values and fill out any that don't have a planned change with a placeholder cty.DynamicVal. This will make references to those output values return an unknown value, instead of raising an error.	2 years ago
Martin Atkins	972f1c0678	stackplan: Track provider config for every resource instance change ... Previously we were still partially relying on the record of this in the embedded planproto object, but since we're now allowing "planned changes" that don't actually include a change in that traditional sense we need to track the provider configuration address separately as a top-level field. As with some of the addresses before, this means we're now storing the provider config address redundantly in two places when there _is_ a planproto change. Hopefully we'll improve that someday, but we don't need to sweat it too much right now because this only affects the internal raw plan format that is not subject to any compatibility promises between releases, so we can freely change it in any future version.	2 years ago
Martin Atkins	7634f00f0f	addrs: AbsProviderConfig.Equal ... As with other Equal (or sometimes "Equals") methods in this package, this returns true if the two given provider configuration addresses represent the same object.	2 years ago
Martin Atkins	4e34e6ebfa	stacks: Track prior state for all resource instance objects ... Previously our raw plan data structure only included information about resource instances that have planned actions, which meant we were missing those which only get their state updated during refresh and don't actually need any real work done. Now we'll track everything, accepting that some "planned changes" for resource instance objects will not actually include a planned change in the typical sense, and instead only represent a state update.	2 years ago
Martin Atkins	e76fb9f3d9	states: State.AllResourceInstanceObjectAddrs is back! ... In the previous commit we renamed AllResourceInstaceObjectAddrs to AllManagedResourceInstanceObjectAddrs to reflect that it filters out any resource instances that aren't "managed". This now adds back in a State.AllResourceInstanceObjectAddrs method which does what the other one sounded like it did before: returns all of the known addresses without any filtering whatsoever.	2 years ago
Martin Atkins	0811395e6f	states: State.AllResourceInstanceObjectAddrs should say "Managed" ... This method used the more appropriate name AllManagedResourceInstanceObjectAddrs in its docstring, but was incorrectly named just AllResourceInstanceObjectAddrs in its actual definition, obscuring the fact that it filters out everything except objects from managed resources. Now we'll use the originally-documented name to make this clearer. This also factors out the implementation into a separate unexported function that takes the filtering criteria as an argument, since in future commits we're going to add other variations of this which return different subsets of the resource instance objects in the state.	2 years ago
Martin Atkins	ac562eeb6f	states: State.AllResourceInstanceObjectAttrs returns a set ... Originally we had this function returning a slice of an anonymous struct type because we hadn't yet recognized "resource instance object" as a first-class address type. We do now have addrs.AbsResourceInstanceObject to represent that sort of thing, and so we'll use that here. This also means we can return a set of addresses rather than a slice, which is semantically better as this set of addresses is not inherently ordered. Some existing callers were depending on the sort.SliceStable we were originally doing to make the result dependable, and so now we have a generic helper for turning addrs.Set into a sorted slice of the same type using our address types' "Less" methods that describe their natural or preferred ordering.	2 years ago
Martin Atkins	61ccc09681	stacks+rpcapi: Wire through the "plan mode" to the stacks runtime ... Previously we were accepting it at the API layer but then silently discarding it and always creating a "normal" plan. Now we'll pass the caller's selected planning mode all the way through to the modules runtime in component plans. This isn't actually quite sufficient to properly support destroy-mode plans since the stacks runtime itself doesn't yet really understand how to deal with things being destroyed, but this is intended to be just enough for those working on callers of this API to see _some_ effect from setting the planning mode, even though it's far from the full effect expected. More work on destroying will follow in subsequent commits.	2 years ago
Martin Atkins	3d8e95d153	rpcapi(Stacks): Support multiple descriptions in a "PlannedChange" ... Previously we had an asymmetry between PlannedChange and AppliedChange, where the latter supported any number of "description" objects whereas the former only supported zero or one. This change then is primarily motivated by making these two APIs consistent, but also has some other benefits: - The generated RPC stub packages will now have a specific type to use to represent _just_ a change description for a plan, so callers can pass around and preserve just those objects in isolation, rather than always having to use the entire PlannedChange message for that purpose. - This gives us a little more flexibility if later we find we need to return more than one description associated with a single logical change in future. For example, if we end up needing to make a breaking change to how we describe a particular kind of change, we might send an object containing both the old form and the new form together, and instruct clients that support the new form to treat it as taking priority over the old form when both are present. We don't yet know of any specific reason to do this, but this protocol is likely to be with us for a long time and so we need to prioritize leaving room for future additions we haven't thought of yet. This is a breaking change to the API from a client's perspective, but that's acceptable at the current stage of development where nothing has been committed in a stable release yet.	2 years ago
Martin Atkins	dd9c0ab98e	rpcapi(Stacks): don't send empty moved/import messages ... The "progress hooks" mechanism was unconditionally allocating messages to represent the moved and imported cases and then conditionally populating the fields inside. That causes an admittedly-small friction on the client side because they need to test whether the string fields inside are non-empty instead of just checking whether the overall message field was populated at all. To smooth that for clients, we'll instead leave these unset unless there's actually something to report. For clients written in Go, that means that the top-level fields will be nil when not relevant, which is easy to test for and relatively idiomatic.	2 years ago
Martin Atkins	af0c4764ef	stackeval: A FIXME about emitting prior state snapshots during apply ... Currently we're not explicitly writing the results from the refresh phase to the raw state or state description, which means that a refresh-only plan wouldn't do anything and even in a normal plan there are some edge-cases where we'll end up not saving the refreshed state. This is just a note to remind us of that in later work.	2 years ago
Martin Atkins	8fd29e9439	stacks: Preserve prior state from plan to apply ... We need to retain the prior state (in a form that Terraform Core's modules runtime can accept) between the plan and apply phases because Terraform uses it to verify that the provider's "final plan" is consistent with the initial plan. We previously tried to do this by reusing the "old" value from the planned change, but that's not really possible in practice because of the historical implementation detail that Terraform wants state information in a JSON format and plan information in MessagePack. This also contains the beginnings of handling the "discarding" of unrecognized keys from the state data structures, although we'll probably need to do more in that area later since this is just the bare minimum.	2 years ago
Martin Atkins	5372d5eb98	stacks+rpcapi(stacks): Model deposed objects for resource instances ... Previously we incorrectly assumed that resource instances only have one "current" object each. However, resource instances can also have deposed objects which we must also keep track of. This is a breaking change to the rpcapi since we're now using a new message type for a resource instance _object_ address in several places. That breakage is intentional here because at the time of writing this commit the rpcapi is not yet in any stable release and we want to force updating our existing internal client to properly handle deposed objects too.	2 years ago
Martin Atkins	2559151d0a	stackruntime: Count actions successfully applied ... Callers expect to be given an updated set of counts for how many of each action were completed, which we'll achieve by having the terraform.Hook implementation keep track of those and then announce them based on what it collected. We can't use the plan for this directly because terraform.Context.Plan seems to modify the plan it's given during its work, and on success the plan ends up being totally empty.	2 years ago
Martin Atkins	1f7dbaf3a8	plans: ResourceInstanceChange.ObjectAddr ... We've now many times made the mistake of thinking that the resource instance address is a suitable unique identifier for a resource instance change in a plan -- the name certainly seems to suggest it -- but really these represent changes to resource instance _objects_ and so it's important to also include the DeposedKey field as part of the unique identifier for a change. Now we'll expose ObjectAddr as a new method that returns the two together for convenient use as a single address value, along with adding some commentary to the fields to make it clearer that Addr is not sufficient. Ideally we'd rename Addr to make it less of an attractive nuisance, but that was too invasive a change for today so will need to wait for another time.	2 years ago
Martin Atkins	8dd08efcde	rpcapi(stacks): Additional OpenTelemetry tracing metadata ... During PlanStackChanges and ApplyStackChanges we'll mark the trace span representing the planning or applying of a particular component instance as either OK or Error status depending on which of the two terminal hooks get called. We'll also emit a trace event for each resource instance we plan. Ideally these would have their own spans too but we're currently limited by the design of the terraform.Hook API which wasn't designed with tracing spans in mind, and so we'll just use point-in-time events for now and rely on manual time-based correlation to try to associate downstream spans with particular resource instance operations if needed.	2 years ago
Martin Atkins	5d9cac85bc	stackruntime: Emit "pending" events for component instance plan/apply ... Previously we did have an event for planning but it wasn't in quite the right place, and so could potentially appear "late" and mess up the event sequence. Now we'll always emit the pending event immediately before the planning or applying event, which is perhaps a little redundant but having these separated means that we might be able to create some space between them later if we find that we need to do some additional work after we know about a component instance but before we're reading to say that we're planning or applying it.	2 years ago
Martin Atkins	0d308110da	rpcapi: Shared hooks for both PlanStackChanges and ApplyStackChanges ... Previously we needed two separate hook implementations because the progress event messages were of different Go interface types for each operation. Now the RPC API shares a single StackChangeProgress message type we can factor out the logic that actually emits the events and have the two phases differ only in how they send those common message types to the client. This also hooks some additional apply-time events that we weren't previously hooking, passing them through as RPC events in the same way as we were doing for the plan phase.	2 years ago
Martin Atkins	947c483c49	rpcapi: Stack plan and apply use the same progress message type ... Previously we had the progress messages directly inlined as events of the PlanStackChanges operation, but that means that there's no common interface type for progress events across both the plan and apply phases, making it hard for Go implementations to share marshaling code between the two phases. Now we'll use a new StackChangeProgress message type to contain all of the progress message situations. This makes constructing and using the progress messages a little more verbose -- an extra layer of message -- but means that we can write code that works generically with the StackChangeProgress generated struct and thus share it between the two phases, leading to less code overall. As of this commit we don't yet have the apply phase generating any progress messages, but we'll extend it in a subsequent commit now that it's possible to share more of that event-generating code between the two phases.	2 years ago
Martin Atkins	1943ed2395	stackruntime: Emit some more progress hooks during the apply phase ... This is not quite comprehensive yet, but echoes the main component start/end events we emit during the planning phase, and some of the resource-instance-related events. The rpcapi layer is not yet consuming all of these events and so most of this will just be no-op right now; we'll add the rpcapi translation layer for these in subsequent commits.	2 years ago
Martin Atkins	c4d7ee93fb	stackeval: Use the right EvalPhase in various places ... In an earlier commit I generalized some code that was originally written for the plan phase to be shared between the plan and apply phases, but missed a few spots where it was still hard-coded for the plan phase. Hopefully this is all of them!	2 years ago
Martin Atkins	69aa51b9ff	stackruntime: Fix some quirks of data sent from plan to apply ... We previously took a few shortcuts just to more quickly produce a proof of concept. This gets us considerably closer to a properly-working handoff from plan to apply, at least for create, update, and replace actions. Destroying stuff that's been removed from the configuration will take some further work since currently everything we do is driven by objects in the configuration. That will follow in later commits.	2 years ago
Martin Atkins	7b41a968a9	core: Allow tests to force a fixed result from plantimestamp ... Tests often need to ensure stable output for ease of comparison with an expected value, so this new planning option allows forcing a particular value for the plantimestamp function to return, when testing behavior related to that.	2 years ago
Martin Atkins	569f0f553b	stackruntime: ComponentInstance considers previous state ... Now that we have access to a previous state snapshot during the planning phase, we can pass the relevant parts to the modules runtime when asking it to plan changes for a component instance. This is not yet complete enough for full support of consecutive plan and apply runs that preserve state, but it gets the needed data plumbed in and we'll continue making this more complete in subsequent commits.	2 years ago
Martin Atkins	aaba750095	rpcapi: Stacks: pass previous state to stack runtime during planning ... As of this commit it doesn't yet do anything with it, but we'll plumb that in more deeply in subsequent commits.	2 years ago
Martin Atkins	49511fe9a8	stackstate: Fuller round-tripping of resource instance object state ... This gets us ever closer to being able to preserve resource instance objects from one run to the next. In subsequent commits we'll make use of the "LoadFromProto" option to load the prior state during the planning phase and take it into account when we're deciding what actions to propose.	2 years ago
Martin Atkins	89776cd2a6	stackstate: Initial work on decoding and the state model ... This is a sketch of the overall structure of the prior state decoder and the model type it populates. Before we can complete this we'll need to slightly rework how the apply phase emits the raw events that this is consuming, and in particular to change the raw state representation to be JSON-based to match with how Terraform Core expects to receive it once reloaded. That will follow in later commits.	2 years ago
Martin Atkins	13e26b60dd	addrs: Adopt DeposedKey as an address type; remove states.Generation ... Previously we had states.DeposedKey as our representation of the unique keys used to differentiate between multiple deposed objects on the same resource instance. That type is useful in various places, but its presence in the "states" package made it troublesome to import for some callers. Since its purpose is as an identifier, this type is more at home in the "addrs" package. The states package retains a small number of aliases to preserve compatibility with a few existing callers for now, until we have the stomach for a more invasive change. This also removes the largely-redundant states.Generation interface type, which initially was intended as a way to distinguish between "current" objects and deposed objects, but didn't really add anything because we already have addrs.NotDeposed to represent that situation. Instead, we just ended up with various bits of boilerplate adapter code converting between the two representations. Everything now uses addrs.DeposedKey and uses addrs.NotDeposed to identify non-deposed objects.	2 years ago
Martin Atkins	54622c68a5	stackstate: Generate "raw state" objects for resource instance objects ... Previously we just had a stub that only generated basic external description objects. Now we'll also emit "raw state" objects for these, and have some initial support for deposed objects too.	2 years ago
Martin Atkins	60431e738e	stackstate: Use the statekeys package for resource instance object keys ... We can now replace our previous interim "tmpKey" with a real key in the syntax decided by our separate statekeys package.	2 years ago
Martin Atkins	8b431f5038	stackstate/statekeys: Representation of stacks state keys ... Our model for state for a stack involves a set of objects that is each identified by an opaque string key. Although those keys are opaque to callers of Terraform Core, we will actually be using them for some meaning in Terraform Core itself, since that will avoid redundantly storing the same information both in the key and in the object associated with the key. This therefore aims to encapsulate the generation and parsing of these keys to help ensure we'll always use them consistently.	2 years ago
Martin Atkins	ce1f8dd22b	states: ParseDeposedKey function ... This is a helper for when we're accepting a deposed key from outside of Terraform, such as in a plan or state file. It doesn't really "parse" the string in the usual sense -- the result is always just the input verbatim if successful -- but it's named that way for consistency with how we usually name functions like this.	2 years ago
Martin Atkins	1cb96ebb32	stackstate: Add HashiCorp copyright comments	2 years ago
Martin Atkins	d1b0632e82	stackruntime: Add HashiCorp copyright comments	2 years ago
Martin Atkins	c67920ce4b	stackplan: Add HashiCorp copyright comments	2 years ago
Martin Atkins	9ccca7acc0	stackconfig: Add HashiCorp copyright comments	2 years ago
Martin Atkins	434e199675	stackaddrs: Add HashiCorp copyright comments	2 years ago
Martin Atkins	ac93c1bb82	rpcapi: Add HashiCorp copyright comments	2 years ago
Martin Atkins	b60aadd1ec	promising: Add HashiCorp copyright comments	2 years ago
Martin Atkins	15c38a1a19	collections: Add HashiCorp copyright comments	2 years ago
Martin Atkins	f347d0a441	stackaddrs: Handle instance keys in embedded stack references ... Previously this logic was wrong, because it had already consumed the two leading traversal steps before checking if there was anything else.	2 years ago
Martin Atkins	d91e5dc9f1	stackstate: Include correct object data when announcing applied resource ... Previously we just had this just stubbed out to always return an empty object because we didn't have the schema information available to transcode the JSON-encoded state data into the form our RPC API wants to produce. An earlier commit added the schema information we need, so we can now transcode just in time to produce the protocol buffers serialization of the applied change description. This is still otherwise just a stub implementation. We'll still need to deal with emitting the "raw" representation of this state for use in future plans, and make sure we're properly handling deposed objects, in future commits.	2 years ago
Martin Atkins	74215a4451	stacks: Include resource type schema in PlannedChange and AppliedChange ... Unfortunately for historical reasons the "terraform" package produces state and plan artifacts that have dynamic data elements pre-encoded in the formats that Terraform CLI's traditional plan and state formats use. Since Stacks uses different plan and state models, and exclusively uses MessagePack encoding for dynamic values in both, we'll sometimes need to transcode from one format to another when generating and decoding the stacks-oriented representations. Transcoding between these serialization formats requires access to the relevant schema because the formats themselves each have an infoset that is only a subset of Terraform's type system. Therefore we'll now annotate the PlannedChange and AppliedChange objects that include provider-specific data types with the schema required to transcode them. This commit does not yet actually arrange to make use of those schemas. That will follow in later commits.	2 years ago
Martin Atkins	48518b8233	stacks: A very basic stub of applied resource instance change announcements ... We'd like to start developing some clients for this part of the RPC API concurrently with remaining work here in Terraform Core, and so this is a bare-minimum implementation of emitting "applied change" events for resource instances just so there's something for client developers to test against. There are various things wrong with this, including but not limited to: - It always reports that the new state for a resource instance is an empty object, rather than including the correct updated object. - It only supports "current" resource instance objects, and ignores deposed ones. - It just uses the resource instance address alone as the key for the state description map, which is fine for now when we only have one kind of description object anyway but will not be sufficient for a real implementation that needs to emit various different kinds of object. We'll need to rework most of this in future commits, but this is hopefully sufficient to start implementing and testing API clients, at least to some extend, despite the crudity of the results.	2 years ago
Martin Atkins	d99985720f	stacks: Carry input variable values from plan to apply ... We need to preserve the input variable values provided during the plan phase so we can reuse them during the apply phase to recalculate derived results based on new information learned during the apply process. We also need to recalculate the input values for individual components during the apply phase, so we can react to previously-unknown values that come from other upstream components.	2 years ago
Martin Atkins	b887d29894	stackeval: NewForApplying must save the ChangeExecResults object ... This was just an oversight in some earlier refactoring. An ApplyPhase instance of Main must store a pointer to the ChangeExecResults object that will capture all of the results of the main side-effects of applying the plan, since we'll need to retrieve those results to evaluate expressions and other similar downstream results based on those outcomes.	2 years ago
Martin Atkins	094ef0a8f3	collections: Better behavior for interactions with uninit maps ... Similar to the built-in map types in Go, the zero value of collections.Map now behaves as if empty on read, and generates actionable panics when attempting to write.	2 years ago
Martin Atkins	de170a9a30	stackeval: InputVariable should respect EvalPhase when deciding value ... Previously it was still hard-coding PlanPhase, which was a cheat we did in earlier work when we were just exploring the overall design of this package. It will now pass ApplyPhase when running in the apply phase, although Main.RootVariableValue doesn't yet have any implementation for retrieving variable values frozen during planning so root input variables will always be unknown values during the apply phase for now. We'll fix that in a future commit.	2 years ago
Martin Atkins	373f39d6ac	stackeval: ApplyPlan's ChangeExec must be in main task ... The ChangeExec call creates new promises, one of which the caller is responsible for resolving by calling the "begin" callback. To correctly follow the rules of the "promising" package we must resolve that promise in the same task that created it or explicitly delegate it to another task. In this case it's fine to just resolve it, but for that to work correctly we need to call "begin" from the same task as ChangeExec was called from.	2 years ago
Martin Atkins	b8923a5f94	stackruntime: Persist a component's plantimestamp in the plan ... We need to preserve the module runtime's chosen plan timestamp through to the apply phase so that plantimestamp function calls during the apply phase will return the same result as during the planning phase. This also means that we will now no longer emit a "component instance" planned change object if the planning operation fails so much that there is no plans.Plan object returned, but that's at least consistent with the fact that we also don't report any resource instance changes in the same situation. Event emission is always best-effort when the result has errors.	2 years ago
Martin Atkins	8f00a7bf5a	stackruntime: Make sure PlanResponse and ApplyResponse channels get closed ... The closure of these channels is how a caller knows that the operation has completed, so we must always close them before we return or the caller is likely to get wedged.	2 years ago
Martin Atkins	f34633577f	stackplan: LoadFromProto must allocate its component map ... Previously this was panicking later on in the function when trying to insert into a nil map.	2 years ago
Alisdair McDiarmid	927350eca8	rpcapi: Add component expansion and change summary ... We want to report the result of component expansion to the client as early as possible, so that it can prepare to receive any following information for each instance. This message is critical when a component's expansion expression evaluates to an empty set: without this message's confirmation of this case, it will be unable to determine that the component will receive no plans for any instances until the plan completes. We also want to report a roll-up of the resource changes upon completion of each component's plan and apply operations, so that a client can summarize the operation changes without performing the roll-up operation on all received change events.	2 years ago
Martin Atkins	3c5a2ea7e5	stackplan: Use comparable version strings for plan version checking ... Previously we were comparing version.String() with version.SemVer.String(), which doesn't match because the latter excludes any prerelease version suffix.	2 years ago
Martin Atkins	385e37cae2	stackaddrs: fix some ParseAbsComponentInstance logic errors	2 years ago
Martin Atkins	e672cd8c28	rpcapi: ApplyStackChanges must actually pass the plan to stackruntime	2 years ago
Martin Atkins	f1ceee5e69	rpcapi: Stacks.ApplyStackChanges initial implementation ... This wires up the stackruntime.ApplyPlan function to the RPC API with a similar strategy as we used for Stacks.PlanStackChanges wrapping stackruntime.Plan. This is not well tested at all and so probably doesn't quite work right. I'm currently optimizing for getting a reasonable stub of everything into place so that we can have multiple people work together on getting the details sorted afterwards.	2 years ago
Martin Atkins	2a0d407eb6	stackruntime: ApplyPlan function ... This is the exported entry-point for applying a plan that was created by an earlier call to stackruntime.Plan.	2 years ago
Martin Atkins	8719f13b58	stackeval: Stubby implementation of applying changes to components ... This implementation is likely naive and will probably need some refinements in future commits; the goal here is just to stub out the general shape of this so that we can potentially have multiple contributors work concurrently on the subsequent improvements.	2 years ago
Martin Atkins	f8d2fef129	stackeval: Factor out the two "walk drivers" ... With validate, plan, and apply now all at least stubbed out we can see that they all share some similar logic for visiting everything that's relevant to an evaluation phase and gathering up diagnostics and other external reports about each object. Since it's important that the phases all visit the same objects so that we can produce consistent results between phases, we'll accept a little inversion-of-control complexity here in return for now having only two "walk-driver" implementations: one for visiting the "static" objects and one for visiting the "dynamic" objects. - The validate phase only visits static objects. - The plan phase visits static objects first, and then dynamic objects only if the static walk doesn't produce any errors. - The apply phase only visits the dynamic objects. This also required some retroactive changes to some of my earlier work on the validation phase since we can no longer assume that the validation walk is the only one which visits the "static" objects. Their methods now take EvalPhase arguments similar to those for the dynamic objects, and we differentiate between the results in each phase so that we can potentially add slight differences between the phases in future if needed. As of this commit, though, the validate phase and the static portion of the plan phase should produce identical results.	2 years ago