95 Commits (stacks-variable-validations)

Author	SHA1	Message	Date
Daniel Banck	4ef9684188	Rework most of the configuration loading ... We previously used a loader -> BuildConfig flow to load configuration. This commit changes most (but not all yet) flows to use the new graph-based approach. Instead of simply recursively loading the modules, we now need to take a stepped approach: 1. Load the root module 2. Collect the variables and their values 3. Build the configuration with the graph-based approach Because this approach relies on different parts from different packages, it can't easliy be done within the `configload` package. So, now we do most of in the backend or command.	4 months ago
Daniel Schmidt	026c935961	move UnparsedVariableValue from backendrun to arguments ... This prevents a cyclic dependency and also makes sense semantically. The arguments package will collect the unparsed variable values and the backendrun helpers will work to collect the values and transform them into terraform.InputValue.	4 months ago
Radek Simko	0fe906fa8c	make copyrightfix	4 months ago
Daniel Schmidt	c9cc64a260	move stray check for deprecation to central location	5 months ago
Samsondeen	0b35407d2c	Allow functions in test mock blocks (#37883)	5 months ago
Sarah French	1dd8e60f15	chore: Bump golang.org/x/crypto dependency, ignore SA4003 linting errors globally (#37901) ... * chore: Upgrade golang.org/x/crypto - `go get golang.org/x/crypto@v0.44.0` * chore: Upgrade golang.org/x/crypto in submodules - `make syncdeps` * chore: Run `make generate` * chore: Ignore SA4003 linter error globally, as `golang.org/x/tools/cmd/stringer` generates code that breaks this rule when it's used with custom types that have unsigned integers or bytes as the underlying type.	7 months ago
Liam Cervante	411f18e6ac	ephemerals: allow root ephemeral outputs in stacks and test (#37813) ... * ephemerals: allow root ephemeral outputs in stacks and test * remember to set new opt for apply stage	8 months ago
Sarah French	6847e763de	Update calls to StateMgr method to use diagnostics more conventionally (#37597) ... * Update call to StateMgr method to use diagnostics more explicitly * Update calling code in remote backend tests * Update calling code in non-test code to handle diags conventionally. * Fix test failure messages from using wrong variable * Additional refactoring	10 months ago
Kristin Laemmert	31a982ec0f	moduletest: fix error return (#37589)	10 months ago
Liam Cervante	551ba2e525	Implement controlling destroy functionality within Terraform Test (#37359) ... * Add ability to parse backend blocks present in a test file's run blocks, validate configuration (#36541) * Add ability to parse backend blocks from a run block * Add validation to avoid multiple backend blocks across run blocks that use the same internal state file. Update tests. * Add validation to avoid multiple backend blocks within a single run block. Update tests. * Remove use of quotes in diagnostic messages * Add validation to avoid backend blocks being used in plan run blocks. Update tests. * Correct local backend blocks in new test fixtures * Add test to show that different test files can use same backend block for same state key. * Add validation to enforce state-storage backend types are used * Remove TODO comment We only need to consider one file at a time when checking if a state_key already has a backend associated with it; parallelism in `terraform test` is scoped down to individual files. * Add validation to assert that the backend block must be in the first apply command for an internal state * Consolidate backend block validation inside a single if statement * Add initial version of validation that ensures a backend isn't re-used within a file * Explicitly set the state_key at the point of parsing the config TODO: What should be done with method (moduletest.Run).GetStateKey? * Update test fixture now that reusing backend configs has been made invalid * Add automated test showing validation of reused configuration blocks * Skip test due to flakiness, minor change to test config naming * Update test so it tolerates non-deterministic order run blocks are evaluated in * Remove unnecessary value assignment to r.StateKey * Replace use of GetStateKey() with accessing the state key that's now set during test config parsing * Fix bug so that run blocks using child modules get the correct state key set at parsing time * Update acceptance test to also cover scenario where root and child module state keys are in use * Update test name * Add newline to regex * Ensure consistent place where repeat backend error is raised from * Write leftover test state(s) to file (#36614) * Add additional validation that the backend used in a run is a supported type (#36648) * Prevent test run when leftover state data is present (#36685) * `test`: Set the initial state for a state files from a backend, allow the run that defines a backend to write state to the backend (#36646) * Allow use of backend block to set initial state for a state key * Note about alternative place to keep 'backend factories' * Allow the run block defining the backend to write state to it * Fix rebase * Change to accessing backend init functions via ContextOpts * Add tests demonstrating how runs containing backend blocks use and update persisted state * Fix test fixture * Address test failure due to trouble opening the state file This problem doesn't happen on MacOS, so I assume is due to the Linux environment of GitHub runners. * Fix issue with paths properly I hope * Fix defect in test assertion * Pivot back to approach introduced in 4afc3d7 * Let failing tests write to persistent state, add test case covering that. I split the acceptance tests into happy/unhappy paths for this, which required some of the helper functions' declarations to be raised up to package-level. * Change how we update internal state files, so that information about the associated backend is never lost * Fix UpdateStateFile * Ensure that the states map set by TestStateTransformer associates a backend with the correct run. * Misc spelling fixes in comments and a log * Replace state get/set functions with existing helpers (#36747) * Replace state get/set functions with existing helpers * Compare to string representation of state * Compare to string representation of state * Terraform Test: Allow skipping cleanup of entire test file or individual run blocks (#36729) * Add validation to enforce skip_cleanup=false cannot be used with backend blocks (#36857) * Integrate use of backend blocks in tests with skip_cleanup feature (#36848) * Fix nil pointer error, update test to not be table-driven * Make using a backend block implicitly set skip_cleanup to true * Stop state artefacts being created when a backend is in use and no cleanup errors have occurred * Return diagnostics so calling code knows if cleanup experienced issues or not * Update tests to show that when cleanup fails a state artefact is created * Add comment about why diag not returned * Bug fix - actually pull in the state from the state manager! * Split and simplify (?) tests to show the backend block can create and/or reuse prior state * Update test to use new fixtures, assert about state artefact. Fix nil pointer * Update test fixture in use, add guardrail for flakiness of forced error during cleanup * Refactor so resource ID set in only one place * Add documentation for using a `backend` block during `test` (#36832) * Add backend as a documented block in a run block * Add documentation about backend blocks in run blocks. * Make the relationship between backends and state keys more clear, other improvements * More test documentation (#36838) * Terraform Test: cleanup command (#36847) * Allow cleanup of states that depend on prior runs outputs (#36902) * terraform test: refactor graph edge calculation * create fake run block nodes during cleanup operation * tidy up TODOs * fix tests * remove old changes * Update internal/moduletest/graph/node_state_cleanup.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> * Improve diagnostics around skip_cleanup conflicts (#37385) * Improve diagnostics around skip_cleanup conflicts * remove unused dynamic node * terraform test: refactor manifest file for simplicity (#37412) * test: refactor apply and plan functions so no run block is needed * terraform test: write and load state manifest files * Terraform Test: Allow skipping cleanup of entire test file or individual run blocks (#36729) * terraform test: add support for skip_cleanup attr * terraform test: add cleanup command * terraform test: add backend blocks * pause * fix tests * remove commented code * terraform test: make controlling destroy functionality experimental (#37419) * address comments * Update internal/moduletest/graph/node_state_cleanup.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> --------- Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> * add experimental changelog entries --------- Co-authored-by: Sarah French <15078782+SarahFrench@users.noreply.github.com> Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> Co-authored-by: Samsondeen Dare <samsondeen.dare@hashicorp.com>	10 months ago
Samsondeen	af8cc7e315	test: create all edges to leaf cleanup nodes (#37564)	10 months ago
Liam Cervante	dec0edfd5a	terraform test: enable deferrals in Terraform Test with the specified flag (#37370)	11 months ago
Liam Cervante	84c2e2f9e5	terraform test: override prevent_destroy meta attribute (#37364)	11 months ago
Liam Cervante	160b6b35b0	terraform test: include expected diagnostics in verbose mode (#37362)	11 months ago
Liam Cervante	0a7701f19a	terraform test: refactor graph edge calculation (#37357) ... * terraform test: refactor graph edge calculation * remove unused attribute	11 months ago
Daniel Schmidt	66001080e6	actions: implement expansion and planning actions ... This implementation is very barebones, but builds the skelleton for all upcoming features in actions. It includes the graph implementation for planning actions including the expansion of actions and dealing with multiple action triggers. We plan all actions as part of their triggering resource for now. All action_triggers depend on the triggering resources plan to determine if an action should run or not, therefore we can plan the actions as part of the triggering resource. This implementation ignores conditions, deferrals, provider arguments and it does not add the actions to the planfile; mostly to enable more people to work on these issues in parallel.	12 months ago
Liam Cervante	2a1b4c3ed4	Update go-cty and HCL for 1.13 release (#37298)	12 months ago
Samsondeen	a1332299a5	Test run Parallelism of 1 should not result in deadlock (#37292)	12 months ago
Samsondeen	f57642d1a6	tf test: parallelize teardown operations (#37220)	1 year ago
Liam Cervante	b4a3e56a19	terraform test: deprecate ambigious variable references in test files (#37218) ... * terraform test: deprecate ambigious variable references in test files * fix panic caused by merge	1 year ago
Liam Cervante	777a916435	terraform test: move providers into the graph (#37216) ... * terraform test: move providers into the graph * consistency checks * remove flaky cycle test * address comments * Update internal/moduletest/graph/node_test_run.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> * go fmt --------- Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com>	1 year ago
Liam Cervante	53afa2cd2f	terraform test: move variable evaluation into the terraform test graph (#37205) ... * terraform test: move variable evaluation into the terraform test graph * make copyrightfix	1 year ago
Liam Cervante	063b7cf62b	terraform test: check specific dependencies before skipping run blocks (#37204) ... * terraform test: check specific dependencies before skipping run blocks * fix data race * complete comment	1 year ago
Liam Cervante	757efaed9a	terraform test: automatically create edges based on references (#37200) ... * terraform test: automatically create edges based on references * address comments * remove unused function	1 year ago
Samsondeen	2a79f5fa53	Use iterators for graph vertices (#36558) ... * Use iterators for graph vertices * use func filter * use type param instead of function filter One type parameter seem to be enough instead of 2	1 year ago
Liam Cervante	f80762d3d5	terraform test: add variable definitions to test files (#37195) ... * terraform test: add variable definitions to test files Currently, `terraform test` attempts to work out the type of any external variables by delaying evaluation until each run block executes so it can use the definitions within the run blocks's module. This means that the values of variables can technically change between run blocks which isn't ideal. This commit is the first in a chain which will move the evaluation of variables into the terraform test graph. We need to give the users the option of specifying the type for external variables within the file as these variables are now going to be assessed outside of the context of a run block. To do this, we introduce the optional variable blocks that are added by this commit. * update comment	1 year ago
Liam Cervante	71af8f8858	terraform test: push context initialisation into the graph (#37194) ... * terraform test: refactor the context initialisation into specific node in the graph * Update internal/moduletest/graph/transform_providers.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> --------- Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com>	1 year ago
Liam Cervante	19540e30fe	terraform test: don't allow diags from executable nodes (#37193)	1 year ago
Liam Cervante	8135fefa62	terraform test: stop transforming the config under test with new variables (#37152)	1 year ago
Samsondeen	c0a7ff235b	Apply some new go collection features (#36818)	1 year ago
Daniel Banck	10c9b64007	Rename schema.Block to Body (#36629)	1 year ago
Samsondeen	3f4ed7e019	Ensure that all states are cleaned up (#36616)	1 year ago
Samsondeen	1bd312f155	Produce detailed diagnostic objects when test run assertions fail (#36522)	1 year ago
Samsondeen	018110c012	No state to cleaup when it is empty (#36526)	1 year ago
James Bardin	e1485f27fc	use go tool to run generate commands	1 year ago
Samsondeen	a3a42979e2	Ensure that the file teardown summary is displayed before the run's teardown summary (#36483)	1 year ago
James Bardin	17f4dcfd8d	Merge pull request #36465 from hashicorp/jbardin/sensitive-order ... Serialize `AttrSensitivePaths` in a consistent order	1 year ago
Samsondeen	2e91113224	Terraform test: Consolidate test execution procedure (#36459)	1 year ago
James Bardin	cbd7435dfc	udpate tests to match core tfdiags ... The local format function for diagnostics didn't match the global formatting for the tfdiags package.	1 year ago
James Bardin	483c179156	use existing path formatter	1 year ago
Samsondeen	4161e88391	Continue test execution after an expected failure (#36447)	1 year ago
Samsondeen	2f0f537e69	Terraform test: Execute eligible test runs in parallel (#36300)	1 year ago
Sarah French	21cafd70ea	Add more details to JUnit `terraform test` output to describe why a test was skipped (#36315) ... * Add ability for TestJUnitXMLFile to access data about whether the test runner was Stopped * Add details to XML describing why a Run was skipped * Fix wording * Code consistency changes * Move all JUnit-related code down to where it's used Away from the Views section of the code where it was relevant before * Move JUnit-related error and warning diags to above where cancellable contexts are created * Fix wording of user feedback * Fix test to match updated skipped message text * Fix test	1 year ago
Sarah French	43f31c43b0	Add start time timestamps to `<testcase>`/Run data in JUnit XML output from `terraform test` (#36295) ... * Record metadata about the start timestamp for a Run/test case * Add RFC3339 format timestamp attributes to `<testcase>` elements in JUnit XML output * Ensure timestamps are saved as and rendered as UTC timestamps	1 year ago
Samsondeen	aec7c3cc89	terraform test: allow computed/mocked values override during planning (#36227)	1 year ago
Liam Cervante	eeacced8f0	Add go vet as a check to the Github commit actions (#31461) ... * Add go vet as a check to the Github commit actions * deliberately break to test * fix after successful test * Use directly * fix more errors	2 years ago
Radek Simko	79a35f3968	lang: Add Extra to avoid duplicate diagnostic in check blocks (#35944)	2 years ago
Daniel Schmidt	b701e152e5	show value is ephemeral in terraform test	2 years ago
Daniel Schmidt	3e7fdd0de9	improve wording of error message	2 years ago
Daniel Schmidt	ae103fc491	test: emit a nice error message if the user uses ephemeral resource in terrafom test	2 years ago