96 Commits (6fdb348b615af5979819f8b6168260f335cfb2e0)

Author	SHA1	Message	Date
Liam Cervante	160b6b35b0	terraform test: include expected diagnostics in verbose mode (#37362)	9 months ago
Liam Cervante	0a7701f19a	terraform test: refactor graph edge calculation (#37357) ... * terraform test: refactor graph edge calculation * remove unused attribute	9 months ago
Sarah French	7199fbd2bb	Update use of `testChdir` to standard library's `t.Chdir`, remove `testChdir` function from codebase (#37334) ... * Replace use of `testChdir` with `t.Chdir` * Update tests to use temporary directories with copied content, instead of using directories in the repo directly. * Remove stacks copy of testChdir function This has been replaced with t.Chdir from the standard library * Replace remaining simple usage of testChdir * Update guidance for using `tempWorkingDir` * Replace use of testChdir in a function reused in a single test * Update comments to no longer recommend using testChdir * Remove testChdir function!	9 months ago
Sarah French	15840b3eae	Replace use of `defer` with `t.Cleanup` in majority of `command` package tests, update `testChdir` to work with t.Cleanup (#37326) ... * Replace use of `defer` with `t.Cleanup` in tests * Fix test failures by making cleanup function permissive of the temp dir being already deleted.	9 months ago
Samsondeen	a1332299a5	Test run Parallelism of 1 should not result in deadlock (#37292)	10 months ago
Samsondeen	f57642d1a6	tf test: parallelize teardown operations (#37220)	10 months ago
Liam Cervante	777a916435	terraform test: move providers into the graph (#37216) ... * terraform test: move providers into the graph * consistency checks * remove flaky cycle test * address comments * Update internal/moduletest/graph/node_test_run.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> * go fmt --------- Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com>	11 months ago
Liam Cervante	53afa2cd2f	terraform test: move variable evaluation into the terraform test graph (#37205) ... * terraform test: move variable evaluation into the terraform test graph * make copyrightfix	11 months ago
Liam Cervante	063b7cf62b	terraform test: check specific dependencies before skipping run blocks (#37204) ... * terraform test: check specific dependencies before skipping run blocks * fix data race * complete comment	11 months ago
Liam Cervante	757efaed9a	terraform test: automatically create edges based on references (#37200) ... * terraform test: automatically create edges based on references * address comments * remove unused function	11 months ago
Liam Cervante	f80762d3d5	terraform test: add variable definitions to test files (#37195) ... * terraform test: add variable definitions to test files Currently, `terraform test` attempts to work out the type of any external variables by delaying evaluation until each run block executes so it can use the definitions within the run blocks's module. This means that the values of variables can technically change between run blocks which isn't ideal. This commit is the first in a chain which will move the evaluation of variables into the terraform test graph. We need to give the users the option of specifying the type for external variables within the file as these variables are now going to be assessed outside of the context of a run block. To do this, we introduce the optional variable blocks that are added by this commit. * update comment	11 months ago
Sarah French	5cc7479e0e	Update test to tolerate durations in scientific notation (very short test durations) (#37041) ... It's not valid to force integer values for `time`, so this is a case where the test needs to be updated and not the code under test.	11 months ago
Sarah French	8d2dffedb3	Fix defects where `done` is called before the command's `Run` method is invoked (#36852) ... * Fix defects where `done` is called before the command's `Run` method is invoked We're attempting to reading output from a command that hasn't been executed yet! * Another one	1 year ago
Liam Cervante	7fadbe34de	write-only attributes: internal providers should set write-only attributes to null (#36824) ... * write-only attributes: internal providers should set write-only attributes to null * add changelog * fix copywrite headers	1 year ago
Liam Cervante	aaf617d6d7	Add test case to ensure no regression of already fixed crash (#36583)	1 year ago
Samsondeen	3f4ed7e019	Ensure that all states are cleaned up (#36616)	1 year ago
Samsondeen	1bd312f155	Produce detailed diagnostic objects when test run assertions fail (#36522)	1 year ago
Liam Cervante	0acd732911	Fix flaky test in test_test.go (#36566)	1 year ago
Samsondeen	85db2264db	skip flaky test (#36540)	1 year ago
Samsondeen	018110c012	No state to cleaup when it is empty (#36526)	1 year ago
Samsondeen	2e91113224	Terraform test: Consolidate test execution procedure (#36459)	1 year ago
Samsondeen	4161e88391	Continue test execution after an expected failure (#36447)	1 year ago
Samsondeen	7f29df96a0	Allow terraform init when only test files are present in directory (#36429)	1 year ago
Samsondeen	2f0f537e69	Terraform test: Execute eligible test runs in parallel (#36300)	1 year ago
Samsondeen	16a34fe47f	Accept CLI option for the number of parallel ops in a test run's plan/apply (#36323)	1 year ago
Sarah French	39224976a8	Make test fail if JUnit feature is experimental (#36348)	1 year ago
Kévin Andrieux	258f91ab0c	feat: add state_key for test run blocks (#36185) ... * feat: add state_alias for test run blocks * fix: move `state_alias` to run block + rename to `state_key` * docs: add state_key changelog entry * docs: add state_key in run block documentation * Revert "docs: add state_key in run block documentation" This reverts commit ccccf62a76. * Revert "docs: add state_key changelog entry" This reverts commit 86e2ad8dc3.	1 year ago
Liam Cervante	408f323efa	test: also allow mock_data and mock_resource blocks to generate data during planning (#36317)	1 year ago
Sarah French	ab6e4f2299	Refactor JUnit XML output of `terraform test` into a new `junit` package (#36304) ... * Refactor JUnit XML output to use new concept of an Artifact * Move JUnit-related code into new `artifact` package * Refactor Artifact's Save method to return diagnostics, update comments Previously TestJUnitXMLFile implemented the View interface, which cannot return errors. Now it's not a View any more we can simplify things. * Make junitXMLTestReport output deterministic by iterating over a slice instead of a map, add test * Provide sources to junitXMLTestReport, allowing complete error messages in the XML We need to ensure that artifact.NewTestJUnitXMLFile is called once the config Loader is available as a non-nil pointer * Whitespace * Add some test coverage for JUnit XML output for `terraform test` * Refactor how file is saved, add tests * Move XML structs definitions outside of `junitXMLTestReport` * Fix nil pointer bug * Add missing file headers * Refactor comparison of byte slices * Rename package to `junit`, rename structs to match * Add a test showing JUnit output when a test is skipped by the user	1 year ago
Samsondeen	aec7c3cc89	terraform test: allow computed/mocked values override during planning (#36227)	1 year ago
Daniel Schmidt	b701e152e5	show value is ephemeral in terraform test	2 years ago
Daniel Schmidt	3e7fdd0de9	improve wording of error message	2 years ago
Daniel Schmidt	ae103fc491	test: emit a nice error message if the user uses ephemeral resource in terrafom test	2 years ago
Daniel Schmidt	97ba67edea	test: ensure ephemeral values are not shown in diagnostics and warning is printed	2 years ago
Daniel Schmidt	725add6a1a	test: pass in variables during apply ... ephemeral variables need to be re-supplied	2 years ago
MicahKimel	3258744166	Remove invalid warning during cleanup phase (#35172) ... * stop invalid warning during cleanup phase * stop invalid warning during cleanup phase * invalid warning during cleanup phase * remove unwanted comment * update GetVariables parameter for including warnings * invalid warning during cleanup phase test case * update invalid warnings in cleanup test case to throw a warning for the validation variable before cleanup phase * remove unwanted warnings	2 years ago
Liam Cervante	448775102c	terraform test: Push evaluation of variables to as late as possible (#35014) ... * terraform test: Push evaluation of variables to as late as possible * Update internal/moduletest/hcl/variable_cache.go Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com> * address comments --------- Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>	2 years ago
Liam Cervante	6963c54bec	terraform test: don't remove sensitive marks from inputs (#35021)	2 years ago
UKEME BASSEY	0ea25e3b40	fixing tests internal/command/init, view and argumento	2 years ago
UKEME BASSEY	370a471ecb	fix json -help display and failing tests	2 years ago
Martin Atkins	14b08c3745	terraform: Move variable validation into its separate node type ... The parent commit added a new node to the graph for each existing input variable node, but that node didn't do anything yet. Now those new nodes are actually responsible for checking the variable's validation rules, instead of that being included in the same graph node that decides the variable's final value. This also refines the validation node graph transformer to only insert a validation node if there's at least one rule to check, which should avoid bloating the graph for anyone who isn't actually using input variable validation. Therefore the plan graph builder tests revert back to their old expected result because that input configuration does not include any validation rules. For the moment this just moves around functionality that already existed and doesn't add anything new, but having the validation handled in a separate graph node means that we can evaluate the validation rules in the EvalContext of the module where the variable was declared instead of where it was defined, and so a future commit will allow -- initially just as an experiment -- validation rules that can refer to other objects in the module where the variable is declared, including the values of other input variables. As a nice bonus, this fixed a small bug in "terraform test" where problems with variable values defined in the test run were reported as problems with the variable declaration, rather than with the expression assigning the value to it. Since it's now NodeRootVariable that decides the definition source range, rather than the validation-rule-checking helper, it has access to the source range provided in a terraform.InputValue object, which was already populated correctly for per-test-run definitions.	2 years ago
Liam Cervante	c6ff573180	Include provider functions in scope used to evaluate test assertions (#34825)	2 years ago
Daniel Schmidt	7ae7a203d8	test: allow using global variables in suite-level variable definitions ... Closes #34534	2 years ago
Liam Cervante	9658f9df6b	mocking overrides: default to concrete empty object when values are missing (#34563)	2 years ago
Liam Cervante	430970c02b	terraform test: Fix crash when file level variables reference variables. (#34531) ... * terraform test: Fix crash when referencing variables within the file level variable block * also allow functions within variables blocks * add tests	2 years ago
Liam Cervante	7e18b1183d	Add test verifying issue #34476 has been fixed (#34478)	2 years ago
Jamie Finnigan	05b0bdac99	adopt hashicorp/cli fork of mitchellh/cli (#34429)	2 years ago
Liam Cervante	5cbf7c5f96	Add docs for automatic var files in test directories (#34349)	2 years ago
Kubo	c98e355ca1	terraform test: Collect variables from default var file within testing directory (#34341) ... * Add logic to collect variables for terrafrom test * Add tests for test variable collection * Update the test variable collection implementation * Update internal/backend/local/test.go Co-authored-by: Liam Cervante <liam.cervante@hashicorp.com> * Update internal/backend/local/test.go Co-authored-by: Liam Cervante <liam.cervante@hashicorp.com> * Move test variables into var file * resolve diff from cross-branch switch * go fmt --------- Co-authored-by: Liam Cervante <liam.cervante@hashicorp.com>	2 years ago
Martin Atkins	0f845adfdb	moduletest: EvalContext lives outside of the "terraform" package ... The initial implementation of the module testing language included some elements deeply embedded inside "package terraform", which is primarily concerned with the traditional Terraform language, which we might now call the "module language" to distinguish it from the test language and the stacks language. That was a reasonable design decision due to the modules runtime needing to evaluate some expressions in a similar way to how the modules language does it, but unfortunately it caused the test language runtime to depend a little too closely on some current implementation details of how the modules runtime deals with expression evaluation, and those details are about to change as we teach Terraform to be able to deal with unknown values in new places, as part of resolving this issue: https://github.com/hashicorp/terraform/issues/30937 This commit therefore refactors the test runtime a little so that it doesn't intrude so deeply into the module runtime's business. Instead of directly extending the module runtime's expression evaluator, we'll instead call into it in a similar way to how the "terraform console" command does, by wrapping the evaluation data produced by the module runtime to extend it with the only two special things that the test runtime needs: references to the output values of previous runs, and references to test-only variables that aren't declared as part of the module under test. This effectively means that the test runtime can now evaluate a superset of the expressions that would be valid in the global scope of the module under test using only the public API of the modules runtime, with no testing-specific evaluation logic embedded inside the modules runtime. This new strategy is also somewhat similar to how the stacks runtime calls into the modules runtime to evaluate components, but the stacks runtime only actually cares about output values and so doesn't need a full evaluation scope. This means the result here is a little more symmetrical with the approach we took in stacks, which will hopefully allow experience working on one to translate better to the other.	2 years ago