36 Commits (27770ee8053da48fbb684fd23d415aae376d2bed)

Author	SHA1	Message	Date
Samsondeen	719aefdb89	tf query: ready list blocks for beta (#37619)	5 months ago
Liam Cervante	551ba2e525	Implement controlling destroy functionality within Terraform Test (#37359) ... * Add ability to parse backend blocks present in a test file's run blocks, validate configuration (#36541) * Add ability to parse backend blocks from a run block * Add validation to avoid multiple backend blocks across run blocks that use the same internal state file. Update tests. * Add validation to avoid multiple backend blocks within a single run block. Update tests. * Remove use of quotes in diagnostic messages * Add validation to avoid backend blocks being used in plan run blocks. Update tests. * Correct local backend blocks in new test fixtures * Add test to show that different test files can use same backend block for same state key. * Add validation to enforce state-storage backend types are used * Remove TODO comment We only need to consider one file at a time when checking if a state_key already has a backend associated with it; parallelism in `terraform test` is scoped down to individual files. * Add validation to assert that the backend block must be in the first apply command for an internal state * Consolidate backend block validation inside a single if statement * Add initial version of validation that ensures a backend isn't re-used within a file * Explicitly set the state_key at the point of parsing the config TODO: What should be done with method (moduletest.Run).GetStateKey? * Update test fixture now that reusing backend configs has been made invalid * Add automated test showing validation of reused configuration blocks * Skip test due to flakiness, minor change to test config naming * Update test so it tolerates non-deterministic order run blocks are evaluated in * Remove unnecessary value assignment to r.StateKey * Replace use of GetStateKey() with accessing the state key that's now set during test config parsing * Fix bug so that run blocks using child modules get the correct state key set at parsing time * Update acceptance test to also cover scenario where root and child module state keys are in use * Update test name * Add newline to regex * Ensure consistent place where repeat backend error is raised from * Write leftover test state(s) to file (#36614) * Add additional validation that the backend used in a run is a supported type (#36648) * Prevent test run when leftover state data is present (#36685) * `test`: Set the initial state for a state files from a backend, allow the run that defines a backend to write state to the backend (#36646) * Allow use of backend block to set initial state for a state key * Note about alternative place to keep 'backend factories' * Allow the run block defining the backend to write state to it * Fix rebase * Change to accessing backend init functions via ContextOpts * Add tests demonstrating how runs containing backend blocks use and update persisted state * Fix test fixture * Address test failure due to trouble opening the state file This problem doesn't happen on MacOS, so I assume is due to the Linux environment of GitHub runners. * Fix issue with paths properly I hope * Fix defect in test assertion * Pivot back to approach introduced in 4afc3d7 * Let failing tests write to persistent state, add test case covering that. I split the acceptance tests into happy/unhappy paths for this, which required some of the helper functions' declarations to be raised up to package-level. * Change how we update internal state files, so that information about the associated backend is never lost * Fix UpdateStateFile * Ensure that the states map set by TestStateTransformer associates a backend with the correct run. * Misc spelling fixes in comments and a log * Replace state get/set functions with existing helpers (#36747) * Replace state get/set functions with existing helpers * Compare to string representation of state * Compare to string representation of state * Terraform Test: Allow skipping cleanup of entire test file or individual run blocks (#36729) * Add validation to enforce skip_cleanup=false cannot be used with backend blocks (#36857) * Integrate use of backend blocks in tests with skip_cleanup feature (#36848) * Fix nil pointer error, update test to not be table-driven * Make using a backend block implicitly set skip_cleanup to true * Stop state artefacts being created when a backend is in use and no cleanup errors have occurred * Return diagnostics so calling code knows if cleanup experienced issues or not * Update tests to show that when cleanup fails a state artefact is created * Add comment about why diag not returned * Bug fix - actually pull in the state from the state manager! * Split and simplify (?) tests to show the backend block can create and/or reuse prior state * Update test to use new fixtures, assert about state artefact. Fix nil pointer * Update test fixture in use, add guardrail for flakiness of forced error during cleanup * Refactor so resource ID set in only one place * Add documentation for using a `backend` block during `test` (#36832) * Add backend as a documented block in a run block * Add documentation about backend blocks in run blocks. * Make the relationship between backends and state keys more clear, other improvements * More test documentation (#36838) * Terraform Test: cleanup command (#36847) * Allow cleanup of states that depend on prior runs outputs (#36902) * terraform test: refactor graph edge calculation * create fake run block nodes during cleanup operation * tidy up TODOs * fix tests * remove old changes * Update internal/moduletest/graph/node_state_cleanup.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> * Improve diagnostics around skip_cleanup conflicts (#37385) * Improve diagnostics around skip_cleanup conflicts * remove unused dynamic node * terraform test: refactor manifest file for simplicity (#37412) * test: refactor apply and plan functions so no run block is needed * terraform test: write and load state manifest files * Terraform Test: Allow skipping cleanup of entire test file or individual run blocks (#36729) * terraform test: add support for skip_cleanup attr * terraform test: add cleanup command * terraform test: add backend blocks * pause * fix tests * remove commented code * terraform test: make controlling destroy functionality experimental (#37419) * address comments * Update internal/moduletest/graph/node_state_cleanup.go Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> --------- Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> * add experimental changelog entries --------- Co-authored-by: Sarah French <15078782+SarahFrench@users.noreply.github.com> Co-authored-by: Samsondeen <40821565+dsa0x@users.noreply.github.com> Co-authored-by: Samsondeen Dare <samsondeen.dare@hashicorp.com>	5 months ago
Kristin Laemmert	9256074c43	Actions in plan/changes (#37320) ... * Add actions to the plans and change * jsonplan - ignoring LinkedResources for now, those are not in the MVP * pausing here: we'll work on the plan rendering later	7 months ago
Daniel Banck	ab04e5cfec	Include list resource schemas in provider json schemas (TF-25497) (#37163) ... * Include list schemas in provider json schemas * Only include list schemas in experimental builds	8 months ago
Sarah French	ab6e4f2299	Refactor JUnit XML output of `terraform test` into a new `junit` package (#36304) ... * Refactor JUnit XML output to use new concept of an Artifact * Move JUnit-related code into new `artifact` package * Refactor Artifact's Save method to return diagnostics, update comments Previously TestJUnitXMLFile implemented the View interface, which cannot return errors. Now it's not a View any more we can simplify things. * Make junitXMLTestReport output deterministic by iterating over a slice instead of a map, add test * Provide sources to junitXMLTestReport, allowing complete error messages in the XML We need to ensure that artifact.NewTestJUnitXMLFile is called once the config Loader is available as a non-nil pointer * Whitespace * Add some test coverage for JUnit XML output for `terraform test` * Refactor how file is saved, add tests * Move XML structs definitions outside of `junitXMLTestReport` * Fix nil pointer bug * Add missing file headers * Refactor comparison of byte slices * Rename package to `junit`, rename structs to match * Add a test showing JUnit output when a test is skipped by the user	1 year ago
Sarah French	43f31c43b0	Add start time timestamps to `<testcase>`/Run data in JUnit XML output from `terraform test` (#36295) ... * Record metadata about the start timestamp for a Run/test case * Add RFC3339 format timestamp attributes to `<testcase>` elements in JUnit XML output * Ensure timestamps are saved as and rendered as UTC timestamps	1 year ago
Bruno Schaatsbergen	a5d8673a41	Revert "Merge pull request #35929 from bschaatsbergen/b/fix-retry-messages-in-cloud-backend-json" ... This reverts commit f8cb25a246, reversing changes made to a25831d6ef.	1 year ago
Bruno Schaatsbergen	6c14e52d8e	test: use Cloud view ... Signed-off-by: Bruno Schaatsbergen <git@bschaatsbergen.com>	1 year ago
Bruno Schaatsbergen	f9797595e3	chore: initialize new view in meta backend ... Signed-off-by: Bruno Schaatsbergen <git@bschaatsbergen.com>	1 year ago
Martin Atkins	c1342f4f6a	terraform test: Include test case execution time in JUnit XML ... Our currently-experimental JUnit XML output didn't previously include the "time" attribute for test runs at all, but from testing with common JUnit XML consumers we've learned that populating this is effectively mandatory because otherwise software just assumes that it's zero, producing a misleading result. We'll now populate it based on the test runner's measure of the execution time, assuming that the test runner actually measured it. If not, we'll leave it omitted and thus get the same behavior as before, which in practice means that most software will treat it as zero anyway. Because JUnit XML output is still experimental with the goal of getting feedback on exactly how we map the Terraform testing model onto that Java-oriented format, this is intentionally not supported for remote test runs in Terraform Cloud yet. It isn't really practical for Terraform Cloud to participate in short-lived Terraform CLI experiments, because Terraform Cloud has a different release schedule than Terraform CLI does. However, if this experiment is successful then we will eventually need to find a way to plumb this additional information through the JSON logs and ensure that it arrives in the locally-generated JUnit XML output.	2 years ago
Martin Atkins	c8611b51d7	terraform test: JUnit XML output sets testcase classname ... JUnit XML output from "terraform test" is currently experimental, because we're still trying to decide how best to map Terraform's testing concepts onto the Java-oriented JUnit XML format. The first round of feedback showed that some JUnit consumers completely ignore test suite names, and instead expect each test case to specify which "class" it belongs to. Those tools then might allow grouping or filtering the tests by "class". Despite the Java-oriented terminology, it seems that "classname" is really just interpreted something like "test group name", where the classname and the name together represent the full test case identifier. With that in mind then, we'll now duplicate the test suite name (which in our case is the filename of the test scenario file) into the classname attribute of each testcase element. For now we'll retain the per-suite names, always containing the same value, because we also have evidence of some consumers preferring to find this information in that location.	2 years ago
Martin Atkins	884e1fb2a4	terraform: Plans can be "complete" and "applyable" ... These ideas are both already implied by some logic elsewhere in the system, but until now we didn't have the decision logic centralized in a single place that could therefore evolve over time without necessarily always updating every caller together. We'll now have the modules runtime produce its own boolean ruling about each characteristic, which callers can rely on for the mechanical decision-making of whether to offer the user an "approve" prompt, and whether to remind the user after apply that it was an incomplete plan that will probably therefore need at least one more plan/apply round to converge. The "Applyable" flag directly replaces the previous method Plan.CanApply, with equivalent logic. Making this a field instead of a method means that we can freeze it as part of a saved plan, rather than recalculating it when we reload the plan, and we can export the field value in our export formats like JSON while ensuring it'll always be consistent with what Terraform is using internally. Callers can (and should) still use other context in the plan to return more tailored messages for specific situations they already know about that might be useful to users, but with these flags as a baseline callers can now just fall back to a generic presentation when encountering a situation they don't yet understand, rather than making the wrong decision and causing something strange to happen. That is: a lack of awareness of a new rule will now cause just a generic message in the UI, rather than incorrect behavior. This commit mostly just deals with populating the flags, and then all of the direct consequences of that on our various tests. Further changes to actually make use of these flags elsewhere in the system will follow in later commits, both in this repository and in other repositories.	2 years ago
Martin Atkins	89a53e9105	command/test: Fix invalid XML processing instruction ... Annoyingly, I seem to have flubbed this slightly while I was adapting to use enc.EncodeToken instead of (as originally written) just writing the literal processing instruction bytes onto the front of the buffer directly. This made the output invalid for any XML processor that interprets the special "xml" processing instruction. It'll now be valid again.	2 years ago
Martin Atkins	4a0d06c13e	terraform test: Report skipped runs in JUnit XML output ... Previously we were just ignoring skipped runs completely, but JUnit XML has some elements/attributes for describing those and so we'll populate those too now.	2 years ago
Martin Atkins	b43ddd8ff6	terraform test: Experimental JUnit XML reporting ... This adds an experimental new option -junit-xml=FILENAME for the "terraform test" command. When specified, it writes a JUnit XML report to the specified filename once the test run is complete, while continuing to report test progress in the UI in the usual way. This is only experimental for now because it remains to be seen if this particular mapping to the JUnit XML schema is actually useful in real software -- this format is woefully underdocumented and implemented slightly differently by each consumer -- and so we might change this significantly before stabilizing it, or remove it altogether if it turns out that there's no useful mapping to JUnit XML here. Hopefully those who are interested in JUnit XML reports will try this experiment against their favorite JUnit XML-consuming software and report back whether the report is presented in a helpful way. It's a de-facto convention for JUnit XML to be reported separately to a file, rather than replacing the normal test run output, since tools that consume this format tend to present its results in a separate and less prominent place than the output of the command itself. This option is designed to follow that convention for consistency with various other software that produces this format. The implementation here is intentionally pretty minimal and simplistic just as a starting point for gathering feedback. The main priority is that it be easy to evolve this based on feedback and to remove it altogether if we decide not to stabilize this at all. If this does become stabilized, it might deserve being factored out into a separate package so that we can minimize the amount of logic embedded directly inside the views package, and it will certainly need some unit tests to represent what we've committed to supporting in future versions.	2 years ago
Martin Atkins	0811395e6f	states: State.AllResourceInstanceObjectAddrs should say "Managed" ... This method used the more appropriate name AllManagedResourceInstanceObjectAddrs in its docstring, but was incorrectly named just AllResourceInstanceObjectAddrs in its actual definition, obscuring the fact that it filters out everything except objects from managed resources. Now we'll use the originally-documented name to make this clearer. This also factors out the implementation into a separate unexported function that takes the filtering criteria as an argument, since in future commits we're going to add other variations of this which return different subsets of the resource instance objects in the state.	2 years ago
Martin Atkins	ac562eeb6f	states: State.AllResourceInstanceObjectAttrs returns a set ... Originally we had this function returning a slice of an anonymous struct type because we hadn't yet recognized "resource instance object" as a first-class address type. We do now have addrs.AbsResourceInstanceObject to represent that sort of thing, and so we'll use that here. This also means we can return a set of addresses rather than a slice, which is semantically better as this set of addresses is not inherently ordered. Some existing callers were depending on the sort.SliceStable we were originally doing to make the result dependable, and so now we have a generic helper for turning addrs.Set into a sorted slice of the same type using our address types' "Less" methods that describe their natural or preferred ordering.	2 years ago
Liam Cervante	01c0480e77	Test framework will produce regular updates in -json mode (#33953) ... * Test framework will produce regular updates in -json mode * add tests * docs * docs tweaking	2 years ago
Liam Cervante	ffbcaf8bef	Implement TFC integration for private module registry test runs (#33710) ... * implement module testing via TFC * ready for review * fix static checks * licence headers	2 years ago
Liam Cervante	91d26b6ea6	Refactor file and run block feedback frequency (#33840) ... * Refactor file and run block feedback frequency * fix consistency checks * make generate	2 years ago
Liam Cervante	2ea141b3f4	Embed the schemas directly in the testing framework JSON verbose outputs (#33755)	3 years ago
Liam Cervante	9bc5c7afbb	Include run block for JSON fatal interrupt test summaries (#33720)	3 years ago
CJ Horton	98b81d3c50	add missing copyright headers	3 years ago
Liam Cervante	4560a83721	test framework: include transformed config before validating config (#33608)	3 years ago
Sebastian Rivera	08e58fd484	Fix saved plan test regressions, fixtures	3 years ago
Liam Cervante	ad26644578	Fix flaky test in views package (#33560)	3 years ago
Liam Cervante	ca85d3bf85	Implement word wrapping in the terraform test view functions (#33547) ... * Implement word wrapping in the terraform test view functions * Update internal/command/views/test.go Co-authored-by: CJ Horton <17039873+radditude@users.noreply.github.com> --------- Co-authored-by: CJ Horton <17039873+radditude@users.noreply.github.com>	3 years ago
Liam Cervante	6c7db16566	testing framework: refactor interrupt logic for immediate exits (#33532) ... * testing framework: refactor interrupt logic * fix formatting	3 years ago
Liam Cervante	6882dd9530	testing framework: introduce test command optional flags (#33504) ... * testing framework: introduce test command optional flags * address consistency checks	3 years ago
Liam Cervante	4862812c94	testing framework: introduce interrupts for stopping tests (#33477) ... * [testing framework] prepare for beta phase of development * [Testing Framework] Add module block to test run blocks * [testing framework] allow tests to define and override providers * testing framework: introduce interrupts for stopping tests * remove panic handling, will do it properly later	3 years ago
Liam Cervante	5acc95dda7	[Testing Framework] Add module block to test run blocks (#33456) ... * [testing framework] prepare for beta phase of development * [Testing Framework] Add module block to test run blocks	3 years ago
Liam Cervante	ed822559e5	[Testing framework] Implement JSON view functionality for test command (#33400)	3 years ago
Liam Cervante	ce8fd2943d	[Testing Framework] Add test structure to views package for rendering test output (#33324) ... * Add test structure to views package for rendering test output * address comments	3 years ago
Liam Cervante	48c818927c	Remove old implementation of terraform test command	3 years ago
hashicorp-copywrite[bot]	325d18262e	[COMPLIANCE] Add Copyright and License Headers	3 years ago
Martin Atkins	ffe056bacb	Move command/ to internal/command/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago