62 Commits (f-testing-eval-prototype)

Author	SHA1	Message	Date
James Bardin	ac2219ba6e	don't lose Private state data during copy ... Fix the scope of the private data copy in DeepCopy. Make sure Dependencies matches nil vs empty so that Equal compares correctly between copied states	7 years ago
James Bardin	1e1dcfd95a	always write the current state terraform_version ... Any time a new state is written, the current terraform version should be recorded.	7 years ago
Radek Simko	8a6d1d62b6	stringer: Regenerate files with latest version	7 years ago
Kristin Laemmert	d7dda4e436	states/statefile: upgrade legacy dependency syntax (#21159) ... We used to allow "foo.1" etc as a reference, but now it's "foo[1]".	7 years ago
Paul Tyng	ab2d09b0cb	godoc typo	7 years ago
Sander van Harmelen	21cca34716	Fixup the docs for WriteStateForMigration	7 years ago
Sander van Harmelen	57f6e01830	backend/local: preserve serial and lineage on failure ... When failing to write the state, the local backend writes the state to a local file called `errrored.tfstate`. Previously it would do so by creating a new state file which would use a new serial and lineage. By exorting the existing state file and directly assigning the new state, the serial and lineage are preserved.	7 years ago
Martin Atkins	c39905e1a8	command: Fix various issues in the "terraform state ..." subcommands ... In earlier refactoring we updated these commands to support the new address and state types, but attempted to partially retain the old-style "StateFilter" abstraction that originally lived in the Terraform package, even though that was no longer being used for any other functionality. Unfortunately the adaptation of the existing filtering to the new types wasn't exact and so these commands ended up having a few bugs that were not covered by the existing tests. Since the old StateFilter behavior was the source of various misbehavior anyway, here it's removed altogether and replaced with some simpler functions in the state_meta.go file that are tailored to the use-cases of these sub-commands. As well as just generally behaving more consistently with the other parts of Terraform that use the new resource address types, this commit fixes the following bugs: - A resource address of aws_instance.foo would previously match an resource of that type and name in any module, which disagreed with the expected interpretation elsewhere of meaning a single resource in the root module. - The "terraform state mv" command was not supporting moves from a single resource address to an indexed address and vice-versa, because the old logic didn't need to make that distinction while they are two separate address types in the new logic. Now we allow resources that do not have count/for_each to be treated as if they are instances for the purposes of this command, which is a better match for likely user intent and for the old behavior. Finally, we also clean up a little some of the usage output from these commands, which hasn't been updated for some time and so had both some stale information and some inaccurate terminology.	7 years ago
Andriy Borodiychuk	a0f3a2daa0	Update doc.go, fix typo (#20529)	7 years ago
Radek Simko	ea05da89cb	states/statemgr: Avoid HTML escaping when printing LockInfo	7 years ago
James Bardin	292535820d	ResourceInstanceObject needs to return a copy	8 years ago
Martin Atkins	e8f9fad0e3	states/statemgr: use -mod=vendor to run the state locking helper ... This ensures that we test using the same source as we're using everywhere else, and more tactically also ensures that when running in Travis-CI we won't try to download all of the dependencies of Terraform during this test. In the long run we will look for a more global solution to this, rather than adding this to all of our embedded "go" command calls directly, but this is intended as a low-risk solution to get the build working again in the mean time.	8 years ago
Martin Atkins	12572e97bc	core: Automatically upgrade resource instance states on read ... If an instance object in state has an earlier schema version number then it is likely that the schema we're holding won't be able to decode the raw data that is stored. Instead, we must ask the provider to upgrade it for us first, which might also include translating it from flatmap form if it was last updated with a Terraform version earlier than v0.12. This ends up being a "seam" between our use of int64 for schema versions in the providers package and uint64 everywhere else. We intend to standardize on int64 everywhere eventually, but for now this remains consistent with existing usage in each layer to keep the type conversion noise contained here and avoid mass-updates to other Terraform components at this time. This also includes a minor change to the test helpers for the backend/local package, which were inexplicably setting a SchemaVersion of 1 on the basic test state but setting the mock schema version to zero, creating an invalid situation where the state would need to be downgraded.	8 years ago
James Bardin	f375691819	add missing key-value from test	8 years ago
Martin Atkins	5255e85238	"go fmt" fixups ... Apparently my editor is still not reliably formatting on save, so I missed a few formatting quirks in these files.	8 years ago
Martin Atkins	6cb3b0f4cf	states/statemgr: Local locks meta is near output path, not input path ... This was a mistake while adapting this code from the old state.LocalState. Since the lock is held on the output file (s.path) the metadata should live adjacent to that rather than being built from the read path (s.readPath) that is used only as the initial snapshot on first instantiation. This also includes more logging, continuing the trend of other recent commits in these files. The local state behavior is sufficiently complex that these trace logs are a great help in debugging issues such as this one with the wrong files being used or actions being taken in the wrong order.	8 years ago
Martin Atkins	48601d261d	states/statemgr: In Filesystem, back up output file, not input file ... The filesystem backend has the option of using a different file for its initial read. Previously we were incorrectly writing the contents of that file out into the backup file, rather than the prior contents of the output file. Now we will always read the output file in RefreshState in order to decide what we will back up but then we will optionally additionally read the input file and prefer its content as the "current" state snapshot. This is verified by command.TestMetaBackend_planLocalStatePath and TestMetaBackend_configureNew, which are both now passing.	8 years ago
Martin Atkins	ec27526cc3	command: Fix TestMetaBackend_configuredChangeCopy_multiToMulti ... This was failing because we now handle the settings for the local backend a little differently as a result of decoding it with the HCL2 machinery. Specifically, the backend.State* fields are now assumed to be what is given in configuration, and any CLI overrides are maintained separately in OverrideState* fields so that they can be imposed "just in time" in StatePaths. This is particularly important because OverrideStatePath (when set) is used regardless of workspace name, while StatePath is a suitable value only for the "default" workspace, with others needing to be constructed from StateWorkspaceDir instead.	8 years ago
Martin Atkins	22c84c71a4	command: Use statemgr.Import and statemgr.Export for state push and pull ... We previously hacked around the import/export functionality being missing in the statemgr layer after refactoring, but now it's been reintroduced to fix functionality elsewhere we should use the centralized Import and Export functions to ensure consistent behavior. In particular, this pushes the logic for checking lineage and serial during push down into the state manager itself, which is better because all other details about lineage and serial are managed within the state managers.	8 years ago
Martin Atkins	985b414dca	states/statemgr: Fix the Filesystem state manager tests ... Now that we're verifying the Terraform version during state loading, we need to force a particular Terraform version to use during these tests.	8 years ago
Martin Atkins	94510bc1b9	states/statemgr: Migrate, Import, and Export functions ... In our recent refactoring of the state manager interfaces we made serial and lineage management the responsibility of the state managers themselves, not exposing them at all to most callers, and allowing for simple state managers that don't implement them at all. However, we do have some specific cases where we need to preserve these properly when available, such as migration between backends, and the "terraform state push" and "terraform state pull" commands. These new functions and their associated optional interface allow the logic here to be captured in one place and access via some simple calls. Separating this from the main interface leaves things simple for the normal uses of state managers. Since these functions are mostly just thin wrappers around other functionality, they are not yet well-tested directly, but will be indirectly tested through the tests of their callers. A subsequent commit will add more unit tests here.	8 years ago
Sander van Harmelen	634430ebb2	Fix wildcard dependencies when upgrading states ... Fixes #19347	8 years ago
Martin Atkins	9ba399bca8	command: Fix TestInit_getProvider ... After all of the refactoring we were no longer checking the Terraform version field in a state file, causing this test to fail. This restores that check, though with a slightly different error message.	8 years ago
Sander van Harmelen	5458a91985	command/state: update and fix the state show command	8 years ago
Sander van Harmelen	7ec3f96e3a	command/state: update and fix the state mv command	8 years ago
Sander van Harmelen	19c1241a50	command/state: update and fix the state rm command	8 years ago
Sander van Harmelen	5e11de460a	Merge pull request #19130 from hashicorp/f-state-push-pull ... command/state: update and fix the state push and pull	8 years ago
Sander van Harmelen	7fbd93b5cd	command/state: update and fix the state push and pull	8 years ago
Sander van Harmelen	af1a471a05	command/state: update and fix the state list command	8 years ago
Martin Atkins	ec57927ea3	build: Take protoc out of the "go generate" path ... Since protoc is not go-gettable, and most development tasks in Terraform won't involve recompiling protoc files anyway, we'll use a separate mechanism for these. This way "go generate" only depends on things we can "go get" in the "make tools" target. In a later commit we should also in some way specify a particular version of protoc to use so that we don't get "flapping" regenerations as developers work with different versions, but the priority here is just to make "make generate" minimally usable again to restore the dev workflow documented in the README. This also includes some updates that resulted from running "make generate" and "make protobuf" after those Makefile changes were in place.	8 years ago
James Bardin	d707049f72	don't make a backup of a nil state ... This makes sure we don't create a backup of an intermediate state if the first read state was empty.	8 years ago
Martin Atkins	66f96cf842	command: Un-stub and reimplement "terraform state rm" ... This was previously targeting the old state manager and state types, so it needed some considerable rework to get it working again with the new state types. Since our new resource address syntax lacks the weird extra .deposed special case we had before, we instead interpret addresses as whole-instance addresses here and remove the deposed objects along with the current one (if present), since this is more likely to match with user expectations because we don't consider deposed objects to be independently addressable in any other situation. With that said, to be more explicit about what is going on we do now have a -dry-run mode and maintain separate counts of current and deposed instances so that we can expose that in the UI where relevant.	8 years ago
Martin Atkins	331cb07a05	states/statefile: Tolerate nil state in statefile.New ... For historical reasons sometimes we have nil state in situations where we'd still like to persist state snapshots to a store. To make life easier for those callers, we'll substitute an empty state if we are given a nil one, thus allowing us to still generate a valid serialization that will load back in as an empty state.	8 years ago
Martin Atkins	b0016e9cf6	command: Allow tests to run to completion without panics or hangs ... There are still 160 test failures as of this commit, but at least the test program can run to completion and list out all the failures.	8 years ago
Martin Atkins	ec2e6cb06f	terraform: Prune resource husks at the end of "terraform destroy" ... When we're being asked to destroy everything, we ideally want to end up with a totally empty state. Normally we will conservatively keep around the "husks" of resources (what's left after all of the instances have been destroyed) unless they are configured without count or for_each, but in this special case we'll prune those out. The implication of this is that in "weird" expression contexts that happen before the next "terraform plan", such as evaluation in "terraform console" or expressions in data resources and provider blocks that get evaluated during the refresh walk, we will see these results as unknown rather than as empty lists of objects. We accept that weirdness for now because in a future release we are likely to remove "refresh" as a separate walk anyway, doing all of that work during the plan walk where we can ensure that these values are properly re-populated before trying to use them.	8 years ago
Martin Atkins	2b5d9506b1	states: Fix TestResourceInstanceDeposeCurrentObject ... This was missed when changing the signature of deposeCurrentObject in an earlier commit.	8 years ago
Martin Atkins	2eea07750a	core: Clean up resource states when they are orphaned ... We previously had mechanisms to clean up only individual instance states, leaving behind empty resource husks in the state after they were all destroyed. This takes care of it in the "orphan" case. It does not yet do it in the "terraform destroy" or "terraform plan -destroy" cases because we don't have anywhere to record in the plan that we're actually destroying and so the resource configurations should be ignored and _everything_ should be cleaned. We'll let the state be not-quite-empty in that case for now, since it doesn't really hurt; cleaning up orphans is the main case because the state will live on afterwards and so leftover cruft will accumulate over the course of many changes.	8 years ago
Martin Atkins	faddb83a92	core: If create leg of create_before_destroy fails, restore deposed ... I misunderstood the logic here on the first pass of porting to the new provider and state types: EvalUndeposeState is supposed to return the deposed object back to being current again, so we can undo the deposing in the case where the create leg fails. If we don't do this, we end up leaving the instance with no current object at all and with its prior object deposed, and then the later destroy node deletes that deposed object, leaving the user with no object at all. For safety we skip this restoration if there _is_ a new current object, since a failed create can still produce a partial result which we need to keep to avoid losing track of any remote objects that were successfully created.	8 years ago
Martin Atkins	334c6f1c2c	core: Be more explicit in how we handle create_before_destroy ... Previously our handling of create_before_destroy -- and of deposed objects in particular -- was rather "implicit" and spread over various different subsystems. We'd quietly just destroy every deposed object during a destroy operation, without any user-visible plan to do so. Here we make things more explicit by tracking each deposed object individually by its pseudorandomly-allocated key. There are two different mechanisms at play here, building on the same concepts: - During a replace operation with create_before_destroy, we *pre-allocate* a DeposedKey to use for the prior object in the "apply" node and then pass that exact id to the destroy node, ensuring that we only destroy the single object we planned to destroy. In the happy path here the user never actually sees the allocated deposed key because we use it and then immediately destroy it within the same operation. However, that destroy may fail, which brings us to the second mechanism: - If any deposed objects are already present in state during _plan_, we insert a destroy change for them into the plan so that it's explicit to the user that we are going to destroy these additional objects, and then create an individual graph node for each one in DiffTransformer. The main motivation here is to be more careful in how we handle these destroys so that from a user's standpoint we never destroy something without the user knowing about it ahead of time. However, this new organization also hopefully makes the code itself a little easier to follow because the connection between the create and destroy steps of a Replace is reprseented in a single place (in DiffTransformer) and deposed instances each have their own explicit graph node rather than being secretly handled as part of the main instance-level graph node.	8 years ago
Martin Atkins	9eb32c4536	core: Reinstaint instance tainting, but without mutating objects ... Our previous mechanism for dealing with tainting relied on directly mutating the InstanceState object to mark it as such. In our new state models we consider the instance objects to be immutable by convention, and so we frequently copy them. As a result, the taint flagging was no longer making it all the way through the apply evaluation process. Here we now implement tainting as a separate step in the evaluation process, creating a copy of the object with a tainted status if there were any errors during creation. This introduces a new behavior where any provider-level errors during creation will also cause an instance to be marked as tainted if any object is returned at all. Create-time errors _normally_ result in no object at all, but the provider might return an object if the failure occurred at a subsequent step of a multi-step creation process and so left behind a remote object that needs to be cleaned up on a future run.	8 years ago
Martin Atkins	d48f3600fe	states: In Module.testString, use incrementing ids for deposed ... In our old world we always used 1-based indices into a slice of deposed objects. The new models instead use a map keyed by pseudorandom strings, so that deposed objects will have a consistent identity across multiple operations. However, having that pseudo-random string in our test comparison output is not helpful, since such strings can never match hard-coded expectation strings. Therefore for the purposes of generating this test comparison output we'll revert back to using 1-based indexes. This should avoid problems for tests that only create one deposed object per instance, but those which create more than one will need to do some more work since the _ordering_ of these objects in the output is still pseudorandom as a result of it coming from a map rather than a slice.	8 years ago
Martin Atkins	83066cd57f	states: Support non-string primitives in state string representation	8 years ago
Martin Atkins	5faf027ea7	states: In State.String, use colon suffix only after all module names	8 years ago
Martin Atkins	60718efc8e	states: DeepCopy for ResourceInstanceObject ... Also a fix for not actually deep-copying "Private", since when this was originally written it was a cty.Value but then later became a []byte.	8 years ago
Martin Atkins	edc0ce6333	states: Prune empty modules after possibly removing resources ... Also includes a new log message for the situation where we _do_ prune, since this seems helpful during debugging.	8 years ago
Martin Atkins	8003b3408f	states: Fix incorrect ResourceInstanceObjectSrc.DeepCopy ... Accidental shadowing of the top-level attrsFlat variable meant that the flatmap portion of these objects was getting lost in the DeepCopy result.	8 years ago
James Bardin	6f429cc81b	make state output match legacy output	8 years ago
Martin Atkins	9af67806fc	core: Prune placeholder objects from state after refresh ... Prior to our refactoring here, we were relying on a lucky coincidence for correct behavior of the plan walk following a refresh in the same run: - The refresh phase created placeholder objects in the state to represent any resource instance pending creation, to allow the interpolator to read attributes from them when evaluating "provider" and "data" blocks. In effect, the refresh walk is creating a partial plan that only covers creation actions, but was immediately discarding the actual diff entries and storing only the planned new state. - It happened that objects pending creation showed up in state with an empty ID value, since that only gets assigned by the provider during apply. - The Refresh function concluded by calling terraform.State.Prune, which deletes from the state any objects that have an empty ID value, which therefore prevented these temporary objects from surviving into the plan phase. After refactoring, we no longer have this special ID field on instance object state, and we instead rely on the Status field for tracking such things. We also no longer have an explicit "prune" step on state, since the state mutation methods themselves keep the structure pruned. To address this, here we introduce a new instance object status "planned", which is equivalent to having an empty ID value in the old world. We also introduce a new method on states.SyncState that deletes from the state any planned objects, which therefore replaces that portion of the old State.prune operation just for this refresh use-case. Finally, we are now expecting the expression evaluator to pull pending objects from the planned changeset rather than from the state directly, and so for correct results these placeholder resource creation changes must also be reported in a throwaway changeset during the refresh walk. The addition of states.ObjectPlanned also permits a previously-missing safety check in the expression evaluator to prevent us from relying on the incomplete value stored in state for a pending object, in the event that some bug prevents the real pending object from being written into the planned changeset.	8 years ago
Martin Atkins	20adb9d9b7	core: Evaluate resource references from plan where possible ... Our state representation is not able to preserve unknown values, so it's not suitable for retaining the transient incomplete values we produce during planning. Instead, we'll discard the unknown values when writing to state and have the expression evaluator prefer an object from the plan where possible. We still use the shape of the transient state to inform things like the resource's "each mode", so the plan only masks the object values themselves.	8 years ago
Martin Atkins	03e6771536	states/statemgr: don't panic if no state file is present on first write	8 years ago