23 Commits (f-debugger)

Author	SHA1	Message	Date
Rémi Lapeyre	5c7008bd89	Merge remote-tracking branch 'origin/master' into consul-force-unlock	6 years ago
Alisdair McDiarmid	e183163739	Merge pull request #25856 from remilapeyre/consul-split-state ... Split the state in chunks when they outgrow the limit of Consul KV store	6 years ago
Rémi Lapeyre	e680211bc0	Split the state in chunks when they outgrow the limit of Consul KV store ... The Consul KV store limits the size of the values in the KV store to 524288 bytes. Once the state reaches this limit Consul will refuse to save it. It is currently possible to try to bypass this limitation by enable Gzip but the issue will manifest itself later. This is particularly inconvenient as it is possible for the state to reach this limit without changing the Terraform configuration as datasources or computed attributes can suddenly return more data than they used to. Several users already had issues with this. To fix the problem once and for all we now split the payload in chunks of 524288 bytes when they are to large and store them separatly in the KV store. A small JSON payload that references all the chunks so we can retrieve them later and concatenate them to reconstruct the payload. While this has the caveat of requiring multiple calls to Consul that cannot be done as a single transaction as those have the same size limit, we use unique paths for the chunks and CAS when setting the last payload so possible issues during calls to Put() should not result in unreadable states. Closes https://github.com/hashicorp/terraform/issues/19182	6 years ago
Rémi Lapeyre	032d339915	Sanitize lock path for the Consul backend when it ends with a / ... When the path ends with / (e.g. `path = "tfstate/"), the lock path used will contain two consecutive slashes (e.g. `tfstate//.lock`) which Consul does not accept. This change the lock path so it is sanitized to `tfstate/.lock`. If the user has two different Terraform project, one with `path = "tfstate"` and the other with `path = "tfstate/"`, the paths for the locks will be the same which will be confusing as locking one project will lock both. I wish it were possible to forbid ending slashes altogether but doing so would require all users currently having an ending slash in the path to manually move their Terraform state and would be a poor user experience. Closes https://github.com/hashicorp/terraform/issues/15747	6 years ago
Rémi Lapeyre	11eb88753d	Fix `terraform force-unlock <lock_id>` for Consul backend ... When locking was enabled with the Consul backend and the lock not properly released, the `terraform force-unlock <lock_id>` command would do nothing as its implementation would exit early in that case. It now destroys the session that created the lock and clean both the lock and the lock-info keys. A regression test is added to TestConsul_destroyLock() to catch the issue if it happends again. Closes https://github.com/hashicorp/terraform/issues/22174	6 years ago
Kristin Laemmert	6621501ae3	state: remove deprecated state package (#25490) ... Most of the state package has been deprecated by the states package. This PR replaces all the references to the old state package that can be done simply - the low-hanging fruit. * states: move state.Locker to statemgr The state.Locker interface was a wrapper around a statemgr.Full, so moving this was relatively straightforward. * command: remove unnecessary use of state package for writing local terraform state files * move state.LocalState into terraform package state.LocalState is responsible for managing terraform.States, so it made sense (to me) to move it into the terraform package. * slight change of heart: move state.LocalState into clistate instead of terraform	6 years ago
James Bardin	12bda53558	Revert "create clistate.Locker interface" ... This reverts commit e88bd74bb7.	8 years ago
James Bardin	e88bd74bb7	create clistate.Locker interface ... Simplify the use of clistate.Lock by creating a clistate.Locker instance, which stores the context of locking a state, to allow unlock to be called without knowledge of how the state was locked. This alows the backend code to bring the needed UI methods to the point where the state is locked, and still unlock the state from an outer scope.	8 years ago
James Bardin	25a8227291	add broken test for lock lost on connection error ... Add a way to inject network errors by setting an immediate deadline on open consul connections. The consul client currently doesn't retry on some errors, and will force us to lose our lock. Once the consul api client is fixed, this test will fail.	9 years ago
James Bardin	d0ecb232ae	record consul session ID in lock info ... This can help correlate TF and consul logs	9 years ago
James Bardin	1070d04b07	Remove consul context adapter for RenewPeriodic ... Updated the vendored consul which no longer requires the channel adapter to convert a `chan stuct{}` to a `<-chan struct{}`. Call testutil.NewTestServerConfigT with the new signature.	9 years ago
James Bardin	bcb11f6d89	have the consul client manage the lock session ... When a consul lock is lost, there is a possibility that the associated session is still active. Most commonly, the long request to watch the lock key may error out, while the session is continually refreshed at a rate of TTL/2. First have the lock monitor retry the lock internally for at least 10 seconds (5 attempts with the default 2 second wait time). In most cases this will reconnect on the first try, keeping the lock channel open. If the consul lock can't recover itself, then cancel the session as soon as possible (terminating the PreiodicRenew will call Session.Destroy), and start over. In the worse case, the consul agents were split, and the session still exists on the leader so we may need to wait for the old session TTL, plus the LockWait time to renew the lock. We use a Context for the cancellation channels here, because that removes the need to worry about double-closes and nil channels. It requires an awkward adapter goroutine for now to convert the Done() `<-chan` to a `chan` for PeriodicRenew, but makes the rest of the code safer in the long run.	9 years ago
James Bardin	3df48bfc27	relock consul when lock is lost ... Consul locks are based on liveness, and may be lost due timeouts, network issued, etc. If the client determines the lock was lost, attempt to reacquire the lock immediately. The client was also not using the `lock` config option. Disable locks if that is not set.	9 years ago
James Bardin	e8330b6f53	use CAS for consul state Put	9 years ago
James Bardin	5621d97925	cleanup consul lock entries ... This matches the consul cli behavior, where locks are cleaned up after use. Return an error from re-locking the state. This isn't required by the Locker interface, but it's an added sanity check for state operations. What was incorrect here was returning an empty ID and error, which would indicate that Lock/Unlock isn't supported.	9 years ago
Christoffer Kylvåg	abfa35db7c	backend/remote-state/consul: Make gzip compression configurable (#8491)	9 years ago
Christoffer Kylvåg	e098c7c24a	backend/remote-state/consul: Add gzip support to consul backend client (#8491)	9 years ago
James Bardin	ec00564be6	Clean up LockInfo and LockError and use them ... Gove LockInfo a Marshal method for easy serialization, and a String method for more readable output. Have the state.Locker implementations use LockError when possible to return LockInfo and an error.	9 years ago
James Bardin	67bbebce08	Have consul state reutrn the lock ID ... The lock ID isn't used because the lock is tied to the client, but return the lock ID to match the behavior of other locks.	9 years ago
James Bardin	cd233fef6a	make consul client pass state.Locker tests	9 years ago
James Bardin	14d965722e	Use single state.LockInfo struct ... Remove redundant structures	9 years ago
James Bardin	54cac349a3	Add state locking to consul backend ... Use consul locks to implement state locking. The lock path is state path + "/.lock" which matches the consul cli default for locks. Lockinfo is stored at path + "/.lockinfo".	9 years ago
Mitchell Hashimoto	1f5d425428	backend/remote-state ... This allows migration of the remote state implementations to a richer experience including input asking.	10 years ago