54 Commits (f-debugger)

Author	SHA1	Message	Date
James Bardin	2635b3b023	update remote state to use legacy types	6 years ago
Rémi Lapeyre	5c7008bd89	Merge remote-tracking branch 'origin/master' into consul-force-unlock	6 years ago
Alisdair McDiarmid	e183163739	Merge pull request #25856 from remilapeyre/consul-split-state ... Split the state in chunks when they outgrow the limit of Consul KV store	6 years ago
Rémi Lapeyre	e680211bc0	Split the state in chunks when they outgrow the limit of Consul KV store ... The Consul KV store limits the size of the values in the KV store to 524288 bytes. Once the state reaches this limit Consul will refuse to save it. It is currently possible to try to bypass this limitation by enable Gzip but the issue will manifest itself later. This is particularly inconvenient as it is possible for the state to reach this limit without changing the Terraform configuration as datasources or computed attributes can suddenly return more data than they used to. Several users already had issues with this. To fix the problem once and for all we now split the payload in chunks of 524288 bytes when they are to large and store them separatly in the KV store. A small JSON payload that references all the chunks so we can retrieve them later and concatenate them to reconstruct the payload. While this has the caveat of requiring multiple calls to Consul that cannot be done as a single transaction as those have the same size limit, we use unique paths for the chunks and CAS when setting the last payload so possible issues during calls to Put() should not result in unreadable states. Closes https://github.com/hashicorp/terraform/issues/19182	6 years ago
Rémi Lapeyre	032d339915	Sanitize lock path for the Consul backend when it ends with a / ... When the path ends with / (e.g. `path = "tfstate/"), the lock path used will contain two consecutive slashes (e.g. `tfstate//.lock`) which Consul does not accept. This change the lock path so it is sanitized to `tfstate/.lock`. If the user has two different Terraform project, one with `path = "tfstate"` and the other with `path = "tfstate/"`, the paths for the locks will be the same which will be confusing as locking one project will lock both. I wish it were possible to forbid ending slashes altogether but doing so would require all users currently having an ending slash in the path to manually move their Terraform state and would be a poor user experience. Closes https://github.com/hashicorp/terraform/issues/15747	6 years ago
Rémi Lapeyre	11eb88753d	Fix `terraform force-unlock <lock_id>` for Consul backend ... When locking was enabled with the Consul backend and the lock not properly released, the `terraform force-unlock <lock_id>` command would do nothing as its implementation would exit early in that case. It now destroys the session that created the lock and clean both the lock and the lock-info keys. A regression test is added to TestConsul_destroyLock() to catch the issue if it happends again. Closes https://github.com/hashicorp/terraform/issues/22174	6 years ago
Kristin Laemmert	6621501ae3	state: remove deprecated state package (#25490) ... Most of the state package has been deprecated by the states package. This PR replaces all the references to the old state package that can be done simply - the low-hanging fruit. * states: move state.Locker to statemgr The state.Locker interface was a wrapper around a statemgr.Full, so moving this was relatively straightforward. * command: remove unnecessary use of state package for writing local terraform state files * move state.LocalState into terraform package state.LocalState is responsible for managing terraform.States, so it made sense (to me) to move it into the terraform package. * slight change of heart: move state.LocalState into clistate instead of terraform	6 years ago
Martin Atkins	aaf405b662	backend/remote-state: Get all the backend tests building again ... The state manager refactoring in an earlier commit was reflected in the implementations of these backends, but not in their tests. This gets us back to a state where the backend tests will compile, and gets _most_ of them passing again, with a few exceptions that will be addressed in a subsequent commit.	8 years ago
Martin Atkins	a3403f2766	terraform: Ugly huge change to weave in new State and Plan types ... Due to how often the state and plan types are referenced throughout Terraform, there isn't a great way to switch them out gradually. As a consequence, this huge commit gets us from the old world to a _compilable_ new world, but still has a large number of known test failures due to key functionality being stubbed out. The stubs here are for anything that interacts with providers, since we now need to do the follow-up work to similarly replace the old terraform.ResourceProvider interface with its replacement in the new "providers" package. That work, along with work to fix the remaining failing tests, will follow in subsequent commits. The aim here was to replace all references to terraform.State and its downstream types with states.State, terraform.Plan with plans.Plan, state.State with statemgr.State, and switch to the new implementations of the state and plan file formats. However, due to the number of times those types are used, this also ended up affecting numerous other parts of core such as terraform.Hook, the backend.Backend interface, and most of the CLI commands. Just as with 5861dbf3fc before, I apologize in advance to the person who inevitably just found this huge commit while spelunking through the commit history.	8 years ago
Martin Atkins	5782357c28	backend: Update interface and implementations for new config loader ... The new config loader requires some steps to happen in a different order, particularly in regard to knowing the schema in order to decode the configuration. Here we lean directly on the configschema package, rather than on helper/schema.Backend as before, because it's generally sufficient for our needs here and this prepares us for the helper/schema package later moving out into its own repository to seed a "plugin SDK".	8 years ago
James Bardin	12bda53558	Revert "create clistate.Locker interface" ... This reverts commit e88bd74bb7.	8 years ago
James Bardin	e88bd74bb7	create clistate.Locker interface ... Simplify the use of clistate.Lock by creating a clistate.Locker instance, which stores the context of locking a state, to allow unlock to be called without knowledge of how the state was locked. This alows the backend code to bring the needed UI methods to the point where the state is locked, and still unlock the state from an outer scope.	8 years ago
James Bardin	57cbcbe99d	update remote state tests to use new test helpers	8 years ago
James Bardin	cf54ca3b0f	update tests for new consul packages ... Reuse the running consul server for all tests. Update the lostLockConnection package, since the api client should no longer lose a lock immediately on network errors.	9 years ago
James Bardin	3a03d3683e	update consul api packages ... This is from a commit just after the v1.0.0 release, because it removes the Porter service dependency for tests. The client api package was not changed.	9 years ago
James Bardin	f5e9a20c66	reset testLockHook	9 years ago
James Bardin	25a8227291	add broken test for lock lost on connection error ... Add a way to inject network errors by setting an immediate deadline on open consul connections. The consul client currently doesn't retry on some errors, and will force us to lose our lock. Once the consul api client is fixed, this test will fail.	9 years ago
James Bardin	fd9adcdb36	only init one consul client, and lower keepalive ... The consul Client is analogous to an http.Client, and we really don't need more than 1. Configure a single client and store it in the backend. Replace the default Transport's Dialer to reduce the KeepAlive setting from 30s to 17s. This avoids racing with the common network timeout value of 30s, and is also coprime to other common intervals.	9 years ago
James Bardin	d0ecb232ae	record consul session ID in lock info ... This can help correlate TF and consul logs	9 years ago
Minkyu Kim	f12a237747	Fix crash when using consul backend	9 years ago
James Bardin	76eb65c911	Fix crash in consul backend ... A TLS config was being assigned to a Transport in a nil http.Client. The Transport is built in the consul config by default, but the http.Client is not built until later in NewClient.	9 years ago
Loïc Albertin	21b287e919	backend/consul: allow specifying various Consul client security options ... These new options are equivalent to the consul CLI arguments -ca-file, -client-cert and -client-key.	9 years ago
James Bardin	1070d04b07	Remove consul context adapter for RenewPeriodic ... Updated the vendored consul which no longer requires the channel adapter to convert a `chan stuct{}` to a `<-chan struct{}`. Call testutil.NewTestServerConfigT with the new signature.	9 years ago
James Bardin	bcb11f6d89	have the consul client manage the lock session ... When a consul lock is lost, there is a possibility that the associated session is still active. Most commonly, the long request to watch the lock key may error out, while the session is continually refreshed at a rate of TTL/2. First have the lock monitor retry the lock internally for at least 10 seconds (5 attempts with the default 2 second wait time). In most cases this will reconnect on the first try, keeping the lock channel open. If the consul lock can't recover itself, then cancel the session as soon as possible (terminating the PreiodicRenew will call Session.Destroy), and start over. In the worse case, the consul agents were split, and the session still exists on the leader so we may need to wait for the old session TTL, plus the LockWait time to renew the lock. We use a Context for the cancellation channels here, because that removes the need to worry about double-closes and nil channels. It requires an awkward adapter goroutine for now to convert the Done() `<-chan` to a `chan` for PeriodicRenew, but makes the rest of the code safer in the long run.	9 years ago
James Bardin	82eba5801d	Test losing and reacquiring a consul lock	9 years ago
James Bardin	3df48bfc27	relock consul when lock is lost ... Consul locks are based on liveness, and may be lost due timeouts, network issued, etc. If the client determines the lock was lost, attempt to reacquire the lock immediately. The client was also not using the `lock` config option. Disable locks if that is not set.	9 years ago
James Bardin	e8330b6f53	use CAS for consul state Put	9 years ago
James Bardin	5621d97925	cleanup consul lock entries ... This matches the consul cli behavior, where locks are cleaned up after use. Return an error from re-locking the state. This isn't required by the Locker interface, but it's an added sanity check for state operations. What was incorrect here was returning an empty ID and error, which would indicate that Lock/Unlock isn't supported.	9 years ago
James Bardin	75458a182d	remove extra state.Locker assertions ... All states are lockers, so get rid of extra asertions.	9 years ago
James Bardin	4980fa20e7	move s3 config from client to backend ... The RemoteClient needs to be configured for the named state, so move the general config to the backend. Rename some fields for consistency.	9 years ago
Mitchell Hashimoto	0cd15176ff	backend/consul: rebase and modify gzip test	9 years ago
Mitchell Hashimoto	1daff7a826	backend/consul: support "lock" option to disable locking ... This adds a "lock" config (default true) to allow users to optionally disable state locking with Consul. This is necessary if the token given doesn't have session permission and is necessary for backwards compatibility.	9 years ago
James Bardin	08d2b44ada	add gzip to consul backend tests	9 years ago
James Bardin	1527eae6b7	Merge pull request #8748 from christoe/christoe/8491-gzip-consul-remote-state ... state/remote: Add gzip support to consul remote state (#8491)	9 years ago
James Bardin	86f711f6fc	Make consul backend tests opt-in ... This way we don't require contributers to have consul installed to run make test.	9 years ago
James Bardin	fa7743b627	quiet the consul server during backend tests ... Don't display logs unless using `-v`	9 years ago
James Bardin	90055c6ae2	convert the consul backend to use consul/testutil ... Start up our own consul server for unit tests.	9 years ago
Christoffer Kylvåg	abfa35db7c	backend/remote-state/consul: Make gzip compression configurable (#8491)	9 years ago
Christoffer Kylvåg	e098c7c24a	backend/remote-state/consul: Add gzip support to consul backend client (#8491)	9 years ago
Mitchell Hashimoto	35afb7bc86	backend/consul: some comments	9 years ago
Mitchell Hashimoto	b842fd0c27	backend/consul: support named states	9 years ago
Mitchell Hashimoto	3db55cf747	backend/consul: build your own backend	9 years ago
Mitchell Hashimoto	549d525487	Merge pull request #12320 from hashicorp/b-legacy-state ... command: fix loading legacy remote state + migration guide	9 years ago
Mitchell Hashimoto	6ea0dc0964	backend/consul: make address have a default to prevent input	9 years ago
James Bardin	f866bb545c	update remote-state/consul	9 years ago
Mitchell Hashimoto	efe754183b	state/remote: export ClientLocker, test for implementation ... This adds unit tests (that will fail at compile time) if various structs don't implement the right interfaces for locking	9 years ago
James Bardin	ec00564be6	Clean up LockInfo and LockError and use them ... Gove LockInfo a Marshal method for easy serialization, and a String method for more readable output. Have the state.Locker implementations use LockError when possible to return LockInfo and an error.	9 years ago
James Bardin	67bbebce08	Have consul state reutrn the lock ID ... The lock ID isn't used because the lock is tied to the client, but return the lock ID to match the behavior of other locks.	9 years ago
James Bardin	cd233fef6a	make consul client pass state.Locker tests	9 years ago
James Bardin	80fab23e04	Don't test consul using demo.consul.io ... We shoudn't require an external service for unit test. TODO: create some proper acceptance tests for consul	9 years ago