74 Commits (5bbd10857250a2c9bf5559ebd9bdc8aa2f144bc1)

Author	SHA1	Message	Date
Chris Arcand	779c958fbf	cloud: Add streamlined 'remote' backend state migration path ... For Terraform Cloud users using the 'remote' backend, the existing 'pattern' prompt should work just fine - but because their workspaces are already present in TFC, the 'migration' here is really just realigning their local workspaces with Terraform Cloud. Instead of forcing users to do the mental gymnastics of what it means to migrate from 'prefix' - and because their remote workspaces probably already exist and already conform to Terraform Cloud's naming concerns - streamline the process for them and calculate the necessary pattern to migrate as-is, without any user intervention necessary.	5 years ago
xiaozhu36	e0a2716601	backend/oss: Removes the ConflictWith tag which on the attributes assume_role* to fix the incompatible error	5 years ago
xiaozhu36	5d19a34f88	backend/oss: Supports the new attribute sts_endpoint	5 years ago
James Bardin	2ebdc099ac	Merge pull request #29307 from hayorov/main ... Flatten `assume_role` block for OSS backend	5 years ago
James Bardin	81e709d185	Merge pull request #29825 from hashicorp/jbardin/no-panicwrap ... Remove the use of panicwrap in the Terraform CLI	5 years ago
Brandon Croft	dbbfae5a1c	refactor ParseVariableValues into separate operations ... 1. ParseDeclaredValues: parses unparsed variables into terraform.InputValues 2. ProbeUndeclaredVariableValues: compares variable declarations with unparsed values to warn/error about undeclared variables	5 years ago
Chris Arcand	1791b71196	cloud: TestCloud_backendWithTags ... Implementing this test was quite a rabbithole, as in order to satisfy backendTestBackendStates() the workspaces returned from backend.Workspaces() must match exactly, and the shortcut taken to test pagination in 3cc58813f0 created an impossible circumstance that got plastered over with the fact that prefix filtering is done clientside, not by the API as it should be. Tagging does not rely on clientside filtering, and expects that the request made to the TFC API returns exactly those workspaces with the given tags. These changes include a better way to test pagination, wherein we actually create over a page worth of valid workspaces in the mock client and implement a simplified pagination behavior to match how the TFC API actually works.	5 years ago
Chris Arcand	471dd479e8	Update go-tfe to 26689e ... These changes include additions to fulfill the interface for the client mock, plus moving all that logic (which needn't be duplicated across both the remote and cloud packages) over to the cloud package under a dedicated mock client file.	5 years ago
Chris Arcand	ea8ad0b15a	Initial commit of 'cloud' package ... The cloud package intends to implement a new integration for Terraform Cloud/Enterprise. The purpose of this integration is to better support TFC users; it will shed some overly generic UX and architecture, behavior changes that are otherwise backwards incompatible in the remote backend, and technical debt - all of which are vestiges from before Terraform Cloud existed. This initial commit is largely a porting of the existing 'remote' backend, which will serve as an underlying implementation detail and not be a typical user-level backend. This is because to re-implement the literal backend interface is orthogonal to the purpose of this integration, and can always be migrated away from later. As this backend is considered an implementation detail, it will not be registered as a declarable backend. Within these changes it is, for easy of initial development and a clean diff.	5 years ago
James Bardin	d03a037567	insert panic handlers	5 years ago
Alex Khaerov	6f1eceb0e2	Revert autoupdate	5 years ago
xiaozhu36	7afaea4cf2	backend/oss: Fixes the nil pointer panic error when missing access key or secret key	5 years ago
Alex Khaerov	14f366dbf4	Update documentation	5 years ago
Alex Khaerov	f32702c5c2	Support deprecated assume_role block	5 years ago
Alex Khaerov	de8810cdd9	Merge branch 'hashicorp:main' into main	5 years ago
Martin Atkins	39779e7021	backend/remote-state/cos: Don't use github.com/likexian/gokit ... We don't use this library anywhere else in Terraform, and this backend was using it only for trivial helpers that are easy to express inline anyway. The new direct code is also type-checkable, whereas these helper functions seem to be written using reflection. This gives us one fewer dependency to worry about and makes the test code for this backend follow a similar assertions style as the rest of this codebase.	5 years ago
James Bardin	ef3c98466d	Merge pull request #29755 from hashicorp/jbardin/first-plan-lineage ... Check for stale plan with no state metadata	5 years ago
James Bardin	5ffa0839f9	only check serial when applying the first plan ... Ensure that we still check for a stale plan even when it was created with no previous state. Create separate errors for incorrect lineage vs incorrect serial. To prevent confusion when applying a first plan multiple times, only report it as a stale plan rather than different lineage.	5 years ago
Martin Atkins	bee7403f3e	command/workspace_delete: Allow deleting a workspace with empty husks ... Previously we would reject attempts to delete a workspace if its state contained any resources at all, even if none of the resources had any resource instance objects associated with it. Nowadays there isn't any situation where the normal Terraform workflow will leave behind resource husks, and so this isn't as problematic as it might've been in the v0.12 era, but nonetheless what we actually care about for this check is whether there might be any remote objects that this state is tracking, and for that it's more precise to look for non-nil resource instance objects, rather than whole resources. This also includes some adjustments to our error messaging to give more information about the problem and to use terminology more consistent with how we currently talk about this situation in our documentation and elsewhere in the UI. We were also using the old State.HasResources method as part of some of our tests. I considered preserving it to avoid changing the behavior of those tests, but the new check seemed close enough to the intent of those tests that it wasn't worth maintaining this method that wouldn't be used in any main code anymore. I've therefore updated those tests to use the new HasResourceInstanceObjects method instead.	5 years ago
Megan Bang	9b9b26a3cd	update error message for invalid json	5 years ago
megan07	7dda3366a6	Update internal/backend/remote-state/gcs/backend.go ... Co-authored-by: appilon <apilon@hashicorp.com>	5 years ago
Megan Bang	81201d69a3	check valid json	5 years ago
Megan Bang	348c761bea	add better error if credentials are invalid	5 years ago
Martin Atkins	df578afd7e	backend/local: Check dependency lock consistency before any operations ... In historical versions of Terraform the responsibility to check this was inside the terraform.NewContext function, along with various other assorted concerns that made that function particularly complicated. More recently, we reduced the responsibility of the "terraform" package only to instantiating particular named plugins, assuming that its caller is responsible for selecting appropriate versions of any providers that _are_ external. However, until this commit we were just assuming that "terraform init" had correctly selected appropriate plugins and recorded them in the lock file, and so nothing was dealing with the problem of ensuring that there haven't been any changes to the lock file or config since the most recent "terraform init" which would cause us to need to re-evaluate those decisions. Part of the game here is to slightly extend the role of the dependency locks object to also carry information about a subset of provider addresses whose lock entries we're intentionally disregarding as part of the various little edge-case features we have for overridding providers: dev_overrides, "unmanaged providers", and the testing overrides in our own unit tests. This is an in-memory-only annotation, never included in the serialized plan files on disk. I had originally intended to create a new package to encapsulate all of this plugin-selection logic, including both the version constraint checking here and also the handling of the provider factory functions, but as an interim step I've just made version constraint consistency checks the responsibility of the backend/local package, which means that we'll always catch problems as part of preparing for local operations, while not imposing these additional checks on commands that _don't_ run local operations, such as "terraform apply" when in remote operations mode.	5 years ago
Martin Atkins	6a98e4720c	plans/planfile: Create takes most arguments via a struct type ... Previously the planfile.Create function had accumulated probably already too many positional arguments, and I'm intending to add another one in a subsequent commit and so this is preparation to make the callsites more readable (subjectively) and make it clearer how we can extend this function's arguments to include further components in a plan file. There's no difference in observable functionality here. This is just passing the same set of arguments in a slightly different way.	5 years ago
Martin Atkins	8d193ad268	core: Simplify and centralize plugin availability checks ... Historically the responsibility for making sure that all of the available providers are of suitable versions and match the appropriate checksums has been split rather inexplicably over multiple different layers, with some of the checks happening as late as creating a terraform.Context. We're gradually iterating towards making that all be handled in one place, but in this step we're just cleaning up some old remnants from the main "terraform" package, which is now no longer responsible for any version or checksum verification and instead just assumes it's been provided with suitable factory functions by its caller. We do still have a pre-check here to make sure that we at least have a factory function for each plugin the configuration seems to depend on, because if we don't do that up front then it ends up getting caught instead deep inside the Terraform runtime, often inside a concurrent graph walk and thus it's not deterministic which codepath will happen to catch it on a particular run. As of this commit, this actually does leave some holes in our checks: the command package is using the dependency lock file to make sure we have exactly the provider packages we expect (exact versions and checksums), which is the most crucial part, but we don't yet have any spot where we make sure that the lock file is consistent with the current configuration, and we are no longer preserving the provider checksums as part of a saved plan. Both of those will come in subsequent commits. While it's unusual to have a series of commits that briefly subtracts functionality and then adds back in equivalent functionality later, the lock file checking is the only part that's crucial for security reasons, with everything else mainly just being to give better feedback when folks seem to be using Terraform incorrectly. The other bits are therefore mostly cosmetic and okay to be absent briefly as we work towards a better design that is clearer about where that responsibility belongs.	5 years ago
James Bardin	8ce2254ad6	Merge pull request #29647 from hashicorp/jbardin/test-temp-cleanup ... temp path clean for some backend tests	5 years ago
Alisdair McDiarmid	57318ef561	backend/remote: Support interop from 0.14 to 1.1 ... The previous conservative guarantee that we would not make backwards incompatible changes to the state file format until at least Terraform 1.1 can now be extended. Terraform 0.14 through 1.1 will be able to interoperably use state files, so we can update the remote backend version compatibility check accordingly.	5 years ago
James Bardin	9c078c27cf	temp path clean for some backend tests	5 years ago
Martin Atkins	f0034beb33	core: refactoring.ImpliedMoveStatements replaces NodeCountBoundary ... Going back a long time we've had a special magic behavior which tries to recognize a situation where a module author either added or removed the "count" argument from a resource that already has instances, and to silently rename the zeroth or no-key instance so that we don't plan to destroy and recreate the associated object. Now we have a more general idea of "move statements", and specifically the idea of "implied" move statements which replicates the same heuristic we used to use for this behavior, we can treat this magic renaming rule as just another "move statement", special only in that Terraform generates it automatically rather than it being written out explicitly in the configuration. In return for wiring that in, we can now remove altogether the NodeCountBoundary graph node type and its associated graph transformer, CountBoundaryTransformer. We handle moves as a preprocessing step before building the plan graph, so we no longer need to include any special nodes in the graph to deal with that situation. The test updates here are mainly for the graph builders themselves, to acknowledge that indeed we're no longer inserting the NodeCountBoundary vertices. The vertices that NodeCountBoundary previously depended on now become dependencies of the special "root" vertex, although in many cases here we don't see that explicitly because of the transitive reduction algorithm, which notices when there's already an equivalent indirect dependency chain and removes the redundant edge. We already have plenty of test coverage for these "count boundary" cases in the context tests whose names start with TestContext2Plan_count and TestContext2Apply_resourceCount, all of which continued to pass here without any modification and so are not visible in the diff. The test functions particularly relevant to this situation are: - TestContext2Plan_countIncreaseFromNotSet - TestContext2Plan_countDecreaseToOne - TestContext2Plan_countOneIndex - TestContext2Apply_countDecreaseToOneCorrupted The last of those in particular deals with the situation where we have both a no-key instance _and_ a zero-key instance in the prior state, which is interesting here because to exercises an intentional interaction between refactoring.ImpliedMoveStatements and refactoring.ApplyMoves, where we intentionally generate an implied move statement that produces a collision and then expect ApplyMoves to deal with it in the same way as it would deal with all other collisions, and thus ensure we handle both the explicit and implied collisions in the same way. This does affect some UI-level tests, because a nice side-effect of this new treatment of this old feature is that we can now report explicitly in the UI that we're assigning new addresses to these objects, whereas before we just said nothing and hoped the user would just guess what had happened and why they therefore weren't seeing a diff. The backend/local plan tests actually had a pre-existing bug where they were using a state with a different instance key than the config called for but getting away with it because we'd previously silently fix it up. That's still fixed up, but now done with an explicit mention in the UI and so I made the state consistent with the configuration here so that the tests would be able to recognize _real_ differences where present, as opposed to the errant difference caused by that inconsistency.	5 years ago
Martin Atkins	332ea1f233	backend/local: TestLocal_plan_context_error to fail terraform.NewContext ... The original intent of this test was to verify that we properly release the state lock if terraform.NewContext fails. This was in response to a bug in an earlier version of Terraform where that wasn't true. In the recent refactoring that made terraform.NewContext no longer responsible for provider constraint/checksum verification, this test began testing a failed plan operation instead, which left the error return path from terraform.NewContext untested. An invalid parallelism value is the one remaining case where terraform.NewContext can return an error, so as a localized fix for this test I've switched it to just intentionally set an invalid parallelism value. This is still not ideal because it's still testing an implementation detail, but I've at least left a comment inline to try to be clearer about what the goal is here so that we can respond in a more appropriate way if future changes cause this test to fail again. In the long run I'd like to move this last remaining check out to be the responsibility of the CLI layer, with terraform.NewContext either just assuming the value correct or panicking when it isn't, but the handling of this CLI option is currently rather awkwardly spread across the command and backend packages so we'll save that refactoring for a later date.	5 years ago
Martin Atkins	718fa3895f	backend: Remove Operation.Parallelism field ... The presence of this field was confusing because in practice the local backend doesn't use it for anything and the remote backend was using it only to return an error if it's set to anything other than the default, under the assumption that it would always match ContextOpts.Parallelism. The "command" package is the one actually responsible for handling this option, and it does so by placing it into the partial ContextOpts which it passes into the backend when preparing for a local operation. To make that clearer, here we remove Operation.Parallelism and change the few uses of it to refer to ContextOpts.Parallelism instead, so that everyone is reading and writing this value from the same place.	5 years ago
James Bardin	d0993b0e80	fix temp directory handling in some tests ... Cleanup some more test fixtures to use t.TempDir Use EvalSymlinks with temp dir paths to help with MacOS errors from various terraform components.	5 years ago
James Bardin	8ed9a270e5	Merge pull request #29559 from hashicorp/jbardin/optional-attrs ... Prevent `ObjectWithOptionalAttrs` from escaping	5 years ago
James Bardin	53a73a8ab6	configs: add ConstraintType to config.Variable ... In order to handle optional attributes, the Variable type needs to keep track of the type constraint for decoding and conversion, as well as the concrete type for creating values and type comparison. Since the Type field is referenced throughout the codebase, and for future refactoring if the handling of optional attributes changes significantly, the constraint is now loaded into an entirely new field called ConstraintType. This prevents types containing ObjectWithOptionalAttrs from escaping the decode/conversion codepaths into the rest of the codebase.	5 years ago
Martin Atkins	343279110a	core: Graph walk loads plugin schemas opportunistically ... Previously our graph walker expected to recieve a data structure containing schemas for all of the provider and provisioner plugins used in the configuration and state. That made sense back when terraform.NewContext was responsible for loading all of the schemas before taking any other action, but it no longer has that responsiblity. Instead, we'll now make sure that the "contextPlugins" object reaches all of the locations where we need schema -- many of which already had access to that object anyway -- and then load the needed schemas just in time. The contextPlugins object memoizes schema lookups, so we can safely call it many times with the same provider address or provisioner type name and know that it'll still only load each distinct plugin once per Context object. As of this commit, the Context.Schemas method is now a public interface only and not used by logic in the "terraform" package at all. However, that does leave us in a rather tenuous situation of relying on the fact that all practical users of terraform.Context end up calling "Schemas" at some point in order to verify that we have all of the expected versions of plugins. That's a non-obvious implicit dependency, and so in subsequent commits we'll gradually move all responsibility for verifying plugin versions into the caller of terraform.NewContext, which'll heal a long-standing architectural wart whereby the caller is responsible for installing and locating the plugin executables but not for verifying that what's installed is conforming to the current configuration and dependency lock file.	5 years ago
James Bardin	863963e7a6	de-linting	5 years ago
Martin Atkins	89b05050ec	core: Functional-style API for terraform.Context ... Previously terraform.Context was built in an unfortunate way where all of the data was provided up front in terraform.NewContext and then mutated directly by subsequent operations. That made the data flow hard to follow, commonly leading to bugs, and also meant that we were forced to take various actions too early in terraform.NewContext, rather than waiting until a more appropriate time during an operation. This (enormous) commit changes terraform.Context so that its fields are broadly just unchanging data about the execution context (current workspace name, available plugins, etc) whereas the main data Terraform works with arrives via individual method arguments and is returned in return values. Specifically, this means that terraform.Context no longer "has-a" config, state, and "planned changes", instead holding on to those only temporarily during an operation. The caller is responsible for propagating the outcome of one step into the next step so that the data flow between operations is actually visible. However, since that's a change to the main entry points in the "terraform" package, this commit also touches every file in the codebase which interacted with those APIs. Most of the noise here is in updating tests to take the same actions using the new API style, but this also affects the main-code callers in the backends and in the command package. My goal here was to refactor without changing observable behavior, but in practice there are a couple externally-visible behavior variations here that seemed okay in service of the broader goal: - The "terraform graph" command is no longer hooked directly into the core graph builders, because that's no longer part of the public API. However, I did include a couple new Context functions whose contract is to produce a UI-oriented graph, and _for now_ those continue to return the physical graph we use for those operations. There's no exported API for generating the "validate" and "eval" graphs, because neither is particularly interesting in its own right, and so "terraform graph" no longer supports those graph types. - terraform.NewContext no longer has the responsibility for collecting all of the provider schemas up front. Instead, we wait until we need them. However, that means that some of our error messages now have a slightly different shape due to unwinding through a differently-shaped call stack. As of this commit we also end up reloading the schemas multiple times in some cases, which is functionally acceptable but likely represents a performance regression. I intend to rework this to use caching, but I'm saving that for a later commit because this one is big enough already. The proximal reason for this change is to resolve the chicken/egg problem whereby there was previously no single point where we could apply "moved" statements to the previous run state before creating a plan. With this change in place, we can now do that as part of Context.Plan, prior to forking the input state into the three separate state artifacts we use during planning. However, this is at least the third project in a row where the previous API design led to piling more functionality into terraform.NewContext and then working around the incorrect order of operations that produces, so I intend that by paying the cost/risk of this large diff now we can in turn reduce the cost/risk of future projects that relate to our main workflow actions.	5 years ago
Alex Khaerov	9d5f1752c8	oss backend: flattern assume_role block	5 years ago
James Bardin	97a2694528	Merge pull request #28838 from remilapeyre/consul-size-limit ... Fix handling large states in the Consul backend	5 years ago
Rémi Lapeyre	da6717761e	Merge remote-tracking branch 'origin/main' into update-consul	5 years ago
James Bardin	570b70b02f	Merge pull request #28078 from jasons42/configure-etcdv3-client-max-request-size ... Expose etcd client MaxCallSendMsgSize config	5 years ago
James Bardin	8dd722ece0	Merge pull request #29167 from xiaozhu36/xiaozhu ... backend/oss: Changes the DescribeEndpoint to DescribeEndpoints to fixes the unsupported sts bug	5 years ago
James Bardin	431aa0280e	Merge pull request #29157 from remilapeyre/unique-constraint ... Add uniqueness constraint on workspaces name for the pg backend	5 years ago
Jason Smith	d1608d7a7f	Expose etcd client MaxCallSendMsgSize config ... The etcdv3 client has a default request send limit of 2.0 MiB. This change exposes the configuration option to increase that limit enabling larger state using the etcdv3 backend. This also requires that the corresponding --max-request-bytes flag be increased on the server side. The default there is 1.5 MiB. Fixes https://github.com/hashicorp/terraform/issues/25745	5 years ago
James Bardin	dd330e5194	Merge pull request #29200 from hashicorp/jbardin/rebase-25554 ... manual rebase of #25554	5 years ago
Kevin Burke	c047958b57	go.mod,backend: update coreos/etcd dependency to release-3.4 branch ... etcd rewrote its import path from coreos/etcd to go.etcd.io/etcd. Changed the imports path in this commit, which also updates the code version. This lets us remove the github.com/ugorji/go/codec dependency, which was pinned to a fairly old version. The net change is a loss of 30,000 lines of code in the vendor directory. (I first noticed this problem because the outdated go/codec dependency was causing a dependency failure when I tried to put Terraform and another project in the same vendor directory.) Note the version shows up funkily in go.mod, but I verified visually it's the same commit as the "release-3.4" tag in github.com/coreos/etcd. The etcd team plans to fix the release version tagging in v3.5, which should be released soon.	5 years ago
James Bardin	9c20ed6185	StateMgr must be able to return with locked state ... The current usage of internal remote state backends requires that `StateMgr` be able to return an instance of `statemgr.Full` even if the state is currently locked.	5 years ago
xiaozhu36	c495caafeb	backend/oss: Changes the DescribeEndpoint to DescribeEndpoints to fixes the unsupported sts bug	5 years ago
Rémi Lapeyre	3c34e15d40	Update the Consul API client ... Some users would want to use Consul namespaces when using the Consul backend but the version of the Consul API client we use is too old and don't support them. In preparation for this change this patch just update it the client and replace testutil.NewTestServerConfig() by testutil.NewTestServerConfigT() in the tests.	5 years ago