43 Commits (f8fb61de12dea9ac94de3060a0d3e97ce9b5480b)

Author	SHA1	Message	Date
Daniel Banck	977a0fb117	Potentially fetch backend variables for most commands ... * modules * providers * providers lock * providers mirror * validate * state mv * state pull * state push * state replace provider * state rm * taint * get	3 months ago
Daniel Banck	4ef9684188	Rework most of the configuration loading ... We previously used a loader -> BuildConfig flow to load configuration. This commit changes most (but not all yet) flows to use the new graph-based approach. Instead of simply recursively loading the modules, we now need to take a stepped approach: 1. Load the root module 2. Collect the variables and their values 3. Build the configuration with the graph-based approach Because this approach relies on different parts from different packages, it can't easliy be done within the `configload` package. So, now we do most of in the backend or command.	4 months ago
Daniel Banck	fb60c15670	Update init to use new graph-based workflow ... The init command and the module installer are now using the new graph-based workflow to install modules instead of using the recursive BuildConfig.	4 months ago
Daniel Schmidt	d4d46b38a2	remove unused Meta.variableArgs	4 months ago
Daniel Schmidt	b003de9c1f	remove unused function	4 months ago
Daniel Schmidt	8faccde780	implement centralized variable collection	4 months ago
Daniel Schmidt	d09d5ea60f	remove now unused method	4 months ago
Radek Simko	0fe906fa8c	make copyrightfix	4 months ago
Sarah French	f591872699	PSS: Allow pluggable state store configuration to be stored in a plan file (#37956) ... * refactor: Rename Meta's backendState field to backendConfigState This helps with navigating ambiguity around the word backend. The new name should indicate that the value represents a `backend` block, not a more general interpretation of what a backend is. * fix: Only set backendConfigState to synthetic object if it's nil due to a lack of data. Don't change it if pluggable state storage is in use. * feat: Enable recording a state store's details in an Operation, and using that data when creating a plan file. * fix: Include provider config when writing a plan file using pluggable state storage * fix: Having `backendConfigState` be nil may be valid, but it definitely isn't valid for `stateStoreConfigState` to be nil When backendConfigState is nil it means that an implied local backend is in use, i.e. there is no backend block in the config. * test: Add integration test showing that a plan command creates a plan file with the expected state_store configuration data * refactor: Apply suggestion from @radeksimko Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com> * fix: Allow panics to occur if an unexpected implementation of `backend.Backend` is encountered when managing a state store * docs: Add code comment explaining the current situation with passing backend config state to downstream logic. In future this should be simplified, either via refactoring or changes affecting the implied local backend --------- Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>	6 months ago
Sarah French	b9522561c8	PSS: Add reusable method for obtaining the provider factory needed for accessing a state store. (#37665) ... * Add method to allow accessing factories from locks that are in memory * Create new getStateStoreProviderFactory method for accessing a factory from config * Update getStateStoreProviderFactory to use in memory locks instead of reading them from the deps lock file. Update calling code to accommodate this. * Add tests for getStateStoreProviderFactory, improve errors returned from method * Update test following schema change in simple providers * Move test case into e2etest package, so we protect against environments where building binaries isn't possible * Fix issues with running test in e2etest package * Update code comments Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com> --------- Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>	9 months ago
Samsondeen	719aefdb89	tf query: ready list blocks for beta (#37619)	9 months ago
Sarah French	b11b54e142	Add code comment to make in-progress migration from Ui to View more obvious (#37492)	10 months ago
Sarah French	2559f0a3db	Update backend state file so it can describe PSS state (#37179) ... * Split code for backend state file vs backend state * Rename BackendState to BackendConfigState * Spelling error * Add `StateStorageConfigState` struct as new implementation of new `ConfigState[T any]` interface. * Split tests for backend state file vs backend config state structs * Rename StateStorageConfigState to StateStoreConfigState * Clarify test name, add comments * Add tests for StateStoreConfigState methods * Add test showing state_store in JSON is parsed correctly * Add detection of malformed backend state files that contain both backend and state_store fields * Add validation that stops a backend state file being written if it will contain state for both backend and state_store blocks * Rename `state_storage` to `state_store` * Rename `state_storage` to `state_store` in filenames * Move`ConfigState` to its own file * Fix test name, remove whitespace * Update `StateStoreConfigState` comment using review suggestion * Update error message to no longer allude to the environment TF is being run in * Update the state_store state to use `version.Version` and an adapted version of `tfaddr.Provider` for marshalling version and source data * Update test helper so it doesn't accidentally supply validation in tests * Add protection against saving an empty backend state file * Remove direct testing of (s *Source) MarshalText() and UnmarshalText() methods * Add Validate method to StateStoreConfigState, use in backend state encoding logic * Refactor to use new features in registry dependency	1 year ago
Samsondeen	c0a7ff235b	Apply some new go collection features (#36818)	1 year ago
Sebastian Rivera	a72d02135b	Define StatePersistInterval as command meta field	2 years ago
UKEME BASSEY	c7bbc09631	move all command args from command/init to args/init	2 years ago
Martin Atkins	88cb4aeea3	backendrun: Separate the types/etc for backends that support operations ... We previously had all of the types and helpers for all kinds of backends together in package backend. That kept things relatively simple, but it also meant that the majority of backends that only deal with remote state storage ended up still indirectly depending on the entire Terraform modules runtime, configuration loader, etc, etc, which brings into scope a bunch of external dependencies that the remote state backends don't really need. Since backends that support operations are a rare exception, we'll move the types and helpers for those into a separate package "backendrun", and then the main package backend can have a much more modest set of types and, more importantly, a modest set of dependencies on other packages in this codebase. This is part of an ongoing effort to reduce the exposure of Terraform Core and CLI code to the remote backends and vice-versa, so that in the long run we can more often treat them as separate for dependency maintenance purposes.	2 years ago
Martin Atkins	06f51dbe04	command/workdir: Model the "backend state" file format ... Historically (before there were "backends") Terraform had a single state file format used both for real state snapshots and for tracking where remote state was stored. Terraform v0.12 caused these two to be split because we adopted state snapshot version 4 for real snapshots but retained a subset of version 3 for tracking the remote backend configuration in the local working directory. Unfortunately we previously kept that working by retaining a snapshot of the entire Terraform v0.11 "terraform" package as legacy/terraform, which happened to still be around because we also needed to retain a copy of the entire legacy SDK to keep the remote state backends working. This now hoists just the tiny slice of legacy terraform package functionality needed to implement the "backend state" file format into package workdir. This package is a good home for it because it's part of the working directory state. Ideally it would be accessed through methods of the workdir.Dir type, but that's too disruptive a refactor to combine into this and so that'll need to wait for another day; for now we'll keep the existing callers doing their access through our "clistate" package that is itself a forked snapshot of what statemgr.Filesystem used to be in Terraform v0.11. This removes all but one of the uses of "legacy/terraform" aside from calls in the other packages under "legacy". We'll clean up the last one in a later commit, because it's not related to the backend state file format.	2 years ago
Jamie Finnigan	05b0bdac99	adopt hashicorp/cli fork of mitchellh/cli (#34429)	3 years ago
hashicorp-copywrite[bot]	53c34ff49c	Update copyright file headers to BUSL-1.1	3 years ago
Martin Atkins	7bee77bdd3	command: Start of propagating OpenTelemetry context ... Several times over the years we've considered adding tracing instrumentation to Terraform, since even when running in isolation as a CLI program it has a "distributed system-like" structure, with lots of concurrent internal work and also some work delegated to provider plugins that are essentially temporarily-running microservices. However, it's always felt a bit overwhelming to do it because much of Terraform predates the Go context.Context idiom and so it's tough to get a clean chain of context.Context values all the way down the stack without disturbing a lot of existing APIs. This commit aims to just get that process started by establishing how a context can propagate from "package main" into the command package, focusing initially on "terraform init" and some other commands that share some underlying functions with that command. OpenTelemetry has emerged as a de-facto industry standard and so this uses its API directly, without any attempt to hide it behind an abstraction. The OpenTelemetry API is itself already an adapter layer, so we should be able to swap in any backend that uses comparable concepts. For now we just discard the tracing reports by default, and allow users to opt in to delivering traces over OTLP by setting an environment variable when running Terraform (the environment variable was established in an earlier commit, so this commit builds on that.) When tracing collection is enabled, every Terraform CLI run will generate at least one overall span representing the command that was run. Some commands might also create child spans, but most currently do not.	3 years ago
Liam Cervante	2793af042c	Revert "Plannable import: Make the streamed logs more consistent during a plan operation (#33155)" ... This reverts commit 3c20f7b340.	3 years ago
Liam Cervante	3c20f7b340	Plannable import: Make the streamed logs more consistent during a plan operation (#33155)	3 years ago
hashicorp-copywrite[bot]	325d18262e	[COMPLIANCE] Add Copyright and License Headers	3 years ago
Martin Atkins	e2380b1038	cliconfig: Allow forcing use of the plugin cache despite the lock file ... Currently Terraform will use an entry from the global plugin cache only if it matches a checksum already recorded in the dependency lock file. This allows Terraform to produce a complete lock file entry on the first encounter with a new provider, whereas using the cache in that case would cause the lock file to only cover the single package in the cache and thereefore be unusable on any other operating system or CPU architecture. This temporary CLI config option is a pragmatic exception to support those who cannot currently correctly use the dependency lock file but who still want to benefit from the plugin cache. With this setting enabled, Terraform has permission to produce a dependency lock file that is only suitable for the current system if that would allow use of an existing entry in the plugin cache. We are introducing this option to resolve a conflict between the needs of folks who are using the dependency lock file as expected and the needs of folks who cannot use the dependency lock file for some reason. The hope then is to give respite to those who need this exception in the meantime while we understand better why they cannot use the dependency lock file and improve its design so that everyone will be able to use it successfully in a future version of Terraform. This option will become a silent no-op in a future version of Terraform, once the dependency lock file behavior is sufficient for all supported Terraform development workflows.	3 years ago
Megan Bang	4d749e2813	add warning to diags and show at the end of each command	4 years ago
Megan Bang	de8bd5826f	first part of code review comments	4 years ago
Megan Bang	b504dd1489	update from code consistency checks	4 years ago
Megan Bang	b572e57fb3	refactor GetSchemas to include an option to pass in a config	4 years ago
Megan Bang	00cc1ea26d	refactor getSchemas	4 years ago
James Bardin	77e6b622f8	Merge pull request #31283 from hashicorp/jbardin/plan-import ... Use plan graph for importing resources	4 years ago
James Bardin	4981942191	remove -allow-missing-config from import	4 years ago
Martin Atkins	fda0579537	Experiments supported only in alpha/dev builds ... We originally introduced the idea of language experiments as a way to get early feedback on not-yet-proven feature ideas, ideally as part of the initial exploration of the solution space rather than only after a solution has become relatively clear. Unfortunately, our tradeoff of making them available in normal releases behind an explicit opt-in in order to make it easier to participate in the feedback process had the unintended side-effect of making it feel okay to use experiments in production and endure the warnings they generate. This in turn has made us reluctant to make use of the experiments feature lest experiments become de-facto production features which we then feel compelled to preserve even though we aren't yet ready to graduate them to stable features. In an attempt to tweak that compromise, here we make the availability of experiments _at all_ a build-time flag which will not be set by default, and therefore experiments will not be available in most release builds. The intent (not yet implemented in this PR) is for our release process to set this flag only when it knows it's building an alpha release or a development snapshot not destined for release at all, which will therefore allow us to still use the alpha releases as a vehicle for giving feedback participants access to a feature (without needing to install a Go toolchain) but will not encourage pretending that these features are production-ready before they graduate from experimental. Only language experiments have an explicit framework for dealing with them which outlives any particular experiment, so most of the changes here are to that generalized mechanism. However, the intent is that non-language experiments, such as experimental CLI commands, would also in future check Meta.AllowExperimentalFeatures and gate the use of those experiments too, so that we can be consistent that experimental features will never be available unless you explicitly choose to use an alpha release or a custom build from source code. Since there are already some experiments active at the time of this commit which were not previously subject to this restriction, we'll pragmatically leave those as exceptions that will remain generally available for now, and so this new approach will apply only to new experiments started in the future. Once those experiments have all concluded, we will be left with no more exceptions unless we explicitly choose to make an exception for some reason we've not imagined yet. It's important that we be able to write tests that rely on experiments either being available or not being available, so here we're using our typical approach of making "package main" deal with the global setting that applies to Terraform CLI executables while making the layers below all support fine-grain selection of this behavior so that tests with different needs can run concurrently without trampling on one another. As a compromise, the integration tests in the terraform package will run with experiments enabled _by default_ since we commonly need to exercise experiments in those tests, but they can selectively opt-out if they need to by overriding the loader setting back to false again.	4 years ago
gabriel376	f5a8608989	cli: Make state commands check required version (#30511)	4 years ago
Alisdair McDiarmid	39de3ebec7	cli: Fix init failure with deleted cache ... The init command needs to initialize a backend, in order to access state, in turn to derive provider requirements from state. The backend initialization step requires building provider factories, which previously would fail if a lockfile was present without a corresponding local provider cache. This commit ensures that in this situation only, errors with the provider factories are temporarily ignored. This allows us to continue to initialize the backend, fetch providers, and then report any errors as necessary.	5 years ago
Martin Atkins	d09510a8fb	command: Early error message for missing cache entries of locked providers ... In the original incarnation of Meta.providerFactories we were returning into a Meta.contextOpts whose signature didn't allow it to return an error directly, and so we had compromised by making the provider factory functions themselves return errors once called. Subsequent work made Meta.contextOpts need to return an error anyway, but at the time we neglected to update our handling of the providerFactories result, having it still defer the error handling until we finally instantiate a provider. Although that did ultimately get the expected result anyway, the error ended up being reported from deep in the guts of a Terraform Core graph walk, in whichever concurrently-visited graph node happened to try to instantiate the plugin first. This meant that the exact phrasing of the error message would vary between runs and the reporting codepath didn't have enough context to given an actionable suggestion on how to proceed. In this commit we make Meta.contextOpts pass through directly any error that Meta.providerFactories produces, and then make Meta.providerFactories produce a special error type so that Meta.Backend can ultimately return a user-friendly diagnostic message containing a specific suggestion to run "terraform init", along with a short explanation of what a provider plugin is. The reliance here on an implied contract between two functions that are not directly connected in the callstack is non-ideal, and so hopefully we'll revisit this further in future work on the overall architecture of the CLI layer. To try to make this robust in the meantime though, I wrote it to use the errors.As function to potentially unwrap a wrapped version of our special error type, in case one of the intervening layers is changed at some point to wrap the downstream error before returning it.	5 years ago
Martin Atkins	8d193ad268	core: Simplify and centralize plugin availability checks ... Historically the responsibility for making sure that all of the available providers are of suitable versions and match the appropriate checksums has been split rather inexplicably over multiple different layers, with some of the checks happening as late as creating a terraform.Context. We're gradually iterating towards making that all be handled in one place, but in this step we're just cleaning up some old remnants from the main "terraform" package, which is now no longer responsible for any version or checksum verification and instead just assumes it's been provided with suitable factory functions by its caller. We do still have a pre-check here to make sure that we at least have a factory function for each plugin the configuration seems to depend on, because if we don't do that up front then it ends up getting caught instead deep inside the Terraform runtime, often inside a concurrent graph walk and thus it's not deterministic which codepath will happen to catch it on a particular run. As of this commit, this actually does leave some holes in our checks: the command package is using the dependency lock file to make sure we have exactly the provider packages we expect (exact versions and checksums), which is the most crucial part, but we don't yet have any spot where we make sure that the lock file is consistent with the current configuration, and we are no longer preserving the provider checksums as part of a saved plan. Both of those will come in subsequent commits. While it's unusual to have a series of commits that briefly subtracts functionality and then adds back in equivalent functionality later, the lock file checking is the only part that's crucial for security reasons, with everything else mainly just being to give better feedback when folks seem to be using Terraform incorrectly. The other bits are therefore mostly cosmetic and okay to be absent briefly as we work towards a better design that is clearer about where that responsibility belongs.	5 years ago
Martin Atkins	65e0c448a0	workdir: Start of a new package for working directory state management ... Thus far our various interactions with the bits of state we keep associated with a working directory have all been implemented directly inside the "command" package -- often in the huge command.Meta type -- and not managed collectively via a single component. There's too many little codepaths reading and writing from the working directory and data directory to refactor it all in one step, but this is an attempt at a first step towards a future where everything that reads and writes from the current working directory would do so via an object that encapsulates the implementation details and offers a high-level API to read and write all of these session-persistent settings. The design here continues our gradual path towards using a dependency injection style where "package main" is solely responsible for directly interacting with the OS command line, the OS environment, the OS working directory, the stdio streams, and the CLI configuration, and then communicating the resulting information to the rest of Terraform by wiring together objects. It seems likely that eventually we'll have enough wiring code in package main to justify a more explicit organization of that code, but for this commit the new "workdir.Dir" object is just wired directly in place of its predecessors, without any significant change of code organization at that top layer. This first commit focuses on the main files and directories we use to find provider plugins, because a subsequent commit will lightly reorganize the separation of concerns for plugin launching with a similar goal of collecting all of the relevant logic together into one spot.	5 years ago
James Bardin	863963e7a6	de-linting	5 years ago
Martin Atkins	89b05050ec	core: Functional-style API for terraform.Context ... Previously terraform.Context was built in an unfortunate way where all of the data was provided up front in terraform.NewContext and then mutated directly by subsequent operations. That made the data flow hard to follow, commonly leading to bugs, and also meant that we were forced to take various actions too early in terraform.NewContext, rather than waiting until a more appropriate time during an operation. This (enormous) commit changes terraform.Context so that its fields are broadly just unchanging data about the execution context (current workspace name, available plugins, etc) whereas the main data Terraform works with arrives via individual method arguments and is returned in return values. Specifically, this means that terraform.Context no longer "has-a" config, state, and "planned changes", instead holding on to those only temporarily during an operation. The caller is responsible for propagating the outcome of one step into the next step so that the data flow between operations is actually visible. However, since that's a change to the main entry points in the "terraform" package, this commit also touches every file in the codebase which interacted with those APIs. Most of the noise here is in updating tests to take the same actions using the new API style, but this also affects the main-code callers in the backends and in the command package. My goal here was to refactor without changing observable behavior, but in practice there are a couple externally-visible behavior variations here that seemed okay in service of the broader goal: - The "terraform graph" command is no longer hooked directly into the core graph builders, because that's no longer part of the public API. However, I did include a couple new Context functions whose contract is to produce a UI-oriented graph, and _for now_ those continue to return the physical graph we use for those operations. There's no exported API for generating the "validate" and "eval" graphs, because neither is particularly interesting in its own right, and so "terraform graph" no longer supports those graph types. - terraform.NewContext no longer has the responsibility for collecting all of the provider schemas up front. Instead, we wait until we need them. However, that means that some of our error messages now have a slightly different shape due to unwinding through a differently-shaped call stack. As of this commit we also end up reloading the schemas multiple times in some cases, which is functionally acceptable but likely represents a performance regression. I intend to rework this to use caching, but I'm saving that for a later commit because this one is big enough already. The proximal reason for this change is to resolve the chicken/egg problem whereby there was previously no single point where we could apply "moved" statements to the previous run state before creating a plan. With this change in place, we can now do that as part of Context.Plan, prior to forking the input state into the three separate state artifacts we use during planning. However, this is at least the third project in a row where the previous API design led to piling more functionality into terraform.NewContext and then working around the incorrect order of operations that produces, so I intend that by paying the cost/risk of this large diff now we can in turn reduce the cost/risk of future projects that relate to our main workflow actions.	5 years ago
Martin Atkins	36d0a50427	Move terraform/ to internal/terraform/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	31349a9c3a	Move configs/ to internal/configs/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	ffe056bacb	Move command/ to internal/command/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago