148 Commits (72de171edde8564cf9e37ca703fe1b4e715be5ec)

Author	SHA1	Message	Date
Martin Atkins	72de171edd	configs: Include context when variable default has nested problem ... Previously this error message was including only the main error message and ignoring any context about what part of a potential nested data structure it arose from. Now we'll use our usual path formatting utility to introduce additional context whenever a problem arises with a nested value.	2 years ago
Liam Cervante	8e47b5907c	stacks: provide more helpful diagnostics when providers types are mismatched (#35422) ... * stacks: provide more helpful diagnostics when providers types are mismatched * remove left over implementation experiments * remove unnecessary fmt.Sprintf * address comments * fix incorrect pluralisation * fix tests	2 years ago
Martin Atkins	6fad32b033	configs: Variables and outputs can be "ephemeral" ... When the ephemeral_values experiment is active, a module author can designate individual input variables and output values as being "ephemeral", which doesn't currently do anything but in future commits will represent that the values can be used only in locations that don't require Terraform to persist the value as part of state snapshots or saved plan files.	2 years ago
James Bardin	acc4c12ac2	variable validation rules must still point to self ... While variable validation rules can now include other references, they also still need to reference the variable itself. If the variable itself wasn't referenced, then the resulting `Invalid value for variable` error would not make sense. Having no self-references will also cause a panic when trying to reinsert the variable name into the evaluation context, because the context variables map could be nil. A check for a nil map does not need to be added however, because ensuring that a self-reference exists means there will always be at least 1 variable in scope.	2 years ago
Liam Cervante	b5d6b13b3b	stacks: convert sourcebundle source into directory for config SourceDir field (#35245)	2 years ago
James Bardin	c1c5e875ed	Merge pull request #35230 from hashicorp/jbardin/removed-provisioners ... Support for provisioners in `removed` blocks	2 years ago
Martin Atkins	3b620bfe69	terraform: Stabilize the variable_validation_crossref experiment ... Previously we introduced a language experiment that would permit variable validation rules to refer to other objects declared in the same module as the variable. Now that experiment is concluded and its behavior is available for all modules. This final version deviates slightly from the experiment: we learned from the experimental implementation that we accidentally made the "validate" command able to validate constant-valued input variables in child modules despite the usual rule that input variables are unknown during validation, because the previous compromise bypassed the main expression evaluator and built its own evaluation context directly. Even though that behavior was not intended, it's a useful behavior that is protected by our compatibility promises and so this commit includes a slightly hacky emulation of that behavior, in eval_variable.go, that fetches the variable value in the same way the old implementation would have and then modifies the hcl evaluation context to include that value, while preserving anything else that our standard evaluation context builder put in there. That narrowly preserves the old behavior for expressions that compare the variable value directly to a constant, while treating all other references (which were previously totally invalid) in the standard way. This quirk was already covered by the existing test TestContext2Validate_variableCustomValidationsFail, which fails if the special workaround is removed.	2 years ago
Martin Atkins	5e545ff427	configs: Provisioners in "removed" blocks ... When the removed_provisioners experiment is active, removed blocks referring to managed resources are allowed to include "connection" and "provisioner" blocks, as long as all of the "provisioner" blocks specify when = destroy to indicate that they should execute as part of the resource's "destroy" action. This commit only deals with parsing the configuration. The logic to react to this during the apply phase will follow in later commits.	2 years ago
Liam Cervante	af4a734262	import: only filter id attribute at root level when generating configuration (#35220)	2 years ago
Liam Cervante	48fc2d3ac7	terraform test: Disallow version constraints within test files (#35161)	2 years ago
Martin Atkins	1df01fdd2b	backend/local: Allow experiments when loading config from snapshot ... If a saved plan is being applied using a Terraform CLI/Core build that allows use of language experiments then that should be allowed when the configuration is being loaded from the snapshot saved in a plan file, in the same way as it would've been handled when preparing to generate that plan file. The local backend doesn't get told directly whether it should allow experiments, and instead its operation comes with a pre-configured configuration loader that may have the experiment flag set internally. When we load from snapshot we can't use the operation's own config loader because that loads from the real filesystem, and so we'll copy over the experiments-allowed flag from the operation's config loader to achieve consistency while still loading the configuration from the snapshot in the plan file. Without this a plan successfully created with experiments would fail with confusing errors during apply, because the snapshot config loader would not accept the same experiments that the plan-time config loader did.	2 years ago
Sebastian Rivera	015f795ff0	Rebrand Terraform Cloud to HCP Terraform	2 years ago
Martin Atkins	30e2fd6525	Handle marks a little more consistently ... In the very first implementation of "sensitive values" we were unfortunately not disciplined about separating the idea of "marked value" from the idea of "sensitive value" (where the latter is a subset of the former). The first implementation just assumed that any marking whatsoever meant "sensitive". We later improved that by adding the marks package and the marks.Sensitive value to standardize on the representation of "sensitive value" as being a value marked with _that specific mark_. However, we did not perform a thorough review of all of the mark-handling codepaths to make sure they all agreed on that definition. In particular, the state and plan models were both designed as if they supported arbitrary marks but then in practice marks other than marks.Sensitive would be handled in various inconsistent ways: dropped entirely, or interpreted as if marks.Sensitive, and possibly do so inconsistently when a value is used only in memory vs. round-tripped through a wire/file format. The goal of this commit is to resolve those oddities so that there are now two possible situations: - General mark handling: some codepaths genuinely handle marks generically, by transporting them from input value to output value in a way consistent with how cty itself deals with marks. This is the ideal case because it means we can add new marks in future and assume these codepaths will handle them correctly without any further modifications. - Sensitive-only mark preservation: the codepaths that interact with our wire protocols and file formats typically have only specialized support for sensitive values in particular, and lack support for any other marks. Those codepaths are now subject to a new rule where they must return an error if asked to deal with any other mark, so that if we introduce new marks in future we'll be forced either to define how we'll avoid those markings reaching the file/wire formats or extend the file/wire formats to support the new marks. Some new helper functions in package marks are intended to standardize how we deal with the "sensitive values only" situations, in the hope that this will make it easier to keep things consistent as the codebase evolves in future. In practice the modules runtime only ever uses marks.Sensitive as a mark today, so all of these checks are effectively covering "should never happen" cases. The only other mark Terraform uses is an implementation detail of "terraform console" and does not interact with any of the codepaths that only support sensitive values in particular.	2 years ago
Martin Atkins	79479f52c9	terraform: Experimental support for variable validation cross-references ... When a module opts in to the variable_validation_crossref experiment, Terraform will treat variable validation rules in a similar way as any other dynamic expression, allowing it to refer to other objects that are declared in the same module as the variable. Because the check for a validation rule referring only to itself is now experiments-sensitive we need to do it only once the full module has been assembled, rather than during assembly of individual files, and so the "badref" test in package configs is now classified under "invalid-modules" rather than "invalid-files". It's otherwise unchanged, and the existing test that a validation rule must refer to at least _something_ is still present because that remains true even when opted in to the experiment. This is experimental for now because although it seems pretty plausible that it should be okay, this is allowing full evaluation in a new place where it wasn't allowed before and so we'd like to ask those interested in this feature to try it out with some of their realistic use-cases and make sure this solution is a good fit before we lock it in as a compatibility constraint.	2 years ago
Martin Atkins	84df5562f2	moduleaddrs: Terraform now owns module source address normalization ... We've been gradually chipping away at how much we use go-getter for source packages, because it's generally been a bit of a nightmare and sharing it with other codebases means that any time someone wants to change something we end up needing to find some way to prevent it breaking Terraform's compatibility promises. Here we make one further step: Terraform owns the "detectors" idea that deals with source address normalization, and now always produces fully-qualified addresses for go-getter to chew on only for the getting and decompressing steps. Retaining go-getter for the actual getting part is helpful because we can then benefit from security fixes upstream, but Terraform owning the first layer of parsing means that we can fix in place the definition of what "module source address" syntax means, and thus we can avoid having everything in this codebase indirectly depend on go-getter just because it wants to parse module source addresses. Now only the module installer actually depends indirectly on go-getter, which finally disconnects go-getter's subtree from all of the remote state backend dependency graphs.	2 years ago
Martin Atkins	621befc3a2	configtesting: Separate package for the SynthBody helper ... This "SynthBody" helper doesn't really have anything to do with package configs except that it's sometimes useful for creating fake instances of the types in there for testing. Moving it into its own package means that package backend no longer needs to depend on package configs, since it doesn't actually have any need for parsing module configurations. The old configs.SynthBody function is still available as a forwarding alias, because most of the existing callers of it are from packages that already need to import package configs for other reasons, and so this reduces the need to churn loads of existing tests.	2 years ago
Martin Atkins	5d87e32eb9	moduleaddrs: Split module source address parsing from package addrs ... Our package addrs ends up getting imported from just about ever other package in Terraform, because it contains the types we use to talk about various different kinds of objects. Therefore we typically try to keep its transitive dependency graph relatively small, because anything it depends on becomes an indirect dependency of nearly everything else. A while back we moved the module source address models into package addrs, which also brought with them the code for parsing strings to produce those addresses. Unfortunately, remote module source addresses are defined using the external dependency go-getter, which is pretty heavy itself and also brings with it numerous other external dependencies such as the AWS SDK, the Google Cloud Platform SDK, etc. Since only relatively few packages actually need to _parse_ source addresses -- with most either not caring about them at all or only consuming addresses that were already parsed by someone else -- we'll move the parser functions into their own package, while keeping the resulting address types in package addrs. This does still retain the package addrs dependency on external module github.com/hashicorp/terraform-registry-address, which is not ideal but that one at least has a relatively shallow dependency subgraph, so there's not so much urgency to tidy that one.	2 years ago
Martin Atkins	545980d664	langrefs: Separate package for expression reference analysis ... Finding references in an expression is a static analysis operation, but we previously had it grouped in with "package lang" that primarily deals with dynamic evaluation of expressions. We need to do static analysis in far more locations than we do full expression evaluation, the static-analysis-only callers don't need anything else from package lang, and the reference-analysis functions are self-contained little wrappers around the address parsing logic in package addrs anyway. Splitting these into a separate package therefore minimizes which packages depend indirectly on package lang, and thus which packages depend indirectly on the implementations of the built-in functions. Some of the built-in functions rely on dependencies we don't use anywhere else, so cutting these out shrinks how many packages indirectly depend on those external dependencies. In particular, they are no longer in the indirect import chain for the "legacy" module.	2 years ago
Martin Atkins	5f2eab6944	providerreqs: Provider requirement/version types in their own package ... As we previously found with "package addrs", any package that contains types used as identifiers or addresses tends to get imported from all over, which means that if we mix real logic with those types in the same package then a lot more packages end up depending indirectly on our external dependencies. In this case, the getproviders package depends on various things we need only during provider installation, such as the library we use for GPG signature verification. Lots of different packages in Terraform need to talk about provider requirements, but only the CLI layer actually needs to do provider installation, and so this commit splits the types we use to talk about requirements into a new package "providerreqs", which package getproviders then depends on. To keep the size of this diff relatively small to start I've left forwarding aliases in package getproviders, which means that not all of the callers needed to be updated all at once. This commit does update all of the packages that the remote state backends depend on though, because that then allows us to reduce the set of indirect dependencies for each of those backends. The nested module go.mod files show that this has disconnected the indirect dependencies on github.com/ProtonMail/go-crypto and on github.com/cloudflare/circl , both of which Terraform CLI uses only for provider installation.	2 years ago
Martin Atkins	1358245617	dependencies: Remove all non-legacy uses of hashicorp/go-multierror ... This functionality is now provided by the "errors" package in the standard library, so the HashiCorp-specific library is obsolete. This also removed our one direct use of github.com/pkg/errors, which we were using in a way that was totally redundant with the stdlib package errors. However, it lives on as an indirect dependency through the Consul backend's dependency on the Consul SDK, which uses that library.	2 years ago
Martin Atkins	d86eee6c1f	go.mod: Eliminate github.com/apparentlymart/go-dump ... I originally wrote this library as a stopgap measure because davecgh/go-spew produced a hard-to-read representation of cty.Value due to exposing the raw internals, whereas my "go-dump" instead just pretty-prints whatever fmt.Sprintf("%#v", ...) would return for a given value. However, in the meantime I wrote ctydebug as a more cty-specialized library for this purpose, and we switched to using go-cmp as our primary library for deep-comparing data structures, and so our few remaining uses of apparentlymart/go-dump were vestigial in tests that haven't been updated recently enough to have been updated to our new standards. This commit therefore swaps out all use of that library with combinations of go-cmp and ctydebug, matching our current standards for how to do these sorts of things in tests. This removes our dependency on "go-dump" altogether. At least one of our dependencies is still using go-dump as part of its own test suite, and so the Go toolchain still includes its declared version in our go.sum file. However, this library is no longer linked in to Terraform either in real builds or in test binaries, and so it gets dropped entirely from our go.mod file.	2 years ago
Martin Atkins	071b6de78a	configs: Don't treat module_variable_optional_attrs as special ... Due to some specific circumstances around how this experiment reached stabilization we made a planned-temporary exception to make it valid to specify even in non-alpha builds of Terraform, just to be able to return a more precise error message. The comment added at the time suggested that we remove this "a release or two after v1.3", and v1.8 is the actively-developed release at the time I'm writing this so we've kept this around far longer than we originally felt we needed to. It's time for it to go. This particular experiment will now get the same treatment as all others when used in a non-alpha release: Terraform will return the general error about experiments only being available in alpha releases.	2 years ago
Liam Cervante	cec4b7b74c	stacks: validate providers based on types instead of local names (#34735) ... * stacks: validate providers based on types instead of local names * fix tests	2 years ago
Alisdair McDiarmid	1459825e53	Merge pull request #34725 from hashicorp/alisdair/configs-sourcebundle-parser ... Add and use a source bundle aware configuration parser for stacks	2 years ago
Alisdair McDiarmid	e8da9ef7c9	configs: Add source bundle aware config loader	2 years ago
Alisdair McDiarmid	f9998ff2a5	configs: Remove unused *Parser.LoadValuesFile	2 years ago
Liam Cervante	9658f9df6b	mocking overrides: default to concrete empty object when values are missing (#34563)	2 years ago
James Bardin	7ccb21d864	Block.Filter can panic with no attrs ... Make sure that we only dereference an attribute is it was set in the filter. While not extremely useful, a provider could have a nested object with no attributes.	2 years ago
Liam Cervante	3a75362cce	terraform test: run block names should be valid HCL identifiers (#34469) ... * terraform test: run block names should be valid HCL identifiers * fix test	2 years ago
James Bardin	8125fe56a4	cannot statically validate an import target moved ... The validation for checking if an import target is the source of a moved block is incorrect. Besides the noted FIXME of the check being incorrect for moved modules (and also resources scoped within modules); an import can only be statically identified at the granularity of a resource block, but moved block can have individual instances in the `from` field. Since we already validate that a resource block must _not_ exist in the config for `moved`, we can rely on that causing `import` to fail with a nonexistant resource block during evaluation if they have the same target.	2 years ago
kmoe	a718f70f85	Add removed block (#34251) ... * terraform: remove redundant code NodeDestroyResourceInstance is never instantiated with a DeposedKey of anything other than states.NotDeposed, so the deleted code is never run. Deposed objects get a NodeDestroyDeposedResourceInstanceObject instead. * tfdiags: add helper func * configs: introduce removed block type * terraform: add forget action * renderer: render forget actions * terraform: deposed objects can be forgotten Deposed objects encountered during planning spawn NodePlanDeposedResourceInstanceObject, which previously generated a destroy change. Now it will generate a forget change if the deposed object is a forget target, and a destroy change otherwise. The apply graph gains a new node type, NodeForgetDeposedResourceInstanceObject, whose execution simply removes the object from the state. * configs: add RemoveTarget address type * terraform: modules can be forgotten * terraform: error if removed obj still in config * tests: better error on restore state fail * Update CHANGELOG.md	2 years ago
Martin Atkins	ff992238c1	core: Allow passing in externally-configured provider instances ... Normally Terraform Core expects what it sees as the root module to be a self-contained root module which defines all of the provider configurations it needs. This commit is the start of making it valid for the apparent root module to actually be a shared module, which means it'd expect to receive provider configurations from its caller rather than defining them itself. The caller of the "root" module is the Terraform Core caller written in Go, so this is essentially a Go equivalent of the "providers" argument in a "module" block in the Terraform language.	2 years ago
Martin Atkins	bd5b49b036	configs: Config.EffectiveRequiredProviderConfigs ... This new helper summarizes all of the provider configuration addresses that a module would require to have passed in or inherited from its caller.	2 years ago
Liam Cervante	ee9aa07e10	terraform test: implement the source attribute for mock providers (#34192) ... * complete implementation * add tests * address comments	2 years ago
Liam Cervante	57edb9c248	testing framework: add warnings for override blocks with invalid targets (#34181) ... * testing framework: add warnings for override blocks with invalid targets * pre-add the override sources for upcoming features * address comments	2 years ago
Liam Cervante	05f877166d	terraform test: parse mocking structures with test files (#34143)	3 years ago
James Bardin	2e11d4a149	data sources cannot be imported ... A data source should not be importable, and will crash if config generation is attempted. Validate the target type early to prevent errors during the plan.	3 years ago
James Bardin	663cc1d1b7	move invalid tests ... Schema attributes must have a type to be valid. These would have panicked anyway when used in other cases, but were passing within the limited test case. The update to hcl makes these panic earlier, so add our own panics to prevent invalid attribute schemas entirely.	3 years ago
Liam Cervante	5bbdc4c3b6	terraform test: allow providers in test files to reference variables (#34069) ... * Process only relevant variables in the testing framework * add support for variables within test provider configurations * remove unused function * fix consistency checks	3 years ago
kmoe	4b88808201	configs/terraform: refactor expansion code, add diags (#33962) ... * configs: remove unneeded code This code inside addProviderRequirements adds an entry to the provider map for each resource targeted by an import block. In the normal case without config generation, a resource block must exist for each import block, so provider requirements will be collected from the resource block. In the config generation case, the code block directly below this deleted one will add the appropriate provider requirements where these must come from the import block. * terraform: diags in DynamicExpand	3 years ago
James Bardin	66637ed228	use new unwrapJSONRefExpr for replace_triggered_by ... Get rid of the duplicated code, and use the new function which also validates the expression result type that was missed in the existing implementation.	3 years ago
James Bardin	a2b189fd71	handle to expression from json files ... Because expression from JSON files are handled differently from HCL, we need to deal with these manually when evaluating the special "to" syntax. We take the same approach as was done for replace_triggered_by, where the raw string is extracted from the json expression, and re-evaluated as an HCL expression.	3 years ago
James Bardin	e130c7b9ab	correct check for duplicate imports	3 years ago
James Bardin	df23c703ce	import block decode ... Update the import configuration block to decode for_each and to into raw expressions, so that we can later evaluate those during expansion.	3 years ago
Jared Baker	649f9831e7	internal/configs: CoerceValue error for omitted attribute requirements	3 years ago
Liam Cervante	041b54fb4b	explicitely disallow duplicate run block names in test files (#33836)	3 years ago
CJ Horton	98b81d3c50	add missing copyright headers	3 years ago
hashicorp-copywrite[bot]	53c34ff49c	Update copyright file headers to BUSL-1.1	3 years ago
kmoe	765c25d2ed	allow interpolation in import block id (#33618) ... The import block id field can now reference variables, attributes, and module outputs, as long as the result is a known non-empty string at plan time. A null or unknown value will result in an error. This commit slightly modifies the legacy CLI terraform import code path to construct a synthetic hcl.Expression from the import id passed in from the command line, with no intended change of functionality.	3 years ago
Liam Cervante	f397954c52	testing framework: allow users to specify deeply nested testing directories (#33584)	3 years ago