60 Commits (94de72c63834acea811c27a585c19eae29374a4b)

Author	SHA1	Message	Date
Alisdair McDiarmid	64cf53bd7e	configs: Fix check block configuration diagnostics ... When validating self-references for resource and data source preconditions and postconditions, we previously did not nil-check the block's condition field, which caused a panic when the block had no condition. While fixing this I noticed that we were not validating that there are no self-references in the error message, so fixed that.	4 years ago
Martin Atkins	fda0579537	Experiments supported only in alpha/dev builds ... We originally introduced the idea of language experiments as a way to get early feedback on not-yet-proven feature ideas, ideally as part of the initial exploration of the solution space rather than only after a solution has become relatively clear. Unfortunately, our tradeoff of making them available in normal releases behind an explicit opt-in in order to make it easier to participate in the feedback process had the unintended side-effect of making it feel okay to use experiments in production and endure the warnings they generate. This in turn has made us reluctant to make use of the experiments feature lest experiments become de-facto production features which we then feel compelled to preserve even though we aren't yet ready to graduate them to stable features. In an attempt to tweak that compromise, here we make the availability of experiments _at all_ a build-time flag which will not be set by default, and therefore experiments will not be available in most release builds. The intent (not yet implemented in this PR) is for our release process to set this flag only when it knows it's building an alpha release or a development snapshot not destined for release at all, which will therefore allow us to still use the alpha releases as a vehicle for giving feedback participants access to a feature (without needing to install a Go toolchain) but will not encourage pretending that these features are production-ready before they graduate from experimental. Only language experiments have an explicit framework for dealing with them which outlives any particular experiment, so most of the changes here are to that generalized mechanism. However, the intent is that non-language experiments, such as experimental CLI commands, would also in future check Meta.AllowExperimentalFeatures and gate the use of those experiments too, so that we can be consistent that experimental features will never be available unless you explicitly choose to use an alpha release or a custom build from source code. Since there are already some experiments active at the time of this commit which were not previously subject to this restriction, we'll pragmatically leave those as exceptions that will remain generally available for now, and so this new approach will apply only to new experiments started in the future. Once those experiments have all concluded, we will be left with no more exceptions unless we explicitly choose to make an exception for some reason we've not imagined yet. It's important that we be able to write tests that rely on experiments either being available or not being available, so here we're using our typical approach of making "package main" deal with the global setting that applies to Terraform CLI executables while making the layers below all support fine-grain selection of this behavior so that tests with different needs can run concurrently without trampling on one another. As a compromise, the integration tests in the terraform package will run with experiments enabled _by default_ since we commonly need to exercise experiments in those tests, but they can selectively opt-out if they need to by overriding the loader setting back to false again.	4 years ago
Alisdair McDiarmid	479c71f93d	Merge pull request #31210 from hashicorp/update-optional-type-attributes ... Edit type constraints docs for style and flow	4 years ago
James Bardin	57c0deb01e	check for duplicate defaults from resource names	4 years ago
Alisdair McDiarmid	922de89be1	Conclude module variable optional attrs experiment	4 years ago
James Bardin	2581bc93cb	Check for duplicate types in required_providers ... Adding multiple local names for the same provider type in required_providers was not prevented, which can lead to ambiguous behavior in Terraform. Providers are always indexed by the providers fully qualified name, so duplicate local names cannot be differentiated.	4 years ago
Alisdair McDiarmid	650380f3ae	configs: Add default argument to optional() ... The optional modifier previously accepted a single argument: the attribute type. This commit adds an optional second argument, which specifies a default value for the attribute. To record the default values for a variable's type, we use a separate parallel structure of `typeexpr.Defaults`, rather than extending `cty.Type` to include a `cty.Value` of defaults (which may in turn include a `cty.Type` with defaults, and so on, and so forth). The new `typeexpr.TypeConstraintWithDefaults` returns a type constraint and defaults value. Defaults will be `nil` unless there are default values specified somewhere in the variable's type.	4 years ago
James Bardin	a6968b5b20	Merge pull request #31060 from hashicorp/alisdair/fix-configload-snapshot-panic ... configs: Fix module loader nil pointer panic	4 years ago
Martin Atkins	ec421fe02d	configs: A test for the regression reported in #31081 ... 5417975946 addressed a regression in the logic for catching when the newer module meta-arguments are used in conjunction with the legacy practice of including provider configurations inside modules, where the check started incorrectly catching situations where the legacy nested provider configuration was in the same module as the child call using one of those meta-arguments. This is a regression test to catch if a similar bug arises in the future. Since it's testing validation rules that apply to an entire configuration tree, it ended up in a rather idiosyncratic location under the "configload" package, rather than directly in "configs". The "configs" package only knows how to load one module at a time, so it's harder to write a test like this in that context. Due to it being further removed from the code it is testing, I included a test for the correct error too in order to increase the chance that we'll learn if future changes in the "configs" package invalidate this regression test. I've verified that this new test fails without the change made in the earlier commit.	4 years ago
kmoe	5417975946	configs: fix module for_each call bug (#31091) ... This fixes a bug introduced in 1879a39 in which initialising a module will fail if that module contains both a provider block and a module call using for_each.	4 years ago
Alisdair McDiarmid	d1e35a3f7c	configs: Fix module loader nil pointer panic ... In configurations which have already been initialized, updating the source of a non-root module call to an invalid value could cause a nil pointer panic. This commit fixes the bug and adds test coverage.	4 years ago
Martin Atkins	289bb60ce1	core: Defer on transitive dependencies for data resources with conditions ... When a data resource is used for the purposes of verifying a condition about an object managed elsewhere (e.g. if the managed resource doesn't directly export all of the information required for the condition) it's important that we defer the data resource read to the apply step if the corresponding managed resource has any changes pending. Typically we'd expect that to come "for free" but unfortunately we have a pragmatic special case in our handling of data resources where we normally defer to the apply step only if a _direct_ dependency of the data resource has a change pending, and allow a plan-time read if there's a pending change in an indirect dependency. This allowed us to preserve some compatibility with the questionable historical behavior of always reading data resources proactively unless the configuration contains unknown values, since the arguably-more-correct behavior would've been a regression for anyone who had been depending on that before. Since preconditions and postconditions didn't exist until now, we are not constrained in the same way by backward compatibility, and so we can adopt the more correct behavior in the case where a data resource has conditions specified. This does unfortunately make the handling of data resources with conditions subtly inconsistent with those that don't, but this is a better situation than the alternative where it would be easy to get into a trapped situation where the remote system is invalid and it's impossible to plan the change that would make it valid again because the conditions evaluate too soon, prior to the fix being applied.	4 years ago
James Bardin	1e79682c24	minor fixes	4 years ago
James Bardin	6eb3264d1a	parse replace_triggered_by in resource configs	4 years ago
Eng Zer Jun	fedd315275	test: use `T.TempDir` to create temporary test directory (#30803) ... This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The directory created by `t.TempDir` is automatically removed when the test and all its subtests complete. Prior to this commit, temporary directory created using `ioutil.TempDir` needs to be removed manually by calling `os.RemoveAll`, which is omitted in some tests. The error handling boilerplate e.g. defer func() { if err := os.RemoveAll(dir); err != nil { t.Fatal(err) } } is also tedious, but `t.TempDir` handles this for us nicely. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	4 years ago
Alisdair McDiarmid	bb35f02c95	Conclude preconditions/postconditions experiment	4 years ago
Martin Atkins	1879a39d2d	configs: Refined error messages for mismatched provider passing ... This set of diagnostic messages is under a number of unusual constraints that make them tough to get right: - They are discussing a couple finicky concepts which authors are likely to be encountering for the first time in these error messages: the idea of "local names" for providers, the relationship between those and provider source addresses, and additional ("aliased") provider configurations. - They are reporting concerns that span across a module call boundary, and so need to take care to be clear about whether they are talking about a problem in the caller or a problem in the callee. - Some of them are effectively deprecation warnings for features that might be in use by a third-party module that the user doesn't control, in which case they have no recourse to address them aside from opening a feature request with the upstream module maintainer. - Terraform has, for backward-compatibility reasons, a lot of implied default behaviors regarding providers and provider configurations, and these errors can arise in situations where Terraform's assumptions don't match the author's intent, and so we need to be careful to explain what Terraform assumed in order to make the messages understandable. After seeing some confusion with these messages in the community, and being somewhat confused by some of them myself, I decided to try to edit them a bit for consistency of terminology (both between the messages and with terminology in our docs), being explicit about caller vs. callee by naming them in the messages, and making explicit what would otherwise be implicit with regard to the correspondences between provider source addresses and local names. My assumed audience for all of these messages is the author of the caller module, because it's the caller who is responsible for creating the relationship between caller and callee. As much as possible I tried to make the messages include specific actions for that author to take to quiet the warning or fix the error, but some of the warnings are only fixable by the callee's maintainer and so those messages are, in effect, a suggestion to send a request to the author to stop using a deprecated feature. I think these new messages are also not ideal by any means, because it's just tough to pack so much information into concise messages while being clear and consistent, but I hope at least this will give users seeing these messages enough context to infer what's going on, possibly with the help of our documentation. I intentionally didn't change which cases Terraform will return warnings or errors -- only the message texts -- although I did highlight in a comment in one of the tests that what it is a asserting seems a bit suspicious to me. I don't intend to address that here; instead, I intend that note to be something to refer to if we later see a bug report that calls that behavior into question. This does actually silence some _unrelated_ warnings and errors in cases where a provider block has an invalid provider local name as its label, because our other functions for dealing with provider addresses are written to panic if given invalid addresses under the assumption that earlier code will have guarded against that. Doing this allowed for the provider configuration validation logic to safely include more information about the configuration as helpful context, without risking tripping over known-invalid configuration and panicking in the process.	4 years ago
Alisdair McDiarmid	b06fe04621	core: Check rule error message expressions ... Error messages for preconditions, postconditions, and custom variable validations have until now been string literals. This commit changes this to treat the field as an HCL expression, which must evaluate to a string. Most commonly this will either be a string literal or a template expression. When the check rule condition is evaluated, we also evaluate the error message. This means that the error message should always evaluate to a string value, even if the condition passes. If it does not, this will result in an error diagnostic. If the condition fails, and the error message also fails to evaluate, we fall back to a default error message. This means that the check rule failure will still be reported, alongside diagnostics explaining why the custom error message failed to render. As part of this change, we also necessarily remove the heuristic about the error message format. This guidance can be readded in future as part of a configuration hint system.	4 years ago
Alisdair McDiarmid	7ded73f266	configs: Validate pre/postcondition self-refs ... Preconditions and postconditions for resources and data sources may not refer to the address of the containing resource or data source. This commit adds a parse-time validation for this rule.	4 years ago
Martin Atkins	9076400436	configs: Decode preconditions and postconditions ... This allows precondition and postcondition checks to be declared for resources and output values as long as the preconditions_postconditions experiment is enabled. Terraform Core doesn't currently know anything about these features, so as of this commit declaring them does nothing at all.	4 years ago
Martin Atkins	4f41a0a1fe	configs: Generalize "VariableValidation" as "CheckRule" ... This construct of a block containing a condition and an error message will be useful for other sorts of blocks defining expectations or contracts, so we'll give it a more generic name in anticipation of it being used in other situations.	4 years ago
James Bardin	0771a214d8	skip provider resolution when there are errors ... If there are errors loading the configuration, don't try to resolve the provider names, as those names may not even be valid.	4 years ago
Martin Atkins	affe2c3295	addrs: Expose the registry address parser's error messages ... Previously we ended up losing all of the error message detail produced by the registry address parser, because we treated any registry address failure as cause to parse the address as a go-getter-style remote address instead. That led to terrible feedback in the situation where the user _was_ trying to write a module address but it was invalid in some way. Although we can't really tighten this up in the default case due to our compatibility promises, it's never been valid to use the "version" argument with anything other than a registry address and so as a compromise here we'll use the presence of "version" as a heuristic for user intent to parse the source address as a registry address, and thus we can return a registry-address-specific error message in that case and thus give more direct feedback about what was wrong. This unfortunately won't help someone trying to install from the registry _without_ a version constraint, but I didn't want to let perfect be the enemy of the good here, particularly since we recommend using version constraints with registry modules anyway; indeed, that's one of the main benefits of using a registry rather than a remote source directly.	4 years ago
Alisdair McDiarmid	fbed52a353	configs: Add sensitive marks for nested attributes ... Object values returned from providers have their attributes marked as sensitive based on the provider schema. This was not fully implemented for nested attribute types, which is corrected in this commit.	4 years ago
Katy Moe	1dabdf0dcd	normalise test names	4 years ago
Katy Moe	8e640ae151	fix typo	5 years ago
Katy Moe	257c24ff75	IgnoreAllChanges should override during merge	5 years ago
Katy Moe	df6dad5070	add failing test for IgnoreAllChanges override	5 years ago
James Bardin	b91d9435ea	Merge pull request #29832 from hashicorp/jbardin/nullable-variable ... configs: explicitly nullable variable values	5 years ago
Martin Atkins	94cbc8fb5d	experiments: config_driven_move has concluded ... Based on feedback during earlier alpha releases, we've decided to move forward with the current design for the first phase of config-driven refactoring. Therefore here we've marked the experiment as concluded with no changes to the most recent incarnation of the functionality. The other changes here are all just updating test fixtures to no longer declare that they are using experimental features.	5 years ago
James Bardin	7b7972ac95	allow nullable override	5 years ago
James Bardin	f0a64eb456	configs: explicitly nullable variable values ... The current behavior of module input variables is to allow users to override a default by assigning `null`, which works contrary to the behavior of resource attributes, and prevents explicitly accepting a default when the input must be defined in the configuration. Add a new variable attribute called `nullable` will allow explicitly defining when a variable can be set to null or not. The current default behavior is that of `nullable=true`. Setting `nullable=false` in a variable block indicates that the variable value can never be null. This either requires a non-null input value, or a non-null default value. In the case of the latter, we also opt-in to the new behavior of a `null` input value taking the default rather than overriding it. In a future language edition where we make `nullable=false` the default, setting `nullable=true` will allow the legacy behavior of `null` overriding a default value. The only future configuration in which this would be required even if the legacy behavior were not desired is when setting an optional+nullable value. In that case `default=null` would also be needed and we could therefor imply `nullable=true` without requiring it in the configuration.	5 years ago
Martin Atkins	f266d1ee82	"Add cloud integration option" ... A more native integration for Terraform Cloud and its CLI-driven run workflow. Instead of a backend, users declare a special block in the top-level terraform settings block to configure Terraform Cloud, then run terraform init. Full documentation will follow in later commits.	5 years ago
Chris Arcand	18d54c1129	Allow cloud block overrides ... These changes allow cloud blocks to be overridden by backend blocks and vice versa; the logic follows the current backend behavior of a block overriding a preceding block in full, with no merges.	5 years ago
Chris Arcand	07b3d015d1	Defer cloud block overrides ... This restriction is temporary. Overrides should be allowed, but have the added complexity of needing to also override a 'backend' block, so this work is being deferred for now.	5 years ago
Chris Arcand	a4c24e3147	Add cloud {} configuration block for Terraform Cloud ... This is a replacement declaration for using Terraform Cloud as a remote backend, leaving the literal backend as an implementation detail and not a user-level concept.	5 years ago
James Bardin	9b62dd6901	Merge pull request #29648 from hashicorp/jbardin/tfproto6.1 ... tfproto version 6.1	5 years ago
Martin Atkins	d76759a6a9	configs/configload: snapshotDir must be used via pointer ... A snapshotDir tracks its current position as part of its state, so we need to use it via pointer rather than value so that Readdirnames can actually update that position, or else we'll just get stuck at position zero. In practice this wasn't hurting anything because we only call Readdir once on our snapshots, to read the whole directory at once. Still nice to fix to avoid a gotcha for future maintenence, though.	5 years ago
Martin Atkins	df578afd7e	backend/local: Check dependency lock consistency before any operations ... In historical versions of Terraform the responsibility to check this was inside the terraform.NewContext function, along with various other assorted concerns that made that function particularly complicated. More recently, we reduced the responsibility of the "terraform" package only to instantiating particular named plugins, assuming that its caller is responsible for selecting appropriate versions of any providers that _are_ external. However, until this commit we were just assuming that "terraform init" had correctly selected appropriate plugins and recorded them in the lock file, and so nothing was dealing with the problem of ensuring that there haven't been any changes to the lock file or config since the most recent "terraform init" which would cause us to need to re-evaluate those decisions. Part of the game here is to slightly extend the role of the dependency locks object to also carry information about a subset of provider addresses whose lock entries we're intentionally disregarding as part of the various little edge-case features we have for overridding providers: dev_overrides, "unmanaged providers", and the testing overrides in our own unit tests. This is an in-memory-only annotation, never included in the serialized plan files on disk. I had originally intended to create a new package to encapsulate all of this plugin-selection logic, including both the version constraint checking here and also the handling of the provider factory functions, but as an interim step I've just made version constraint consistency checks the responsibility of the backend/local package, which means that we'll always catch problems as part of preparing for local operations, while not imposing these additional checks on commands that _don't_ run local operations, such as "terraform apply" when in remote operations mode.	5 years ago
James Bardin	a53faf43f6	return partial config from LoadConfig with errors ... LoadConfig should return any parsed configuration in order for the caller to verify `required_version`.	5 years ago
James Bardin	9847eaa9cf	remove usage of MinItems/MaxItems ... MinItems and MaxItems are not used on nested types in the protocol, so remove their usage in Terraform to prevent future confusion.	5 years ago
James Bardin	9029870613	handle NestedTypes in Block.CoerceValue ... The CoerceValue code was not updated to handle NestedTypes, and while none of the new codepaths make use of this method, there are still some internal uses.	5 years ago
James Bardin	53a73a8ab6	configs: add ConstraintType to config.Variable ... In order to handle optional attributes, the Variable type needs to keep track of the type constraint for decoding and conversion, as well as the concrete type for creating values and type comparison. Since the Type field is referenced throughout the codebase, and for future refactoring if the handling of optional attributes changes significantly, the constraint is now loaded into an entirely new field called ConstraintType. This prevents types containing ObjectWithOptionalAttrs from escaping the decode/conversion codepaths into the rest of the codebase.	5 years ago
James Bardin	af4f4540a9	configschema: do not expose optional attributes ... Objects with optional attributes are only used for the decoding of HCL, and those types should never be exposed elsewhere within terraform. Separate the external ImpliedType method from the cty.Type generated internally for the decoder spec. This unfortunately causes our ImpliedType method to return a different type than the hcldec.ImpliedType function, but the former is only used within terraform for concrete values, while the latter is used to decode HCL. Renaming the ImpliedType methods could be done to further differentiate them, but that does cause fairly large diff in the codebase that does not seem worth the effort at this time.	5 years ago
Alisdair McDiarmid	06956773dc	configs: Disable experiment warnings at link time ... This package level variable can be overridden at link time to allow temporarily disabling the UI warning when experimental features are enabled. This makes it easier to understand how UI will render when the feature is no longer experimental. This change is only for those developing Terraform.	5 years ago
James Bardin	863963e7a6	de-linting	5 years ago
Martin Atkins	383bbdeebc	Upgrade to Go 1.17 ... This includes the addition of the new "//go:build" comment form in addition to the legacy "// +build" notation, as produced by gofmt to ensure consistent behavior between Go versions. The new directives are all equivalent to what was present before, so there's no change in behavior. Go 1.17 continues to use the Unicode 13 tables as in Go 1.16, so this upgrade does not require also upgrading our Unicode-related dependencies. This upgrade includes the following breaking changes which will also appear as breaking changes for Terraform users, but that are consistent with the Terraform v1.0 compatibility promises. - On MacOS, Terraform now requires macOS 10.13 High Sierra or later. This upgrade also includes the following breaking changes which will appear as breaking changes for Terraform users that are inconsistent with our compatibility promises, but have justified exceptions as follows: - cidrsubnet, cidrhost, and cidrnetmask will now reject IPv4 CIDR addresses whose decimal components have leading zeros, where previously they would just silently ignore those leading zeros. This is a security-motivated exception to our compatibility promises, because some external systems interpret zero-prefixed octets as octal numbers rather than decimal, and thus the previous lenient parsing could lead to a different interpretation of the address between systems, and thus potentially allow bypassing policy when configuring firewall rules etc. This upgrade also includes the following breaking changes which could _potentially_ appear as breaking changes for Terraform users, but that do not in practice for the reasons given: - The Go net/url package no longer allows query strings with pairs separated by semicolons instead of ampersands. This primarily affects HTTP servers written in Go, and Terraform includes a special temporary HTTP server as part of its implementation of OAuth for "terraform login", but that server only needs to accept URLs created by Terraform itself and Terraform does not generate any URLs that would be rejected.	5 years ago
Kristin Laemmert	0729e9fdd7	configs/configschema: extend block.AttributeByPath to descend into Objects (#29222) ... * configs/configschema: extend block.AttributeByPath to descend into Objects This commit adds a recursive Object.AttributeByPath function which will step through Attributes with NestedTypes. If an Attribute without a NestedType is encountered while there is still more to the path, it will return nil.	5 years ago
James Bardin	aaf03d3251	update test error for hclv2.10.1	5 years ago
Kristin Laemmert	0d80a74539	configs/configschema: fix missing "computed" attributes from NestedObject's ImpliedType (#29172) ... * configs/configschema: fix missing "computed" attributes from NestedObject's ImpliedType listOptionalAttrsFromObject was not including "computed" attributes in the list of optional object attributes. This is now fixed. I've also added some tests and fixed some panics and otherwise bad behavior when bad input is given. One natable change is in ImpliedType, which was panicking on an invalid nesting mode. The comment expressly states that it will return a result even when the schema is inconsistent, so I removed the panic and instead return an empty object.	5 years ago