77 Commits (7e767490c87bae2fa2b60da33ecefa1b7a220afa)

Author	SHA1	Message	Date
Radek Simko	e312ffc2d8	lang/functions: Add test for ephemeralasnull (#35652)	2 years ago
kmoe	fab3673f8d	lang: stabilise ephemeral values experiment (#35636) ... * lang: stabilise ephemeral values experiment Stabilise the ephemeral values experiment and ephemeralasnull experimental function. * simplify handling of apply time input vars As the final step in stabilising the ephemeral values experiment, we can remove the separate code path for handling input variables supplied via -var and -var-file during apply. The intent here is conveyed in the tests: when applying a saved plan, all ephemeral variables set in the plan (the "apply-time variables"), must be set, and no others. As per the TODO copied from the prototype, there is some more work to be done here in making the handling of undeclared variables during apply as sophisticated as that during plan, emitting helpful warnings (for example) when input variables are supplied unnecessarily via environment variables. * experiments: officially conclude EV experiment --------- Co-authored-by: Radek Simko <radek.simko@gmail.com>	2 years ago
Martin Atkins	71d14e78fd	lang: Experimental "ephemeralasnull" function ... This is another part of the existing ephemeral_values experiment, taking a value of any type that might have ephemeral values in it and returning a value of the same type which has any ephemeral value replaced with a null value. The primary purpose of this is to allow a module to conveniently return an object that would normally contain nested ephemeral values -- such as an instance of a managed resource type that has a write-only attribute -- through an output value that isn't declared as ephemeral. This would then expose all of the non-ephemeral parts of the object but withhold the ephemeral parts. In the case of write-only attributes, it exposes the normal attributes while withholding the write-only ones. The name of this function could potentially change before stabilization, because it's quite long and clunky. I did originally consider "nonephemeral" to match with the existing "nonsensitive", but that didn't feel right because "nonsensitive" removes the sensitive mark while preserving the underlying value while this function removes the mark and the real value at the same time. (It would not be appropriate to have a function that just removes the ephemeral mark while preserving the value, because correct handling of ephemerality is important for correctness while sensitivity is primarily a UI concern so we don't need to be quite so picky about it.)	2 years ago
Martin Atkins	6f84623cb6	terraform: Initial partial support for ephemeral values ... This is a checkpoint commit on the path to supporting ephemeral values as a cross-cutting concern in the Terraform language. An ephemeral value is one that lives only in RAM during a single phase and so cannot persist from the plan phase to the apply phase, or between consecutive plan/apply rounds. Terraform tracks whether each value is ephemeral using the cty "marks" concept, thus achieving the same dynamic analysis as we already employ for the concept of "sensitive values" that prevents displaying a value in the user interface. This commit is just a starting point which gets some of the basics into place: - input variables and output values can both be statically declared as being ephemeral. Only ephemeral inputs/outputs can have ephemeral values assigned to them, and the recipient of the value sees it as ephemeral even if the assigned value was not already ephemeral. This creates a dynamic analysis cutoff point at module boundaries so that it's possible to determine in isolation whether a single module is using ephemeral values correctly, without having to test it in every possible calling context. - Managed and data resources cannot have ephemeral values assigned into their configurations because Terraform and providers both expect the resource attributes to persist between phases. - Ephemeral values _can_ be used in provider and provisioner configurations, because both of those effectively meet the definition of the new "ephemeral" concept despite it not previously having a name. - Ephemeral markings propagate through all of the built-in language features for dynamic analysis purposes, largely relying on cty's efforts to do that in similar vein as for sensitive marks. In particular, it's possible to define a local value whose expression produces an ephemeral result, and passing ephemeral values to functions should propagate the ephemeral mark to the results when appropriate. (I've not yet thoroughly reviewed all of the built-in functions for correct marks handling though, so there may be some gaps to address in later commits.) The next step for this work will be to change the modules runtime to have support for a workflow involving ephemeral _root_ input variables, where their values must be re-supplied during the apply phase. That will then achieve (in experimental capacity) the first goal of ephemeral values: to be able to provide non-persistent settings such as time-limited API tokens to use in provider configuration blocks.	2 years ago
Martin Atkins	5e2281a9e0	lang: A new mark type for ephemeral values ... This isn't actually used anywhere yet. We'll introduce uses of it in subsequent commits.	2 years ago
Martin Atkins	e23e6b84c7	lang/funcs: Allow some more expression types in templatestring ... The templatestring function has some special constraints on its first argument that are included to add some intentional friction for those who are new to Terraform, want to do some simple template rendering, but have only found the templatestring function so far. We know from previous experience with the hashicorp/template provider that this sort of functionality tends to attract those who haven't yet learned that the Terraform language has built-in support for string templates (without calling any function), who would then get confused by the need for an extra level of escaping to render a template string only indirectly through this function. However, this rule is not intended to be onerous and require writing the rest of the containing module in an unnatural way to work around it, so here we loosen the rule to allow some additional forms: - An index expression whose collection operand meets these rules. - A relative traversal whose source operand meets these rules. In particular this makes it possible to write an expression like: data.example.example[each.key].result ...which is a relative traversal from an index from a scope traversal, and is a very reasonable thing to write if you've retrieved multiple templates using a data resource that uses for_each. This also treats splat expressions in the same way as index expressions at the static check stage, but that's only to allow us to reach the dynamic type check that will ultimately report that a string is required, because the result of a splat expression is a tuple. The type-related error message is (subjectively) more helpful/relevant than the syntax-related one for this case. Finally, this includes some revisions to the documentation for this function to correct some editing errors from the first pass and to slightly loosen the language about what's allowed. It's still a little vague about what exactly is allowed, but I'm doubtful that a precise definition in terms of HCL's expression types would be very enlightening for a typical reader anyway. We can tweak the specificity of the language here if we start to see repeated questions about what is and is not valid.	2 years ago
kmoe	1e2d4a2ecc	lang: stabilise templatestring func experiment (#35224) ... * lang: stabilise templatestring func experiment * command/jsonfunction: marshal templatestring * docs: add templatestring	2 years ago
Martin Atkins	9dd28fc2e5	funcs: Don't panic if templatefile path is sensitive ... Previously we were partially propagating any marks from the path, but not going all the way so we still ran into trouble when trying to use the string containing the file contents. Now we'll have loadTmpl also return the marks it had to read through to actually parse the template, and then we'll use those (instead of the original path marks) to mark the result. In practice the pathMarks and the tmplMarks should always match today, but this is intentionally structured to make the data flow clearer -- the marks always travel along with whatever they related to -- so we're less likely to break this accidentally under future maintenence.	2 years ago
Martin Atkins	edf335e7eb	lang/funcs: Experimental "templatestring" function ... This function complements the existing "templatefile" to deal with the unusual situation of rendering a template that comes from somewhere outside of the current module's source code, such as from a data resource result. We have some historical experience with the now-deprecated hashicorp/template provider and its template_file data source, where we found that new authors would find it via web search and assume it was "the way" to render templates in Terraform, and then get frustrated dealing with the confusing situation of writing a string template that generates another string template for a second round of template rendering. To try to support those who have this unusual need without creating another attractive nuisance that would derail new authors, this function imposes the artificial extra rule that its template argument may only be populated using a single reference to a symbol defined elsewhere in the same module. This is intended to entice folks trying to use this function for something other than its intended purpose to refer to its documentation (once written) and then hopefully learn what other Terraform language feature they ought to have used instead. The syntax restriction only goes one level deep, so particularly-determined authors can still intentionally misuse this function by adding one level of indirection, such as by building template source code in a local value and then passing that local value as the template argument. The restriction is in place only to reduce the chances of someone _misunderstanding_ the purpose of this function; we don't intend to prevent someone from actively deciding to misuse it, if they have a good reason to do so. This new function inherits the same restriction as templatefile where it does not allow recursively calling other template-rendering functions. This is to dissuade from trying to use Terraform templates "at large", since Terraform's template language is not designed for such uses. It would be better to build a Terraform provider that wraps a more featureful template system like Gonja if someone really does need advanced templating, beyond Terraform's basic goals of being able to build small configuration files, etc. Because this function's intended purpose is rendering templates obtained from elsewhere, this function also blocks calls to any of Terraform's functions that would read from the filesystem of the computer where Terraform is running. This is a small additional measure of isolation to reduce the risk of an attacker somehow modifying a dynamically-fetched template to inspire Terraform to write sensitive data from the host computer into a location accessible to the same attacker, or similar. This is currently only a language experiment and so will not yet be available in stable releases of Terraform. Before stabilizing this and committing to supporting it indefinitely we'll want to gather feedback on whether this function actually meets the intended narrow set of use-cases around dynamic template rendering.	2 years ago
Martin Atkins	b5a4860a36	lang/marks: MarkPaths helper function ... This helper is a slightly more convenient wrapper around cty's own "MarkWithPaths" function that applies a single given mark at zero or more paths in a given value. In several parts of Terraform we preserve sensitive marks in particular for serialization and cannot preserve anything else, and so this helper is useful for reapplying the saved sensitive paths back to the stored value during decoding.	2 years ago
Martin Atkins	2334fc867c	lang/marks: PathsWithMark helper function ... This helper deals with the somewhat-common situation where a particular codepath can handle only one kind of marking, such as in our various wire and file format serializers which have special support for serializing sensitive value paths but cannot preserve any other marks. The assumption here is that the caller will probably return some sort of error if the second return value is non-empty, but we leave thec caller to decide exactly what to do because different situations might warrant different treatment of the problem, such as whether it's considered to be a Terraform language authoring error or to be a bug in Terraform.	2 years ago
Martin Atkins	545980d664	langrefs: Separate package for expression reference analysis ... Finding references in an expression is a static analysis operation, but we previously had it grouped in with "package lang" that primarily deals with dynamic evaluation of expressions. We need to do static analysis in far more locations than we do full expression evaluation, the static-analysis-only callers don't need anything else from package lang, and the reference-analysis functions are self-contained little wrappers around the address parsing logic in package addrs anyway. Splitting these into a separate package therefore minimizes which packages depend indirectly on package lang, and thus which packages depend indirectly on the implementations of the built-in functions. Some of the built-in functions rely on dependencies we don't use anywhere else, so cutting these out shrinks how many packages indirectly depend on those external dependencies. In particular, they are no longer in the indirect import chain for the "legacy" module.	2 years ago
James Bardin	336145c384	update diagnostic message	2 years ago
James Bardin	bac2f892c6	more helpful provider function diagnostics ... Use the new FunctionCallUnknownDiagExtra feature from hcl to help guide users through problems with provider function calls. Now that we can detect diagnostics from unknown function calls, we can correlate the namespace and names with certain problems specific to Terraform.	2 years ago
Ansgar Mertens	cd1306430b	fix(docs): Include product in relative link (#34510) ... CDKTF uses these definitions to generate docs which end up on developer.hashicorp.com There are content checks that fail for this link as the product (terraform) is required to be explicitly stated now	2 years ago
Daniel Schmidt	3a77065e69	add `issensitive` function ... This function can be used to check if a value is marked as sensitive or not within terraform. Closes #34572	2 years ago
James Bardin	1644c2f63b	check function descriptions directly ... Now that we dynamically add scoped versions to the global functions we can't check the static Descriptions list. This should be OK since the function descriptions are taken directly from Descriptions list, and if any other scoped functions were added in the future, they may not be coming from the current global set of functions anyway.	2 years ago
Martin Atkins	17d7f7ff24	lang: Scope accepts "External Functions" ... This is intended as a bridge to functions defined outside of the core language, which will live in other namespaces in order to avoid ambiguity with any built-in functions. For now we only accept provider-contributed external functions. The lang package doesn't really know anything special about providers, so its only responsibility here is to enforce the naming scheme that any provider-contributed functions always live under the namespace prefix "provider::NAME::", where NAME is a name decided by whatever codepath instantiated the state as a unique identifier for a particular provider. This commit also includes updates to the core package to make it possible to pass in ExternalFuncs in principle, although in practice we don't yet have any codepaths which do so, and so this is effectively a no-op commit.	2 years ago
Martin Atkins	d1df979115	lang: Expose built-in functions under core:: namespace ... This essentially doubles up the registrations for all of the built-in functions to have both non-namespaced and core::-namespaced versions of each function. This is in preparation for later commits introducing other namespaces, such as a provider:: namespace which could hold functions that were contributed by providers that are currently in scope.	2 years ago
Martin Atkins	a2ca1208a4	lang: EvalCheckErrorMessage is language-agnostic ... We have a design goal of reusing the same design patterns for all of our "condition-like" configuration constructs across all of Terraform's languages, including custom conditions, validation rules, and check blocks. Now that we have the stacks language and the test language in addition to what I guess we'll now call the "modules language", we'll eventually want to hoist much of the handling of such blocks into a language-agnostic place where all three languages can use it. This is a very meagre start towards that goal, just factoring out the one function for evaluating and validating the "error_message" argument associated with a condition. I've factored it out into package "lang" because that is currently the closest thing we have to a language-agnostic place for this. It's not _quite_ right because lang.Scope is quite specific to the modules language, but we're already importing things from this package in both the test runtime and the stacks runtime and so it's a good enough place until in future we find a more complete design to represent the common elements across all three languages.	2 years ago
Liam Cervante	b9f6b23ba3	testing framework: add support for functions in variables and providers (#34204)	2 years ago
Joao C Costa	47beda901e	nonsensitive: no longer produces error when applied to nonsensitive values (#33856) ... * Proposing fix to #31693 (not always possible to apply nonsensitive) https://github.com/hashicorp/terraform/issues/31693 * hashicorp#31693 not always possible to apply nonsensitive updated documentation * Update sensitive_test.go Fixing tests * Corrections as per https://github.com/hashicorp/terraform/pull/33856#discussion_r1330042125	2 years ago
CJ Horton	98b81d3c50	add missing copyright headers	3 years ago
Liam Cervante	9742f22c4e	Introduce 'run' keyword for referencing outputs from earlier run blocks (#33683) ... * introduce 'run' keyword for referencing outputs from earlier run blocks * fix code consistency	3 years ago
hashicorp-copywrite[bot]	53c34ff49c	Update copyright file headers to BUSL-1.1	3 years ago
James Bardin	ea1144995f	update callers of InstallModules ... Update callers of InstallModules and have `get` return only install errors.	3 years ago
Liam Cervante	6882dd9530	testing framework: introduce test command optional flags (#33504) ... * testing framework: introduce test command optional flags * address consistency checks	3 years ago
James Bardin	cf5ebaf685	s/Schemas/ProviderSchema/	3 years ago
James Bardin	4e8dd5866c	update schemas type	3 years ago
Liam Cervante	212ae6c4ba	Introduce separate testing scope for reference validation (#33339)	3 years ago
Martin Atkins	81c15f987e	lang/funcs: startswith considers string prefix refinement ... If the string to be tested is an unknown value that's been refined with a prefix and the prefix we're being asked to test is in turn a prefix of that known prefix then we can return a known answer despite the inputs not being fully known. There are also some other similar deductions we can make about other combinations of inputs. This extra analysis could be useful in a custom condition check that requires a string with a particular prefix, since it can allow the condition to fail even on partially-unknown input, thereby giving earlier feedback about a problem.	3 years ago
Martin Atkins	c912970153	lang/funcs: Non-null refinements for various functions ... cty's new "refinements" concept allows us to reduce the range of unknown values from our functions. This initial changeset focuses only on declaring which functions are guaranteed to return a non-null result, which is a helpful baseline refinement because it allows "== null" and "!= null" tests to produce known results even when the given value is otherwise unknown. This commit also includes some updates to test results that are now refined based on cty's own built-in refinement behaviors, just as a result of us having updated cty in the previous commit.	3 years ago
hashicorp-copywrite[bot]	325d18262e	[COMPLIANCE] Add Copyright and License Headers	3 years ago
Zac Clifton	3bb6a5896b	Feat: Add strcontains function and documentation (#33069) ... * add strcontains function and documentation	3 years ago
Liam Cervante	aae6990d38	Add native `plantimestamp` function to Terraform (#32980) ... * add plantimestamp function * documentation * add missing links * fix typo	3 years ago
Martin Atkins	9f827f57ae	go.mod: Revert cty minor version v1.13 upgrade ... We inadvertently incorporated the new minor release of cty into the 1.4 branch, and that's introduced some more refined handling of unknown values that is too much of a change to introduce in a patch release. Therefore this reverts back to the previous minor release for the v1.4 series, and then we'll separately get the main branch ready to work correctly with the new cty before Terraform v1.5. This reverts just the upgrade and the corresponding test changes from #32775, while retaining the HCL upgrade and the new test case it introduced for that bug it was trying to fix. That new test is still passing so it seems that the cty upgrade is not crucial to that fix.	3 years ago
Liam Cervante	af05cbb645	Add support for scoped resources (#32732)	3 years ago
Liam Cervante	32f151f20b	Update HCL to handle type mismatches when applying defaults (#32775) ... * add test reproducing terraform crash * pull latest hcl * add missing go.sum file * also fix tests broken by go-cty update	3 years ago
Alisdair McDiarmid	8df065a2fe	initwd: Switch from earlyconfig to configs ... This is a mostly mechanical refactor with a handful of changes which are necessary due to the semantic difference between earlyconfig and configs. When parsing root and descendant modules in the module installer, we now check the core version requirements inline. If the Terraform version is incompatible, we drop any other module loader diagnostics. This ensures that future language additions don't clutter the output and confuse the user. We also add two new checks during the module load process: * Don't try to load a module with a `nil` source address. This is a necessary change due to the move away from earlyconfig. * Don't try to load a module with a blank name (i.e. `module ""`). Because our module loading manifest uses the stringified module path as its map key, this causes a collision with the root module, and a later panic. This is the bug which triggered this refactor in the first place.	3 years ago
Daniel Banck	f29156cdef	Replace HTML entities in function descriptions (#32710)	3 years ago
Daniel Banck	4fd8322802	Add function descriptions (#32453) ... * Add consolidated function description list * Add function parameter descriptions * Add descriptions to all functions * Add sanity test for function descriptions * Apply suggestions from code review Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com> Co-authored-by: kmoe <5575356+kmoe@users.noreply.github.com>	3 years ago
Martin Atkins	8253821e56	go get github.com/zclconf/go-cty-yaml@v1.0.3 ... This includes the fix for a bug in what Terraform calls the "yamldecode" function, where it was not correctly handling any situation where the decode result is a null value. It was previously returning an unknown value in that case, whereas now it returns a null value as expected.	3 years ago
Martin Atkins	2ee9589650	lang/funcs: "timecmp" function ... This is a complement to "timestamp" and "timeadd" which allows establishing the ordering of two different timestamps while taking into account their timezone offsets, which isn't otherwise possible using the existing primitives in the Terraform language.	4 years ago
Martin Atkins	783a07d9e8	build: Use Go 1.19 ... Go 1.19's "fmt" has some awareness of the new doc comment formatting conventions and adjusts the presentation of the source comments to make it clearer how godoc would interpret them. Therefore this commit includes various updates made by "go fmt" to acheve that. In line with our usual convention that we make stylistic/grammar/spelling tweaks typically only when we're "in the area" changing something else anyway, I also took this opportunity to review most of the comments that this updated to see if there were any other opportunities to improve them.	4 years ago
James Bardin	893a5336d8	don't lose warnings from static validation ... Warnings were dropped from static reference validation if there weren't also errors in the configuration.	4 years ago
Joshua Barton	fd3798b276	tests: add test cases around empty values	4 years ago
Joshua Barton	c73c0ad63c	feat: add startswith and endswith funcs	4 years ago
Alisdair McDiarmid	718b0875ef	lang: Remove defaults function ... Now that we are able to specify optional object attribute defaults inline in a type constraint, the separate `defaults` function is no longer needed.	4 years ago
Martin Atkins	d4776e8ef1	lang/funcs: type conversion functions can convert null values ... We had intended these functions to attempt to convert any given value, but there is a special behavior in the function system where functions must opt in to being able to handle dynamically-typed arguments so that we don't need to repeat the special case for that inside every function implementation. In this case we _do_ want to specially handle dynamically-typed values, because the keyword "null" in HCL produces cty.NullVal(cty.DynamicPseudoType) and we want the conversion function to convert it to a null of a more specific type. These conversion functions are already just a thin wrapper around the underlying type conversion functionality anyway, and that already supports converting dynamic-typed values in the expected way, so we can just opt in to allowing dynamically-typed values and let the conversion functionality do the expected work. Fixing this allows module authors to use type conversion functions to give additional type information to Terraform in situations that are too ambiguous to be handled automatically by the type inference/unification process. Previously tostring(null) was effectively a no-op, totally ignoring the author's request to treat the null as a string.	4 years ago
Martin Atkins	49d7c879ac	Fix problems caught by staticcheck v0.3.0 ... This will allow us to upgrade to this version in a later commit without causing the our build checks to fail.	4 years ago