80 Commits (da521707970109f4866b2c064afdc716632dd016)

Author	SHA1	Message	Date
Martin Atkins	003317d7c8	lang: Detect references when a list/set attr is defined using blocks ... For compatibility with documented patterns from existing providers we are now allowing (via a pre-processing step) any attribute whose type is a list-of-object or set-of-object type to optionally be assigned using one or more blocks whose type is the attribute name. The pre-processing functionality was implemented in previous commits but we were not correctly detecting references within these blocks that are, from the perspective of the primary schema, invalid. Now we'll use an alternative implementation of variable detection that is able to apply the same schema rewriting technique we used to implement the transform and thus can find all of the references as if they were already in their final locations.	7 years ago
Martin Atkins	8746e9e8ad	lang/blocktoattr: ExpandedVariables function ... Because we handle FixUpBlockAttrs after dynamic block expansion, when resolving variables we unfortunately need to consider the possibility of both dynamic block expansion _and_ the block attrs fixup. To accommodate this we have a variant of dynblock.VariablesHCLDec that instead walks using the configschema.Block representation of the schema and applies the same opportunistic schema rewrite used by FixUpBlockAttrs at each body encountered during the walk.	7 years ago
Martin Atkins	87786484ea	lang/eval: Apply attr-as-nested-block fixup in EvalBlock ... For any block content we evaluate dynamically via this API, we'll make a special allowance for users to optionally write members of a list attribute instead as a sequence of nested blocks, thus allowing some existing provider features that were assuming this capability to continue to support it after v0.12. This should not be used for any new provider features, and should ideally be eventually phased out so that there aren't two similar-but-slightly-different syntaxes for saying the same thing.	7 years ago
Martin Atkins	6dcf8195b8	lang/blocktoattr: Selectively allow block syntax to be used for attributes ... This preprocessing step allows users to use nested block syntax to specify elements of an attribute that is defined as being a list or set of an object type. This restores part of the unintended flexibility permitted in Terraform v0.11 so that we can work around a few tricky edges where provider implementations were relying on Terraform's failure to validate this in earlier versions. For any body that is pre-processed using this new helper, we will recognize when a configuration author uses nested block syntax with a name that is specified in the schema as an attribute of a suitable type and tweak the schema just in time before decoding to expect that usage and then fix up the result on the way out to conform to the original schema. Achieving this requires an abstraction inversion because only Terraform's high-level schema has enough information to decide how to rewrite the incoming low-level schema. We must therefore here implement HCL's lowest-level API interface in terms of the higher-level abstractions of hcldec and Terraform's configschema. Because of the abstraction inversion this fixup mechanism cannot be used generally for arbitrary HCL bodies but we can use it carefully inside the lang package where its own API can guarantee the necessary invariants for this to work.	7 years ago
Martin Atkins	096b1bb87b	lang/funcs: Port the "reverse" function from the old functions set ... This has the same functionality as the "reverse" function that was implemented in the "config" package, but adapted to the new language type system.	7 years ago
Martin Atkins	2b1e650df2	lang/funcs: Fix out-of-bounds ArgError in templatefile function ... The templatefile function only has two arguments, so ArgErrorf can be called with only zero or one as the argument index. If we are out of bounds then HCL itself will panic trying to build the error message for this call when called as an HCL function. Unfortunately there isn't really a great layer in Terraform to test for this class of bug systematically, because we are currently testing these functions directly rather than going through HCL to do it. For the moment we'll just live with that, but if we see this class of error arise again we might consider either reworking the tests in this package to work with HCL expression source code instead of direct calls or adding some additional tests elsewhere that do so.	7 years ago
Martin Atkins	50a101afbd	lang: Consider "dynamic" blocks when resolving references ... The hcldec package has no awareness of the dynamic block extension, so the hcldec.Variables function misses any variables declared inside dynamic blocks. dynblock.VariablesHCLDec is a drop-in replacement for hcldec.Variables that _is_ aware of dynamic blocks, returning all of the same variables that hcldec would find naturally plus also any variables used inside the dynamic block "for_each" and "labels" arguments and inside the nested "content" block.	7 years ago
Martin Atkins	838a42d218	vendor: go get github.com/hashicorp/hcl2@master ... This includes improved functionality for HCL's "dynamic block extension", which will allow us (in a subsequent commit) to properly detect dependencies inside nested "dynamic" blocks, where currently they get missed. For this commit though, we just upgrade HCL to a version that includes it and make a small change to our "lang" package to align with an upstream renaming.	7 years ago
Brian Flad	398fe8652d	config: Mention other file hashing functions when file() detects unsupported contents ... Reference: https://github.com/hashicorp/terraform/issues/20664	7 years ago
Brian Flad	81bdaa8c38	lang/funcs: Fix filebase64sha256 function hashing algorithm ... Reference: https://github.com/hashicorp/terraform/issues/20652 The implementation and testing were incorrectly referencing SHA-512 instead of SHA-256.	7 years ago
Martin Atkins	f8a6f66be4	lang/funcs: Fix panic in "join" when an element is null ... It is now a proper error message.	7 years ago
Martin Atkins	954d38e870	lang: New file-hashing functions ... In prior versions, we recommended using hash functions in conjunction with the file function as an idiom for detecting changes to upstream blobs without fetching and comparing the whole blob. That approach relied on us being able to return raw binary data from file(...). Since Terraform strings pass through intermediate representations that are not binary-safe (e.g. the JSON state), there was a risk of string corruption in prior versions which we have avoided for 0.12 by requiring that file(...) be used only with UTF-8 text files. The specific case of returning a string and immediately passing it into another function was not actually subject to that corruption risk, since the HIL interpreter would just pass the string through verbatim, but this is still now forbidden as a result of the stricter handling of file(...). To avoid breaking these use-cases, here we introduce variants of the hash functions a with "file" prefix that take a filename for a disk file to hash rather than hashing the given string directly. The configuration upgrade tool also now includes a rule to detect the documented idiom and rewrite it into a single function call for one of these new functions. This does cause a bit of function sprawl, but that seems preferable to introducing more complex rules for when file(...) can and cannot read binary files, making the behavior of these various functions easier to understand in isolation.	7 years ago
Martin Atkins	ecaaa91da9	lang/funcs: Factor out the various hash function implementations ... These all follow the pattern of creating a hash and converting it to a string using some encoding function, so we can write this implementation only once and parameterize it with a hash factory function and an encoding function. This also includes a new test for the sha512 function, which was previously missing a test and, it turns out, actually computing sha256 instead.	7 years ago
Martin Atkins	2f8f7d6f4d	lang/funcs: Type conversion functions ... It's not normally necessary to make explicit type conversions in Terraform because the language implicitly converts as necessary, but explicit conversions are useful in a few specialized cases: - When defining output values for a reusable module, it may be desirable to force a "cleaner" output type than would naturally arise from a computation, such as forcing a string containing digits into a number. - Our 0.12upgrade mechanism will use some of these to replace use of the undocumented, hidden type conversion functions in HIL, and force particular type interpretations in some tricky cases. - We've found that type conversion functions can be useful as _temporary_ workarounds for bugs in Terraform and in providers where implicit type conversion isn't working correctly or a type constraint isn't specified precisely enough for the automatic conversion behavior. These all follow the same convention of being named "to" followed by a short type name. Since we've had a long-standing convention of running all the words together in lowercase in function names, we stick to that here even though some of these names are quite strange, because these should be rarely-used functions anyway.	7 years ago
Martin Atkins	da51e72cbb	lang/functions: set functions from cty ... The sethaselement, setintersection, and setunion functions are defined in the cty stdlib. Making them available in Terraform will make it easier to work with sets, and complement the currently-Terraform-specific setproduct function. In the long run setproduct should probably move into the cty stdlib too, but since it was submitted as a Terraform function originally we'll leave it here now for simplicity's sake and reorganize later.	7 years ago
Martin Atkins	edb5f82de1	lang/funcs: Convert the "setproduct" function to the new approach ... In our new world it produces either a set of a tuple type or a list of a tuple type, depending on the given argument types. The resulting collection's element tuple type is decided by the element types of the given collections, allowing type information to propagate even if unknown values are present.	7 years ago
Martin Atkins	d0e6a4c69a	lang: Add "formatdate" function ... We missed this one on a previous pass of bringing in most of the cty stdlib functions. This will resolve #17625 by allowing conversion from Terraform's conventional RFC 3339 timestamps into various other formats.	7 years ago
Martin Atkins	c753df6a93	lang/funcs: templatefile function ... This function is similar to the template_file data source offered by the template provider, but having it built in to the language makes it more convenient to use, allowing templates to be rendered from files anywhere an inline template would normally be allowed: user_data = templatefile("${path.module}/userdata.tmpl", { hostname = format("petserver%02d", count.index) }) Unlike the template_file data source, this function allows values of any type in its variables map, passing them through verbatim to the template. Its tighter integration with Terraform also allows it to return better error messages with source location information from the template itself. The template_file data source was originally created to work around the fact that HIL didn't have any support for map values at the time, and even once map support was added it wasn't very usable. With HCL2 expressions, there's little reason left to use a data source to render a template; the only remaining reason left to use template_file is to render a template that is constructed dynamically during the Terraform run, which is a very rare need.	7 years ago
Martin Atkins	2be524d6ac	core: Validate depends_on and ignore_changes traversals ... Both depends_on and ignore_changes contain references to objects that we can validate. Historically Terraform has not validated these, instead just ignoring references to non-existent objects. Since there is no reason to refer to something that doesn't exist, we'll now verify this and return errors so that users get explicit feedback on any typos they may have made, rather than just wondering why what they added seems to have no effect. This is particularly important for ignore_changes because users have historically used strange values here to try to exploit the fact that Terraform was resolving ignore_changes against a flatmap. This will give them explicit feedback for any odd constructs that the configuration upgrade tool doesn't know how to detect and fix.	7 years ago
Martin Atkins	e63a1dfb96	lang: EvalExpr only convert if wantType is not dynamic ... This actually seems to be a bug in the underlying cty Convert function since converting to cty.DynamicPseudoType should always just return the input verbatim, but it seems like it's actually converting unknown values of any type to be cty.DynamicVal, losing the type information. We should eventually fix this in cty too, but having this extra check in the Terraform layer is harmless and allows us to make progress without context-switching.	7 years ago
Martin Atkins	30497bbfb7	lang/funcs: zipmap accepts tuple of values and produces object ... Now that our language supports tuple/object types in addition to list/map types, it's convenient for zipmap to be able to produce an object type given a tuple, since this makes it symmetrical with "keys" and "values" such the the following identity holds for any map or object value "a" a == zipmap(keys(a), values(a)) When the values sequence is a tuple, the result has an object type whose attribute types correspond to the given tuple types. Since an object type has attribute names as part of its definition, there is the additional constraint here that the result has an unknown type (represented by the dynamic pseudo-type) if the given values is a tuple and the given keys contains any unknown values. This isn't true for values as a list because we can predict the resulting map element type using just the list element type.	7 years ago
Martin Atkins	3b49028b77	core: Static-validate resource references against schemas ... In the initial move to HCL2 we started relying only on full expression evaluation to catch attribute errors, but that's not sufficient for resource attributes in practice because during validation we can't know yet whether a resource reference evaluates to a single object or to a list of objects (if count is set). To address this, here we reinstate some static validation of resource references by analyzing directly the reference objects, disregarding any instance index if present, and produce errors if the remaining subsequent traversal steps do not correspond to items within the resource type schema. This also allows us to produce some more specialized error messages for certain situations. In particular, we can recognize a reference like aws_instance.foo.count, which in 0.11 and prior was a weird special case for determining the count value of a resource block, and offer a helpful error showing the new length(aws_instance.foo) usage pattern. This eventually delegates to the static traversal validation logic that was added to the configschema package in a previous commit, which also includes some specialized error messages that distinguish between attributes and block types in the schema so that the errors relate more directly to constructs the user can see in the configuration. In future we could potentially move more of the checks from the dynamic schema construction step to the static validation step, but resources are the reference type that most needs this immediately due to the ambiguity caused by the instance indexing syntax. We can safely refactor other reference types to be statically validated in later releases. This is verified by two pre-existing context validate tests which we temporarily disabled during earlier work (now re-enabled) and also by a new validate test aimed specifically at the special case for the "count" attribute.	7 years ago
Martin Atkins	ecc42b838c	lang/funcs: Fix crash and improve precision of keys/values functions ... The "values" function wasn't producing consistently-ordered keys in its result, leading to crashes. This fixes #19204. While working on these functions anyway, this also improves slightly their precision when working with object types, where we can produce a more complete result for unknown values because the attribute names are part of the type. We can also produce results for known maps that have unknown elements; these unknowns will also appear in the values(...) result, allowing them to propagate through expressions. Finally, this adds a few more test cases to try different permutations of empty and unknown values.	8 years ago
Martin Atkins	8c01cf7293	lang/funcs: Fix broken test for lookup function ... When the value we're looking in has an object type, we need to know the key in order to decide the result type. Therefore an object lookup with an unknown key must produce cty.DynamicVal, not an unknown value with a known type.	8 years ago
Martin Atkins	8f578c365f	lang/funcs: Permit object types in the "length" function ... The implementation already allowed this, so this was just an oversight in the type checking function. This fixes #19278.	8 years ago
Radek Simko	b41cda7d7a	Return cty.DynamicPseudoType instead	8 years ago
Radek Simko	4856d81300	lang: Fix crash in lookup function	8 years ago
James Bardin	9e3c23d890	verify that all LengthInt calls are known ... There were some funcs with LengthInt calls on unknown values	8 years ago
Radek Simko	edaa4bbc82	lang: Add fileexists function	8 years ago
Kristin Laemmert	fd77e56fd6	lookup will return a tuple type when passed an object	8 years ago
Kristin Laemmert	f54ee830d3	lang/funcs: update values to accept object types	8 years ago
Kristin Laemmert	4ec904bca7	funcs/lang lookup: validate that argument is map or object type	8 years ago
Kristin Laemmert	d1d0ede069	lang/funcs: return default value if provided when object lookup fails to find attr	8 years ago
Kristin Laemmert	46e168a682	lang/funcs: update lookup() to accept object-typed values for "map" arg	8 years ago
Martin Atkins	c990c9f36d	lang/funcs: Don't panic if coalescelist gets an unknown list	8 years ago
Martin Atkins	db58b88c2d	lang/funcs: short-circuit with unknown index and tuple collection ... Since we need to know the index to know the result type for a tuple, we need a special case here to deal with that situation and return cty.DynamicVal; we can't predict the result type exactly until we know the element type.	8 years ago
Martin Atkins	efe631d9ec	lang/funcs: in "sort", don't panic if given a null string ... It is incorrect to use a null string, but that should be reported as an error rather than a panic.	8 years ago
Martin Atkins	479c6b2466	move "configschema" from "config" to "configs" ... The "config" package is no longer used and will be removed as part of the 0.12 release cleanup. Since configschema is part of the "new world" of configuration modelling, it makes more sense for it to live as a subdirectory of the newer "configs" package.	8 years ago
Martin Atkins	ccd90bcf35	lang/funcs: never include the private key in error output ... This is based on c811440188 made against the old "config" package implementations, but also catches a few other cases where we would previously have printed the private key into the error messages.	8 years ago
Kristin Laemmert	c23a971ed1	minor fixes	8 years ago
Kristin Laemmert	0dbecc54c0	functions: ValuesFunc - cleanup return type	8 years ago
Kristin Laemmert	a213c4a648	functions: add tests and support for unknown values	8 years ago
Kristin Laemmert	d802d5c624	functions: pr feedback fixes	8 years ago
Kristin Laemmert	ff4b3d763b	functions: fix lookup()'s handling of numberical defaults	8 years ago
Kristin Laemmert	4f5c03339a	functions: ZipmapFunc	8 years ago
Kristin Laemmert	605e23db6b	various code review fixes	8 years ago
Kristin Laemmert	6463dd90e9	functions: TransposeFunc, SliceFunc	8 years ago
Kristin Laemmert	30671d85ad	functions: MergeFunc	8 years ago
Kristin Laemmert	aecd7b2e62	functions: LookupFunc	8 years ago
Kristin Laemmert	4d8c398f8e	functions: KeysFunc	8 years ago