54 Commits (a15a4acf2f0e4e8a749d9cb9a46ac09f224d5e90)

Author	SHA1	Message	Date
Kristin Laemmert	a15a4acf2f	configs/configupgrade: detect possible relative module sources (#20646) ... * configs/configupgrade: detect possible relative module sources If a module source appears to be a relative local path but does not have a preceding ./, print a #TODO message for the user. * internal/initwd: limit go-getter detectors to those supported by terraform * internal/initwd: move isMaybeRelativeLocalPath check into getWithGoGetter To avoid making two calls to getter.Detect, which potentially makes non-trivial API calls, the "isMaybeRelativeLocalPath" check was moved to a later step and a custom error type was added so user-friendly diagnostics could be displayed in the event that a possible relative local path was detected.	7 years ago
Martin Atkins	b217624d83	config/configupgrade: Test to show that list unwrapping works for sets ... This was already working but we had no tests to prove it.	7 years ago
Martin Atkins	dd43926761	configs/configupgrade: Fix up uses of the .count pseudo-attribute ... Terraform 0.11 and prior had an odd special case where a resource attribute access for "count" would be resolved as the count for the whole resource, rather than as an attribute of an individual instance as for all other attributes. Because Terraform 0.12 makes test_instance.foo appear as a list when count is set (so it can be used in other expressions), it's no longer possible to have an attribute in that position: lists don't have attributes. Fortunately we don't really need that special case anymore since it doesn't do anything we can't now do with the length(...) function. This upgrade rule, then, detects references like test_instance.foo.count and rewrites to length(test_instance.foo). As a special case, if test_instance.foo doesn't have "count" set then it just rewrites as the constant 1, which mimics what would've happened in that case in Terraform 0.11.	7 years ago
Martin Atkins	966eb39427	configs/configupgrade: Default arguments in "connection" blocks ... Prior to Terraform v0.12 it was possible for a provider to secretly set some default arguments for the "connection" block, which most commonly included a hard-coded type of "ssh" and a value from "host". In the interests of "explicit is better than implicit", Terraform 0.12 no longer has this feature and instead requires connection settings to be written explicitly in terms of the resource's exported attributes. For compatibility though, the upgrade tool will insert expressions that are as close as possible to the logic the provider formerly implemented, or in a few rare cases a TF-UPGRADE-TODO comment to fix it up manually. Some of the existing resource type implementations have incredibly complicated implementations of selecting a single host IP address to use and don't expose the result of that as an attribute, so for now we handle those via a complicated Terraform language expression achieving the same result. Ideally these providers would introduce a new attribute that exports the same address formerly exported as the hostname before their initial v0.12-compatible release, in which case we can simplify these to just reference the attribute in question. That would be preferable also because it would allow use of that exported attribute in other contexts, such as in a null_resource provisioner somewhere else or in an output to allow a caller to deal with the SSH part itself.	7 years ago
Martin Atkins	ac6e0e42dc	configs/configupgrade: Upgrade the bodies of "connection" blocks ... This uses the fixed "superset" schema from the main terraform package to apply our standard expression mapping, with the exception of "type" where interpolation sequences are not supported due to the type being evaluated early to retrieve the schema for decoding the rest.	7 years ago
Martin Atkins	e2ef51800a	configs/configupgrade: Upgrade the bodies of "provisioner" blocks ... Aside from the two special meta-arguments "connection" and "provisioner" this is just our standard mapping from schema to conversion rules, using the provisioner's configuration schema.	7 years ago
Martin Atkins	cdca8fbfe8	configs/configupgrade: Correct ignore_changes error message ... Due to a copy-paste error, this was using the message from the providers map in a "module" block. This new message is not particularly helpful, but we should only see it for a configuration that wouldn't have been valid in 0.11 either, and so it's unlikely to be displayed.	7 years ago
Martin Atkins	0095a944cd	configs/configupgrade: Include provisioner schemas in analysis ... We'll need these to migrate any "provisioner" blocks we find in the input configuration.	7 years ago
Martin Atkins	fa0d6484df	configs/configupgrade: Detect and fix element(...) usage with sets ... Although sets do not have indexed elements, in Terraform 0.11 and earlier element(...) would work with sets because we'd automatically convert them to lists on entry to HIL -- with an arbitrary-but-consistent ordering -- and this return an arbitrary-but-consistent element from the list. The element(...) function in Terraform 0.12 does not allow this because it is not safe in general, but there was an existing pattern relying on this in Terraform 0.11 configs which this upgrade rule is intended to preserve: resource "example" "example" { count = "${length(any_set_attribute)}" foo = "${element(any_set_attribute, count.index}" } The above works because the exact indices assigned in the conversion are irrelevant: we're just asking Terraform to create one resource for each distinct element in the set. This upgrade rule therefore inserts an explicit conversion to list if it is able to successfully provide that the given expression will return a set type: foo = "${element(tolist(any_set_attribute), count.index}" This makes the conversion explicit, allowing users to decide if it is safe and rework the configuration if not. Since our static type analysis functionality focuses mainly on resource type attributes, in practice this rule will only apply when the given expression is a statically-checkable resource reference. Since sets are an SDK-only concept in Terraform 0.11 and earlier anyway, in practice that works out just right: it's not possible for sets to appear anywhere else in older versions anyway.	7 years ago
Martin Atkins	085ac6d8ca	configs/configupgrade: Test for removing commas between block arguments ... The comma-separated syntax is now reserved only for object constructor expressions in attribute values, so the upgrade tool rewrites block arguments to be newline-separated instead. This was already working but we didn't have an explicit test for it until now.	7 years ago
Martin Atkins	54bb0b1e25	configs/configupgrade: Silently ignore and trim .% .# in ignore_changes ... Prior to Terraform 0.12, ignore_changes was implemented in a flatmap-oriented fashion and so users found that they could (and in fact, were often forced to) use the internal .% and .# suffixes flatmap uses to ignore changes to the number of elements in a list or map. Terraform 0.12 no longer uses that representation, so we'll interpret ignoring changes to the length as ignoring changes to the entire collection. While this is not a totally-equivalent change, in practice this pattern was most often used in conjunction with specific keys from a map in order to _effectively_ ignore the entire map, even though Terraform didn't really support that.	7 years ago
Martin Atkins	1d35233a03	configs/configupgrade: Fix up internal HIL conversion functions ... HIL implemented its type conversions by rewriting its AST to include calls to some undocumented builtin functions. Unfortunately those functions were still explicitly callable if you could figure out the name for them, and so they may have been used in the wild. In particular, __builtin_StringToFloat was used as part of a workaround for a HIL design flaw where it would prefer to convert strings to integers rather than floats when performing arithmetic operations. This issue was, indeed, the main reason for unifying int ant float into a single number type in HCL. Since we published that as a suggested workaround, the upgrade tool ought to fix it up. The other cases have never been documented as a workaround, so they are less likely to appear in the wild, but we might as well fix them up anyway since we already have the conversion functions required to get the same result in the new language. To be safe/conservative, most of these convert to _two_ function calls rather than just one, which ensures that these new expressions retain the behavior of implicitly converting to the source type before running the conversion. The new conversion functions only specify target type, and so cannot guarantee identical results if the argument type does not exactly match what was previously given as the parameter type in HIL.	7 years ago
Martin Atkins	154911688a	configs/configupgrade: upgrade expressions inside heredocs ... HEREDOC tokens are a little more fussy than normal string sequences because we need to preserve the whitespace within them along with the start and end markers while we upgrade any interpolated expressions inside. We need to do some work locally here because the HCL heredoc processing "does too much" and throws away information we need to do a faithful upgrade. We also need to contend with the fact that Terraform <=0.11 had an older version of HCL that accidentally permitted a degenerate form of heredoc where the marker was at the end of the final line, like this: degenerate = <<EOT this should never have workedEOT When we migrate this, we'll introduce the additional newline that is now required, which will unfortunately slightly change the result string to include a newline when parsed by 0.12, and so we'll need to call this out as a caveat in the upgrade guide.	7 years ago
Kristin Laemmert	5f8916b4fd	configs/configupgrade: do not panic on HEREDOCs. (#20281) ... Previously, configupgrade would panic if it encountered a HEREDOC. For the time being, we will simply print out the HEREDOC as-is. Unfortunately, we discovered that terraform 0.11's version of HCL allowed for HEREDOCs with the termination delimiter inline (instead of on a newline, which is technically correct). Since 0.12configupgrade needs to be bug-compatible with terraform 0.11, we must roll back to the same version of HCL used in terraform 0.11.	7 years ago
Martin Atkins	954d38e870	lang: New file-hashing functions ... In prior versions, we recommended using hash functions in conjunction with the file function as an idiom for detecting changes to upstream blobs without fetching and comparing the whole blob. That approach relied on us being able to return raw binary data from file(...). Since Terraform strings pass through intermediate representations that are not binary-safe (e.g. the JSON state), there was a risk of string corruption in prior versions which we have avoided for 0.12 by requiring that file(...) be used only with UTF-8 text files. The specific case of returning a string and immediately passing it into another function was not actually subject to that corruption risk, since the HIL interpreter would just pass the string through verbatim, but this is still now forbidden as a result of the stricter handling of file(...). To avoid breaking these use-cases, here we introduce variants of the hash functions a with "file" prefix that take a filename for a disk file to hash rather than hashing the given string directly. The configuration upgrade tool also now includes a rule to detect the documented idiom and rewrite it into a single function call for one of these new functions. This does cause a bit of function sprawl, but that seems preferable to introducing more complex rules for when file(...) can and cannot read binary files, making the behavior of these various functions easier to understand in isolation.	7 years ago
Martin Atkins	f93f7e5b5c	configs/configupgrade: Remove redundant list brackets ... In early versions of Terraform where the interpolation language didn't have any real list support, list brackets around a single string was the signal to split the string on a special uuid separator to produce a list just in time for processing, giving expressions like this: foo = ["${test_instance.foo.*.id}"] Logically this is weird because it looks like it should produce a list of lists of strings. When we added real list support in Terraform 0.7 we retained support for this behavior by trimming off extra levels of list during evaluation, and inadvertently continued relying on this notation for correct type checking. During the Terraform 0.10 line we fixed the type checker bugs (a few remaining issues notwithstanding) so that it was finally possible to use the more intuitive form: foo = "${test_instance.foo.*.id}" ...but we continued trimming off extra levels of list for backward compatibility. Terraform 0.12 finally removes that compatibility shim, causing redundant list brackets to be interpreted as a list of lists. This upgrade rule attempts to identify situations that are relying on the old compatibility behavior and trim off the redundant extra brackets. It's not possible to do this fully-generally using only static analysis, but we can gather enough information through or partial type inference mechanism here to deal with the most common situations automatically and produce a TF-UPGRADE-TODO comment for more complex scenarios where the user intent isn't decidable with only static analysis. In particular, this handles by far the most common situation of wrapping list brackets around a splat expression like the first example above. After this and the other upgrade rules are applied, the first example above will become: foo = test_instance.foo.*.id	7 years ago
Martin Atkins	d9603d5bc5	configs/configupgrade: Do type inference with input variables ... By collecting information about the input variables during analysis, we can return approximate type information for any references to those variables in expressions. Since Terraform 0.11 allowed maps of maps and lists of lists in certain circumstances even though this was documented as forbidden, we conservatively return collection types whose element types are unknown here, which allows us to do shallow inference on them but will cause us to get an incomplete result if any operations are performed on elements of the list or map value.	7 years ago
Martin Atkins	a2d9634dbf	configs/configupgrade: Expression type inference ... Although we can't do fully-precise type inference with access only to a single module's configuration, we can do some approximate inference using some clues within the module along with our resource type schemas. This depends on HCL's ability to pass through type information even if the input values are unknown, mapping our partial input type information into partial output type information by evaluating the same expressions. This will allow us to do some upgrades that require dynamic analysis to fully decide, by giving us three outcomes: needed, not needed, or unknown. If it's unknown then that'll be our prompt to emit a warning for the user to make a decision.	7 years ago
Martin Atkins	8112f589c1	configs/configupgrade: Pass through connection and provisioner blocks ... This is a temporary implementation of these rules just so that these can be passed through verbatim (rather than generating an error) while we do testing of other features. A subsequent commit will finish these with their own custom rulesets.	7 years ago
Martin Atkins	028b5ba34e	configs/configupgrade: Upgrade depends_on in resources and outputs	7 years ago
Martin Atkins	ef017345f1	configs/configupgrade: Upgrade the resource "lifecycle" nested block ... The main tricky thing here is ignore_changes, which contains strings that are better given as naked traversals in 0.12. We also handle here mapping the old special case ["*"] value to the new "all" keyword.	7 years ago
Martin Atkins	4b52148262	configs/configupgrade: Upgrade provider addresses ... Both resource blocks and module blocks contain references to providers that are expressed as short-form provider addresses ("aws.foo" rather than "provider.aws.foo"). These rules call for those to be unwrapped as naked identifiers during upgrade, rather than appearing as quoted strings. This also introduces some further rules for other simpler meta-arguments that are required for the test fixtures for this feature.	7 years ago
Martin Atkins	ea3b8b364c	configs/configupgrade: Initial passthrough mapping for module blocks ... Some further rules are required here to deal with the meta-arguments we accept inside these blocks, but this is good enough to pass through most module blocks using the standard attribute-expression-based mapping.	7 years ago
Martin Atkins	4b5d31d35d	configs/configupgrade: Rules-based upgrade for "locals" block ... Previously we were handling this one as a special case, effectively duplicating most of the logic from upgradeBlockBody. By doing some prior analysis of the block we can produce a "rules" that just passes through all of the attributes as-is, allowing us to reuse upgradeBlockBody. This is a little weird for the locals block since everything in it is user-selected names, but this facility will also be useful in a future commit for dealing with module blocks, which contain a mixture of user-chosen and reserved argument names.	7 years ago
Martin Atkins	76221a3a4a	configs/configupgrade: Retain any .tf.json files unchanged ... We don't change JSON files at all and instead just emit a warning about them since JSON files are usually generated rather than hand-written and so any updates need to happen in the generator program rather than in its output. However, we do still need to copy them verbatim into the output map so that we can keep track of them through any subsequent steps.	7 years ago
Martin Atkins	8490fc36f7	configs/configupgrade: Fix up references to counted/uncounted resources ... Prior to v0.12 Terraform was liberal about these and allowed them to mismatch, but now it's important to get this right so that resources and resource instances can be used directly as object values, and so we'll fix up any sloppy existing references so things keep working as expected. This is particularly important for the pattern of using count to create conditional resources, since previously the "true" case would create one instance and Terraform would accept an unindexed reference to that.	7 years ago
Martin Atkins	ceedeb69a9	configs/configupgrade: Normalize and upgrade reference expressions ... The reference syntax is not significantly changed, but there are some minor additional restrictions on identifiers in HCL2 and as a special case we need to rewrite references to data.terraform_remote_state . Along with those mandatory upgrades, we will also switch references to using normal index syntax where it's safe to do so, as part of de-emphasizing the old strange integer attribute syntax (like foo.0.bar).	7 years ago
Martin Atkins	e83976c008	configs/configupgrade: Print trailing comments inside blocks ... Previously we were erroneously moving these out of their original block into the surrounding body. Now we'll make sure to collect up any remaining ad-hoc comments inside a nested block body before closing it.	7 years ago
Martin Atkins	de0eb9ea30	configs/configupgrade: Distinguish data resources in analysis ... Early on it looked like we wouldn't need to distinguish these since we were only analyzing for provider types, but we're now leaning directly on the resource addresses later on and so we need to make sure we produce valid ones when data resources are present.	7 years ago
Martin Atkins	6596d031d9	configs/configupgrade: Convert block-as-attr to dynamic blocks ... Users discovered that they could exploit some missing validation in Terraform v0.11 and prior to treat block types as if they were attributes and assign dynamic expressions to them, with some significant caveats and gotchas resulting from the fact that this was never intended to work. However, since such patterns are in use in the wild we'll convert them to a dynamic block during upgrade. With only static analysis we must unfortunately generate a very conservative, ugly dynamic block with every possible argument set. Users ought to then clean up the generated configuration after confirming which arguments are actually required.	7 years ago
Martin Atkins	f96d702d4f	configs/configupgrade: Upgrading of simple nested blocks	7 years ago
Martin Atkins	39c3e7112f	configs/configupgrade: Use break to cancel default function rendering ... We're using break elsewhere in here so it was weird to have a small set of situations that return instead, which could then cause confusion for future maintenance if a reader doesn't notice that control doesn't always leave the outer switch statement.	7 years ago
Martin Atkins	48f1245e6b	configs/configupgrade: Replace lookup(...) with index syntax ... If lookup is being used with only two arguments then it is equivalent to index syntax and more readable that way, so we'll replace it. Ideally we'd do similarly for element(...) here but sadly we cannot because we can't prove in static analysis that the user is not relying on the modulo wraparound behavior of that function.	7 years ago
Martin Atkins	4927a4105b	configs/configupgrade: Replace calls to list() and map() ... We now have native language features for declaring tuples and objects, which are the idiomatic way to construct sequence and mapping values that can then be converted to list, set, and map types as needed.	7 years ago
Martin Atkins	8cf024d45a	configs/configupgrade: Upgrade expressions nested in HCL list/object ... In the old world, lists and maps could be created either using functions in HIL or list/object constructs in HCL. Here we ensure that in the HCL case we'll apply any required expression transformations to the individual items within HCL's compound constructs.	7 years ago
Martin Atkins	e49d993d89	configs/configupgrade: Decide on blank lines by ends of items ... Previously we were using the line count difference between the start of one item and the next to decide whether to insert a blank line between two items, but that is incorrect for multi-line items. Instead, we'll now count the difference from the final line of the previous item to the first line of the next, as best we can with the limited position info recorded by the HCL1 parser.	7 years ago
Martin Atkins	bdb724562c	configs/configupgrade: Test that map attrs as blocks are fixed ... The old parser was forgiving in allowing the use of block syntax where a map attribute was expected, but the new parser is not (in order to allow for dynamic map keys, for expressions, etc) and so the upgrade tool must fix these to use attribute syntax.	7 years ago
Martin Atkins	1aa368d0d8	configs/configupgrade: Add some logging and enable it for tests	7 years ago
Martin Atkins	e8999eefdc	configs/configupgrade: Populate the test provider schema ... This will allow us to test some schema-sensitive migration rules.	7 years ago
Martin Atkins	8e594f32aa	configs/configupgrade: Upgrade rules for the "terraform" block type ... This includes the backend configuration, so we need to load the requested backend and migrate the included configuration per that schema.	7 years ago
Martin Atkins	302b29557f	configs/configupgrade: Pass through diagnostics from body upgrades	7 years ago
Martin Atkins	c755745285	configs/configupgrade: Generalize migration of block bodies ... The main area of interest in upgrading is dealing with special cases for individual block items, so this generalization allows us to use the same overall body-processing logic for everything but to specialize just how individual items are dealt with, which we match by their names as given in the original input source code.	7 years ago
Martin Atkins	cf52e224f6	configs/configupgrade: Basic migration of provider blocks ... This involved some refactoring of how block bodies are migrated, which still needs some additional work to deal with meta-arguments but is now at least partially generalized to support both resource and provider blocks.	7 years ago
Martin Atkins	1a654e9e1c	configs/configupgrade: Disable the tests for now ... The tests in here are illustrating that this package is not yet finished, but we plan to run a release before we finish this and so we'll skip those tests for now with the intent of reinstating this again once we return to finish this up.	8 years ago
Martin Atkins	961056c08d	configs/configupgrade: Use mock provider instead of test provider ... The test provider comes with a lot of baggage since it's designed to be used as a plugin, so instead we'll just use the mock provider implementation directly, and so we can (in a later commit) configure it appropriately for what our tests need here.	8 years ago
Martin Atkins	85aa8769db	configs/configupgrade: Partially fix TestUpgradeValid ... This is still not compileable because the test provider needs to be updated to the new provider interface, but all the rest of the types are now correct so we can update the test provider in a later commit to make this work again.	8 years ago
Martin Atkins	44bc7519a6	terraform: More wiring in of new provider types ... This doesn't actually work yet, but it builds and then panics in a pretty satisfying way.	8 years ago
Martin Atkins	479c6b2466	move "configschema" from "config" to "configs" ... The "config" package is no longer used and will be removed as part of the 0.12 release cleanup. Since configschema is part of the "new world" of configuration modelling, it makes more sense for it to live as a subdirectory of the newer "configs" package.	8 years ago
Martin Atkins	adb88eaa16	configupgrade: Analysis of input configuration ... In order to properly migrate the contents of resource, data, provider and provisioner blocks we will need the provider's schema in order to understand what is expected, so we can resolve some ambiguities inherent in the legacy HCL AST. This includes an initial prototype of migrating the content of resource blocks just to verify that the information is being gathered correctly. As with the rest of the upgrade_native.go file, this will be reorganized significantly once the basic end-to-end flow is established and we can see how to organize this code better.	8 years ago
Martin Atkins	ffe5f7c4e6	command: 0.12upgrade command ... This is the frontend to the work-in-progress codepath for upgrading the source code for a module written for Terraform v0.11 or earlier to use the new syntax and idiom of v0.12. The underlying upgrade code is not yet complete as of this commit, and so the command is not yet very useful. We will continue to iterate on the upgrade code in subsequent commits.	8 years ago