236 Commits (073fd0d183a6fa68965bdfcedbf459e2fbc78f8d)

Author	SHA1	Message	Date
James Bardin	a286ef6d9c	remove depends_on from helper test shims	6 years ago
Kristin Laemmert	c4873778c8	Mildwonkey/tests (#24548) ... * helper/resource: remove provider resolver test * repl tests passing * helper/resource: add some extra shimming to ShimLegacyState Some of the tests in helper/resource have to shim between legacy and modern states. Terraform expects modern states to have the Provider set for each resource (and not be a legacy provider). This PR adds a wrapper around terraform.ShimLegacyState which iterates through the resources in the state (after the shim) and adds the provider FQN.	6 years ago
Martin Atkins	549aede792	Remove terraform.ResourceProvider, use providercache.Installer instead ... Back when we first introduced provider versioning in Terraform 0.10, we did the provider version resolution in terraform.NewContext because we weren't sure yet how exactly our versioning model was going to play out (whether different versions could be selected per provider configuration, for example) and because we were building around the limitations of our existing filesystem-based plugin discovery model. However, the new installer codepath is new able to do all of the selections up front during installation, so we don't need such a heavy inversion of control abstraction to get this done: the command package can select the exact provider versions and pass their factories directly to terraform.NewContext as a simple static map. The result of this commit is that CLI commands other than "init" are now able to consume the local cache directory and selections produced by the installation process in "terraform init", passing all of the selected providers down to the terraform.NewContext function for use in implementing the main operations. This commit is just enough to get the providers passing into the terraform.Context. There's still plenty more to do here, including to repair all of the tests this change has additionally broken.	6 years ago
Kristin Laemmert	c8d64846ad	Mildwonkey/ps import (#24412) ... * import: remove Config from ImportOpts `Config` in ImportOpts was any provider configuration provided by the user on the command line. This option has already been removed in favor of only taking the provider from the configuration loaded in the current context. * terrafrom: add Config to ImportStateTransformer and refactor Transform to get the resource provider FQN from the Config	6 years ago
James Bardin	a8b9547e0d	fixup states.Resource change throughout packages	6 years ago
James Bardin	e13eecbc5b	finish provider ModuleInstance replacement	6 years ago
Kristin Laemmert	47a16b0937	addrs: embed Provider in AbsProviderConfig instead of Type ... a large refactor to addrs.AbsProviderConfig, embedding the addrs.Provider instead of a Type string. I've added and updated tests, added some Legacy functions to support older state formats and shims, and added a normalization step when reading v4 (current) state files (not the added tests under states/statefile/roundtrip which work with both current and legacy-style AbsProviderConfig strings). The remaining 'fixme' and 'todo' comments are mostly going to be addressed in a subsequent PR and involve looking up a given local provider config's FQN. This is fine for now as we are only working with default assumption.	6 years ago
Martin Atkins	8b511524d6	Initial steps towards AbsProviderConfig/LocalProviderConfig separation (#23978) ... * Introduce "Local" terminology for non-absolute provider config addresses In a future change AbsProviderConfig and LocalProviderConfig are going to become two entirely distinct types, rather than Abs embedding Local as written here. This naming change is in preparation for that subsequent work, which will also include introducing a new "ProviderConfig" type that is an interface that AbsProviderConfig and LocalProviderConfig both implement. This is intended to be largely just a naming change to get started, so we can deal with all of the messy renaming. However, this did also require a slight change in modeling where the Resource.DefaultProviderConfig method has become Resource.DefaultProvider returning a Provider address directly, because this method doesn't have enough information to construct a true and accurate LocalProviderConfig -- it would need to refer to the configuration to know what this module is calling the provider it has selected. In order to leave a trail to follow for subsequent work, all of the changes here are intended to ensure that remaining work will become obvious via compile-time errors when all of the following changes happen: - The concept of "legacy" provider addresses is removed from the addrs package, including removing addrs.NewLegacyProvider and addrs.Provider.LegacyString. - addrs.AbsProviderConfig stops having addrs.LocalProviderConfig embedded in it and has an addrs.Provider and a string alias directly instead. - The provider-schema-handling parts of Terraform core are updated to work with addrs.Provider to identify providers, rather than legacy strings. In particular, there are still several codepaths here making legacy provider address assumptions (in order to limit the scope of this change) but I've made sure each one is doing something that relies on at least one of the above changes not having been made yet. * addrs: ProviderConfig interface In a (very) few special situations in the main "terraform" package we need to make runtime decisions about whether a provider config is absolute or local. We currently do that by exploiting the fact that AbsProviderConfig has LocalProviderConfig nested inside of it and so in the local case we can just ignore the wrapping AbsProviderConfig and use the embedded value. In a future change we'll be moving away from that embedding and making these two types distinct in order to represent that mapping between them requires consulting a lookup table in the configuration, and so here we introduce a new interface type ProviderConfig that can represent either AbsProviderConfig or LocalProviderConfig decided dynamically at runtime. This also includes the Config.ResolveAbsProviderAddr method that will eventually be responsible for that local-to-absolute translation, so that callers with access to the configuration can normalize to an addrs.AbsProviderConfig given a non-nil addrs.ProviderConfig. That's currently unused because existing callers are still relying on the simplistic structural transform, but we'll switch them over in a later commit. * rename LocalType to LocalName Co-authored-by: Kristin Laemmert <mildwonkey@users.noreply.github.com>	6 years ago
Kristin Laemmert	6541775ce4	addrs: roll back change to Type field in ProviderConfig (#23937)	6 years ago
Kristin Laemmert	e3416124cc	addrs: replace "Type string" with "Type Provider" in ProviderConfig ... * huge change to weave new addrs.Provider into addrs.ProviderConfig * terraform: do not include an empty string in the returned Providers / Provisioners - Fixed a minor bug where results included an extra empty string	6 years ago
Martin Atkins	30bf83cdeb	helper/logging: Bring the LevelFilter into our own codebase ... In order to make this work reasonably we can't avoid using some funny heuristics, which are somewhat reasonable to apply within the context of Terraform itself but would not be good to add to the general "logutils". Specifically, this is adding the additional heuristic that lines starting with spaces are continuation lines and so should inherit the log level of the most recent non-continuation line.	6 years ago
Kristin Laemmert	9891d0354a	providers: use addrs.Provider as map keys for provider.Factory (#23548) ... * terraform/context: use new addrs.Provider as map key in provider factories * added NewLegacyProviderType and LegacyString funcs to make it explicit that these are temporary placeholders This PR introduces a new concept, provider fully-qualified name (FQN), encapsulated by the `addrs.Provider` struct.	6 years ago
James Bardin	16b368e292	only need to handle depends_on to shim old state	7 years ago
Martin Atkins	39e609d5fd	vendor: switch to HCL 2.0 in the HCL repository ... Previously we were using the experimental HCL 2 repository, but now we'll shift over to the v2 import path within the main HCL repository as part of actually releasing HCL 2.0 as stable. This is a mechanical search/replace to the new import paths. It also switches to the v2.0.0 release of HCL, which includes some new code that Terraform didn't previously have but should not change any behavior that matters for Terraform's purposes. For the moment the experimental HCL2 repository is still an indirect dependency via terraform-config-inspect, so it remains in our go.sum and vendor directories for the moment. Because terraform-config-inspect uses a much smaller subset of the HCL2 functionality, this does still manage to prune the vendor directory a little. A subsequent release of terraform-config-inspect should allow us to completely remove that old repository in a future commit.	7 years ago
James Bardin	a3c374b227	don't lose track of private data in the ACC tests ... If the resource schema version is > 0, the metadata containing the timeout values was overwritten during acceptance tests.	7 years ago
Radek Simko	a08ad08169	Merge pull request #18277 from ndmckinley/patch-1 ... helper/resource: Minor docs tweak on ImportStateVerify	7 years ago
Alex Pilon	4bf43efcfd	move hcl2shim package to configs	7 years ago
Alex Pilon	83aa07f907	prune NewResourceConfig and update tests	7 years ago
Alex Pilon	7f8f198719	remove UnknownVariabeValue from config and update references to shim	7 years ago
James Bardin	4cb6ebe22c	maintain private data through provider ACC tests ... The private->meta data was lost in the test harness.	7 years ago
James Bardin	9a2a6d14bd	Stop grpc server when running ACC tests ... The grpc server does not shutdown when the listener is closed. Since tests aren't run through go-plugin, which has a separate RPC Shutdown channel to stop the server, we need to track and stop the server directly.	7 years ago
Martin Atkins	bd1a215580	helper/resource: Ignore Removed attributes for ImportStateVerify ... Due to the lossiness of our legacy models for diff and state, shimming a diff and then creating a state from it produces a different result than shimming a state directly. That means that ImportStateVerify no longer works as expected if there are any Computed attributes in the schema where d.Set isn't called during Read. Fixing that for every case would require some risky changes to the shim behavior, so we're instead going to ask provider developers to address it by adding `d.Set` calls where needed, since that is the contract for "Computed" anyway -- a default value should be produced during Create, and thus by extension during Import. However, since a common situation where this occurs is attributes marked as "Removed", since all of the code that deals with them has generally been deleted, we'll avoid problems in that case here by treating Removed attributes as ignored for the purposes of ImportStateVerify. This required exporting some functionality that was formerly unexported in helper/schema, but it's a relatively harmless schema introspection function so shouldn't be a big deal to export it.	7 years ago
James Bardin	1a9c06d0f5	Revert "helper/schema: Implementation of the AsSingle mechanism" ... This reverts commit 1987a92386.	7 years ago
James Bardin	bb62aba651	add (forces new resource) to provider test diffs ... Add the (forces new resource) annotation to the diff output for provider tests failures when we can. This helps providers narrow down what might be triggering changes when encountering test failures with the new SDK.	7 years ago
Justin Downing	1e32ae243c	grammatical updates to comments and docs (#20195)	7 years ago
Martin Atkins	35df450dc0	helper/resource: Preserve provider address when shimming to old state ... It's important to preserve the provider address because during the destroy phase of provider tests we'll use the references in the state to determine which providers are required, and so without this attempts to override the provider using the "provider" meta-argument can cause failures at destroy time when the wrong provider gets selected. (This is particularly acute in the google-beta provider tests because that provider is _always_ used with provider = "google-beta" to override the default behavior of using the normal "google" provider.)	7 years ago
Martin Atkins	1987a92386	helper/schema: Implementation of the AsSingle mechanism ... The previous commit added a new flag to schema.Schema which is documented to make a list with MaxItems: 1 be presented to Terraform Core as a single value instead, giving a way to switch to non-list nested resources without it being a breaking change for Terraform v0.11 users as long as it's done prior to a provider's first v0.12-compatible release. This is the implementation of that mechanism. It's intentionally implemented as a suite of extra fixups rather than direct modifications to existing shim code because we want to ensure that this has no effect whatsoever on the result of a resource type that _isn't_ using AsSingle. Although there is some small unit test coverage of the fixup steps here, the primary testing for this is in the test provider since the integration of all of these fixup steps in the correct order is the more important result than any of the intermediate fixup steps.	7 years ago
James Bardin	c02f1d7256	allow 0 and unset to be equal in count tests ... This was changed in the single attribute test cases, but the AttrPair test is used a lot for data source. As far as tests are concerned, 0 and unset should be treated equally for flatmapped collections.	7 years ago
Martin Atkins	4c99864dad	helper/resource: TestCheckResourceAttrPair allow nonexist ... This checking helper is frequently used in provider tests for data sources, as a shorthand to verify that an attribute of the data source matches with the corresponding attribute on a managed resource. Since we now leave empty collections null in more cases, this function is sometimes effectively asked to verify that a given attribute is _unset_ in both the data source and the resource, so here we slightly adjust the definition of the check to consider two nulls to be equal to one another, which at this layer manifests as the keys not being present in the state attributes map at all. This check function didn't previously have tests, so this commit also adds a basic suite of tests, including coverage for the new behavior.	7 years ago
Martin Atkins	477da57a92	helper/plugin: Honor resource type overrides in import ... One quirky aspect of our import feature is that we allow the importer to produce additional resources alongside the one that was imported, such as to create separate rules for each rule of an imported security group. Providers need to be able to set the types of these other resources since they may not match the "main" resource type. They do this by calling ResourceData.SetType, which in turn sets InstanceState.Ephemeral.Type. In our shims here we therefore need to copy that out into our new TypeName field so that the new core import code can see it and create the right type in the state. Testing this required a minor change to the test harness to allow the ImportStateCheck function to see the resource type.	7 years ago
Martin Atkins	86c02d5c35	command: "terraform init" can partially initialize for 0.12upgrade ... There are a few constructs from 0.11 and prior that cause 0.12 parsing to fail altogether, which previously created a chicken/egg problem because we need to install the providers in order to run "terraform 0.12upgrade" and thus fix the problem. This changes "terraform init" to use the new "early configuration" loader for module and provider installation. This is built on the more permissive parser in the terraform-config-inspect package, and so it allows us to read out the top-level blocks from the configuration while accepting legacy HCL syntax. In the long run this will let us do version compatibility detection before attempting a "real" config load, giving us better error messages for any future syntax additions, but in the short term the key thing is that it allows us to install the dependencies even if the configuration isn't fully valid. Because backend init still requires full configuration, this introduces a new mode of terraform init where it detects heuristically if it seems like we need to do a configuration upgrade and does a partial init if so, before finally directing the user to run "terraform 0.12upgrade" before running any other commands. The heuristic here is based on two assumptions: - If the "early" loader finds no errors but the normal loader does, the configuration is likely to be valid for Terraform 0.11 but not 0.12. - If there's already a version constraint in the configuration that excludes Terraform versions prior to v0.12 then the configuration is probably _already_ upgraded and so it's just a normal syntax error, even if the early loader didn't detect it. Once the upgrade process is removed in 0.13.0 (users will be required to go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can be simplified to remove that special mode, but the idea of doing the dependency version checks against the liberal parser will remain valuable to increase our chances of reporting version-based incompatibilities rather than syntax errors as we add new features in future.	7 years ago
Martin Atkins	0c0a437bcb	Move module install functionality over to internal/initwd	7 years ago
James Bardin	041ed67e46	type names don't imply the resource mode ... The addr type doesn't imply the resource mode, so data sources and managed resources with the same type name could shim incorrectly.	7 years ago
James Bardin	a7b399cb4c	use actual schema.Resources for state shims ... Provider tests often rely on checking values contained within sets, by directly accessing their flatmapped representation. In order to provider the test harness with the expected set hashes, the sets must be generated by the schema.Resource itself. During the test we now build a fixed map of the providers, which should only contain schema.Provider instances, and pass them into each TestStep. The individual schema.Resource instances can then be pulled from the providers, and used to recreate the state from the cty.Value returned by the core operations.	7 years ago
James Bardin	7973872524	allow TestCheckNoResourceAttr for empty containers ... Stricter type handling in the new shims may add empty containers into the state where they were previously elided. Since the detection of missing and empty containers in the legacy state was never reliable, allow TestCheckNoResourceAttr to succeed if the key is a container count index, and the value is "0"	7 years ago
James Bardin	c63040c737	have TestCheckResourceAttr accept missing counts ... Missing containers were often erroneously kept in the state, but since the addition of the new provider shims, they can often be correctly eliminated. There are however many tests that check for a "0" count in the flatmap state when there shouldn't be a key at all. This addition looks for a container count key and "0" pair, and allows for the key to be missing. There may be some tests negatively effected by this which were legitimately checking for empty containers, but those were also not reliably detected, and there should be much fewer tests involved.	7 years ago
Martin Atkins	cdad78d69b	helper/resource: Allow multiple providers in a single TestCase ... Due to incorrect use of a loop iterator variable inside a closure, all of the given providers were ending up with the same factory function. Now we copy the factory function to a local within the loop first so that each iteration has its own variable. This is the second round of similar bugs in this function, so we'll also add a test case for it to reduce the risk of future regressions given that most real callers don't exercise this with multiple providers in practice.	7 years ago
Martin Atkins	b190d3b4f2	helper/resource: Shim back to old state must preserve schema version ... We use a shim to convert from the new state model back to the old because the provider test API is still using the old API throughout. However, the shim was not preserving the schema version recorded in the new-style state and so a round-trip through this shim would cause the schema versions to all revert to zero. This can cause trouble with the destroy phase of provider tests because (for API legacy reasons) we round-trip from old state back to new again before the destroy phase and thus causing the providers to try to upgrade from state version zero even though the data was already latest, which can cause errors because state upgrades are generally not idempotent.	7 years ago
Martin Atkins	f9fef56167	helper/resource: print full diagnostics for operation errors in tests ... This causes the output to include additional helpful context such as the values of variables referenced in the config, etc. The output is in the same format as normal Terraform CLI error output, though we don't retain a source code cache in this codepath so it will not include a source code snippet.	7 years ago
Martin Atkins	55469cd416	helper/resource: Get schemas from Terraform context ... Previously the test harness was preloading schemas from the providers before running any test steps. Since terraform.NewContext already deals with loading provider schemas, we can instead just use the schemas it loaded for our shimming needs, avoiding the need to reimplement the schema lookup behavior and thus the need to create a throwaway provider instance with which to do it.	7 years ago
Martin Atkins	a4991c5780	helper/resource: Create a separate provider instance each call ... Previously we were running the factory function only once when constructing the provider resolver, which means that all contexts created from that resolver share the same provider instance. Instead now we will call the given factory function once for each instantiation, ensuring that each caller ends up with a separate object as would be the case in real-world use.	7 years ago
James Bardin	98870fadb1	Merge pull request #19544 from hashicorp/jbardin/import-tests ... Fix provider import tests	7 years ago
James Bardin	7d296f752c	don't add numeric indexes to resources with a count of 0	7 years ago
James Bardin	484d67028a	handle shim errors in provider tests ... These should be rare, and even though it's likely a shim bug, the error is probably easier for provider developers to deal with than the panic	7 years ago
James Bardin	547d63bcde	remove empty flatmap containers from test states ... Don't compare attributes with zero-length flatmap continers in tests.	7 years ago
Martin Atkins	4fe9632f09	plugin: Establish our current plugin protocol as version 5 ... The main significant change here is that the package name for the proto definition is "tfplugin5", which is important because this name is part of the wire protocol for references to types defined in our package. Along with that, we also move the generated package into "internal" to make it explicit that importing the generated Go package from elsewhere is not the right approach for externally-implemented SDKs, which should instead vendor the proto definition they are using and generate their own stubs to ensure that the wire protocol is the only hard dependency between Terraform Core and plugins. After this is merged, any provider binaries built against our helper/schema package will need to be rebuilt so that they use the new "tfplugin5" package name instead of "proto". In a future commit we will include more elaborate and organized documentation on how an external codebase might make use of our RPC interface definition to implement an SDK, but the primary concern here is to ensure we have the right wire package name before release.	8 years ago
James Bardin	3716db3865	Merge pull request #19384 from hashicorp/jbardin/nested-sets ... New Attribute and Diff handling in shims	8 years ago
James Bardin	ce5d7ff6d0	spelling	8 years ago
Dana Hoffman	7edbb3c8bf	return state even if cfg is invalid	8 years ago
James Bardin	a5ef403dfd	skip resource tests for now ... These aren't going to be fixed in the immediate future, and are preventing the CI tests from being helpful.	8 years ago