173 Commits (6440c839474f052d8997bb5cec19c2743e5108e6)

Author	SHA1	Message	Date
Martin Atkins	72e279e6b2	providers: Consistently use int64 for schema versions ... The rest of Terraform is still using uint64 for this in various spots, but we'll update that gradually later. We use int64 here because that matches what's used in our protobuf definition, and unsigned integers are not portable across all of the protobuf target languages anyway.	7 years ago
findkim	5e06e39fcc	Use registry alias to fetch providers	7 years ago
Justin Campbell	06825bf46d	plugin/discovery: Add friendly gpg err msg ... When GPG verification fails, display a helpful message to the user instead of the generic openpgp error.	7 years ago
Justin Campbell	9a8a74b9bb	plugin/discovery: Print name before verification ... This is so that any errors output from the checksum/signature verification show up in the expected place in the output.	7 years ago
Justin Campbell	495826444b	plugin/discovery: Use GPG keys from Registry ... When verifying the signature of the SHA256SUMS file, we have been hardcoding HashiCorp's public GPG key and using it as the keyring. Going forward, Terraform will get a list of valid public keys for a provider from the Terraform Registry (registry.terraform.io), and use them as the keyring for the openpgp verification func.	7 years ago
Martin Atkins	4fe9632f09	plugin: Establish our current plugin protocol as version 5 ... The main significant change here is that the package name for the proto definition is "tfplugin5", which is important because this name is part of the wire protocol for references to types defined in our package. Along with that, we also move the generated package into "internal" to make it explicit that importing the generated Go package from elsewhere is not the right approach for externally-implemented SDKs, which should instead vendor the proto definition they are using and generate their own stubs to ensure that the wire protocol is the only hard dependency between Terraform Core and plugins. After this is merged, any provider binaries built against our helper/schema package will need to be rebuilt so that they use the new "tfplugin5" package name instead of "proto". In a future commit we will include more elaborate and organized documentation on how an external codebase might make use of our RPC interface definition to implement an SDK, but the primary concern here is to ensure we have the right wire package name before release.	7 years ago
Martin Atkins	ec57927ea3	build: Take protoc out of the "go generate" path ... Since protoc is not go-gettable, and most development tasks in Terraform won't involve recompiling protoc files anyway, we'll use a separate mechanism for these. This way "go generate" only depends on things we can "go get" in the "make tools" target. In a later commit we should also in some way specify a particular version of protoc to use so that we don't get "flapping" regenerations as developers work with different versions, but the priority here is just to make "make generate" minimally usable again to restore the dev workflow documented in the README. This also includes some updates that resulted from running "make generate" and "make protobuf" after those Makefile changes were in place.	7 years ago
James Bardin	3bdd3b2a6e	update plugin client with PrepareProviderConfig	7 years ago
James Bardin	b173037eb3	update mock for PrepareProviderConfig	7 years ago
James Bardin	ba472c7059	Update proto with new PrepareProviderConfig method	7 years ago
James Bardin	3a0c2f1b67	close grpc TestListener	7 years ago
Martin Atkins	e25f79ed28	plugin/convert: Show approximate location context for all provider errors ... Even if a provider doesn't indicate a specific attribute as the cause of a resource operation error, we know the error relates to some aspect of the resource, so we'll include that approximate information in the result so that we don't produce user-hostile error messages with no context whatsoever. Later we can hopefully refine this to place the source range on the header of the configuration block rather than on an empty part of the body, but that'll require some more complex rework here and so for now we'll just accept this as an interim state so that the user can at least figure out which resource block the error is coming from.	7 years ago
Martin Atkins	3eaf2c308d	plugin/discovery: Use mockos_mockarch when testing installation ... If we don't override these then the tests can only pass on one platform.	7 years ago
James Bardin	0d4d572c39	start work on helper/resource test fixtures ... The helper resource tests won't pass for now, as they use a terraform.MockProvider which can't be used in the schema.Provider shims.	7 years ago
James Bardin	52c0032aed	update provisioners for multiple processes ... The "internal" provisioners are still run in a separate process, and need to be updated to restart on each walk.	7 years ago
James Bardin	a14fe7ae0c	missing diagnostics in pugin protocol ... Some call diagnostics were being missed	7 years ago
James Bardin	795161f615	update to start a new process for each plugin ... Modify the plugin factories to create a new plugin process for each individual plugin.	7 years ago
James Bardin	35b375d3ee	check responses for nil values ... While providers shouldn't return nil values, we need to protect against them. Always assign valid null cty values when the response contains a nil.	7 years ago
Martin Atkins	88551181d5	plugin: Fix tests ... An earlier change made the tests not compile here. We now need to use the legacyPluginMap function, since pluginMap has now been replaced with helpers to produce new-style plugin _sets_.	7 years ago
Martin Atkins	a7342de274	core: Properly handle no-op changes in plan ... Previously we just left these out of the plan altogether, but in the new plan types we intentionally include change information for every resource instance, even if no changes are actually planned, to allow alternative plan file viewers to show what isn't changing as well as what is.	7 years ago
Martin Atkins	9ec00468f8	plugin/convert: Convert cty.PathError to diagnostics with attribute path	7 years ago
Martin Atkins	5d744e6824	plugin: TRACE log for each GRPC-remoted plugin call	7 years ago
Martin Atkins	4bf19797f3	plugin: Fix deadlock in getSchema where mutex is never unlocked ... On the second call here this would fail to release the lock.	7 years ago
Martin Atkins	5559d21855	plugin: Register v5 plugins only if present ... Previously we would construct both provisioner and the provider objects if either callback was set, but this is incorrect because a plugin should actually set only one of these at a time, depending on what kind of plugin it is.	7 years ago
Martin Atkins	44bc7519a6	terraform: More wiring in of new provider types ... This doesn't actually work yet, but it builds and then panics in a pretty satisfying way.	7 years ago
James Bardin	c07ce1cd4b	move conversion functions into separate package ... Managing which function need to be shared between the terraform plugin and the helper plugin without creating cycles was becoming difficult. Move all functions related to converting between terraform and proto type into plugin/convert.	7 years ago
James Bardin	88bfbeb9c5	schema version changed type	7 years ago
James Bardin	b403023841	serve the new version 5 grpc plugins ... Use the new go-plugin version negotiation to server th appropriate plugin type when the client requests protocol version 5.	7 years ago
James Bardin	ff7d51a9b4	update tests to match new test func signature	7 years ago
Kristin Laemmert	5e7956f440	registry/response: rename Collection to ProviderVersionCollection for clarity	7 years ago
James Bardin	c9d91dc64e	remove old comment	7 years ago
Kristin Laemmert	bdaf8290b4	registry/client: added a specific error if the registry client does not ... support the requested service.	7 years ago
James Bardin	c1d4a63fae	GRPCProvider and GRPCProvisioner clients ... Here we add the GRPCProvisioner and GRPCProvider which implement the core provisioners.Interface and providers.Interface, and translate betweeen the core types and the grpc protocol.	7 years ago
James Bardin	81bd3b09d6	generate mocks for some of the grpc interfaces ... These will allow easier testing of the grpc endpoints in isolation. Mocks are generated for ProviderClient, ProvisionerClient, Provisioner_ProvisionResourceClient, and Provisioner_ProvisionResourceServer using `go generate`	7 years ago
James Bardin	e29053432e	grpc proto fixes ... Add missing connection field for ProvisionResource. Add diagnostics to the ProvisionResource stream.	7 years ago
Kristin Laemmert	ae015e0f75	plugin/discovery: add function to filter list of versions by those which ... support the requested platform. If the newest version of a provider which matches the version constraints does not support the requested platform, filter the list of available versions by platform support and try again.	7 years ago
Kristin Laemmert	f83d5866fe	plugin/discovery: removing deprecated functions	7 years ago
Kristin Laemmert	ce5e66e178	plugin/discovery provider installer: download providers from the registry ... Terraform will query the public registry at https://registry.terraform.io for providers, instead of https://releases.hashicorp.com.	7 years ago
James Bardin	8f51b2fba5	add grpc service definitions ... This add the grpc protobuf service definitions for the new plugin protocol, as well as the generated code.	7 years ago
Martin Atkins	4ed06a9227	terraform: HCL2-flavored module dependency resolver ... For the moment this is just a lightly-adapted copy of ModuleTreeDependencies named ConfigTreeDependencies, with the goal that the two can live concurrently for the moment while not all callers are yet updated and then we can drop ModuleTreeDependencies and its helper functions altogether in a later commit. This can then be used to make "terraform init" and "terraform providers" work properly with the HCL2-powered configuration loader.	7 years ago
James Bardin	0a6938e90a	update tests to match pre-release constraint rules	7 years ago
Sean Chittenden	d749420a25	Fix drift caused from gofmt when running make dev and go 1.11. ... A fresh checkout of `origin/master` does not build atm using the `dev` target because `master` has not been formatted using `gofmt` from Go 1.11 (tis has been the case for a while if you've been running devel). None of the drift in question is especially new but now that Go 1.11 has been released and gofmt's formatting guidelines have been updated, it would be *really* nice if the code in `master` reflected the current tooling in order to avoid having to fight this drift locally. * 8mo: https://github.com/hashicorp/terraform/blame/master/backend/remote-state/s3/backend_test.go#L260-L261 * 6mo: https://github.com/hashicorp/terraform/blame/master/builtin/provisioners/chef/linux_provisioner_test.go#L124 * 1yr: 7cfeffe36b/command/init.go (L75-L76) * 12d: 7cfeffe36b/command/meta_backend_test.go (L1437) * 2yr: 7cfeffe36b/helper/schema/resource_timeout_test.go (L26) * 4yr: 7cfeffe36b/helper/schema/schema_test.go (L2059) * 1yr: 7cfeffe36b/plugin/discovery/get_test.go (L151)	8 years ago
Paul Tyng	c868092d2d	Standardize http.Client creation with User-Agent	8 years ago
Masayuki Morita	f440dba137	Standardize on log level "WARN" rather than "WARNING"	8 years ago
James Bardin	6cb4e14cf8	Use pooled http client for fetching providers ... While the TLS handshakes are a fairly small overhead compared to downloading the providers, clients in some situation are failing to complete the TLS handshake in a timely manner. It's unclear if this is because of heavily constrained clients are stalling while doing the major crpto operations, or the edge servers are throttling repeated requests from the same IPs. This should allow reusing the open TLS connection to the release edge servers during init.	8 years ago
Martin Atkins	183833affc	core: terraform.ResourceProvider.GetSchema method ... In order to parse provider, resource and data source configuration from HCL2 config files, we need to know the relevant configuration schema. This new method allows Terraform Core to request these from a provider. This is a breaking change to this interface, so all of its implementers in this package are updated too. This includes concrete implementations of the new method in helper/schema that use the schema conversion code added in an earlier commit to produce a configschema.Block automatically. Plugins compiled against prior versions of helper/schema will not have support for this method, and so calls to them will fail. Callers of this new method will therefore need to sniff for support using the SchemaAvailable field added to both ResourceType and DataSource. This careful handling will need to persist until next time we increment the plugin protocol version, at which point we can make the breaking change of requiring this information to be available.	9 years ago
Martin Atkins	879899d434	plugin/discovery: plugin caching mechanism ... For users that have metered or slow internet connections it is annoying to have Terraform constantly re-downloading the same files when they initialize many separate directories. To help such users, here we add an opt-in mechanism to use a local directory as a read-through cache. When enabled, any plugin download will be skipped if a suitable file already exists in the cache directory. If the desired plugin isn't in the cache, it will be downloaded into the cache for use next time. This mechanism also serves to reduce total disk usage by allowing plugin files to be shared between many configurations, as long as the target system isn't Windows and supports either hardlinks or symlinks.	9 years ago
Martin Atkins	12d6bc8c30	plugin/discovery: ignore non-files when discovering ... If we encounter something that isn't a file -- for example, a dangling symlink whose referent has been deleted -- we'll ignore it so that we can either later produce a "no such plugin" error or auto-install a plugin that will actually work.	9 years ago
James Bardin	3dd0f93aed	Merge pull request #15816 from hashicorp/jbardin/plugin-logger ... specify a logger for go-plugin	9 years ago
James Bardin	714df97a02	specify a logger for go-plugin ... The go-plugin package now uses hclog. The default Logger has a level set to Info, but all plugin output is relayed via Debug. Create a new named logger for plugins with the level set to Trace so that all output comes through.	9 years ago
Radek Simko	28b33c9299	plugin: Display version + source when initializing plugins	9 years ago
Radek Simko	de3015cebf	Remove overly verbose DEBUG log	9 years ago
Martin Atkins	ac250d2792	plugin/discovery: trim off .exe suffix when parsing filenames ... On Windows systems the plugin binaries use a .exe suffix, which we were misparsing as part of either the "v" or "x" parts of the filename. This fixes #15578.	9 years ago
James Bardin	2b84e786e9	allow missing x-terraform-protocol-version ... If the release site is missing the "x-terraform-protocol-version" header, we should fetch the latest spec'ed release. Downloading the wrong protocol version can't do any damage, and the version present is more than likely compatible.	9 years ago
Martin Atkins	610fcb605e	plugin/discovery: allow customizing the OS/arch for auto-install ... Previously we forced only installing for the current GOOS and GOARCH. Now we allow this to be optionally overridden, which allows building tools that can, for example, populate a directory with plugins to run on a Linux server while working on a Mac.	9 years ago
James Bardin	beebd5cee3	improve discovery logging ... Display whether we're looking for a provider or provisioner in the log messages.	9 years ago
James Bardin	e059eb80d1	add test for signature mismatch	9 years ago
James Bardin	2cca6f10fa	return sig error when fetching a plugin ... A signature error was being ignored while fetching a plugin	9 years ago
James Bardin	c10f5caf05	Merge pull request #15345 from hashicorp/jbardin/plugin-sig ... verify automatically downloaded plugins	9 years ago
Martin Atkins	af2111f24e	plugin/discovery: sentinel error values for Get errors ... Some errors from Get are essentially user error, so we want to be able to recognize them and give the user good feedback on how to proceed. Although sentinel values are not an ideal solution to this, it's something reasonably simple we can do to get this done without lots of refactoring.	9 years ago
James Bardin	13d835f996	test signature verification ... use a published provider's checksum file and signature to check the verification.	9 years ago
James Bardin	415d562d36	add signature verification ... Fetch the SHA256SUMS file and verify it's signature before downloading any plugins. This embeds the hashicorp public key in the binary. If the publickey is replaced, new releases will need to be cut anyway. A --verify-plugin=false flag will be added to skip signature verification in these cases.	9 years ago
James Bardin	2e57d284cb	Merge pull request #15323 from hashicorp/jbardin/constraint-suggestion ... remove "~> 0.0" constraint suggestions	9 years ago
James Bardin	ec99b6910b	remove "~> 0.0" constraint suggestions ... Don't suggest constraints when the available plugin isn't versioned. Add zero version const for comparisons.	9 years ago
James Bardin	6faace287d	remove restriction on unversioned plugins ... Discover unversioned plugins regarless of location.	9 years ago
James Bardin	ba5b0dc609	mostly remove OS_ARCH knowledge from discovery ... Since the command package also needs to know about the specific OS_ARCH directories, remove the logic fom the discovery package. This doesn't completely remove the knowledge of the path from discovery, in order to maintain the current behavior of skipping legacy plugin names within a new-style path.	9 years ago
James Bardin	f723270e3e	search the vendor directory for plugins ... The default location for users to manually add plugins will be ./terraform.d/plugins/	9 years ago
Martin Atkins	af4c82d151	plugin/discovery: Installer.PurgeUnused method ... Given a map describing the chosen plugin for each provider name, this method should purge any other plugins present in the local cache directory.	9 years ago
Martin Atkins	f753974bb3	plugin/discovery: Installer interface, and provider implementation ... Previously we had a "getProvider" function type used to implement plugin fetching. Here we replace that with an interface type, initially with just a "Get" function. For now this just simplifies the interface by allowing the target directory and protocol version to be members of the struct rather than passed as arguments. A later change will extend this interface to also include a method to purge unused plugins, so that upgrading frequently doesn't leave behind a trail of unused executable files.	9 years ago
James Bardin	840978b2d5	udpate to plugin name convention ... Names are case-insensitive, using lowercase by default.	9 years ago
James Bardin	1b201e67ea	fix releases path and protocol header ... Last minute change to the location of the binaries	9 years ago
James Bardin	283ae0c7c4	fix comment and add some logging in discovery	9 years ago
James Bardin	ea3d9c86ba	sort constraints for consistent output	9 years ago
Martin Atkins	4ba20f9c1c	command init: show suggested constraints for unconstrained providers ... When running "terraform init" with providers that are unconstrained, we will now produce information to help the user update configuration to constrain for the particular providers that were chosen, to prevent inadvertently drifting onto a newer major release that might contain breaking changes. A ~> constraint is used here because pinning to a single specific version is expected to create dependency hell when using child modules. By using this constraint mode, which allows minor version upgrades, we avoid the need for users to constantly adjust version constraints across many modules, but make major version upgrades still be opt-in. Any constraint at all in the configuration will prevent the display of these suggestions, so users are free to use stronger or weaker constraints if desired, ignoring the recommendation.	9 years ago
James Bardin	dbbafbd43f	clean up plugin fetching ... We can filter the allowed versions and sort them before checking the protocol version, that way we can just return the first one found reducing network requests.	9 years ago
James Bardin	48d37131e0	more tests for fetching providers ... Extend the test reslease server to return the protocol version header and a dummy zip file for the provider. Test filtering the plugins by plugin protocol version and add a full GetProvder test.	9 years ago
James Bardin	5f053a2e64	refactor GetProvider ... Get provider needs to be provided with the plugin protocol version, because it can't be imported here directly. The plugin url types and methods were confusing; replace them with a few functions to format the urls.	9 years ago
James Bardin	8ad67991a5	check protocol version on plugins ... Verify that the plugin we're requesting has a compatible protocol version.	9 years ago
James Bardin	fdeb3d929c	Add Versions.Sort ... Sort versions from newest to oldest.	9 years ago
Martin Atkins	04bcece59c	plugin/discovery: handle the -Xn suffix used by auto-installed plugins ... This is used to mark the plugin protocol version. Currently we actually just ignore this entirely, since only one protocol version exists anyway. Later we will need to add checks here to ensure that we only pay attention to plugins of the right version.	9 years ago
Martin Atkins	e3401947a6	plugin/discovery: PluginRequirements can specify SHA256 digests ... As well as constraining plugins by version number, we also want to be able to pin plugins to use specific executables so that we can detect drift in available plugins between commands. This commit allows such requirements to be specified, but doesn't yet specify any such requirements, nor validate them.	9 years ago
James Bardin	044ad5ef59	rename some Constraints methods per code review	9 years ago
James Bardin	211f5b5d6e	add test for newestVersion	9 years ago
James Bardin	46190590cb	change []*Version to []Version ... Versions are used as values, so don't keep them as pointers here	9 years ago
James Bardin	a547e7c2f0	add releases version listing test	9 years ago
James Bardin	2749946f5c	basic plugin getter ... Add discovery.GetProviders to fetch plugins from the relases site. This is an early version, with no tests, that only (probably) fetches plugins from the default location. The URLs are still subject to change, and since there are no plugin releases, it doesn't work at all yet.	9 years ago
James Bardin	fa49c69793	expose Version.NewerThan ... NewerThan is useful outside of the package	9 years ago
James Bardin	7d2d951f27	Rename VersionSet to Constraints ... VersionSet is a wrapper around version.Contraints, so rename it it as such.	9 years ago
James Bardin	718ede0636	have Meta.Backend use a Config rather than loading ... Instead of providing the a path in BackendOpts, provide a loaded *config.Config instead. This reduces the number of places where configuration is loaded.	9 years ago
Martin Atkins	cfc08ae7e3	plugin/discovery: provide an AllVersions set ... Various implied dependencies will accept all versions, so this is convenient for populating dependency data structures for such implied dependencies.	9 years ago
Martin Atkins	a1e29ae290	plugin/discovery: use go-version instead of semver ... The semver library we were using doesn't have support for a "pessimistic constraint" where e.g. the user wants to accept only minor or patch version upgrades. This is important for providers since users will generally want to pin their dependencies to not inadvertantly accept breaking changes. So here we switch to hashicorp's home-grown go-version library, which has the ~> constraint operator for this sort of constraint. Given how much the old version object was already intruding into the interface and creating dependency noise in callers, this also now wraps the "raw" go-version objects in package-local structs, thus keeping the details encapsulated and allowing callers to deal just with this package's own types.	9 years ago
Martin Atkins	9b4f15c261	plugin: move Client function into plugin, from plugin/discovery ... Having this as a method of PluginMeta felt most natural, but unfortunately that means that discovery must depend on plugin and plugin in turn depends on core Terraform, thus making the discovery package hard to use without creating dependency cycles. To resolve this, we invert the dependency and make the plugin package be responsible for instantiating clients given a meta, using a top-level function.	9 years ago
Martin Atkins	bcd395e6bd	plugin/discovery: Allow overriding the paths of certain plugin names ... The .terraformrc file allows the user to override the executable location for certain plugins. This mechanism allows us to retain that behavior for a deprecation period by treating such executables as an unversioned plugin for the given name and excluding all other candidates for that name, thus ensuring that the override will "win". Users must eventually transition away from using this mechanism and use vendor directories instead, because these unversioned overrides will never match for a provider referenced with non-zero version constraints.	9 years ago
Martin Atkins	551bc9c4a4	plugin/discovery: build plugin clients ... These new methods ClientConfig and Client provide the bridge into the main plugin infrastructure by configuring and instantiating (respectively) a client object for the referenced plugin. This stops short of getting the proxy object from the client since that then requires referencing the interface for the plugin kind, which would then create a dependency on the main terraform package which we'd rather avoid here. It'll be the responsibility of the caller in the command package to do the final wiring to get a provider instance out of a provider plugin client.	9 years ago
Martin Atkins	0a08214a74	plugin/discovery: SHA256() method to get the hash of each plugin ... This will be used later to verify that installed plugins match what's available on the releases server.	9 years ago
Martin Atkins	a94e9a4362	plugin/discovery: find plugins in a given set of directories ... For now this supports both our old and new directory layouts, so we can preserve compatibility with existing configurations until a future major release where support for the old paths will be removed. Currently this allows both styles in all given directories, which means we support more variants than we actually intend to but this is accepted to keep the interface simple such that we can easily remove the legacy exception later. The documentation will reflect only the subset of path layouts that we actually intend to support.	9 years ago
Martin Atkins	39deb1ff6b	plugin/discovery: helpers for wrangling plugin versions ... With forthcoming support for versioned plugins we need to be able to answer questions like what versions of plugins are currently installed, what's the newest version of a given plugin available, etc. PluginMetaSet gives us a building block for this sort of plugin version wrangling.	9 years ago
Mitchell Hashimoto	700c74bdd0	plugin: bump protocol version ... We changed the type of InstanceState.Meta and this will break pre-existing plugins. They simply have to recompile to work.	9 years ago
Mitchell Hashimoto	96884ec42d	plugin: bump the protocol version due to Provisioner change	9 years ago
Mitchell Hashimoto	2e894c4ef7	plugin: add ResourceProvisioner.Stop API	9 years ago