50 Commits (449ea72e87da7e697ef7b68283a852cc9cc09045)

Author	SHA1	Message	Date
Martin Atkins	8eff19e48f	internal/getproviders: Initial implementation of FilesystemMirrorSource ... This is a basic implementation of FilesystemMirrorSource for now aimed only at the specific use-case of scanning the cache of provider plugins Terraform will keep under the ".terraform" directory, as part of our interim provider installer implementation for Terraform 0.13. The full functionality of this will grow out in later work when we implement explicit local filesystem mirrors, but for now the goal is to use this just to inspect the work done by the automatic installer once we switch it to the new provider-FQN-aware directory structure. The various FIXME comments in this are justified by the limited intended scope of this initial implementation, and they should be resolved by later work to use FilesystemMirrorSource explicitly for user-specified provider package mirrors.	6 years ago
Martin Atkins	c073db09ea	internal/getproviders: Sorting and filtering for lists of PackageMeta ... These are utility functions to ease processing of lists of PackageMeta elsewhere, once we have functionality that works with multiple packages at once. The local filesystem mirror source will be the first example of this, so these methods are motivated mainly by its needs.	6 years ago
Martin Atkins	d82a36b6f5	internal/getproviders: ParsePlatform method ... This is just to have a centralized set of logic for converting from a platform string (like "linux_amd64") to a Platform object, so we can do normalization and validation consistently.	6 years ago
Martin Atkins	3078d21c5f	internal/getproviders: Include Provider and Version in PackageMeta ... Although we tend to return these in contexts where at least one of these values is implied, being explicit means that PackageMeta values are self-contained and less reliant on such external context.	6 years ago
Pam Selle	c249943360	Module Expansion: Part 2 (#24154) ... * WIP: dynamic expand * WIP: add variable and local support * WIP: outputs * WIP: Add referencer * String representation, fixing tests it impacts * Fixes TestContext2Apply_outputOrphanModule * Fix TestContext2Apply_plannedDestroyInterpolatedCount * Update DestroyOutputTransformer and associated types to reflect PlannableOutputs * Remove comment about locals * Remove module count enablement * Removes allowing count for modules, and reverts the test, while adding a Skip()'d test that works when you re-enable the config * update TargetDownstream signature to match master * remove unnecessary method Co-authored-by: James Bardin <j.bardin@gmail.com>	6 years ago
Martin Atkins	86f0b5191c	addrs: Stronger validation and normalization of provider namespace/type ... The provider FQN is becoming our primary identifier for a provider, so it's important that we are clear about the equality rules for these addresses and what characters are valid within them. We previously had a basic regex permitting ASCII letters and digits for validation and no normalization at all. We need to do at least case folding and UTF-8 normalization because these names will appear in file and directory names in case-insensitive filesystems and in repository names such as on GitHub. Since we're already using DNS-style normalization and validation rules for the hostname part, rather than defining an entirely new set of rules here we'll just treat the provider namespace and type as if they were single labels in a DNS name. Aside from some internal consistency, that also works out nicely because systems like GitHub use organization and repository names as part of hostnames (e.g. with GitHub Pages) and so tend to apply comparable constraints themselves. This introduces the possibility of names containing letters from alphabets other than the latin alphabet, and for latin letters with diacritics. That's consistent with our introduction of similar support for identifiers in the language in Terraform 0.12, and is intended to be more friendly to Terraform users throughout the world that might prefer to name their products using a different alphabet. This is also a further justification for using the DNS normalization rules: modern companies tend to choose product names that make good domain names, and now such names will be usable as Terraform provider names too.	6 years ago
Kristin Laemmert	7eed30595a	moduledeps: replace ProviderInstance with addrs.Provider (#24017) ... * addrs: add ParseProviderSourceString function to parse fqns from tfconfig-inspect * moduledeps: use addrs.Provider instead of ProviderInstance	6 years ago
Pam Selle	9f374eb1fc	Merge pull request #23865 from loslocostacos/fix-win-tfe-pathing ... Fix compatibility issue between Windows CLI and remote plan and apply	6 years ago
Martin Atkins	4d7122a0dd	internal/getproviders: LookupLegacyProvider ... This is a temporary helper so that we can potentially ship the new provider installer without making a breaking change by relying on the old default namespace lookup API on the default registry to find a proper FQN for a legacy provider provider address during installation. If it's given a non-legacy provider address then it just returns the given address verbatim, so any codepath using it will also correctly handle explicit full provider addresses. This also means it will automatically self-disable once we stop using addrs.NewLegacyProvider in the config loader, because there will therefore no longer be any legacy provider addresses in the config to resolve. (They'll be "default" provider addresses instead, assumed to be under registry.terraform.io/hashicorp/* ) It's not decided yet whether we will actually introduce the new provider in a minor release, but even if we don't this API function will likely be useful for a hypothetical automatic upgrade tool to introduce explicit full provider addresses into existing modules that currently rely on the equivalent to this lookup in the current provider installer. This is dead code for now, but my intent is that it would either be called as part of new provider installation to produce an address suitable to pass to Source.AvailableVersions, or it would be called from the aforementioned hypothetical upgrade tool. Whatever happens, these functions can be removed no later than one whole major release after the new provider installer is introduced, when everyone's had the opportunity to update their legacy unqualified addresses.	6 years ago
Steve Burns	26305a83a3	Remove unecessary conditions for module manifest read and write functions	6 years ago
Steve Burns	43b855743b	Uniformity for module manifest dir read/write	6 years ago
Steven Burns	98e612ee50	Add condition for conversion of module path in manifest	6 years ago
Steve Burns	8bc0ef8fdc	Attempt to change the output of module manifest	6 years ago
Martin Atkins	2aac8cf812	internal/getproviders: Distinguished packed vs. unpacked local packages ... Our local filesystem mirror mechanism will allow provider packages to be given either in packed form as an archive directly downloaded to disk or in an unpacked form where the archive is extracted. Distinguishing these two cases in the concrete Location types will allow callers to reliably select the mode chosen by the selected installation source and handle it appropriately, rather than resorting to out-of-band heuristics like checking whether the object is a directory or a file.	6 years ago
Martin Atkins	92a58ec438	internal/getproviders: Stub out the two mirror sources ... In a future commit, these implementations of Source will allow finding and retrieving provider packages via local mirrors, both in the local filesystem and over the network using an HTTP-based protocol.	6 years ago
Martin Atkins	773e38b49c	internal/getproviders: Stub of MultiSource ... This is an API stub for a component that will be added in a future commit to support considering a number of different installation sources for each provider. These will eventually be configurable in the CLI configuration, allowing users to e.g. mirror certain providers within their own infrastructure while still being able to go upstream for those that aren't mirrored, or permit locally-mirrored providers only, etc.	6 years ago
Martin Atkins	d695956061	internal/getproviders: MemoizeSource for local caching of source responses ... Some sources make network requests that are likely to be slow, so this wrapper type can cache previous responses for its lifetime in order to speed up repeated requests for the same information.	6 years ago
Martin Atkins	a77bc59c44	internal/getproviders: Package URL should always be absolute ... Registries backed by static files are likely to use relative paths to their archives for simplicity's sake, but we'll normalize them to be absolute before returning because the caller wouldn't otherwise know what to resolve the URLs relative to.	6 years ago
Martin Atkins	c8f7223adb	internal/getproviders: Source interface for generalization ... We intend to support installation both directly from origin registries and from mirrors in the local filesystem or over the network. This Source interface will serve as our abstraction over those three options, allowing calling code to treat them all the same.	6 years ago
Martin Atkins	c76260e957	internal/getproviders: Query a provider registry ... Our existing provider installer was originally built to work with releases.hashicorp.com and later retrofitted to talk to the official Terraform Registry. It also assumes a flat namespace of providers. We're starting a new one here, copying and adapting code from the old one as necessary, so that we can build out this new API while retaining all of the existing functionality and then cut over to this new implementation in a later step. Here we're creating a foundational component for the new installer, which is a mechanism to query for the available versions and download locations of a particular provider. Subsequent commits in this package will introduce other Source implementations for installing from network and filesystem mirrors.	6 years ago
Kristin Laemmert	18dd1bb4d6	Mildwonkey/tfconfig upgrade (#23670) ... * deps: bump terraform-config-inspect library * configs: parse `version` in new required_providers block With the latest version of `terraform-config-inspect`, the required_providers attribute can now be a string or an object with attributes "source" and "version". This change allows parsing the version constraint from the new object while ignoring any given source attribute.	6 years ago
Kristin Laemmert	4045a6e5d0	initwd: cache registry responses for module versions and download URL (#23727) ... * initwd: cache registry responses for module versions and download URL Closes #23544	6 years ago
Martin Atkins	2b2ac1f6de	configs: use local set of go-getter detectors ... In an earlier change we switched to defining our own sets of detectors, getters, etc for go-getter in order to insulate us from upstream changes to those sets that might otherwise change the user-visible behavior of Terraform's module installer. However, we apparently neglected to actually refer to our local set of detectors, and continued to refer to the upstream set. Here we catch up with the latest detectors from upstream (taken from the version of go-getter we currently have vendored) and start using that fixed set. Currently we are maintaining these custom go-getter sets in two places due to the configload vs. initwd distinction. That was already true for goGetterGetters and goGetterDecompressors, and so I've preserved that for now just to keep this change relatively simple; in later change it would be nice to factor these "get with go getter" functions out into a shared location which we can call from both configload and initwd.	6 years ago
Kristin Laemmert	68dfc3046d	backend/remote-state: etcdv3, oss, and manta acc tests should fail ... faster The acceptance tests for etcdv3, oss and manta were not validating required env variablea, chosing to assume that if one was running acceptance tests they had already configured the credentials. It was not always clear if this was a bug in the tests or the provider, so I opted to make the tests fail faster when required attributes were unset (or "").	7 years ago
Kristin Laemmert	69b6791ef1	internal/initwd: fix issues with tests and symlinks ... filepath.EvalSymlinks is our friend! The code already does this, the tests needed to be updated to do the same.	7 years ago
Radek Simko	3d94baecf6	Regenerate protobuf files under latest versions ... Used protobuf 3.9.1 with protoc-gen-go 1.3.2	7 years ago
Radek Simko	5b9f2fafc8	Standardise directory name for test data	7 years ago
Pam Selle	59c5cc4788	Merge pull request #21254 from davewongillies/gcs ... Add GCS source support for modules	7 years ago
James Bardin	fa1f9be539	add tfplugin proto docs files ... Add versioned tfplugin proto files to the docs directory, for easier reference. The latest version starts as a symlink to the current file used for generated the tfplugin package in ./internal/tfplugin5. When changing the protocol version, the old file must be copied to ./docs/plugin-protocol/, and a new symlink created for the latest version.	7 years ago
James Bardin	863bfdce06	set proto version 5.1 ... With the addition of the private field to ReadResource, we need to bump the proto version to 5.1.	7 years ago
James Bardin	e2b2f1bbbc	add Private fields to ReadResource ... Private data was previously created during Plan, and sent back to the provider during Apply. This data also needs to be persisteded accross Read calls, but rather than rely on core for that we can send the data to the provider during Read to allow for more flexibilty.	7 years ago
Martin Atkins	c5285021fa	internal/initwd: Test that version is rejected for local modules ... This was already working, but since that codepath is separate from the go-getter install codepath it's helpful to have a separate test for it, in addition to the existing one for go-getter modules.	7 years ago
David Gillies	8b45443b21	Add GCS source support for modules	7 years ago
Matt Morrison	cbebb7cdf1	command/init: Don't panic if go-getter-fetched module has version constraint	7 years ago
Martin Atkins	f5a7f45ffb	internal/initwd: Fix module installation error ... The "err" variable in the MaybeRelativePathErr condition was masking the original err with nil in the "else" case of this branch, causing the error message to be incomplete. While here, also tweaked the wording to say "Could not download" rather than "Error attempting to download", both to say the same thing in fewer words and because the summary line above already starts with "Error:" when we print out this message, so it looks weird to have both lines start with the same word.	7 years ago
Kristin Laemmert	9327eedb04	internal/initwd: follow local module path symlink (#21063) ... * internal/initwd: follow local module path symlink Fixes #21060 While a previous commit fixed a problem when the local module directory contained a symlink, it did not account for the possibility that the entire directory was a symlink.	7 years ago
Martin Atkins	88e76fa9ef	configs/configschema: Introduce the NestingGroup mode for blocks ... In study of existing providers we've found a pattern we werent previously accounting for of using a nested block type to represent a group of arguments that relate to a particular feature that is always enabled but where it improves configuration readability to group all of its settings together in a nested block. The existing NestingSingle was not a good fit for this because it is designed under the assumption that the presence or absence of the block has some significance in enabling or disabling the relevant feature, and so for these always-active cases we'd generate a misleading plan where the settings for the feature appear totally absent, rather than showing the default values that will be selected. NestingGroup is, therefore, a slight variation of NestingSingle where presence vs. absence of the block is not distinguishable (it's never null) and instead its contents are treated as unset when the block is absent. This then in turn causes any default values associated with the nested arguments to be honored and displayed in the plan whenever the block is not explicitly configured. The current SDK cannot activate this mode, but that's okay because its "legacy type system" opt-out flag allows it to force a block to be processed in this way anyway. We're adding this now so that we can introduce the feature in a future SDK without causing a breaking change to the protocol, since the set of possible block nesting modes is not extensible.	7 years ago
Justin Downing	1e32ae243c	grammatical updates to comments and docs (#20195)	7 years ago
Kristin Laemmert	a15a4acf2f	configs/configupgrade: detect possible relative module sources (#20646) ... * configs/configupgrade: detect possible relative module sources If a module source appears to be a relative local path but does not have a preceding ./, print a #TODO message for the user. * internal/initwd: limit go-getter detectors to those supported by terraform * internal/initwd: move isMaybeRelativeLocalPath check into getWithGoGetter To avoid making two calls to getter.Detect, which potentially makes non-trivial API calls, the "isMaybeRelativeLocalPath" check was moved to a later step and a custom error type was added so user-friendly diagnostics could be displayed in the event that a possible relative local path was detected.	7 years ago
Kristin Laemmert	bf04503f04	internal/initwd: identify possible relative module sources ... Identify module sources that look like relative paths ("child" instead of "./child", for example) and surface a helpful error. Previously, such module sources would be passed to go-getter, which would fail because it was expecting an absolute, or properly relative, path. This commit moves the check for improper relative paths sooner so a user-friendly error can be displayed.	7 years ago
Kristin Laemmert	0c5fd835ce	copyDir: detect if the module install path is a symlink to a directory (#20603) ... configs/configload and internal/initwd both had a copyDir function that would fail if the source directory contained a symlinked directory, because the os.FileMode.IsDir() returns false for symlinks. This PR adds a check for a symlink and copies that symlink in the target directory. It handles symlinks for both files and directories (with included tests). Fixes #20539	7 years ago
Martin Atkins	31299e688d	core: Allow legacy SDK to opt out of plan-time safety checks ... Due to the inprecision of our shimming from the legacy SDK type system to the new Terraform Core type system, the legacy SDK produces a number of inconsistencies that produce only minor quirky behavior or broken edge-cases. To retain compatibility with those existing weird behaviors, the legacy SDK opts out of our safety checks. The intent here is to allow existing providers to continue to do their previous unsafe behaviors for now, accepting that this will allow certain quirky bugs from previous releases to persist, and then gradually migrate away from the legacy SDK and remove this opt-out on a per-resource basis over time. As with the apply-time safety check opt-out, this is reserved only for the legacy SDK and must not be used in any new SDK implementations. We still include any inconsistencies as warnings in the logs as an aid to anyone debugging weird behavior, so that they can see situations where blame may be misplaced in the user-visible error messages.	7 years ago
Martin Atkins	1530fe52f7	core: Legacy SDK providers opt out of our new apply result check ... The shim layer for the legacy SDK type system is not precise enough to guarantee it will produce identical results between plan and apply. In particular, values that are null during plan will often become zero-valued during apply. To avoid breaking those existing providers while still allowing us to introduce this check in the future, we'll introduce a rather-hacky new flag that allows the legacy SDK to signal that it is the legacy SDK and thus disable the check. Once we start phasing out the legacy SDK in favor of one that natively understands our new type system, we can stop setting this flag and thus get the additional safety of this check without breaking any previously-released providers. No other SDK is permitted to set this flag, and we will remove it if we ever introduce protocol version 6 in future, assuming that any provider supporting that protocol will always produce consistent results.	7 years ago
Martin Atkins	86c02d5c35	command: "terraform init" can partially initialize for 0.12upgrade ... There are a few constructs from 0.11 and prior that cause 0.12 parsing to fail altogether, which previously created a chicken/egg problem because we need to install the providers in order to run "terraform 0.12upgrade" and thus fix the problem. This changes "terraform init" to use the new "early configuration" loader for module and provider installation. This is built on the more permissive parser in the terraform-config-inspect package, and so it allows us to read out the top-level blocks from the configuration while accepting legacy HCL syntax. In the long run this will let us do version compatibility detection before attempting a "real" config load, giving us better error messages for any future syntax additions, but in the short term the key thing is that it allows us to install the dependencies even if the configuration isn't fully valid. Because backend init still requires full configuration, this introduces a new mode of terraform init where it detects heuristically if it seems like we need to do a configuration upgrade and does a partial init if so, before finally directing the user to run "terraform 0.12upgrade" before running any other commands. The heuristic here is based on two assumptions: - If the "early" loader finds no errors but the normal loader does, the configuration is likely to be valid for Terraform 0.11 but not 0.12. - If there's already a version constraint in the configuration that excludes Terraform versions prior to v0.12 then the configuration is probably _already_ upgraded and so it's just a normal syntax error, even if the early loader didn't detect it. Once the upgrade process is removed in 0.13.0 (users will be required to go stepwise 0.11 -> 0.12 -> 0.13 to upgrade after that), some of this can be simplified to remove that special mode, but the idea of doing the dependency version checks against the liberal parser will remain valuable to increase our chances of reporting version-based incompatibilities rather than syntax errors as we add new features in future.	7 years ago
Martin Atkins	0c0a437bcb	Move module install functionality over to internal/initwd	7 years ago
Martin Atkins	e27e0ddc9e	internal/init: Module installation functionality ... This is an adaptation of the installation code from configs/configload, now using the "earlyconfig" package instead of the "configs" package. Module installation is an initialization-only process, with all other commands assuming an already-initialized directory. Having it here can therefore simplify the API of configs/configload, which can now focus only on the problem of loading modules that have already been installed. The old installer code in configs/configload is still in place for now because the caller in "terraform init" isn't yet updated to use this.	7 years ago
Martin Atkins	53ee858851	internal/init: Package for helpers of the "terraform init" command ... "terraform init" is quite a complex beast in relation to other commands since almost everything it does is unique to it and thus not factored out into other packages. To get some of that sprawl out of the "command" package, this new internal package will give us somewhere to put this init functionality that is also useful for test code that needs to mimic the initialization behavior against fixture directories.	7 years ago
Martin Atkins	0ce2add59f	internal/modsdir: Factor out module manifest models to separate package	7 years ago
Martin Atkins	8ca1fcec51	internal/earlyconfig: Liberal config parsing for init ... This is an alternative to the full config loader in the "configs" package that is good for "early" use-cases like "terraform init", where we want to find out what our dependencies are without getting tripped up on any other errors that might be present.	7 years ago
Martin Atkins	4fe9632f09	plugin: Establish our current plugin protocol as version 5 ... The main significant change here is that the package name for the proto definition is "tfplugin5", which is important because this name is part of the wire protocol for references to types defined in our package. Along with that, we also move the generated package into "internal" to make it explicit that importing the generated Go package from elsewhere is not the right approach for externally-implemented SDKs, which should instead vendor the proto definition they are using and generate their own stubs to ensure that the wire protocol is the only hard dependency between Terraform Core and plugins. After this is merged, any provider binaries built against our helper/schema package will need to be rebuilt so that they use the new "tfplugin5" package name instead of "proto". In a future commit we will include more elaborate and organized documentation on how an external codebase might make use of our RPC interface definition to implement an SDK, but the primary concern here is to ensure we have the right wire package name before release.	7 years ago