103 Commits (main)

Author	SHA1	Message	Date
Sarah French	c05eaaf20d	PSS: Refactor how we handle unmanaged providers, add support for using dev_override (#38269) ... * feat: Enable use of developer override providers when using PSS * feat: Store how a provider was supplied to Terraform in the backend state file. This allows triggering a prompt for state migration if there's a change between using a managed provider versus an unmanaged/override provider. * test: Update all PSS-related backend state files in test fixtures to include new, necessary provider_supply_mode data. * fix: Avoid duplicate dev_override warnings * refactor: Change how we determine how a provider is supplied to Terraform. By using supply mode data we prevent multiple parts of the code checking if a given provider is reattached/builtin/etc... * test: Add separate tests showing a built-in or dev override provider being used for the main workflow using PSS * test: Add E2E tests that characterise current behaviour when how a provider is supplied is changed partway through different workflows * fix: Tolerate missing default workspace when using PSS and init handles minor changes in state store configuration. * feat: `init`-related code will prompt a user to migrate state if Terraform detects a swap between managed and unmanaged providers being used for PSS. Internally, what happens is the different version data (null versus a value being present) impacts the hash of the state store. This then triggers logic where TF checks if a migration is needed or if the new hash value should be silently updated in the backend state file. We always prompt for migration. * refactor: Add `NotManagedByTerraform` method to `ProviderSupplyMode` * Apply suggestions from code review Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com> --------- Co-authored-by: Radek Simko <radeksimko@users.noreply.github.com>	1 week ago
mickael-hc	332a94d9a1	update gpg with latest version from website (#38271) ... * update gpg with latest version updated expiry to 2030-03-01, key can be found at https://www.hashicorp.com/.well-known/pgp-key.txt * update another reference * include both old and new pubkeys ensures the certification of the old key is still present	1 month ago
Sarah French	9f5f21f1d8	test: Enable mocking http provider sources in tests (#38206) ... * test: Update`MockSource` to be able to present providers as being installed via HTTP, add `newMockProviderSourceViaHTTP` for direct use in tests. Compare newMockProviderSourceViaHTTP to the existing newMockProviderSource. newMockProviderSource uses FakeInstallablePackageMeta, which creates metadata for providers that says it's being sourced from a local archive. In newMockProviderSourceViaHTTP the metadata reports the provider is downloaded via HTTP. The URL is created using a base URL that should be obtained from a test http server created in the test. * test: Add `newMockProviderSourceUsingTestHttpServer` helper, to abstract away making the http test server. Add test showing it in use. * refactor: Update test helpers to call t.Cleanup internally	2 months ago
Radek Simko	0fe906fa8c	make copyrightfix	2 months ago
Sarah French	6b73f710f8	PSS: Implement initialisation of new working directory (or use of `-reconfigure` flag) while using `state_store` (#37732) ... * Minor fixes in diagnostics This can only be done once modules have been parsed and the required providers data is available. There are multiple places where config is parsed, into either Config or Module structs, so this needs to be implemented in multiple places. * Rename test to make it specific to use of backend block in config * Update initBackend to accept whole initArgs collection * Only process --backend-config data, when setting up a `backend`, if that data isn't empty * Simplify how mock provider factories are made in tests * Update mock provider's default logic to track and manage existing workspaces * Add `ProviderSchema` method to `Pluggable` structs. This allows calling code to access the provider schema when using provider configuration data. * Add function for converting a providerreqs.Version to a hashicorp/go-version Version. This is needed for using locks when creating the backend state file. * Implement initial version of init new working directories using `stateStore_C_s`. Default to creating the default workspace if no workspaces exist. * Update test fixtures to match the hashicorp/test mock provider used in PSS tests * Allow tests to obtain locks that include `testingOverrides` providers. The `testingOverrides` field will only be set in tests, so this should not impact end users. * Add tests showing TF can initialize a working directory for the first time (and do the same when forced by -reconfigure flag). Remove replaced tests. * Add -create-default-workspace flag, to be used to disable creating the default workspace by default when -input=false (i.e for use in CI). Refactor creation of default workspace logic. Add tests. * Allow reattached providers to be used during init for PSS * Rename variable to `backendHash` so relation to `backend` is clearer * Allow `(m *Meta) Backend` to return warning diagnostics * Protect against nil testingOverrides in providerFactoriesFromLocks * Add test case seeing what happens if default workspace selected, doesn't exist, but other workspaces do exist. The consequences here are due to using `selectWorkspace` in `stateStore_C_s`, matching what's done in `backend_C_r_s`. * Address code consistency check failure on PR * Refactor use of mock in test that's experiencing EOF error... * Remove test that requires test to supply input for user prompt This test passes when run in isolation but fails when run alongside other tests, even when skipping all other tests using `testStdinPipe`. I don't think the value of this test is great enough to start changing how we test stdin input. * Allow -create-default-workspace to be used regardless of whether input is enabled or disabled * Add TF_SKIP_CREATE_DEFAULT_WORKSPACE environment variable * Responses to feedback, including making testStdinPipe helper log details of errors copying data to stdin. Note: We cannot call t.Fatal from a non-test goroutine. * Use Errorf instead * Allow backend state files to not include version data when a builtin or reattached provider is in use. * Add clarifying comment about re-attached providers when finding the matching entry in required_providers * Report that the default workspace was created to the view * Refactor: use error comparison via `errors.Is` to identify when no workspaces exist. * Move handling of TF_ENABLE_PLUGGABLE_STATE_STORAGE into init's ParseInit func. * Validate that PSS-related flags can only be used when experiments are enabled, enforce coupling of PSS-related flags when in use. * Slight rewording of output message about default workspace * Update test to assert new output about default workspace	6 months ago
Sarah French	0dfa115d7d	First step to enable use of TF_REATTACH_PROVIDERS with PSS (#37634) ... * Add test coverage for `parseReattachProviders` * Add unhappy path test cases to parseReattachProviders test * Add `isProviderReattached` function, to help identify when a reattached provider is being used for PSS * Move reattach config-related code into its own package * Make calling code not need to know what the ENV is for reattach providers * Refactor IsProviderReattached to use pre-existing logic * Add test case, make spell check happy with US English * Add headers * Make calling code responsible for reading in the related ENV * Add godoc comment to TF_REATTACH_PROVIDERS const * Update internal/getproviders/reattach/reattach.go	7 months ago
Samsondeen	ede46c0b81	Allow successful init when constraint matches at least one valid version (#37137)	11 months ago
Martin Atkins	5f2eab6944	providerreqs: Provider requirement/version types in their own package ... As we previously found with "package addrs", any package that contains types used as identifiers or addresses tends to get imported from all over, which means that if we mix real logic with those types in the same package then a lot more packages end up depending indirectly on our external dependencies. In this case, the getproviders package depends on various things we need only during provider installation, such as the library we use for GPG signature verification. Lots of different packages in Terraform need to talk about provider requirements, but only the CLI layer actually needs to do provider installation, and so this commit splits the types we use to talk about requirements into a new package "providerreqs", which package getproviders then depends on. To keep the size of this diff relatively small to start I've left forwarding aliases in package getproviders, which means that not all of the callers needed to be updated all at once. This commit does update all of the packages that the remote state backends depend on though, because that then allows us to reduce the set of indirect dependencies for each of those backends. The nested module go.mod files show that this has disconnected the indirect dependencies on github.com/ProtonMail/go-crypto and on github.com/cloudflare/circl , both of which Terraform CLI uses only for provider installation.	2 years ago
thirdkeyword	8284cde302	remove repetitive words ... Signed-off-by: thirdkeyword <fliterdashen@gmail.com>	2 years ago
James Bardin	0d4a29f7f3	ignore expired provider signing keys ... Provider developers are not currently required to keep the signing keys stored in the registry up to date, and older releases may be signed with a key which has since expired. For our purposes here however, we are validating the key and signature used at the time of publishing, and given that the registry has previously vouched for the validity of the key used, we can continue to trust that key returned by the registry for installation.	3 years ago
James Bardin	86667375fd	remove errant debug print	3 years ago
hashicorp-copywrite[bot]	53c34ff49c	Update copyright file headers to BUSL-1.1	3 years ago
Liam Cervante	1fe57d457d	upgrade golang.org/x/crypto/openpgp to github.com/ProtoMail/go-crypto (#33406)	3 years ago
hashicorp-copywrite[bot]	325d18262e	[COMPLIANCE] Add Copyright and License Headers	3 years ago
Martin Atkins	783a07d9e8	build: Use Go 1.19 ... Go 1.19's "fmt" has some awareness of the new doc comment formatting conventions and adjusts the presentation of the source comments to make it clearer how godoc would interpret them. Therefore this commit includes various updates made by "go fmt" to acheve that. In line with our usual convention that we make stylistic/grammar/spelling tweaks typically only when we're "in the area" changing something else anyway, I also took this opportunity to review most of the comments that this updated to see if there were any other opportunities to improve them.	4 years ago
Radek Simko	b82eee6c9f	getproviders: account for occasionally missing Host header in errors (#31542)	4 years ago
Liam Cervante	9f0d1d0630	terraform init: add link to documentation when a checksum is missing from the lock file (#31408) ... * terraform init: add suggested fix for when a checksum is missing from the lock file * improve error message * add link to the documentation * cleanup leftovers from previous attempt * fix tests * s/,/; * fix imports	4 years ago
Radek Simko	7feef1c4aa	Use hashicorp/terraform-registry-address as a decoupled library (#28338) ... * refactor: Use tfaddr for provider address parsing * refactor: Use tfaddr for module address parsing * deps: introduce hashicorp/terraform-registry-address	4 years ago
Radek Simko	746af015ea	internal/getproviders: Add URL to error message for clarity (#30810) ... * internal/getproviders: Add URL to error message for clarity Occasionally `terraform init` on some providers may return the following error message: Error while installing citrix/citrixadc v1.13.0: could not query provider registry for registry.terraform.io/citrix/citrixadc: failed to retrieve authentication checksums for provider: 403 Forbidden The 403 is most often returned from GitHub (rather than Registry API) and this change makes it more obvious. * Use Host instead of full URL	4 years ago
Krista LaFentres	2fc4e06890	Fix failing lint check for deprecated x/crypto/openpgp library	4 years ago
Alisdair McDiarmid	fb58f9e6d2	cli: Fix flaky init cancel test ... There is a race between the MockSource and ShutdownCh which sometimes causes this test to fail. Add a HangingSource implementation of Source which hangs until the context is cancelled, so that there is always time for a user-initiated shutdown to trigger the cancellation code path under test.	5 years ago
Radek Simko	2496bc2b1e	internal/registry: Add URL to error message for clarity (#29298)	5 years ago
Paddy	8bdea502ab	Add support for protocol 6 providers during init. (#29153) ... Update the version constraints for what providers will be downloaded from the registry, allowing protocol 6 providers to be downloaded from the registry.	5 years ago
Martin Atkins	e5f52e56f8	addrs: Rename DefaultRegistryHost to DefaultProviderRegistryHost ... As the comment notes, this hostname is the default for provide source addresses. We'll shortly be adding some address types to represent module source addresses too, and so we'll also have DefaultModuleRegistryHost for that situation. (They'll actually both contain the the same hostname, but that's a coincidence rather than a requirement.)	5 years ago
Kristin Laemmert	649095c602	providers subcommand tests (#28744) ... * getproviders ParsePlatform: add check for invalid platform strings with too many parts The existing logic would not catch things like a platform string containing multiple underscores. I've added an explicit check for exactly 2 parts and some basic tests to prove it. * command/providers-lock: add tests This commit adds some simple tests for the providers lock command. While adding this test I noticed that there was a mis-copied error message, so I replaced that with a more specific message. I also added .terraform.lock.hcl to our gitignore for hopefully obvious reasons. getproviders.ParsePlatform: use parts in place of slice range, since it's available * command: Providers mirror tests The providers mirror command is already well tested in e2e tests, so this includes only the most absolutely basic test case.	5 years ago
Martin Atkins	b9a93a0fe7	Move addrs/ to internal/addrs/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
Martin Atkins	4c254cc2be	Move httpclient/ to internal/httpclient/ ... This is part of a general effort to move all of Terraform's non-library package surface under internal in order to reinforce that these are for internal use within Terraform only. If you were previously importing packages under this prefix into an external codebase, you could pin to an earlier release tag as an interim solution until you've make a plan to achieve the same functionality some other way.	5 years ago
James Bardin	a4a6b40bee	update getproviders tests for new key	5 years ago
James Bardin	e1d790be61	update HC PublicKey	5 years ago
Pam Selle	242f319638	Give suggestions & remind users to use required_providers when provider not in registry (#28014) ... * Add helper suggestion when failed registry err When someone has a failed registry error on init, remind them that they should have required_providers in every module * Give suggestion for a provider based on reqs Suggest another provider on a registry error, from the list of requirements we have on init. This skips the legacy lookup process if there is a similar provider existing in requirements.	5 years ago
Alex Chan	4ec80fa18d	Correct the spelling of heirarchy/hierarchy throughout	5 years ago
Romain Philibert	5ae5c99ba0	fix: do not fail when provider symlink is broken. Only print a warning. ... fix https://github.com/hashicorp/terraform/issues/27445	5 years ago
Martin Atkins	4b3e237668	command/init: Hint about providers in other namespaces ... If a user forgets to specify the source address for a provider, Terraform will assume they meant a provider in the registry.terraform.io/hashicorp/ namespace. If that ultimately doesn't exist, we'll now try to see if there's some other provider source address recorded in the registry's legacy provider lookup table, and suggest it if so. The error message here is a terse one addressed primarily to folks who are already somewhat familiar with provider source addresses and how to specify them. Terraform v0.13 had a more elaborate version of this error message which directed the user to try the v0.13 automatic upgrade tool, but we no longer have that available in v0.14 and later so the user must make the fix themselves.	5 years ago
James Bardin	f0c3ddf5a1	internal/getproviders: staticcheck	5 years ago
Alisdair McDiarmid	b1bc0e5d92	getproviders: Normalize versions before dedupe ... When rendering a set of version constraints to a string, we normalize partially-constrained versions. This means converting a version like 2.68.* to 2.68.0. Prior to this commit, this normalization was done after deduplication. This could result in a version constraints string with duplicate entries, if multiple partially-constrained versions are equivalent. This commit fixes this by normalizing before deduplicating and sorting.	6 years ago
Martin Atkins	430318e262	getproviders: Consistent ordering of terms in VersionConstraintsString ... An earlier commit made this remove duplicates, which set the precedent that this function is trying to canonically represent the _meaning_ of the version constraints rather than exactly how they were expressed in the configuration. Continuing in that vein, now we'll also apply a consistent (though perhaps often rather arbitrary) ordering to the terms, so that it doesn't change due to irrelevant details like declarations being written in a different order in the configuration. The ordering here is intended to be reasonably intuitive for simple cases, but constraint strings with many different constraints are hard to interpret no matter how we order them so the main goal is consistency, so those watching how the constraints change over time (e.g. in logs of Terraform output, or in the dependency log file) will see fewer noisy changes that don't actually mean anything.	6 years ago
Alisdair McDiarmid	fe9a9fadfb	internal: Suppress duplicate version constraints ... A set of version constraints can contain duplicates. This can happen if multiple identical constraints are specified throughout a configuration. When rendering the set, it is confusing to display redundant constraints. This commit changes the string renderer to only show the first instance of a given constraint, and adds unit tests for this function to cover this change. This also fixes a bug with the locks file generation: previously, a configuration with redundant constraints would result in this error on second init: Error: Invalid provider version constraints on .terraform.lock.hcl line 6: (source code not available) The recorded version constraints for provider registry.terraform.io/hashicorp/random must be written in normalized form: "3.0.0".	6 years ago
Alisdair McDiarmid	7a31e56cb7	Merge pull request #26637 from hashicorp/alisdair/fix-locksfile-unconstrained-versions ... internal: Fix lockfile constraint output for 1.2.*	6 years ago
Martin Atkins	1fe314b70b	getproviders: Don't log if EvalSymlinks changes nothing ... Previously this codepath was generating a confusing message in the absense of any symlinks, because filepath.EvalSymlinks returns a successful result if the target isn't a symlink. Now we'll emit the log line only if filepath.EvalSymlinks returns a result that's different in a way that isn't purely syntactic (which filepath.Clean would "fix"). The new message is a little more generic because technically we've not actually ensured that a difference here was caused by a symlink and so we shouldn't over-promise and generate something potentially misleading.	6 years ago
Alisdair McDiarmid	9576a5b2d8	internal: Fix lockfile constraint output for 1.2.* ... If a configuration requires a partial provider version (with some parts unspecified), Terraform considers this as a constrained-to-zero version. For example, a version constraint of 1.2 will result in an attempt to install version 1.2.0, even if 1.2.1 is available. When writing the dependency locks file, we previously would write 1.2.*, as this is the in-memory representation of 1.2. This would then cause an error on re-reading the locks file, as this is not a valid constraint format. Instead, we now explicitly convert the constraint to its zero-filled representation before writing the locks file. This ensures that it correctly round-trips. Because this change is made in getproviders.VersionConstraintsString, it also affects the output of the providers sub-command.	6 years ago
James Bardin	0b31ffa587	use a single log writer ... Use a single log writer instance for all std library logging. Setup the std log writer in the logging package, and remove boilerplate from test packages.	6 years ago
James Bardin	6ca477f042	move helper/logging to internal ... remove a dead code file too	6 years ago
Martin Atkins	b3f5c7f1e6	command/init: Read, respect, and update provider dependency locks ... This changes the approach used by the provider installer to remember between runs which selections it has previously made, using the lock file format implemented in internal/depsfile. This means that version constraints in the configuration are considered only for providers we've not seen before or when -upgrade mode is active.	6 years ago
Martin Atkins	59b116f7bf	command/init: Remove support for legacy provider addresses ... We no longer need to support 0.12-and-earlier-style provider addresses because users should've upgraded their existing configurations and states on Terraform 0.13 already. For now this is only checked in the "init" command, because various test shims are still relying on the idea of legacy providers the core layer. However, rejecting these during init is sufficient grounds to avoid supporting legacy provider addresses in the new dependency lock file format, and thus sets the stage for a more severe removal of legacy provider support in a later commit.	6 years ago
Martin Atkins	0b734a2803	command: Make provider installation interruptible ... In earlier commits we started to make the installation codepath context-aware so that it could be canceled in the event of a SIGINT, but we didn't complete wiring that through the API of the getproviders package. Here we make the getproviders.Source interface methods, along with some other functions that can make network requests, take a context.Context argument and act appropriately if that context is cancelled. The main providercache.Installer.EnsureProviderVersions method now also has some context-awareness so that it can abort its work early if its context reports any sort of error. That avoids waiting for the process to wind through all of the remaining iterations of the various loops, logging each request failure separately, and instead returns just a single aggregate "canceled" error. We can then set things up in the "terraform init" and "terraform providers mirror" commands so that the context will be cancelled if we get an interrupt signal, allowing provider installation to abort early while still atomically completing any local-side effects that may have started.	6 years ago
Martin Atkins	ef64df950c	getproviders: Prepare for having multiple valid hashes per package ... As we continue iterating towards saving valid hashes for a package in a depsfile lock file after installation and verifying them on future installation, this prepares getproviders for the possibility of having multiple valid hashes per package. This will arise in future commits for two reasons: - We will need to support both the legacy "zip hash" hashing scheme and the new-style content-based hashing scheme because currently the registry protocol is only able to produce the legacy scheme, but our other installation sources prefer the content-based scheme. Therefore packages will typically have a mixture of hashes of both types. - Installing from an upstream registry will save the hashes for the packages across all supported platforms, rather than just the current platform, and we'll consider all of those valid for future installation if we see both successful matching of the current platform checksum and a signature verification for the checksums file as a whole. This also includes some more preparation for the second case above in that signatureAuthentication now supports AcceptableHashes and returns all of the zip-based hashes it can find in the checksums file. This is a bit of an abstraction leak because previously that authenticator considered its "document" to just be opaque bytes, but we want to make sure that we can only end up trusting _all_ of the hashes if we've verified that the document is signed. Hopefully we'll make this better in a future commit with some refactoring, but that's deferred for now in order to minimize disruption to existing codepaths while we work towards a provider locking MVP.	6 years ago
Martin Atkins	6694cfaa0e	getproviders: Add a real type Hash for package hashes ... The logic for what constitutes a valid hash and how different hash schemes are represented was starting to get sprawled over many different files and packages. Consistently with other cases where we've used named types to gather the definition of a particular string into a single place and have the Go compiler help us use it properly, this introduces both getproviders.Hash representing a hash value and getproviders.HashScheme representing the idea of a particular hash scheme. Most of this changeset is updating existing uses of primitive strings to uses of getproviders.Hash. The new type definitions are in internal/getproviders/hash.go.	6 years ago
Martin Atkins	264a3cf031	depsfile: Flatten the "hashes" locks to a single set of strings ... Although origin registries return specific [filename, hash] pairs, our various different installation methods can't produce a structured mapping from platform to hash without breaking changes. Therefore, as a compromise, we'll continue to do platform-specific checks against upstream data in the cases where that's possible (installation from origin registry or network mirror) but we'll treat the lock file as just a flat set of equally-valid hashes, at least one of which must match after we've completed whatever checks we've made against the upstream-provided checksums/signatures. This includes only the minimal internal/getproviders updates required to make this compile. A subsequent commit will update that package to actually support the idea of verifying against multiple hashes.	6 years ago
Martin Atkins	b2c0ccdf96	internal/getproviders: Allow PackageMeta to carry acceptable hashes ... The "acceptable hashes" for a package is a set of hashes that the upstream source considers to be good hashes for checking whether future installs of the same provider version are considered to match this one. Because the acceptable hashes are a package authentication concern and they already need to be known (at least in part) to implement the authenticators, here we add AcceptableHashes as an optional extra method that an authenticator can implement. Because these are hashes chosen by the upstream system, the caller must make its own determination about their trustworthiness. The result of authentication is likely to be an input to that, for example by distrusting hashes produced by an authenticator that succeeds but doesn't report having validated anything.	6 years ago
Martin Atkins	e843097e52	internal/getproviders: Formalize the "ziphash" hashing scheme ... This is the pre-existing hashing scheme that was initially built for releases.hashicorp.com and then later reused for the provider registry protocol, which takes a SHA256 hash of the official distribution .zip file and formats it as lowercase hex. This is a non-ideal hash scheme because it works only for PackageLocalArchive locations, and so we can't verify package directories on local disk against such hashes. However, the registry protocol is now a compatibility constraint and so we're going to need to support this hashing scheme for the foreseeable future.	6 years ago