14 Commits (27770ee8053da48fbb684fd23d415aae376d2bed)

Author	SHA1	Message	Date
Bruno Schaatsbergen	51376781ec	getmodules: organize go-getter getter sets ... - Grouped decompressor extensions (bz2, gzip, xz, zip). - Grouped getters (protocol-based, cloud storage, version control, file-based). Signed-off-by: Bruno Schaatsbergen <git@bschaatsbergen.com>	1 year ago
Bruno Schaatsbergen	3ff7f17b0b	getmodules: support `tbz2` module archives ... The `tbz2` extension, a shorthand for `tar.bz2`, is supported for downloading and handling compressed archives in go-getter, which already treats both `tbz2` and `tar.bz2` as using the same bzip2 decompressor. This commit adds support for the `tbz2` decompressor when fetching modules from a source. Signed-off-by: Bruno Schaatsbergen <git@bschaatsbergen.com>	1 year ago
James Bardin	04b442d419	allow slashes in git url parameters ... Git url refs often contain slashes, which were incorrectly being split and appended to the path as a go-getter subdir.	2 years ago
Martin Atkins	84df5562f2	moduleaddrs: Terraform now owns module source address normalization ... We've been gradually chipping away at how much we use go-getter for source packages, because it's generally been a bit of a nightmare and sharing it with other codebases means that any time someone wants to change something we end up needing to find some way to prevent it breaking Terraform's compatibility promises. Here we make one further step: Terraform owns the "detectors" idea that deals with source address normalization, and now always produces fully-qualified addresses for go-getter to chew on only for the getting and decompressing steps. Retaining go-getter for the actual getting part is helpful because we can then benefit from security fixes upstream, but Terraform owning the first layer of parsing means that we can fix in place the definition of what "module source address" syntax means, and thus we can avoid having everything in this codebase indirectly depend on go-getter just because it wants to parse module source addresses. Now only the module installer actually depends indirectly on go-getter, which finally disconnects go-getter's subtree from all of the remote state backend dependency graphs.	2 years ago
Martin Atkins	5d87e32eb9	moduleaddrs: Split module source address parsing from package addrs ... Our package addrs ends up getting imported from just about ever other package in Terraform, because it contains the types we use to talk about various different kinds of objects. Therefore we typically try to keep its transitive dependency graph relatively small, because anything it depends on becomes an indirect dependency of nearly everything else. A while back we moved the module source address models into package addrs, which also brought with them the code for parsing strings to produce those addresses. Unfortunately, remote module source addresses are defined using the external dependency go-getter, which is pretty heavy itself and also brings with it numerous other external dependencies such as the AWS SDK, the Google Cloud Platform SDK, etc. Since only relatively few packages actually need to _parse_ source addresses -- with most either not caring about them at all or only consuming addresses that were already parsed by someone else -- we'll move the parser functions into their own package, while keeping the resulting address types in package addrs. This does still retain the package addrs dependency on external module github.com/hashicorp/terraform-registry-address, which is not ideal but that one at least has a relatively shallow dependency subgraph, so there's not so much urgency to tidy that one.	2 years ago
hashicorp-copywrite[bot]	53c34ff49c	Update copyright file headers to BUSL-1.1	3 years ago
hashicorp-copywrite[bot]	325d18262e	[COMPLIANCE] Add Copyright and License Headers	3 years ago
James Bardin	47f9850f02	add XTerraformGetLimit to prevent redirect loops	4 years ago
Martin Atkins	74761b2f8b	getmodules: Use go-getter v1.5.10 and return to upstream GitGetter ... There was an unintended regression in go-getter v1.5.9's GitGetter which caused us to temporarily fork that particular getter into Terraform to expedite a fix. However, upstream v1.5.10 now includes a functionally-equivalent fix and so we can heal that fork by upgrading. We'd also neglected to update the Module Sources docs when upgrading to go-getter v1.5.9 originally and so we were missing documentation about the new "depth" argument to enable shadow cloning, which I've added retroactively here along with documenting its restriction of only supporting named refs. This new go-getter release also introduces a new credentials-passing method for the Google Cloud Storage getter, and so we must incorporate that into the Terraform-level documentation about module sources.	4 years ago
Martin Atkins	c4d46e7c6b	getmodules: Re-allow git:: source with ref=COMMIT_ID ... Earlier versions of this code allowed "ref" to take any value that would be accepted by "git checkout" as a valid target of a symbolic ref. We inadvertently accepted a breaking change to upstream go-getter that broke that as part of introducing a shallow clone optimization, because shallow clone requires selecting a single branch. To restore the previous capabilities while retaining the "depth" argument, here we accept a compromise where "ref" has the stronger requirement of being a valid named ref in the remote repository if and only if "depth" is set to a value greater than zero. If depth isn't set or is less than one, we will do the old behavior of just cloning all of the refs in the remote repository in full and then switching to refer to the selected branch, tag, or naked commit ID as a separate step. This includes a heuristic to generate an additional error message hint if we get an error from "git clone" and it looks like the user might've been trying to use "depth" and "ref=COMMIT" together. We can't recognize that error accurately because it's only reported as human-oriented git command output, but this heuristic should hopefully minimize situations where we show it inappropriately. For now this is a change in the Terraform repository directly, so that we can expedite the fix to an already-reported regression. After this is released I tend to also submit a similar set of changes to upstream go-getter, at which point we can revert Terraform to using the upstream getter.GitGetter instead of our own local fork.	4 years ago
Martin Atkins	b0ff17ef2a	getmodules: Inline our own fork of getter.GitGetter ... This is a pragmatic temporary solution to allow us to more quickly resolve an upstream regression in go-getter locally within Terraform, so that the work to upstream it for other callers can happen asynchronously and with less time pressure. This commit doesn't yet include any changes to address the bug, and instead aims to be functionally equivalent to getter.GitGetter. A subsequent commit will then address the regression, so that the diff of that commit will be easier to apply later to the upstream to get the same effect there.	4 years ago
kmoe	40ec62c139	command: make module installation interruptible ... Earlier work to make "terraform init" interruptible made the getproviders package context-aware in order to allow provider installation to be cancelled. Here we make a similar change for module installation, which is now also cancellable with SIGINT. This involves plumbing context through initwd and getmodules. Functions which can make network requests now include a context parameter whose cancellation cancels those requests. Since the module installation code is shared, "terraform get" is now also interruptible during module installation.	4 years ago
Martin Atkins	1a8da65314	Refactoring of module source addresses and module installation ... It's been a long while since we gave close attention to the codepaths for module source address parsing and external module package installation. Due to their age, these codepaths often diverged from our modern practices such as representing address types in the addrs package, and encapsulating package installation details only in a particular location. In particular, this refactor makes source address parsing a separate step from module installation, which therefore makes the result of that parsing available to other Terraform subsystems which work with the configuration representation objects. This also presented the opportunity to better encapsulate our use of go-getter into a new package "getmodules" (echoing "getproviders"), which is intended to be the only part of Terraform that directly interacts with go-getter. This is largely just a refactor of the existing functionality into a new code organization, but there is one notable change in behavior here: the source address parsing now happens during configuration loading rather than module installation, which may cause errors about invalid addresses to be returned in different situations than before. That counts as backward compatible because we only promise to remain compatible with configurations that are _valid_, which means that they can be initialized, planned, and applied without any errors. This doesn't introduce any new error cases, and instead just makes a pre-existing error case be detected earlier. Our module registry client is still using its own special module address type from registry/regsrc for now, with a small shim from the new addrs.ModuleSourceRegistry type. Hopefully in a later commit we'll also rework the registry client to work with the new address type, but this commit is already big enough as it is.	5 years ago
Martin Atkins	f25935649a	getmodules: Beginnings of a new package about Terraform module packages ... This new package aims to encapsulate all of our interactions with go-getter to fetch remote module packages, to ensure that the rest of Terraform will only use the small subset of go-getter functionality that our modern module installer uses. In older versions of Terraform, go-getter was the entire implementation of module installation, but along the way we found that several aspects of its design are poor fit for Terraform's needs, and so now we're using it as just an implementation detail of Terraform's handling of remote module packages only, hiding it behind this wrapper API which exposes only the services that our module installer needs. This new package isn't actually used yet, but in a later commit we will change all of the other callers to go-getter to only work indirectly through this package, so that this will be the only package that actually imports the go-getter packages.	5 years ago