website: Reconcile docs about plugin discovery and downloading (#18973)

website: Reconcile docs about plugin discovery and downloading

I'm attempting to keep things simple for normal users while making sure we've
got the full behavior written down somewhere for plugin developers.

This commit doesn't stand alone; it's paired with a commit in the
terraform-website repo, to deal with some related content in the "extend"
section.

tools/terraform-bundle/README.md

@@ -9,6 +9,10 @@ work with a particular configuration, but sometimes Terraform is deployed in
 a network that, for one reason or another, cannot access the official
 plugin repository for automatic download.
 
+In some cases, this can be solved by installing provider plugins into the
+[user plugins directory](https://www.terraform.io/docs/configuration/providers.html#third-party-plugins).
+However, this doesn't always meet the needs of automated deployments.
+
 `terraform-bundle` provides an alternative, by allowing the auto-download
 process to be run out-of-band on a separate machine that _does_ have access
 to the repository. The result is a zip file that can be extracted onto the
@@ -54,7 +58,7 @@ providers {
   # the bundle archive.
   google = ["~> 1.0", "~> 2.0"]
 
-  # Include a custom plugin to the bundle. Will search for the plugin in the 
+  # Include a custom plugin to the bundle. Will search for the plugin in the
   # plugins directory, and package it with the bundle archive. Plugin must have
   # a name of the form: terraform-provider-*, and must be build with the operating
   # system and architecture that terraform enterprise is running, e.g. linux and amd64
@@ -107,11 +111,12 @@ distinguished from official release archives and from each other when multiple
 bundles contain the same core Terraform version.
 
 To include custom plugins in the bundle file, create a local directory "./plugins"
-and put all the plugins you want to include there. Optionally, you can use the 
+and put all the plugins you want to include there. Optionally, you can use the
 `-plugin-dir` flag to specify a location where to find the plugins. To be recognized
-as a valid plugin, the file must have a name of the form: "terraform-provider-*-v*". In 
-addition, ensure that the plugin is build using the same operating system and 
-architecture used for terraform enterprise. Typically this will be linux and amd64.
+as a valid plugin, the file must have a name of the form
+`terraform-provider-<NAME>-v<VERSION>`. In
+addition, ensure that the plugin is built using the same operating system and
+architecture used for Terraform Enterprise. Typically this will be `linux` and `amd64`.
 
 ## Provider Resolution Behavior
 
@@ -123,7 +128,12 @@ Therefore if automatic installation is not desired, it is important to ensure
 that version constraints within Terraform configurations do not exclude all
 of the versions available from the bundle. If a suitable version cannot be
 found in the bundle, Terraform _will_ attempt to satisfy that dependency by
-automatic installation from the official repository.
+automatic installation from the official repository. If you want
+`terraform init` to explicitly fail instead of contacting the repository, pass
+the `-get-plugins=false` option.
+
+For full details about provider resolution, see
+[How Terraform Works: Plugin Discovery](https://www.terraform.io/docs/extend/how-terraform-works.html#discovery).
 
 The downloaded provider archives are verified using the same signature check
 that is used for auto-installed plugins, using Hashicorp's release key. At
@@ -133,21 +143,22 @@ that may change in a future version of this tool.
 ## Installing a Bundle in On-premises Terraform Enterprise
 
 If using a private install of Terraform Enterprise in an "air-gapped"
-environment, this tool can produce a custom _tool package_ for Terraform, which
+environment, this tool can produce a custom Terraform version package, which
 includes a set of provider plugins along with core Terraform.
 
 To create a suitable bundle, use the `-os` and `-arch` options as described
 above to produce a bundle targeting `linux_amd64`. You can then place this
 archive on an HTTP server reachable by the Terraform Enterprise hosts and
 install it as per
-[Managing Tool Versions](https://github.com/hashicorp/terraform-enterprise-modules/blob/master/docs/managing-tool-versions.md).
-
-After choosing the "Add Tool Version" button, be sure to set the Tool to
-"terraform" and then enter as the Version the generated bundle version from
-the bundle filename, which will be of the form `N.N.N-bundleYYYYMMDDHH`.
-Enter the URL at which the generated bundle archive can be found, and the
-SHA256 hash of the file which can be determined by running the tool
-`sha256sum` with the given file.
+[Administration: Managing Terraform Versions](https://www.terraform.io/docs/enterprise/private/admin/resources.html#managing-terraform-versions).
+
+After clicking the "Add Terraform Version" button:
+
+1. In the "Version" field, enter the generated bundle version from the bundle
+   filename, which will be of the form `N.N.N-bundleYYYYMMDDHH`.
+2. In the "URL" field, enter the URL where the generated bundle archive can be found.
+3. In the "SHA256 Checksum" field, enter the SHA256 hash of the file, which can
+   be found by running `sha256sum <FILE>` or `shasum -a256 <FILE>`.
 
 The new bundle version can then be selected as the Terraform version for
 any workspace. When selected, configurations that require only plugins

website/docs/commands/init.html.markdown

@@ -116,37 +116,42 @@ initialized with its child modules.
 
 ## Plugin Installation
 
-During init, the configuration is searched for both direct and indirect
-references to [providers](/docs/configuration/providers.html), and the plugins
-for the providers are retrieved from the plugin repository. The downloaded
-plugins are installed to a subdirectory of the working directory, and are thus
-local to that working directory.
-
-Re-running init with plugins already installed will install plugins only for
-any providers that were added to the configuration since the last init. Use
-`-upgrade` to additionally update already-installed plugins to the latest
-versions that comply with the version constraints given in configuration.
-
-To skip plugin installation, use `-get-plugins=false`.
-
-The automatic plugin installation behavior can be overridden by extracting
-the desired providers into a local directory and using the additional option
-`-plugin-dir=PATH`. When this option is specified, _only_ the given directory
-is consulted, which prevents Terraform from making requests to the plugin
-repository or looking for plugins in other local directories. Passing an empty
-string to `-plugin-dir` removes any previously recorded paths.
-
-Custom plugins can be used along with automatically installed plugins by
-placing them in `terraform.d/plugins/OS_ARCH/` inside the directory being
-initialized. Plugins found here will take precedence if they meet the required
-constraints in the configuration. The `init` command will continue to
-automatically download other plugins as needed.
-
-When plugins are automatically downloaded and installed, by default the
-contents are verified against an official HashiCorp release signature to
-ensure that they were not corrupted or tampered with during download. It is
-recommended to allow Terraform to make these checks, but if desired they may
-be disabled using the option `-verify-plugins=false`.
+During init, Terraform searches the configuration for both direct and indirect
+references to providers and attempts to load the required plugins.
+
+For [providers distributed by HashiCorp](/docs/providers/index.html),
+init will automatically download and install plugins if necessary. Plugins
+can also be manually installed in the user plugins directory, located at
+`~/.terraform.d/plugins` on most operating systems and
+`<APPLICATION DATA>\plugins` on Windows.
+
+For more information about configuring and installing providers, see
+[Configuration: Providers](/docs/configuration/providers.html).
+
+On subsequent runs, init only installs providers without acceptable versions
+installed. (This includes newly added providers, and providers whose installed
+versions can't meet the current version constraints.) Use `-upgrade` if you want
+to update _all_ providers to the newest acceptable version.
+
+You can modify `terraform init`'s plugin behavior with the following options:
+
+- `-upgrade` — Update all previously installed plugins to the newest version
+  that complies with the configuration's version constraints. This option does
+  not apply to manually installed plugins.
+- `-get-plugins=false` — Skips plugin installation. Terraform will use plugins
+  installed in the user plugins directory, and any plugins already installed
+  for the current working directory. If the installed plugins aren't sufficient
+  for the configuration, init fails.
+- `-plugin-dir=PATH` — Skips plugin installation and loads plugins _only_ from
+  the specified directory. This ignores the user plugins directory and any
+  plugins already installed in the current working directory. To restore the
+  default behavior after using this option, run init again and pass an empty
+  string to `-plugin-dir`.
+- `-verify-plugins=false` — Skips release signature validation when
+  installing downloaded plugins (not recommended). Official plugin releases are
+  digitally signed by HashiCorp, and Terraform verifies these signatures when
+  automatically downloading plugins. This option disables that verification.
+  (Terraform does not check signatures for manually installed plugins.)
 
 ## Running `terraform init` in automation
 

website/docs/configuration/providers.html.md

@@ -66,6 +66,14 @@ Provider initialization is one of the actions of `terraform init`. Running
 this command will download and initialize any providers that are not already
 initialized.
 
+Providers downloaded by `terraform init` are only installed for the current
+working directory; other working directories can have their own installed
+provider versions.
+
+Note that `terraform init` cannot automatically download providers that are not
+distributed by HashiCorp. See [Third-party Plugins](#third-party-plugins) below
+for installation instructions.
+
 For more information, see
 [the `terraform init` command](/docs/commands/init.html).
 
@@ -207,24 +215,67 @@ may also be used, but currently may cause errors when running `terraform destroy
 
 ## Third-party Plugins
 
-At present Terraform can automatically install only the providers distributed
-by HashiCorp. Third-party providers can be manually installed by placing
-their plugin executables in one of the following locations depending on the
-host operating system:
+Anyone can develop and distribute their own Terraform providers. (See
+[Writing Custom Providers](/docs/extend/writing-custom-providers.html) for more
+about provider development.) These third-party providers must be manually
+installed, since `terraform init` cannot automatically download them.
+
+Install third-party providers by placing their plugin executables in the user
+plugins directory. The user plugins directory is in one of the following
+locations, depending on the host operating system:
+
+Operating system  | User plugins directory
+------------------|-----------------------
+Windows           | `terraform.d\plugins` in your user's "Application Data" directory
+All other systems | `.terraform.d/plugins` in your user's home directory
+
+Once a plugin is installed, `terraform init` can initialize it normally.
 
-* On Windows, in the sub-path `terraform.d/plugins` beneath your user's
-  "Application Data" directory.
-* On all other systems, in the sub-path `.terraform.d/plugins` in your
-  user's home directory.
+Providers distributed by HashiCorp can also go in the user plugins directory. If
+a manually installed version meets the configuration's version constraints,
+Terraform will use it instead of downloading that provider. This is useful in
+airgapped environments and when testing pre-release provider builds.
 
-`terraform init` will search this directory for additional plugins during
-plugin initialization.
+### Plugin Names and Versions
 
-The naming scheme for provider plugins is `terraform-provider-NAME_vX.Y.Z`,
+The naming scheme for provider plugins is `terraform-provider-<NAME>_vX.Y.Z`,
 and Terraform uses the name to understand the name and version of a particular
-provider binary. Third-party plugins will often be distributed with an
-appropriate filename already set in the distribution archive so that it can
-be extracted directly into the plugin directory described above.
+provider binary.
+
+If multiple versions of a plugin are installed, Terraform will use the newest
+version that meets the configuration's version constraints.
+
+Third-party plugins are often distributed with an appropriate filename already
+set in the distribution archive, so that they can be extracted directly into the
+user plugins directory.
+
+### OS and Architecture Directories
+
+Terraform plugins are compiled for a specific operating system and architecture,
+and any plugins in the root of the user plugins directory must be compiled for
+the current system.
+
+If you use the same plugins directory on multiple systems, you can install
+plugins into subdirectories with a naming scheme of `<OS>_<ARCH>` (for example,
+`darwin_amd64`). Terraform uses plugins from the root of the plugins directory
+and from the subdirectory that corresponds to the current system, ignoring
+other subdirectories.
+
+Terraform's OS and architecture strings are the standard ones used by the Go
+language. The following are the most common:
+
+* `darwin_amd64`
+* `freebsd_386`
+* `freebsd_amd64`
+* `freebsd_arm`
+* `linux_386`
+* `linux_amd64`
+* `linux_arm`
+* `openbsd_386`
+* `openbsd_amd64`
+* `solaris_amd64`
+* `windows_386`
+* `windows_amd64`
 
 ## Provider Plugin Cache
 

website/docs/plugins/basics.html.md

@@ -10,7 +10,7 @@ description: |-
 
 ~> **Advanced topic!** Plugin development is a highly advanced
 topic in Terraform, and is not required knowledge for day-to-day usage.
-If you don't plan on writing any plugins, this section of the documentation is 
+If you don't plan on writing any plugins, this section of the documentation is
 not necessary to read. For general use of Terraform, please see our
 [Intro to Terraform](/intro/index.html) and [Getting
 Started](/intro/getting-started/install.html) guides.
@@ -34,15 +34,22 @@ The network communication and RPC is handled automatically by higher-level
 Terraform libraries. The exact interface to implement is documented
 in its respective documentation section.
 
-## Installing a Plugin
+## Installing Plugins
 
-To install a plugin distributed by a third party developer, place the binary
-(extracted from any containing zip file) in
-[the third-party plugins directory](/docs/configuration/providers.html#third-party-plugins).
+The [provider plugins distributed by HashiCorp](/docs/providers/index.html) are
+automatically installed by `terraform init`. Third-party plugins (both
+providers and provisioners) can be manually installed into the user plugins
+directory, located at `<APPLICATION DATA>\terraform.d\plugins` on Windows and
+`~/.terraform.d/plugins` on other systems.
 
-Provider plugin binaries are named with the prefix `terraform-provider-`,
-while provisioner plugins have the prefix `terraform-provisioner-`. Both
-are placed in the same directory.
+For more information, see:
+
+- [Configuring Providers](/docs/configuration/providers.html)
+- [Configuring Providers: Third-party Plugins](/docs/configuration/providers.html#third-party-plugins)
+
+For developer-centric documentation, see:
+
+- [How Terraform Works: Plugin Discovery](/docs/extend/how-terraform-works.html#discovery)
 
 ## Developing a Plugin
 

website/guides/running-terraform-in-automation.html.md

@@ -156,7 +156,7 @@ applied, as opposed to some later plan that also exists.
 Different orchestration tools address this in different ways, but generally
 this is implemented via a _build pipeline_ feature, where different steps
 can be applied in sequence, with later steps having access to data produced
-by earlier steps. 
+by earlier steps.
 
 The recommended approach is to allow only one plan to be outstanding at a
 time. When a plan is applied, any other existing plans that were produced
@@ -285,14 +285,14 @@ $ ls -lah /usr/lib/custom-terraform-plugins
 ```
 
 The version information at the end of the filenames is important so that
-Terraform can infer the version number of each plugin. It is allowed to
-concurrently install multiple versions of the same provider plugin,
-which will then be used to satisfy
+Terraform can infer the version number of each plugin. Multiple versions of the
+same provider plugin can be installed, and Terraform will use the newest one
+that matches the
 [provider version constraints](/docs/configuration/providers.html#provider-versions)
-from Terraform configurations.
+in the Terraform configuration.
 
 With this directory populated, the usual auto-download and
-[plugin discovery](/docs/plugins/basics.html#installing-a-plugin)
+[plugin discovery](/docs/extend/how-terraform-works.html#discovery)
 behavior can be bypassed using the `-plugin-dir` option to `terraform init`:
 
 * `terraform init -input=false -plugin-dir=/usr/lib/custom-terraform-plugins`
@@ -307,7 +307,7 @@ unique constraints within each organization.
 Plugins can also be provided along with the configuration by creating a
 `terraform.d/plugins/OS_ARCH` directory, which will be searched before
 automatically downloading additional plugins. The `-get-plugins=false` flag can
-be used to prevent Terraform from automatically downloading additional plugins. 
+be used to prevent Terraform from automatically downloading additional plugins.
 
 ## Terraform Enterprise