website: guide for using the S3 backend with multiple AWS accounts

Users commonly ask how the S3 backend can be used in an organization that
splits its infrastructure across many AWS accounts.

We've traditionally shied away from making specific recommendations here
because we can't possibly anticipate the different standards and
regulations that constrain each user. This new section attempts to
describe one possible approach that works well with Terraform's workflow,
with the goal that users make adjustments to it taking into account their
unique needs.

Since we are intentionally not being prescriptive here -- instead
considering this just one of many approaches -- it deviates from our usual
active writing style in several places to avoid giving the impression that
these are instructions to be followed exactly, which in some cases
requires the use of passive voice even though that is contrary to our
documentation style guide. For similar reasons, this section is also
light on specific code examples, since we do not wish to encourage users
to just copy-paste the examples without thinking through the consequences.

website/docs/backends/types/s3.html.md

@@ -147,3 +147,221 @@ The following configuration options or environment variables are supported:
  * `skip_region_validation` - (Optional) Skip validation of provided region name.
  * `skip_requesting_account_id` - (Optional) Skip requesting the account ID.
  * `skip_metadata_api_check` - (Optional) Skip the AWS Metadata API check.
+
+## Multi-account AWS Architecture
+
+A common architectural pattern is for an organization to use a number of
+separate AWS accounts to isolate different teams and environments. For example,
+a "staging" system will often be deployed into a separate AWS account than
+its corresponding "production" system, to minimize the risk of the staging
+environment affecting production infrastructure, whether via rate limiting,
+misconfigured access controls, or other unintended interactions.
+
+The S3 backend can be used in a number of different ways that make different
+tradeoffs between convenience, security, and isolation in such an organization.
+This section describes one such approach that aims to find a good compromise
+between these tradeoffs, allowing use of
+[Terraform's workspaces feature](/docs/state/workspaces.html) to switch
+conveniently between multiple isolated deployments of the same configuration.
+
+Use this section as a starting-point for your approach, but note that
+you will probably need to make adjustments for the unique standards and
+regulations that apply to your organization. You will also need to make some
+adjustments to this approach to account for _existing_ practices within your
+organization, if for example other tools have previously been used to manage
+infrastructure.
+
+Terraform is an administrative tool that manages your infrastructure, and so
+ideally the infrastructure that is used by Terraform should exist outside of
+the infrastructure that Terraform manages. This can be achieved by creating a
+separate _administrative_ AWS account which contains the user accounts used by
+human operators and any infrastructure and tools used to manage the the other
+accounts. Isolating shared administrative tools from your main environments
+has a number of advantages, such as avoiding accidentally damaging the
+administrative infrastructure while changing the target infrastructure, and
+reducing the risk that an attacker might abuse production infrastructure to
+gain access to the (usually more privileged) administrative infrastructure.
+
+### Administrative Account Setup
+
+Your administrative AWS account will contain at least the following items:
+
+* One or more [IAM user](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html)
+  for system administrators that will log in to maintain infrastructure in
+  the other accounts.
+* Optionally, one or more [IAM groups](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups.html)
+  to differentiate between different groups of users that have different
+  levels of access to the other AWS accounts.
+* An [S3 bucket](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html)
+  that will contain the Terraform state files for each workspace.
+* A [DynamoDB table](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html#HowItWorks.CoreComponents.TablesItemsAttributes)
+  that will be used for locking to prevent concurrent operations on a single
+  workspace.
+
+Provide the S3 bucket name and DynamoDB table name to Terraform within the
+S3 backend configuration using the `bucket` and `lock_table` arguments
+respectively, and configure a suitable `workspace_key_prefix` to contain
+the states of the various workspaces that will subsequently be created for
+this configuration.
+
+### Environment Account Setup
+
+For the sake of this section, the term "environment account" refers to one
+of the accounts whose contents are managed by Terraform, separate from the
+administrative account described above.
+
+Your environment accounts will eventually contain your own product-specific
+infrastructure. Along with this it must contain one or more
+[IAM roles](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)
+that grant sufficient access for Terraform to perform the desired management
+tasks.
+
+### Delegating Access
+
+Each Administrator will run Terraform using credentials for their IAM user
+in the administrative account.
+[IAM Role Delegation](http://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html)
+is used to grant these users access to the roles created in each environment
+account.
+
+Full details on role delegation are covered in the AWS documentation linked
+above. The most important details are:
+
+* Each role's _Assume Role Policy_ must grant access to the administrative AWS
+  account, which creates a trust relationship with the administrative AWS
+  account so that its users may assume the role.
+* The users or groups within the administrative account must also have a
+  policy that creates the converse relationship, allowing these users or groups
+  to assume that role.
+
+Since the purpose of the administrative account is only to host tools for
+managing other accounts, it is useful to give the administrative accounts
+restricted access only to the specific operations needed to assume the
+environment account role and access the Terraform state. By blocking all
+other access, you remove the risk that user error will lead to staging or
+production resources being created in the administrative account by mistake.
+
+When configuring Terraform, use either environment variables or the standard
+credentials file `~/.aws/credentials` to provide the administrator user's
+IAM credentials within the administrative account to both the S3 backend _and_
+to Terraform's AWS provider.
+
+Use conditional configuration to pass a different `assume_role` value to
+the AWS provider depending on the selected workspace. For example:
+
+```hcl
+variable "workspace_iam_roles" {
+  default = {
+    staging    = "arn:aws:iam::STAGING-ACCOUNT-ID:role/Terraform"
+    production = "arn:aws:iam::PRODUCTION-ACCOUNT-ID:role/Terraform"
+  }
+}
+
+provider "aws" {
+  # No credentials explicitly set here because they come from either the
+  # environment or the global credentials file.
+
+  assume_role = "${var.workspace_iam_roles[terraform.workspace]}"
+}
+```
+
+If workspace IAM roles are centrally managed and shared across many separate
+Terraform configurations, the role ARNs could also be obtained via a data
+source such as [`terraform_remote_state`](/docs/providers/terraform/d/remote_state.html)
+to avoid repeating these values.
+
+### Creating and Selecting Workspaces
+
+With the necessary objects created and the backend configured, run
+`terraform init` to initialize the backend and establish an initial workspace
+called "default". This workspace will not be used, but is created automatically
+by Terraform as a convenience for users who are not using the workspaces
+feature.
+
+Create a workspace corresponding to each key given in the `workspace_iam_roles`
+variable value above:
+
+```
+$ terraform worspace new staging
+Created and switched to workspace "staging"!
+
+...
+
+$ terraform workspace new production
+Created and switched to workspace "production"!
+
+...
+```
+
+Due to the `assume_role` setting in the AWS provider configuration, any
+management operations for AWS resources will be performed via the configured
+role in the appropriate environment AWS account. The backend operations, such
+as reading and writing the state from S3, will be performed directly as the
+administrator's own user within the administrative account.
+
+```
+$ terraform workspace select staging
+$ terraform apply
+...
+```
+
+### Running Terraform in Amazon EC2
+
+Teams that make extensive use of Terraform for infrastructure management
+often [run Terraform in automation](/guides/running-terraform-in-automation.html)
+to ensure a consistent operating environment and to limit access to the
+various secrets and other sensitive information that Terraform configurations
+tend to require.
+
+When running Terraform in an automation tool running on an Amazon EC2 instance,
+consider running this instance in the administrative account and using an
+[instance profile](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html)
+in place of the various administrator IAM users suggested above. An IAM
+instance profile can also be granted cross-account delegation access via
+an IAM policy, giving this instance the access it needs to run Terraform.
+
+To isolate access to different environment accounts, use a separate EC2
+instance for each target account so that its access can be limited only to
+the single account.
+
+Similar approaches can be taken with equivalent features in other AWS compute
+services, such as ECS.
+
+### Protecting Access to Workspace State
+
+In a simple implementation of the pattern described in the prior sections,
+all users have access to read and write states for all workspaces. In many
+cases it is desirable to apply more precise access constraints to the
+Terraform state objects in S3, so that for example only trusted administrators
+are allowed to modify the production state, or to control _reading_ of a state
+that contains sensitive information.
+
+Amazon S3 supports fine-grained access control on a per-object-path basis
+using IAM policy. A full description of S3's access control mechanism is
+beyond the scope of this guide, but an example IAM policy granting access
+to only a single state object within an S3 bucket is shown below:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": "s3:ListBucket",
+      "Resource": "arn:aws:s3:::myorg-terraform-states"
+    },
+    {
+      "Effect": "Allow",
+      "Action": ["s3:GetObject", "s3:PutObject"],
+      "Resource": "arn:aws:s3:::myorg-terraform-states/myapp/production/tfstate"
+    }
+  ]
+}
+```
+
+It is not possible to apply such fine-grained access control to the DynamoDB
+table used for locking, so it is possible for any user with Terraform access
+to lock any workspace state, even if they do not have access to read or write
+that state. If a malicious user has such access they could block attempts to
+use Terraform against some or all of your workspaces as long as locking is
+enabled in the backend configuration.