Merge remote-tracking branch 'origin/master' into azr_pick_keygen_type

CHANGELOG.md

@@ -4,6 +4,8 @@
 
 * builder/azure-chroot: Fix typo in option `exlude_from_latest` to
     `exclude_from_latest`. Old name will still be respected. [GH-10034]
+* builder/openstack: Fix source image validation regression when using filters.
+    [GH-10065]
 * core/hcl2: Packer HCL's "Coalesce" function now behaves same way as
     Terraform's. [GH-10016]
 * core/HCL: Hide sensitive variables from output. [GH-10031]

builder/azure/arm/azure_error_response_test.TestAzureErrorNestedShouldFormat.approved.txt

@@ -1,4 +1,4 @@
-ERROR: -> DeploymentFailed : At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.
-ERROR:   -> BadRequest
-ERROR:   -> InvalidRequestFormat : Cannot parse the request.
-ERROR:     -> InvalidJson : Error converting value "playground" to type 'Microsoft.WindowsAzure.Networking.Nrp.Frontend.Contract.Csm.Public.IpAllocationMethod'. Path 'properties.publicIPAllocationMethod', line 1, position 130.
+ERROR: -> DeploymentFailed : At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.
+ERROR:   -> BadRequest
+ERROR:   -> InvalidRequestFormat : Cannot parse the request.
+ERROR:     -> InvalidJson : Error converting value "playground" to type 'Microsoft.WindowsAzure.Networking.Nrp.Frontend.Contract.Csm.Public.IpAllocationMethod'. Path 'properties.publicIPAllocationMethod', line 1, position 130.

builder/azure/arm/azure_error_response_test.TestAzureErrorSimpleShouldFormat.approved.txt

@@ -1 +1 @@
-ERROR: -> ResourceNotFound : The Resource 'Microsoft.Compute/images/PackerUbuntuImage' under resource group 'packer-test00' was not found.
+ERROR: -> ResourceNotFound : The Resource 'Microsoft.Compute/images/PackerUbuntuImage' under resource group 'packer-test00' was not found.

builder/qemu/config.go

@@ -208,19 +208,27 @@ type Config struct {
 	// the builder. By default this is output-BUILDNAME where "BUILDNAME" is the
 	// name of the build.
 	OutputDir string `mapstructure:"output_directory" required:"false"`
-	// Allows complete control over the qemu command line (though not, at this
-	// time, qemu-img). Each array of strings makes up a command line switch
+	// Allows complete control over the qemu command line (though not qemu-img).
+	// Each array of strings makes up a command line switch
 	// that overrides matching default switch/value pairs. Any value specified
 	// as an empty string is ignored. All values after the switch are
 	// concatenated with no separator.
 	//
 	// ~> **Warning:** The qemu command line allows extreme flexibility, so
-	// beware of conflicting arguments causing failures of your run. For
-	// instance, using --no-acpi could break the ability to send power signal
-	// type commands (e.g., shutdown -P now) to the virtual machine, thus
-	// preventing proper shutdown. To see the defaults, look in the packer.log
-	// file and search for the qemu-system-x86 command. The arguments are all
-	// printed for review.
+	// beware of conflicting arguments causing failures of your run.
+	// For instance adding a "--drive" or "--device" override will mean that
+	// none of the default configuration Packer sets will be used. To see the
+	// defaults that Packer sets, look in your packer.log
+	// file (set PACKER_LOG=1 to get verbose logging) and search for the
+	// qemu-system-x86 command. The arguments are all printed for review, and
+	// you can use those arguments along with the template engines allowed
+	// by qemu-args to set up a working configuration that includes both the
+	// Packer defaults and your extra arguments.
+	//
+	// Another pitfall could be setting arguments like --no-acpi, which could
+	// break the ability to send power signal type commands
+	// (e.g., shutdown -P now) to the virtual machine, thus preventing proper
+	// shutdown.
 	//
 	// The following shows a sample usage:
 	//

website/pages/docs/provisioners/ansible.mdx

@@ -25,6 +25,10 @@ accept jinja2 `{{ function }}` macro syntax in a way that can be preserved to
 the Ansible run. If you need to set variables using Ansible macros, you need to
 do so inside your playbooks or inventory files.
 
+
+Please see the [Debugging](#debugging), [Limitations](#limitations), or [Troubleshooting](#troubleshooting) if you are having trouble
+getting started.
+
 ## Basic Example
 
 This is a fully functional template that will provision an image on
@@ -575,8 +579,7 @@ Example Packer template:
       "groups": [ "webserver" ],
       "playbook_file": "./webserver.yml",
       "extra_arguments": [
-        "--extra-vars",
-        "ansible_host={{user `ansible_host`}} ansible_connection={{user `ansible_connection`}}"
+        "--extra-vars", "ansible_host={{user `ansible_host`}} ansible_connection={{user `ansible_connection`}}"
       ]
     }
   ]
@@ -630,6 +633,147 @@ Example playbook:
         name: httpd
 ```
 
+### Amazon Session Manager
+
+When trying to use Ansible with Amazon's Session Manager, you may run into an error where Ansible
+is unable to connect to the remote Amazon instance if the local proxy adapter for Ansible [use_proxy](#use_proxy) is false.
+
+The error may look something like the following:
+
+```
+amazon-ebs: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 127.0.0.1 port 8362: Connection timed out", "unreachable": true}
+```
+
+The error is caused by a limitation on using Amazon's SSM default Port Forwarding session which only allows for one
+remote connection on the forwarded port. Since Ansible's SSH communication is not using the local proxy adapter
+it will try to make a new SSH connection to the same forwarded localhost port and fail.
+
+In order to workaround this issue Ansible can be configured via a custom inventory file to use the AWS session-manager-plugin
+directly to create a new session, separate from the one created by Packer, at runtime to connect and remotely provision the instance.
+
+-> **Warning:** Please note that the default region configured for the `aws` cli must match the build region where the instance is being
+provisioned otherwise you may run into a TargetNotConnected error. Users can use `AWS_DEFAULT_REGION` to temporarily override
+their configured region.
+
+<Tabs>
+<Tab heading="JSON">
+
+```json
+  "provisioners": [
+    {
+      "type": "ansible",
+      "use_proxy": false,
+      "ansible_env_vars": ["PACKER_BUILD_NAME={{ build_name }}"],
+      "playbook_file":   "./playbooks/playbook_remote.yml",
+      "inventory_file_template": "{{ .HostAlias }} ansible_host={{ .ID }} ansible_user={{ .User }} ansible_ssh_common_args='-o StrictHostKeyChecking=no -o ProxyCommand=\"sh -c \\\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p\\\"\"'\n"
+    }
+  ]
+```
+
+</Tab>
+<Tab heading="HCL2">
+
+```hcl
+  provisioner "ansible" {
+      use_proxy               =  false
+      playbook_file           =    "./playbooks/playbook_remote.yml"
+      ansible_env_vars        =  ["PACKER_BUILD_NAME={{ build_name }}"]
+      inventory_file_template =  "{{ .HostAlias }} ansible_host={{ .ID }} ansible_user={{ .User }} ansible_ssh_common_args='-o StrictHostKeyChecking=no -o ProxyCommand=\"sh -c \\\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p\\\"\"'\n"
+    }
+```
+</Tab>
+</Tabs>
+
+Full Packer template example:
+
+<Tabs>
+<Tab heading="JSON">
+
+```json
+{
+  "variables": {
+    "instance_role": "SSMInstanceProfile"
+  },
+
+  "builders": [
+      {
+        "type": "amazon-ebs",
+        "region": "us-east-1",
+        "ami_name": "packer-ami-ansible",
+        "instance_type": "t2.micro",
+        "source_ami_filter": {
+            "filters": {
+                "virtualization-type": "hvm",
+                "name": "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*",
+                "root-device-type": "ebs"
+            },
+            "owners": [
+                "099720109477"
+            ],
+            "most_recent": true
+        },
+        "communicator": "ssh",
+        "ssh_username": "ubuntu",
+        "ssh_interface": "session_manager",
+        "iam_instance_profile":"{{user `instance_role`}}"
+      }
+  ],
+  "provisioners": [
+    {
+      "type": "ansible",
+      "use_proxy": false,
+      "ansible_env_vars": ["PACKER_BUILD_NAME={{ build_name }}"],
+      "playbook_file":   "./playbooks/playbook_remote.yml",
+      "inventory_file_template": "{{ .HostAlias }} ansible_host={{ .ID }} ansible_user={{ .User }} ansible_ssh_common_args='-o StrictHostKeyChecking=no -o ProxyCommand=\"sh -c \\\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p\\\"\"'\n"
+    }
+	]
+}
+
+```
+
+</Tab>
+<Tab heading="HCL2">
+
+```hcl
+
+variables {
+  instance_role = "SSMInstanceProfile"
+}
+
+source "amazon-ebs" "ansible-example" {
+  region        = "us-east-1"
+  ami_name      = "packer-ami-ansible"
+  instance_type = "t2.micro"
+  source_ami_filter {
+      filters = {
+          name                = "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*"
+          virtualization-type = "hvm"
+          root-device-type    = "ebs"
+      }
+      owners      = [ "099720109477" ]
+      most_recent =  true
+  }
+  communicator         = "ssh"
+  ssh_username         = "ubuntu"
+  ssh_interface        = "session_manager"
+  iam_instance_profile = var.instance_role
+}
+
+build {
+  sources = ["source.amazon-ebs.ansible-example"]
+
+  provisioner "ansible" {
+      use_proxy               =  false
+      playbook_file           =    "./playbooks/playbook_remote.yml"
+      ansible_env_vars        =  ["PACKER_BUILD_NAME={{ build_name }}"]
+      inventory_file_template =  "{{ .HostAlias }} ansible_host={{ .ID }} ansible_user={{ .User }} ansible_ssh_common_args='-o StrictHostKeyChecking=no -o ProxyCommand=\"sh -c \\\"aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p\\\"\"'\n"
+    }
+}
+```
+
+</Tab>
+</Tabs>
+
 ### Troubleshooting
 
 If you are using an Ansible version >= 2.8 and Packer hangs in the

website/pages/guides/hcl/from-json-v1.mdx

@@ -13,8 +13,22 @@ description: |-
 
 @include 'guides/hcl2-beta-note.mdx'
 
-We will soon provide a programatic way to transpose a v1 buildfile to a v1.5
-HCL file. In the meantime we will show how to manually do it.
+As of v1.6.4, Packer provides a tool to help you convert legacy JSON files to
+HCL2 files. To run it, you can use the `hcl2_upgrade` command.
+
+for example,
+
+```sh
+packer hcl2_upgrade mytemplate.json
+```
+
+will convert your packer template to a new HCL2 file in your current working
+directory named mytemplate.json.pkr.hcl. It is not a perfect converter yet;
+please open an issue if you find a problem with the conversion. Packer will not
+destroy your legacy json template, so this is not a risky command to call.
+
+Following is an explanation of how to manually upgrade a JSON template to an
+HCL2 template.
 
 The following file :
 
@@ -143,6 +157,7 @@ repeatable blocks with the same identifier. For example:
         }
       ],
     }
+]
 ```
 
 Becomes:
@@ -162,19 +177,19 @@ source "amazon-ebs" "example" {
         delete_on_termination = true
         encrypted = true
     }
+}
 ```
 
 There is soon going to be a PR to drop the `s` at the end of these fields.
 
 ### Deprecation
 
-The current layout of buildfiles will be supported until we and the community
-love the new format. Only then the v1 format will be carefully deprecated.
+As we become more confident in the new templates, we may begin to add new
+features that are HCL2-only; one of our major motivations to moving to the new
+template format is that HCL2 provides us with the flexibility to implement some
+features which would be very difficult to add to the legacy JSON templates.
 
--> **Note:** The HCL parsing library can read JSON and if it is your
-configuration format of predilection, you will still be able to do it. You will
-have to tweak a few things in order to use future versions of Packer that have
-deprecated the current format. Sorry about that! Because the HCL reading code
-is generated from the JSON parsing settings; every builder, provisioner and
-post-processor setting should look and work the same. A config file transposer
-is currently in the making.
+However, the Packer team will continue to support the main functionality of the
+current "legacy JSON" packer templates alongside the new HCL2 templates until
+we and the community love the new templates. Only then the v1 format will be
+deprecated. We do not anticipate this happening until late 2021 at the earliest.

website/pages/partials/builder/qemu/Config-not-required.mdx

@@ -136,19 +136,27 @@
   the builder. By default this is output-BUILDNAME where "BUILDNAME" is the
   name of the build.
 
-- `qemuargs` ([][]string) - Allows complete control over the qemu command line (though not, at this
-  time, qemu-img). Each array of strings makes up a command line switch
+- `qemuargs` ([][]string) - Allows complete control over the qemu command line (though not qemu-img).
+  Each array of strings makes up a command line switch
   that overrides matching default switch/value pairs. Any value specified
   as an empty string is ignored. All values after the switch are
   concatenated with no separator.
   
   ~> **Warning:** The qemu command line allows extreme flexibility, so
-  beware of conflicting arguments causing failures of your run. For
-  instance, using --no-acpi could break the ability to send power signal
-  type commands (e.g., shutdown -P now) to the virtual machine, thus
-  preventing proper shutdown. To see the defaults, look in the packer.log
-  file and search for the qemu-system-x86 command. The arguments are all
-  printed for review.
+  beware of conflicting arguments causing failures of your run.
+  For instance adding a "--drive" or "--device" override will mean that
+  none of the default configuration Packer sets will be used. To see the
+  defaults that Packer sets, look in your packer.log
+  file (set PACKER_LOG=1 to get verbose logging) and search for the
+  qemu-system-x86 command. The arguments are all printed for review, and
+  you can use those arguments along with the template engines allowed
+  by qemu-args to set up a working configuration that includes both the
+  Packer defaults and your extra arguments.
+  
+  Another pitfall could be setting arguments like --no-acpi, which could
+  break the ability to send power signal type commands
+  (e.g., shutdown -P now) to the virtual machine, thus preventing proper
+  shutdown.
   
   The following shows a sample usage: