369 Commits (update_version)

Author	SHA1	Message	Date
anshulSharma	e8d5ddb38a	formatting multiple hcl files (#13362) ... * formatting multiple hcl files * recursive formating * test case fixing * test case fix * removed not required code * existing test code fix * go formatting * more test cases for new cases * doc changes * added additional check on error message in negative test case	10 months ago
karthik P	c109e73f12	Merging main	10 months ago
Karthik P	8e36b92ce3	Preserve User-Defined Variable File Order (#13350) ... * modifying the for loop to preserver user inputted variable files preference * adding test cases * pr comments + refactoring the for loop for better readability. * pr comments + refactoring the for loop for better readability. * pr comments | handling default case * adding additional test case * fixing test cases * fixing test cases. creating a subdirectory to add a test case for both hcl and json auto var files * adding test case for json auto var file	10 months ago
Karthik P	51eeadba3d	feat: add sum, startswith and endswith functions ... This commit adds 3 new HCL2 functions: * `sum`: computes the sum of a collection of numerical values * `startswith`: checks if a string has another as prefix * `endswith`: checks if a string has another as suffix	10 months ago
Martin Grogan	b1417b7f5c	hcp: use GetHCPPackerRegistryBlock in the hcl module	12 months ago
Martin Grogan	5e6e1a2217	hcl: add get GetHCPPackerRegistry method to PackerConfig ... This method will be used to validate and fetch the correct HCPPackerRefistry block to be used in HCP build hcl: check for multiple HCP Packer registry block hcl: support for top level HCP Packer Registry hcl: deprecation warning for build block based hcp config	12 months ago
Martin Grogan	86c8902b81	hcl: add packer_registry at root schema	12 months ago
Lucas Bajolet	98198b5af8	packer: make GetBuilds return *[]CoreBuild ... The GetBuilds function, available on both HCL2 and legacy JSON configuration objects, used to return the Build interface. This typing by interface is not useful in this instance, since all the uses of `GetBuilds' are self-contained within Packer, and we're never using any other implementation for it than `*CoreBuild`. We've been relying on the dynamic type for all the builds being *CoreBuild in some places of the code, so to avoid potential surprises in the future, we'll change the signature now so it returns only concrete types.	1 year ago
Devashish	a353260f5d	packer: add hcp-sbom provisioner ... The hcp-sbom provisioner is a provisioner that acts essentially like a download-only file provisioner, which also verifies the file downloaded is a SPDX/CycloneDX JSON-encoded SBOM file, and sets up its upload to HCP Packer later on.	1 year ago
Lucas Bajolet	9f3e32b9fc	hcl2template: add support for raw aws secrets ... As the SDK now supports it in the context of legacy templating engine, we add support in HCL2 for the aws_secretsmanager_raw function, which gets the raw value of a secret from aws secrets manager.	1 year ago
Martin Grogan	cb4965d53a	hcl2template: add anytrue function ... this function add the hcl2 anytrue function which takes a collection and return true if any of the element is true.	1 year ago
Martin Grogan	476ddc3810	hcl2template: add alltrue function ... add an hcl2 function that return true if all the value in a collection are true, this function was derived from terraform codebase	1 year ago
Gustavo Cruz	7f64ca11f6	fix: `packer validate` unsupported type error ... `packer validate` would output the same error message four times per unsupported root block type found in a template (e.g., 'src' instead of 'source'). This behavior was due to a function being called four times for each file on each stage of the parsing.	1 year ago
Lucas Bajolet	ac899c44c2	hcl2template: don't error on empty bucket slug ... When a user defines a `hcp_packer_registry` block in their `build` without a `bucket_name`, but they define it in their environment, Packer should not report the bucket_name being wrong.	1 year ago
Martin Grogan	0ddcbaf18f	hcl2template: add strcontains function ... The strcontains function check if a sub string is a indeed a subset of a given string. hcl2template: add strcontains function The strcontains function check if a sub string is a indeed a subset of a given string.	1 year ago
Lucas Bajolet	da06110cc3	hcl2template: add filebase64 function ... The filebase64 function aims to read and encode a file's contents into base64. This is mostly to support reading the content of a file that is not valid UTF-8, as is the case with the `file` function.	1 year ago
Lucas Bajolet	8d6b8da996	hcl2template: locals evaluation returns a variable ... Evaluating local variables used to be directly written to the PackerConfig while each variable was created. This was somewhat of an issue with testing, as we have a bunch of tests that relied on `PackerConfig.Variables` being set only when we actually write something. This is not really a concern for normal use, just for testing, but to limit the number of changes to the tests in hcl2template, I opted to change how variables' values are retained, so that evaluating a single variable returns a Variable in addition to hcl.Diagnostics, so we can reify the approach and only create the map of variables if there's something evaluated.	1 year ago
Lucas Bajolet	d28982c29a	hcl2template: fix datasource dependency detection ... When preparing the datasources to add into the DAG for evaluating the build prerequisites, we ended-up in a weird situation in which the datasources for each vertex pointed to the same one. This is because of the loop semantics of Go, where the same object is reused over and over again during each loop, so in the end every datasource vertex pointed to the same instance of a datasource block. To avoid this, we instead grab them through their reference, making the reference to the datasource purely local, and pointing to the actual datasource block, not the one scoped to the function.	1 year ago
Lucas Bajolet	0dcb8c02c2	hcl2template: don't use Walk for walking on DAG ... Walk uses a reverse topological order to walk on the graph, doing that visit concurrently if possible. This is nice as we can speed-up execution of datasources and locals, however since the `Variables` map stored in the config, and the production of the context for it, are not meant to be used concurrently, this means that we end-up in cases where Packer crashes because of concurrent accesses to that map. So until we can change this behaviour, we will fallback to using the sequential visit algorithm for those vertexes, therefore limiting the risk of those conflicts.	1 year ago
Lucas Bajolet	4a4b837386	hcl2template: rename `Name`->`LocalName` for local ... Local variables had an attribute called Name with the name of the local variable. However, when producing an error while walking the DAG of local/datasources, if an error is encountered during validation, the raw structure of the vertex was printed out, making the error message produced hard to understand. Therefore in order to clean it up, we rename the `Name` attribute for Local variables as `LocalName`, and introduce a `Name()` function for that block so that the complete name of the variable is clearly reported.	1 year ago
Lucas Bajolet	09774aaeb8	internal/dag: adapt Validate to not check for Root ... The implementation of the DAG as extracted from Terraform relied on a Root vertex being injected into the graph as the last node to visit. This is used as a sanity check for Terraform, but doesn't apply to our use-case for now, as we are always executing everything and have no need for this root node. Instead, we change how Validate operates so it does not error in case there is no valid root node for the graph, but enables us calling it to check for self-referencing edges, and circular dependencies.	1 year ago
Lucas Bajolet	586762564b	hcl2template: intro and add UseSequential init opt ... Following up on the DAG work, this commit adds a new option for initialisation that disables DAG on request. By default we are going to use the DAG approach, with an option to fallback to using the older algorithm for evaluation in case users end-up in an edge-case that prevents them from building a template.	1 year ago
Lucas Bajolet	56e08c2eff	hcl2template: add DAG-based eval for local/data ... As we have finished setting-up the codebase for it, this commit adds the logic that uses the internal DAG package, and is able to orchestrate evaluation of datasources and locals in a non-phased way. Instead, this code acts by first detecting the dependencies for those components, builds a graph from them, with edges representing the dependency links between them, and finally walking on the graph breadth-first to evaluate those components. This can act as a drop-in replacement for the current phased logic, but both should be supported until we are confident that the approach works, and that there are little to no bugs left to squash.	1 year ago
Lucas Bajolet	9c3ec44379	hcl2template: use refString for local/data deps ... When registering dependencies for datasources and locals, we now use refString. This allows for the functions that detect dependencies to not only be able to register the same types as dependencies, but instead generalises it to any type that refString supports, so data, local and var. This can then be leveraged for orchestrating evaluation of those components in a non-phased way (i.e. with a DAG for dependency management).	1 year ago
Lucas Bajolet	8ba998a5d0	hcl2template: add function to eval one datasource ... As with variables, this commit introduces a new function whose purpose is evaluating the datasource directly, without looking at its dependencies, or recursively trying to execute them. Instead, we rely on the DAG to determine when it is safe to execute it, or if using the phased approach, the current logic will apply.	1 year ago
Lucas Bajolet	8f1da2fa79	hcl2template: split local evaluation functions ... Since we are in the process of integrating a new way to orchestrate dependency management and evaluation for datsources and local variables, we need to split the current function that manages the evaluation of said local variables, so that recursion and the actual evaluation of a local variable are two separate functions. This will allow for evaluating a single variable once the dag is ready to be introduced.	1 year ago
Lucas Bajolet	e1b8edf8d8	hcl2template: rename Datasource.Name to DSName ... The `Name` function is used by the DAG library to determine which vertex is associated to a component, so the `Name` attribute/function needs to replicate the combination of type and name, so the vertex can be accurately fetched from the graph.	1 year ago
Lucas Bajolet	8e21597e30	hcl2template: add refstring type for generic refs ... The hcl2template package contains references already, but these are linked to a particular type. This becomes problematic if we want to support cross-type references, so this commit adds a new abstraction: refString. A refString contains the component type, its type (if applicable), and its name, so that the combination of those points to a cty object that can be linked to a block in the configuration. Right now, only `var`, `local` and `data` are supported, but the type is extensible enough that anything else that fits this model such as sources can be supported in the future potentially.	1 year ago
Wenfeng Pan	1b160e5df0	Add base64gzip function support to Packer template	2 years ago
Lucas Bajolet	3e3b136f3c	packer: address unused lint errors	2 years ago
Lucas Bajolet	36e43e30ee	hcl2template: detect duplicate locals during parse ... Previously duplicate detection for local variables happened during `Initialise`, through a call to `checkForDuplicateLocalDefinition`. This works in a majority of cases, but for commands like `console`, this was not detected as the return diagnostics for `Initialise` are ignored. That check can be done as early as during parsing however, as the names of blocks are not dynamic in the slightest (no interpolation possible), so we move that detection logic into `Parse`, so that the behaviour is coherent between all commands.	2 years ago
Lucas Bajolet	14cf4b40d4	hcl2template: recursively evaluate local variables ... The logic for evaluating local variables used to rely on their definition order, with some edge cases. Typically `locals` blocks define multiple local variables, which don't necessarily appear in the same order at evaluation as within the template, leading to inconsistent behaviour, as the order in which those are added to the list of local variables is non-deterministic. To avoid this problem, we change how local variables are evaluated, and we're adopting a workflow similar to datasources, where the local variables first build a list of direct dependencies. Then when evaluation happens, we evaluate all the dependencies recursively for each local variable, which takes care of this issue. As with Datasources, we add a cap to the recursion: 10. I.e. if the evaluation of a single variable provokes an infinite recursion, we stop at that point and return an error to the user, telling them to fix their template.	2 years ago
Lucas Bajolet	5673af108f	hcl2template: split GetVarsByType in two functions ... GetVarsByType is a function that gets a list of Traversals from a hcl Block. This approach works when what we are visiting is indeed one, however when we can get an immediate list of Traversals, but want to filter them based on their roots, we have to reimplement parts of that function. Therefore, we split this function in two, GetVarsByType still keeps its current behaviour, but the filtering step is exposed as another function now: FilterTraversalsByType, so we can reuse it elsewhere.	2 years ago
Lucas Bajolet	b4a843c4e2	commands: reject constraints with pre-releases (#12999) ... When remotely installing a plugin, constraints are used by Packer to determine which version of a plugin to install. These constraints can be arbitrarily complex, including operators and ranges in which to look for valid versions. However, the versions specified in those constraints should always be final releases, and not a pre-release since we don't explicitly support remotely installing pre-releases. This commit therefore addds checks to make sure these are reported ASAP, even before the source is contacted to list releases and picking one to install.	2 years ago
Lucas Bajolet	accbe97e1e	hcl2template: add text(encode|decode)base64 funcs (#12997) ... Compared to Terraform, Packer was lacking a capability to encode/decode strings to/from base64-encoded text encoded with another encoding. This could be problematic in some cases, mainly when working with Windows, as most of the OS uses UTF-16LE as its standard encoding for many operations. Therefore, we take a page from Terraform here, and add those functions to what Packer supports in an HCL2 context.	2 years ago
Lucas Bajolet	ec682afc4a	addrs: require at least 3 parts to source URIs	2 years ago
Lucas Bajolet	c7f2508373	addrs: return error on ParsePluginSource ... The ParsePluginSource function can be invoked from either a HCL2 context (when parsing a required_plugins block), or from the command-line itself. While in the first context a hcl.Diagnostics is coherent, in case the source to parse is a command-line argument, for example when installing or removing a plugin, the error message cannot have an HCL context, leading to errors that are incorrectly prefixed by a <nil> string dure to the lack of a reference to attach the diagnostic to. Therefore, in order to fix this behaviour, the logic that parses plugin sources now returns an error, and attaching the error to an HCL subject is done independently, if needed.	2 years ago
Lucas Bajolet	9b38e0eb5c	addrs: limit source component length to 16 ... When specifying/installing plugins, a source URI is required for Packer to be able to locate or install a plugin to the local plugin hierarchy. The plugin hierarchy is based on the plugin source, where each component in this hierarchy will become a directory. In order to avoid sources with too many levels of nesting, causing a lot of mkdirs, we limit the number of sources to 16 in this commit, this should be long enough for most of our users.	2 years ago
Lucas Bajolet	d2ade9a783	packer: register plugin components only once ... When running a packer command on an HCL2 template, depending on whether or not there are required_plugin blocks defined, Packer may need to discover and register a plugin's components multiple times. This is the behaviour ever since those blocks were introduced to Packer, but given we are doing the operation multiple times, this is suboptimal. This commit changes the way things works by first doing the restricted discovery of plugins (as dictated by required_plugins), then proceeding to the global discovery, with the change that subsequent component discoveries will not have precedence over those pre-discovered anymore. This allows us to invert the call order of both discovery phases safely, and maintains the constraints described in the templates.	2 years ago
Lucas Bajolet	6fc1d154bd	packer: relax constraints on sources ... The source parsing logic was heavily directed towards Github compatible source URIs, however if we want to support more cases, we need to make sure we are able to specify those URIs, and to load plugins installed from those sources. Right now, since the getters available are only github.com, we will not support remotely instlling plugins from sources other than github.com, with the same set of constraints as before. However, we do support now installing from a local plugin binary to any kind of source, and we support loading them, including if a template wants this plugin installed locally with version constraints.	2 years ago
Lucas Bajolet	1bf3e86ee1	hcl2template: fix invalid_inexplicit_source_2 test ... The invalid_inexplicit_source_2 subtest for parse used to fail in previous versions of Packer, but with the latest changes to plugin source management, it won't, at least at parsing time, instead it fails later when the Github getter (the only available for now) cannot get the plugin binary from the source, as the source is not a valid Github URI.	2 years ago
Lucas Bajolet	93bec31ec9	version: embed version/VERSION as version string ... Since we have both version/version.go and version/VERSION to specify version strings, both are a bit redundant. As version/VERSION is supposed to be the source of truth now, we are using it to derive the version informaiton we used to rely on in Packer and its subcommands. Note: doing this prevents us from changing the version/prerelease through ldflags though as we derive Version/VersionPrerelease from the rawVersion variable.	2 years ago
Devashish	595b45e67c	Packer tracks Version and Plugins Metadata (#12860)	2 years ago
Lucas Bajolet	ea1e798c47	fmt: print parse errors on error ... The fmt command reformats HCL2 templates, provided it can parse the file and reformat its contents according to the standards set by the HCL library's formatters. However, if the file is malformed for some reason, the command will fail with a parse error, but while the parse error message is shown, the actual errors in the template(s) are not forwarded, making it hard for users to understand what went wrong with the contents of the file they're trying to format. In order to be more helpful with those errors, we now forward those parsing errors to the UI.	2 years ago
Lucas Bajolet	9f7e6ca52b	plugins: add a release only flag to Discover ... Since we now support loading pre-releases, we also want Packer to be able to ignore them by user demand, so we put in place the infrastructure to modulate this.	2 years ago
Lucas Bajolet	e099c5c661	main: move Discover to DetectPluginBinaries ... When Packer is loaded, we used to perform plugin discovery. This was done for every call to Packer, including when it is executed as a plugin, arguably against what the comments document. Doing this as early in the loading process makes it harder to change this behaviour, as we'd need to introduce flags aside from the rest, and handle them manually, which is not optimal. Therefore, we change this: now when Packer starts executing, it will not attempt to discover installed plugins anymore, and instead will only try to load them when a configuration has been parsed, and is being used to perform actions (typically build/validate).	2 years ago
Lucas Bajolet	b51c350bd6	packer: fix plugin version sorting and pickup ... When Packer orders the plugins by their version, we'd assumed it was ordered from highest to lowest. However, this was the case because our implementation of `Less' was actually returning `>=', which is not what it was supposed to do. This commit therefore fixes the implementation of `Less' to do what it is documented to do, and ensures the behaviour is correct through additional testing. Changing this requires some changes to the loading process as well because of the aforementioned assumption with regards to ordering.	2 years ago
Lucas Bajolet	fd5f668ee9	plugin: consolidate loading logic ... Right now we had two paths for discovering installed plugins, i.e. through plugin-getter's `ListInstallations' function, or through the `Discover' call, which relied on a glob to list the installations. This was required since we allowed plugins to be installed in multiple locations, and with different constraints. Now that we force a certain convention, we can consolidate the logic into ListInstallations, and rely on that logic in `Discover' to load a plugin into the PluginConfig for the current Packer run.	2 years ago
Lucas Bajolet	4b00a81bf3	packer: remove directory slices from structs ... Since we'll only look in the plugin directory, and not from multiple sources now, for installing/listing plugins, we can simplify the structures which used to accept multiple directories so they only accept one.	2 years ago
Lucas Bajolet	94aaff44a1	packer: only support one directory for plugins ... Since we're removing the alternative plugin installation directories in favour of only supporting installing them in the PACKER_PLUGIN_PATH directory, we only return one directory when getting the known plugin directories.	2 years ago