353 Commits (6c40bfb0cdfd4e3ae3132d240387999cfb4500a2)

Author	SHA1	Message	Date
Lucas Bajolet	8d6b8da996	hcl2template: locals evaluation returns a variable ... Evaluating local variables used to be directly written to the PackerConfig while each variable was created. This was somewhat of an issue with testing, as we have a bunch of tests that relied on `PackerConfig.Variables` being set only when we actually write something. This is not really a concern for normal use, just for testing, but to limit the number of changes to the tests in hcl2template, I opted to change how variables' values are retained, so that evaluating a single variable returns a Variable in addition to hcl.Diagnostics, so we can reify the approach and only create the map of variables if there's something evaluated.	1 year ago
Lucas Bajolet	d28982c29a	hcl2template: fix datasource dependency detection ... When preparing the datasources to add into the DAG for evaluating the build prerequisites, we ended-up in a weird situation in which the datasources for each vertex pointed to the same one. This is because of the loop semantics of Go, where the same object is reused over and over again during each loop, so in the end every datasource vertex pointed to the same instance of a datasource block. To avoid this, we instead grab them through their reference, making the reference to the datasource purely local, and pointing to the actual datasource block, not the one scoped to the function.	1 year ago
Lucas Bajolet	0dcb8c02c2	hcl2template: don't use Walk for walking on DAG ... Walk uses a reverse topological order to walk on the graph, doing that visit concurrently if possible. This is nice as we can speed-up execution of datasources and locals, however since the `Variables` map stored in the config, and the production of the context for it, are not meant to be used concurrently, this means that we end-up in cases where Packer crashes because of concurrent accesses to that map. So until we can change this behaviour, we will fallback to using the sequential visit algorithm for those vertexes, therefore limiting the risk of those conflicts.	1 year ago
Lucas Bajolet	4a4b837386	hcl2template: rename `Name`->`LocalName` for local ... Local variables had an attribute called Name with the name of the local variable. However, when producing an error while walking the DAG of local/datasources, if an error is encountered during validation, the raw structure of the vertex was printed out, making the error message produced hard to understand. Therefore in order to clean it up, we rename the `Name` attribute for Local variables as `LocalName`, and introduce a `Name()` function for that block so that the complete name of the variable is clearly reported.	1 year ago
Lucas Bajolet	09774aaeb8	internal/dag: adapt Validate to not check for Root ... The implementation of the DAG as extracted from Terraform relied on a Root vertex being injected into the graph as the last node to visit. This is used as a sanity check for Terraform, but doesn't apply to our use-case for now, as we are always executing everything and have no need for this root node. Instead, we change how Validate operates so it does not error in case there is no valid root node for the graph, but enables us calling it to check for self-referencing edges, and circular dependencies.	1 year ago
Lucas Bajolet	586762564b	hcl2template: intro and add UseSequential init opt ... Following up on the DAG work, this commit adds a new option for initialisation that disables DAG on request. By default we are going to use the DAG approach, with an option to fallback to using the older algorithm for evaluation in case users end-up in an edge-case that prevents them from building a template.	1 year ago
Lucas Bajolet	56e08c2eff	hcl2template: add DAG-based eval for local/data ... As we have finished setting-up the codebase for it, this commit adds the logic that uses the internal DAG package, and is able to orchestrate evaluation of datasources and locals in a non-phased way. Instead, this code acts by first detecting the dependencies for those components, builds a graph from them, with edges representing the dependency links between them, and finally walking on the graph breadth-first to evaluate those components. This can act as a drop-in replacement for the current phased logic, but both should be supported until we are confident that the approach works, and that there are little to no bugs left to squash.	1 year ago
Lucas Bajolet	9c3ec44379	hcl2template: use refString for local/data deps ... When registering dependencies for datasources and locals, we now use refString. This allows for the functions that detect dependencies to not only be able to register the same types as dependencies, but instead generalises it to any type that refString supports, so data, local and var. This can then be leveraged for orchestrating evaluation of those components in a non-phased way (i.e. with a DAG for dependency management).	1 year ago
Lucas Bajolet	8ba998a5d0	hcl2template: add function to eval one datasource ... As with variables, this commit introduces a new function whose purpose is evaluating the datasource directly, without looking at its dependencies, or recursively trying to execute them. Instead, we rely on the DAG to determine when it is safe to execute it, or if using the phased approach, the current logic will apply.	1 year ago
Lucas Bajolet	8f1da2fa79	hcl2template: split local evaluation functions ... Since we are in the process of integrating a new way to orchestrate dependency management and evaluation for datsources and local variables, we need to split the current function that manages the evaluation of said local variables, so that recursion and the actual evaluation of a local variable are two separate functions. This will allow for evaluating a single variable once the dag is ready to be introduced.	1 year ago
Lucas Bajolet	e1b8edf8d8	hcl2template: rename Datasource.Name to DSName ... The `Name` function is used by the DAG library to determine which vertex is associated to a component, so the `Name` attribute/function needs to replicate the combination of type and name, so the vertex can be accurately fetched from the graph.	1 year ago
Lucas Bajolet	8e21597e30	hcl2template: add refstring type for generic refs ... The hcl2template package contains references already, but these are linked to a particular type. This becomes problematic if we want to support cross-type references, so this commit adds a new abstraction: refString. A refString contains the component type, its type (if applicable), and its name, so that the combination of those points to a cty object that can be linked to a block in the configuration. Right now, only `var`, `local` and `data` are supported, but the type is extensible enough that anything else that fits this model such as sources can be supported in the future potentially.	1 year ago
Wenfeng Pan	1b160e5df0	Add base64gzip function support to Packer template	2 years ago
Lucas Bajolet	3e3b136f3c	packer: address unused lint errors	2 years ago
Lucas Bajolet	36e43e30ee	hcl2template: detect duplicate locals during parse ... Previously duplicate detection for local variables happened during `Initialise`, through a call to `checkForDuplicateLocalDefinition`. This works in a majority of cases, but for commands like `console`, this was not detected as the return diagnostics for `Initialise` are ignored. That check can be done as early as during parsing however, as the names of blocks are not dynamic in the slightest (no interpolation possible), so we move that detection logic into `Parse`, so that the behaviour is coherent between all commands.	2 years ago
Lucas Bajolet	14cf4b40d4	hcl2template: recursively evaluate local variables ... The logic for evaluating local variables used to rely on their definition order, with some edge cases. Typically `locals` blocks define multiple local variables, which don't necessarily appear in the same order at evaluation as within the template, leading to inconsistent behaviour, as the order in which those are added to the list of local variables is non-deterministic. To avoid this problem, we change how local variables are evaluated, and we're adopting a workflow similar to datasources, where the local variables first build a list of direct dependencies. Then when evaluation happens, we evaluate all the dependencies recursively for each local variable, which takes care of this issue. As with Datasources, we add a cap to the recursion: 10. I.e. if the evaluation of a single variable provokes an infinite recursion, we stop at that point and return an error to the user, telling them to fix their template.	2 years ago
Lucas Bajolet	5673af108f	hcl2template: split GetVarsByType in two functions ... GetVarsByType is a function that gets a list of Traversals from a hcl Block. This approach works when what we are visiting is indeed one, however when we can get an immediate list of Traversals, but want to filter them based on their roots, we have to reimplement parts of that function. Therefore, we split this function in two, GetVarsByType still keeps its current behaviour, but the filtering step is exposed as another function now: FilterTraversalsByType, so we can reuse it elsewhere.	2 years ago
Lucas Bajolet	b4a843c4e2	commands: reject constraints with pre-releases (#12999) ... When remotely installing a plugin, constraints are used by Packer to determine which version of a plugin to install. These constraints can be arbitrarily complex, including operators and ranges in which to look for valid versions. However, the versions specified in those constraints should always be final releases, and not a pre-release since we don't explicitly support remotely installing pre-releases. This commit therefore addds checks to make sure these are reported ASAP, even before the source is contacted to list releases and picking one to install.	2 years ago
Lucas Bajolet	accbe97e1e	hcl2template: add text(encode|decode)base64 funcs (#12997) ... Compared to Terraform, Packer was lacking a capability to encode/decode strings to/from base64-encoded text encoded with another encoding. This could be problematic in some cases, mainly when working with Windows, as most of the OS uses UTF-16LE as its standard encoding for many operations. Therefore, we take a page from Terraform here, and add those functions to what Packer supports in an HCL2 context.	2 years ago
Lucas Bajolet	ec682afc4a	addrs: require at least 3 parts to source URIs	2 years ago
Lucas Bajolet	c7f2508373	addrs: return error on ParsePluginSource ... The ParsePluginSource function can be invoked from either a HCL2 context (when parsing a required_plugins block), or from the command-line itself. While in the first context a hcl.Diagnostics is coherent, in case the source to parse is a command-line argument, for example when installing or removing a plugin, the error message cannot have an HCL context, leading to errors that are incorrectly prefixed by a <nil> string dure to the lack of a reference to attach the diagnostic to. Therefore, in order to fix this behaviour, the logic that parses plugin sources now returns an error, and attaching the error to an HCL subject is done independently, if needed.	2 years ago
Lucas Bajolet	9b38e0eb5c	addrs: limit source component length to 16 ... When specifying/installing plugins, a source URI is required for Packer to be able to locate or install a plugin to the local plugin hierarchy. The plugin hierarchy is based on the plugin source, where each component in this hierarchy will become a directory. In order to avoid sources with too many levels of nesting, causing a lot of mkdirs, we limit the number of sources to 16 in this commit, this should be long enough for most of our users.	2 years ago
Lucas Bajolet	d2ade9a783	packer: register plugin components only once ... When running a packer command on an HCL2 template, depending on whether or not there are required_plugin blocks defined, Packer may need to discover and register a plugin's components multiple times. This is the behaviour ever since those blocks were introduced to Packer, but given we are doing the operation multiple times, this is suboptimal. This commit changes the way things works by first doing the restricted discovery of plugins (as dictated by required_plugins), then proceeding to the global discovery, with the change that subsequent component discoveries will not have precedence over those pre-discovered anymore. This allows us to invert the call order of both discovery phases safely, and maintains the constraints described in the templates.	2 years ago
Lucas Bajolet	6fc1d154bd	packer: relax constraints on sources ... The source parsing logic was heavily directed towards Github compatible source URIs, however if we want to support more cases, we need to make sure we are able to specify those URIs, and to load plugins installed from those sources. Right now, since the getters available are only github.com, we will not support remotely instlling plugins from sources other than github.com, with the same set of constraints as before. However, we do support now installing from a local plugin binary to any kind of source, and we support loading them, including if a template wants this plugin installed locally with version constraints.	2 years ago
Lucas Bajolet	1bf3e86ee1	hcl2template: fix invalid_inexplicit_source_2 test ... The invalid_inexplicit_source_2 subtest for parse used to fail in previous versions of Packer, but with the latest changes to plugin source management, it won't, at least at parsing time, instead it fails later when the Github getter (the only available for now) cannot get the plugin binary from the source, as the source is not a valid Github URI.	2 years ago
Lucas Bajolet	93bec31ec9	version: embed version/VERSION as version string ... Since we have both version/version.go and version/VERSION to specify version strings, both are a bit redundant. As version/VERSION is supposed to be the source of truth now, we are using it to derive the version informaiton we used to rely on in Packer and its subcommands. Note: doing this prevents us from changing the version/prerelease through ldflags though as we derive Version/VersionPrerelease from the rawVersion variable.	2 years ago
Devashish	595b45e67c	Packer tracks Version and Plugins Metadata (#12860)	2 years ago
Lucas Bajolet	ea1e798c47	fmt: print parse errors on error ... The fmt command reformats HCL2 templates, provided it can parse the file and reformat its contents according to the standards set by the HCL library's formatters. However, if the file is malformed for some reason, the command will fail with a parse error, but while the parse error message is shown, the actual errors in the template(s) are not forwarded, making it hard for users to understand what went wrong with the contents of the file they're trying to format. In order to be more helpful with those errors, we now forward those parsing errors to the UI.	2 years ago
Lucas Bajolet	9f7e6ca52b	plugins: add a release only flag to Discover ... Since we now support loading pre-releases, we also want Packer to be able to ignore them by user demand, so we put in place the infrastructure to modulate this.	2 years ago
Lucas Bajolet	e099c5c661	main: move Discover to DetectPluginBinaries ... When Packer is loaded, we used to perform plugin discovery. This was done for every call to Packer, including when it is executed as a plugin, arguably against what the comments document. Doing this as early in the loading process makes it harder to change this behaviour, as we'd need to introduce flags aside from the rest, and handle them manually, which is not optimal. Therefore, we change this: now when Packer starts executing, it will not attempt to discover installed plugins anymore, and instead will only try to load them when a configuration has been parsed, and is being used to perform actions (typically build/validate).	2 years ago
Lucas Bajolet	b51c350bd6	packer: fix plugin version sorting and pickup ... When Packer orders the plugins by their version, we'd assumed it was ordered from highest to lowest. However, this was the case because our implementation of `Less' was actually returning `>=', which is not what it was supposed to do. This commit therefore fixes the implementation of `Less' to do what it is documented to do, and ensures the behaviour is correct through additional testing. Changing this requires some changes to the loading process as well because of the aforementioned assumption with regards to ordering.	2 years ago
Lucas Bajolet	fd5f668ee9	plugin: consolidate loading logic ... Right now we had two paths for discovering installed plugins, i.e. through plugin-getter's `ListInstallations' function, or through the `Discover' call, which relied on a glob to list the installations. This was required since we allowed plugins to be installed in multiple locations, and with different constraints. Now that we force a certain convention, we can consolidate the logic into ListInstallations, and rely on that logic in `Discover' to load a plugin into the PluginConfig for the current Packer run.	2 years ago
Lucas Bajolet	4b00a81bf3	packer: remove directory slices from structs ... Since we'll only look in the plugin directory, and not from multiple sources now, for installing/listing plugins, we can simplify the structures which used to accept multiple directories so they only accept one.	2 years ago
Lucas Bajolet	94aaff44a1	packer: only support one directory for plugins ... Since we're removing the alternative plugin installation directories in favour of only supporting installing them in the PACKER_PLUGIN_PATH directory, we only return one directory when getting the known plugin directories.	2 years ago
Lucas Bajolet	dff49df129	hcl2template: check bucket name at parse-time ... Not validating the bucket's name during parse leads to configurations being marked as valid, even if the bucket name is not, which will fail during a real build afterwards. To avoid this problem and fail with an error as quickly as possible, we add a check during parsing, so that it gets reported for validate as well.	2 years ago
Devashish	682eca2877	Remove repeated code	2 years ago
Devashish	4b591d6937	Add PR suggestions	2 years ago
Devashish	b77d581cda	Add the deprecation warning and update docs	2 years ago
Devashish	cbc16cfed5	Add versionFingerprint to EvalContext	2 years ago
Wilken Rivera	263821ac25	hcl2template/functions: Add Non-null refinements for various functions ... cty's new "refinements" concept allows us to reduce the range of unknown values from our functions. This initial changeset focuses only on declaring which functions are guaranteed to return a non-null result, which is a helpful baseline refinement because it allows "== null" and "!= null" tests to produce known results even when the given value is otherwise unknown. This commit also includes some updates to test results that are now refined based on cty's own built-in refinement behaviors, just as a result of us having updated cty in the previous commit.	2 years ago
Lucas Bajolet	8c7b97331b	hcl2: don't show an empty source on var errors ... If a variable is defined and overriden in the packer command-line, but there's a problem during the evaluation of this override (type error typically), we show an error message which details the problem. This message points to a temporary in-memory HCL2 "file" that we use for parsing and evaluating the expression for the variable, but since it's virtual, there's no point in using this as the source for the error, as it will always yield "line 0" and no contents. So, in order to limit confusion here, we remove the source for this error message.	2 years ago
Lucas Bajolet	efe182b221	packer: link to docs if a component is missing ... When a user invokes packer for a build or validation, the template being processed needs components to be present for Packer to process it without error. If the component cannot be found from the plugins loaded (or from the components bundled with Packer), Packer errors, and the command fails. This is expected, but the error message does not suggest anything to fix the error, potantially leaving users confused at the problem. This commit suggests either a replacement (in case of a typo), or points to the web documentation for Packer, specifically the integrations, so they can look for the plugin they're missing, and install it, so subsequent invocations of Packer work.	2 years ago
Lucas Bajolet	434a1637f8	hcl2template: fix func to get vars from a config ... The previous implementation of the GetVarsByType function worked only on top-level attributes, ignoring the nested blocks in the structure. This implies that if a datasource depends on another through an expression within a nested block, we may not execute it first, and then executing this datasource before its dependent is possible, resulting in an error in the end. This commit is an attempt at making this more reliable for HCL configs, but only works on configs lifted from HCL files for now. We need to make this more reliable for later iterations.	2 years ago
Lucas Bajolet	51ec78622d	hcl2template: simplify startDatasource function	2 years ago
Lucas Bajolet	d9259d6d12	hcl2template: extract attr filter code from ds ... Datasources use their attribute's expressions to determine whether or not they depend on another datasource, in order to get the list of dependencies and execute them before executing a datasource. This code may be useful later on for figuring out the dependencies for any block, so we move this code to the utils.go file, and use this for datasources.	2 years ago
Lucas Bajolet	14b6ad10fe	hcl2template: report localtion for cycle detection ... When a datasource fails to be evaluated because a cycle has been detected, we point out one of the links of the chain now so that users have a better idea of what to look at.	2 years ago
Lucas Bajolet	2a6ccaccab	hcl2template: simplify datasource evaluation ... Since datasources are recursively evaluated depending on their dependencies, we don't need to pre-execute those that depend on nothing, as the recursive traversal of the datasources will take care of that for us.	2 years ago
Lucas Bajolet	8438d81d4d	hcl2template: remove unused shouldContinue bool ... Not sure why this was defined and returned, but the value was set, but never used, as such this is not useful to keep in the code, so let's simplify this now.	2 years ago
guoguangwu	7eb6a45cd4	chore: remove refs to deprecated io/ioutil ... Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>	2 years ago
guoguangwu	324e628b86	chore: unnecessary use of fmt.Sprintf ... Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>	2 years ago