13 Commits (add-missing-variable-loading)

Author	SHA1	Message	Date
Radek Simko	0fe906fa8c	make copyrightfix	2 months ago
Liam Cervante	f82ae44484	stacks: make promises identify themselves (#36749) ... * stacks: refactor promises so they declare id when starting * stacks: add some missing perEval promises * formatting	1 year ago
Martin Atkins	170cca1332	stacks+promising: Better error messages for promise resolution failure ... We were previously just treating promising.ErrUnresolved as a singleton that doesn't carry any context along with it. This particular error always represents a bug in the stack runtime rather than a problem with the input, because resolving all of the promises is a key part of the contract for package promising, but it's still helpful to know a little more about what exactly has failed when debugging it. Therefore promising.ErrUnresolved is now a type rather than a value, and its carries with it a set of promise ids that were left unresolved. The stackeval package then recognizes this error type and uses its own records of what each promise was supposed to produce to hopefully return a useful error message that would narrow down what part of the system is buggy. Since this message always reflects a bug in Terraform the error message will probably just be copy-pasted into a bug report, and so this particular diagnostic type (unlike the one we use for promise self-references) only has one presentation form that we use regardless of how many promises we have to return.	2 years ago
Martin Atkins	d46f0ed015	promising: A readme giving an overview of how this package works	2 years ago
Martin Atkins	b60aadd1ec	promising: Add HashiCorp copyright comments	2 years ago
Martin Atkins	03c981d9d0	promising,rpcapi,stackruntime: Quiet some staticcheck errors ... This code is still under development and so we have some stuff that's there to support further work despite not actually being used yet. We should reconsider these changes later on once the work here has settled.	2 years ago
Martin Atkins	55e744f3ed	promising: Immediately fail all promises in a self-dependency chain ... Previously we only directly signalled an error to whichever task was unlucky enough to be the one completing the dependency cycle in the promise-task graph. Although that's technically correct if our only goal is to prevent the formation of a blocking chain, in practice callers have their own concerns that can be left unresolved if we rely on the self-dependency-related errors to propagate organically as the callers each unwind themselves. In particular, our "stackeval" package design relies on the idea of each object only directly reporting its own errors and not reporting downstream errors from dependent objects (since they should report themselves), but that assumption is not valid if only one callstack directly sees the ErrSelfDependent error, since the arbitrarily-selected observer might be an indirect caller which discards errors on the assumption that another observer will be the one to return them. Instead then, here we introduce the idea of a promise having an "forced error", which is generated by the promise system itself rather than by the responsible task, and put all of the promises in a self-dependency chain into that state as soon as the self-dependency is detected. This causes the error to propagate up through _all_ of the calling stacks, which means that at least one of them is likely to reach the top caller but it also means that this particular class of error is likely to be duplicated by multiple return paths. Seeing multiple instances of the error is preferable to not seeing it at all, so we'll accept that compromise and expect the caller to deduplicate at the top join point if the multiple errors prove to be bothersome. This also includes some additional tracing events that happened to be useful while developing and debugging this change. For now the promising package is intentionally very chatty in its trace output since it's useful while the stackeval caller is under heavy development, but we will probably tone this down some in later commits once things have settled more.	2 years ago
Martin Atkins	3050d11989	promising: Once.Do must be called in a task context ... Previously we tried to make this optional by wrapping the behavior of this function in a call to promising.MainTask, but that's incorrect because it causes the calling task to implicitly depend on the created main task without any promise connecting them, which leads to a gap in the promise-task bipartite graph. (Tasks can only depend on each other indirectly through promises.) It turns out that we don't actually currently even need this convenience because all of the callers to promising.Once are in task context anyway. Therefore we'll just remove this unnecessary extra wrapper and have the calling task directly wait for the completion of the promise, thereby causing the promise-task graph to be complete so we can avoid Once-related deadlocks.	2 years ago
Martin Atkins	6d64943f1b	promising: Don't start a trace span for MainTask ... MainTask callbacks just run to completion synchronously and so there's no value in wrapping that call in a separate span. Instead, we can just use the caller's active span (if any) and thus have the promise blocking events etc attributed back to a more useful span name.	2 years ago
Martin Atkins	b0497293ed	promising: AssertContextInTask ... This is just a helper for functions that require that they run in a task context, to panic if not.	2 years ago
Martin Atkins	844234811e	promising: Emit tracing spans for tasks and promises ... Tasks and promises don't fit into the typical tree structure that OpenTracing spans normally form, and so this follows the documented conventions for coalesced batch processing making each task and promise appear as a separate span but with links between the spans and events to show when and how the tasks interacted with the promises. Each promise is a span just because it acts as a convenient container to attach an event log to. It doesn't actually do any work itself and so a promise span will never have any children. Tasks as spans is a more conventional use, allowing us to watch all of the activities a task does in preparation for resolving each of the promises that it's responsible for. We use the span IDs as de-facto ids for the tasks and promises themselves, since they have the appropriate amount of uniqueness for our purposes and this saves us from devising our own separate ID scheme.	2 years ago
Martin Atkins	51d6ad3382	promising: "Once" helper ... A common need is to coalesce multiple calls for the same data into a single background job. It's relatively straightforward to implement that requirement in terms of promises, but it requires some boilerplate code that would obscure author intent if inlined into every function that needs this capability. With that in mind, promising.Once is a reusable implementation of this pattern which has a usage pattern similar to the standard library's sync.Once, but is built with promises and can therefore detect and report self-dependency errors instead of deadlocking.	2 years ago
Martin Atkins	a9f3ad5a68	promising: A new utility for concurrent, inter-dependent data lookups ... For the main Terraform language runtime we've traditionally used an explicit directed acyclic graph to manage the order of execution, which is effective and proven but leads to an unfortunate inversion-of-control programming style both for graph construction and graph walking that can be difficult to read and debug. For the stacks language runtime we're going to use a slightly different approach of writing some relatively normal-looking functions that each appear to be normal straight-through code but internally are running as collaborating goroutines. This should make the control flow and data flow somewhat clearer. An important building block for this is a way for these different functions to asynchronously provide data to one another without the risk of deadlock. This new package "promising" provides that by modeling a bipartite graph of tasks and promises, where each promise has a single task responsible for resolving it and each task has zero or one promises it's waiting for. If any task attempts a wait that would introduce a cycle into the graph then the get operation immediately fails to avoid a deadlock.	2 years ago