stackruntime: An extensible "hooks" API

Following a similar model that we've previously used in other parts of
Terraform to expose UI-oriented progress events, here we define
stackruntime.Hooks as a collection of optional callbacks a caller can
use to subscribe to events, and the supporting utility code to help us
concisely and correctly call those hook functions from the main evaluation
code without adding too much extra noise to the code.

Although the primary focus of this API is driving realtime UI updates,
the API design also aims to accommodate distributed tracing use-cases by
allowing callers to propagate arbitrary values between related event
notifications and to inject additional values into the context.Context
used for downstream operations that might not be under the direct
supervision of stackruntime/stackeval.

internal/stacks/stackruntime/hooks.go

@@ -0,0 +1,46 @@
+package stackruntime
+
+import (
+	"context"
+
+	"github.com/hashicorp/terraform/internal/stacks/stackruntime/internal/stackeval"
+)
+
+// This file exposes a small part of the API surface of "stackeval" to external
+// callers. We need to orient it this way because package stackeval cannot
+// itself depend on stackruntime.
+
+// Hooks is an optional mechanism for callers to get streaming notifications
+// of various kinds of events that can occur during plan and apply operations.
+//
+// To use this, construct a Hooks object and then wrap it in a [context.Context]
+// using the [ContextWithHooks] function, and then use that context (or another
+// context derived from it that inherits the values) when calling into
+// [Plan] or [Apply].
+//
+// All of the callback fields in Hooks are optional and may be left as nil
+// if the caller has no interest in a particular event.
+//
+// The events exposed by Hooks are intended for ancillary use-cases like
+// realtime UI updates, and so a caller that is only concerned with the primary
+// results of an operation can safely ignore this and just consume the direct
+// results from the [Plan] and [Apply] functions as described in their
+// own documentation.
+//
+// Hook functions should typically run to completion quickly to avoid noticable
+// delays to the progress of the operation being monitored. In particular,
+// if a hook implementation is sending data to a network service then the
+// actual transmission of the events should be decoupled from the notifications,
+// such as by using a buffered channel as a FIFO queue and ideally transmitting
+// the events in batches where possible.
+type Hooks = stackeval.Hooks
+
+// ContextWithHooks returns a context that carries the given [Hooks] as
+// one of its values.
+//
+// Pass the resulting context -- or a descendent that preserves the values --
+// to [Plan] or [Apply] to be notified when the different hookable events
+// occur during that plan or apply process.
+func ContextWithHooks(parent context.Context, hooks *Hooks) context.Context {
+	return stackeval.ContextWithHooks(parent, hooks)
+}

internal/stacks/stackruntime/hooks/callbacks.go

@@ -0,0 +1,63 @@
+package hooks
+
+import (
+	"context"
+)
+
+// BeginFunc is the signature of a callback for a hook which begins a
+// series of related events.
+//
+// The given context is guaranteed to preserve the values from whichever
+// context was passed to the top-level [stackruntime.Plan] or
+// [stackruntime.Apply] call.
+//
+// The hook callback may return any value, and that value will be passed
+// verbatim to the corresponding [MoreFunc]. A typical use for that
+// extra arbitrary value would be to begin a tracing span in "Begin" and then
+// either adding events to or ending that span in "More".
+//
+// If a particular "begin" hook isn't implemented but one of its "more" hooks
+// is implemented then the extra tracking value will always be nil when
+// the first "more" hook runs.
+type BeginFunc[Msg any] func(context.Context, Msg) any
+
+// MoreFunc is the signature of a callback for a hook which reports
+// ongoing process or completion of a multi-step process previously reported
+// using a [HookFuncBegin] callback.
+//
+// The given context is guaranteed to preserve the values from whichever
+// context was passed to the top-level [stackruntime.Plan] or
+// [stackruntime.Apply] call.
+//
+// The hook callback recieves an additional argument which is guaranteed to be
+// the same value returned from the corresponding [BeginFunc]. See
+// [BeginFunc]'s documentation for more information.
+//
+// If the overall hooks also defines a [ContextAttachFunc] then a context
+// descended from its result will be passed into the [MoreFunc] for any events
+// related to the operation previously signalled by [BeginFunc].
+//
+// The hook callback may optionally return a new arbitrary tracking value. If
+// the return value is non-nil then it replaces the original value for future
+// hooks belonging to the same context. If it's nil then the previous value
+// is retained.
+//
+// MoreFunc is also sometimes used in isolation for one-shot events,
+// in which case the extra value will always be nil unless stated otherwise
+// in a particular hook's documentation.
+type MoreFunc[Msg any] func(context.Context, any, Msg) any
+
+// ContextAttachFunc is the signature of an optional callback that knows
+// how to bind an arbitrary tracking value previously returned by a [BeginFunc]
+// to the values of a [context.Context] so that the tracking value can be
+// made available to downstream operations outside the direct scope of the
+// stack runtime, such as external HTTP requests.
+//
+// Use this if your [BeginFunc]s return something that should be visible to
+// all context-aware operations within the scope of the operation that was
+// begun.
+//
+// If you use this then your related [MoreFunc] callbacks for the same event
+// should always return nil, because there is no way to mutate the context
+// with a new tracking value after the fact.
+type ContextAttachFunc func(parent context.Context, tracking any) context.Context

internal/stacks/stackruntime/hooks/doc.go

@@ -0,0 +1,8 @@
+// Package hooks is part of an optional API for callers to get realtime
+// notifications of various events during the stack runtime's plan and apply
+// processes.
+//
+// [stackruntime.Hooks] is the main entry-point into this API. This package
+// contains supporting types and functions that hook implementers will typically
+// need.
+package hooks

internal/stacks/stackruntime/internal/stackeval/hooks.go

@@ -0,0 +1,128 @@
+package stackeval
+
+import (
+	"context"
+	"sync"
+
+	"github.com/hashicorp/terraform/internal/stacks/stackruntime/hooks"
+)
+
+// Hooks is an optional API for external callers to be notified about various
+// progress events during plan and apply operations.
+//
+// This type is exposed to external callers through a type alias in package
+// stackruntime, and so it is part of the public API of that package despite
+// being defined in here.
+type Hooks struct {
+
+	// ContextAttach is an optional callback for wrapping a non-nil value
+	// returned by a [hooks.BeginFunc] into a [context.Context] to be passed
+	// to other context-aware operations that descend from the operation that
+	// was begun.
+	//
+	// See the docs for [hooks.ContextAttachFunc] for more information.
+	ContextAttach hooks.ContextAttachFunc
+}
+
+// A do-nothing default Hooks that we use when the caller doesn't provide one.
+var noHooks = &Hooks{}
+
+// ContextWithHooks returns a context that carries the given [Hooks] as
+// one of its values.
+func ContextWithHooks(parent context.Context, hooks *Hooks) context.Context {
+	return context.WithValue(parent, hooksContextKey{}, hooks)
+}
+
+func hooksFromContext(ctx context.Context) *Hooks {
+	hooks, ok := ctx.Value(hooksContextKey{}).(*Hooks)
+	if !ok {
+		return noHooks
+	}
+	return hooks
+}
+
+type hooksContextKey struct{}
+
+// hookSeq is a small helper for keeping track of a sequence of hooks related
+// to the same multi-step action.
+//
+// It retains the hook implementer's arbitrary tracking values between calls
+// so as to reduce the visual noise and complexity of our main evaluation code.
+// Once a hook sequence has begun using a "begin" callback, it's safe to run
+// subsequent hooks concurrently from multiple goroutines, although from
+// the caller's perspective that will make the propagation of changes to their
+// tracking values appear unpredictable.
+type hookSeq struct {
+	tracking any
+	mu       sync.Mutex
+}
+
+// hookBegin begins a hook sequence by calling a [hooks.BeginFunc] callback.
+//
+// The result can be used with [hookMore] to report ongoing progress or
+// completion of whatever multi-step process has begun.
+//
+// This function also deals with the optional [hook.ContextAttachFunc] that
+// hook implementers may provide. If it's non-nil then the returned context
+// is the result of that function. Otherwise it is the same context provided
+// by the caller.
+
+// Callers should use the returned context for all subsequent context-aware
+// calls that are related to whatever multi-step operation this hook sequence
+// represents, so that the hook subscriber can use this mechanism to propagate
+// distributed tracing spans to downstream operations. Callers MUST also use
+// descendents of the resulting context for any subsequent calls to
+// [runHookBegin] using the returned [hookSeq].
+func hookBegin[Msg any](ctx context.Context, cb hooks.BeginFunc[Msg], ctxCb hooks.ContextAttachFunc, msg Msg) (*hookSeq, context.Context) {
+	tracking := runHookBegin(ctx, cb, msg)
+	if ctxCb != nil {
+		ctx = ctxCb(ctx, tracking)
+	}
+	return &hookSeq{
+		tracking: tracking,
+	}, ctx
+}
+
+// hookMore continues a hook sequence by calling a [hooks.MoreFunc] callback
+// using the tracking state retained by the given [hookSeq].
+//
+// It's safe to use [hookMore] with the same [hookSeq] from multiple goroutines
+// concurrently, and it's guaranteed that no two hooks will run concurrently
+// within the same sequence, but it'll be unpredictable from the caller's
+// standpoint which order the hooks will occur.
+func hookMore[Msg any](ctx context.Context, seq *hookSeq, cb hooks.MoreFunc[Msg], msg Msg) {
+	// We hold the lock throughout the hook call so that callers don't need
+	// to worry about concurrent calls to their hooks and so that the
+	// propagation of the arbitrary "tracking" values from one hook to the
+	// next will always exact follow the sequence of the calls.
+	seq.mu.Lock()
+	seq.tracking = runHookMore(ctx, cb, seq.tracking, msg)
+	seq.mu.Unlock()
+}
+
+// runHookBegin is a lower-level helper that just directly runs a given
+// callback if it isn't nil and returns its result. If the given callback is
+// nil then runHookBegin immediately returns nil.
+func runHookBegin[Msg any](ctx context.Context, cb hooks.BeginFunc[Msg], msg Msg) any {
+	if cb == nil {
+		return nil
+	}
+	return cb(ctx, msg)
+}
+
+// runHookMore is a lower-level helper that just directly runs a given
+// callback if it isn't nil and returns the effective new tracking value,
+// which may or may not be the same value passed as "tracking".
+// If the given callback is nil then runHookMore immediately returns the given
+// tracking value.
+func runHookMore[Msg any](ctx context.Context, cb hooks.MoreFunc[Msg], tracking any, msg Msg) any {
+	if cb == nil {
+		// We'll retain any existing tracking value, then.
+		return tracking
+	}
+	newTracking := cb(ctx, tracking, msg)
+	if newTracking != nil {
+		return newTracking
+	}
+	return tracking
+}