Architecture Flows

Auto-generated

Rendered from docs/architecture/calm/architecture.json by the CALM CLI (calm template). Do not edit this file by hand — edit the architecture JSON or the Handlebars template at docs/architecture/calm/templates/mermaid/flows.md.hbs and regenerate with make calm-diagrams.

Each business flow defined in the CALM architecture is rendered below as its own Mermaid flowchart TD — one diagram per flow, linking the transitions in sequence order.

Schedule Activation (machine enters window)

When the current time enters a ScheduledMachine's active window, 5-Spot creates the bootstrap + infrastructure resources and a CAPI Machine so that CAPI provisions the physical node and joins it to the workload cluster.

flowchart TD t1["1. Operator applies a ScheduledMachine CR declaring schedule, bootstrap spec, infrastructure spec, and cluster name."] t2["2. Controller watch observes the CR and the provider object named by spec.schedule; the provider's status.active (e.g. TimeBasedSpotSchedule evaluating daysOfWeek/hoursOfDay in its timezone) reports the machine should be active (ADR 0009)."] t3["3. Controller creates K0sWorkerConfig (bootstrap), RemoteMachine (infra), and a CAPI Machine referencing both; phase -> Scheduled."] t4["4. CAPI core reconciles the Machine and hands off to bootstrap + infra providers."] t5["5. Infrastructure provider SSHes to the physical node, installs k0s with bootstrap data, and joins the workload cluster; phase -> Active."] t1 --> t2 --> t3 --> t4 --> t5

_{Source: flow flow-schedule-activation in architecture.json.}

Schedule Deactivation (machine exits window or kill switch)

When the window closes (or killSwitch is set), the controller cordons the node, evicts pods within nodeDrainTimeout, then deletes the CAPI Machine so the infra provider tears it down.

flowchart TD t1["1. Scheduler tick detects out-of-window (or killSwitch=true); phase -> Removing; gracefulShutdownTimeout starts."] t2["2. Controller cordons the node (patch spec.unschedulable=true) and creates pod eviction requests, respecting nodeDrainTimeout."] t3["3. Controller deletes the CAPI Machine (and associated bootstrap/infra objects if owned); phase -> Inactive / UnScheduled."] t4["4. CAPI core propagates deletion to providers; physical node is removed from the workload cluster."] t1 --> t2 --> t3 --> t4

_{Source: flow flow-schedule-deactivation in architecture.json.}

Emergency Reclaim (process-match eject)

A matched process on a workload node triggers an ASAP eject that skips graceful drain. The 7-step ordering contract is load-bearing: spec.enabled=false (step 5, ADR 0009) must be written BEFORE the annotation clear (step 6) so a crash between them is replay-safe. Full narrative in docs/src/concepts/emergency-reclaim.md.

flowchart TD t1["1. Agent detects a process matching spec.killIfCommands via /proc scan and PATCHes the reclaim annotation triple (5spot.finos.org/reclaim-requested=#quot;true#quot;, /reclaim-reason, /reclaim-requested-at) onto its own Node via the kubelet node-scoped token. Field manager: 5spot-reclaim-agent."] t2["2. Controller Node watch fires. check_emergency_reclaim reads the annotation, transitions status.phase = EmergencyRemove, and emits Event Reason: EmergencyReclaim with the annotation's reason string."] t3["3. Best-effort kubectl drain --grace-period=0 --force --disable-eviction against the workload cluster, bounded by EMERGENCY_DRAIN_TIMEOUT_SECS=60. Failure is log-and-continue — the eject has already been committed to."] t4["4. Controller deletes the CAPI Machine on the management cluster — no PDB respect, no graceful shutdown. CAPI propagates deletion to providers and the physical node leaves the workload cluster."] t5["5. Controller PATCHes ScheduledMachine spec.enabled=false (the loop-breaker: without this, the provider's next active window re-adds the node and the agent re-fires forever) and emits Event Reason: EmergencyReclaimDisabledSchedule. spec.enabled is SM-scoped (ADR 0009) — the provider object is never patched, since it may be shared by other SMs. Load-bearing: must run BEFORE step 6."] t6["6. Controller PATCHes the Node to null all three reclaim annotations. Best-effort: a crash between steps 5 and 6 is replay-safe because the agent re-annotates on the next poll cycle if the matched process is still running."] t7["7. Controller transitions ScheduledMachine status.phase = Disabled. Re-enable is a manual operator action (kubectl patch ... spec.enabled=true)."] t1 --> t2 --> t3 --> t4 --> t5 --> t6 --> t7

_{Source: flow flow-emergency-reclaim in architecture.json.}

Kata Config Delivery (per-node drop-in + k0s restart)

A ScheduledMachine with spec.kata set drives delivery of a Kata containerd drop-in onto its backing node and a one-shot host k0s restart so containerd reloads it. The applied-hash annotation (step 4, written BEFORE the restart in step 5) is the load-bearing restart-loop guard: it ensures the agent restarts at most once per distinct config change even though the restart SIGKILLs the agent. Decisions: ADR 0002 (contract + projection) and ADR 0003 (host write + nsenter restart).

flowchart TD t1["1. Controller (Active phase, best-effort, workload-cluster-scoped) reads (get) the source Secret/ConfigMap named in spec.kata in spec.kata.namespace (default 5spot-system) on the workload cluster. If present, it stamps 5spot.finos.org/kata-config=enabled plus the 5spot.finos.org/kata-config-ref annotation on the backing Node; if absent, it clears the opt-in and reports a fail-fast status reason (SourceNotFound / TargetNamespaceMissing)."] t2["2. The kata config agent DaemonSet (scheduled onto the node by the opt-in label) watches its per-node ConfigMap and reads the projected drop-in content at /etc/5spot/kata-config/#lt;key#gt;."] t3["3. Agent compares the SHA-256 of the source content against the fixed host drop-in path (/etc/k0s/containerd.d/kata.toml under the /host mount) and the kata-config-applied annotation. On mismatch it atomically writes the drop-in (temp-file + rename, mode 0644, root:root) to that fixed path. If the source key is absent it unlinks the host file (GitOps delete)."] t4["4. Agent records the applied hash (bare content SHA-256, or 'absent' after tear-down) in the 5spot.finos.org/kata-config-applied annotation on its own Node via the kubelet node-scoped token. Load-bearing: written BEFORE the restart so a crash/SIGKILL during restart does not re-trigger on the next loop."] t5["5. Agent runs `nsenter -t 1 -m -u -i -n -p -- systemctl restart #lt;restartService#gt;` to restart the host k0s service; containerd reloads the new drop-in. The restart SIGKILLs the agent pod; systemd still completes the D-Bus job. Kubelet restarts the agent; on the next loop hashes match the applied annotation, so it is a no-op. Single-cycle convergence."] t1 --> t2 --> t3 --> t4 --> t5

_{Source: flow flow-kata-config-delivery in architecture.json.}

Spot Schedule Activation (provider-driven active/inactive)

Every ScheduledMachine delegates its active/inactive decision to the single provider named by spec.schedule (ADR 0009) via that provider's duck-typed status.active — the default being the first-party TimeBasedSpotSchedule. The controller watches the provider CR event-driven; the provider verdict IS the should-be-active decision (no inline schedule to AND with after ADR 0009; killSwitch always wins). An unresolved/unready provider never flaps machines: the controller surfaces SpotScheduleResolved=False and holds last-known state (fail-inactive when never resolved). ADR 0006 §4, ADR 0009.

flowchart TD t1["1. External provider controller evaluates its own spec (e.g. exchange calendar/holidays/early-closes) and PATCHes status.active (and the Ready condition) onto its CR. 5-Spot never reads the provider spec — only this status."] t2["2. Controller's dynamic watch on the provider GVK fires; the reverse index maps the event to the referencing ScheduledMachine(s) and enqueues them. The resolver reads status.active and the Ready condition defensively (missing CRD/object/status or Ready=False ⇒ Unresolved, surfaced as SpotScheduleResolved=False, not inactive)."] t3["3. Controller composes the provider verdict with evaluate_schedule (AND when spec.schedule is also set; killSwitch overrides). If the composite verdict is active and the machine is not yet up, it proceeds into flow-schedule-activation (create bootstrap/infra + CAPI Machine)."] t4["4. On a provider transition to status.active=false (composite verdict inactive), the controller proceeds into flow-schedule-deactivation (cordon, drain within nodeDrainTimeout, delete the CAPI Machine). Provider Unresolved does NOT trigger this — last-known state is held."] t1 --> t2 --> t3 --> t4

_{Source: flow flow-spot-schedule-activation in architecture.json.}

Schedule-Gated Capacity (host stays in the cluster, a slice is conceded)

The non-handover pattern (ADR 0011). The machine never leaves the cluster: when the schedule says active, 5-Spot writes a bounded capacity value onto a foreign object so a consumer can size ephemeral microVM workloads on the host's spare cores, and when it says inactive 5-Spot writes zero and waits for the consumer to hand the slice back. Contrast flow-schedule-activation, where the whole physical node is added to and removed from the cluster.

flowchart TD t1["1. Operator applies a ScheduledCapacity CR naming the schedule, the target object, the field to write (spec.capacity.path), the active value, and optionally the drain signal and the governed node."] t2["2. The controller resolves spec.schedule through resolve_spot_schedule and reads the provider's status.active. Unresolved holds the last written value rather than zeroing it."] t3["3. Guard clauses in precedence order: killSwitch, then spec.enabled, then the provider verdict. If spec.nodeName is also the status.nodeRef of a ScheduledMachine in this namespace, the controller sets HostGovernanceConflict and drives capacity to ZERO (ADR 0014), because one host cannot both be handed over and shared and zero is the safe direction. The active value is never written to a conflicted object."] t4["4. Pre-flight SelfSubjectAccessReview for patch on the resolved target resource, so an RBAC gap is reported as a condition rather than failing mid-write."] t5["5. Active: the controller merge-patches spec.capacity.activeValue onto the validated path (for example spec.warmReplicas of a VirtualMachinePool). Phase becomes Active, CapacityWritten=True. The consumer sizes its guests under its own cgroup budget; 5-Spot knows nothing about guests."] t6["6. The schedule flips inactive (for example market open on a CapitalMarketsSchedule). The dynamic provider watch wakes the reconcile at watch latency, not on a timer."] t7["7. Handback begins: the controller writes zero to the same path and moves to HandingBack, recording a deadline from spec.handback.timeout. It does not delete the object and does not force anything."] t8["8. The controller watches spec.handback.drainedPath on the target (for example status.claimed). Reaching zero means the consumer has released the slice: phase Inactive, HandbackComplete=True. If the deadline passes first, the controller HOLDS the written value, reports HandbackTimedOut loudly and increments a metric, because a missed handover is visible and recoverable while killing in-flight agent work is neither."] t1 --> t2 --> t3 --> t4 --> t5 --> t6 --> t7 --> t8

_{Source: flow flow-schedule-gated-capacity in architecture.json.}