Architecture Flows

Auto-generated

Rendered from docs/architecture/calm/architecture.json by the CALM CLI (calm template). Do not edit this file by hand — edit the architecture JSON or the Handlebars template at docs/architecture/calm/templates/mermaid/flows.md.hbs and regenerate with make calm-diagrams.

Each business flow defined in the CALM architecture is rendered below as its own Mermaid flowchart TD — one diagram per flow, linking the transitions in sequence order.

Schedule Activation (machine enters window)

When the current time enters a ScheduledMachine's active window, 5-Spot creates the bootstrap + infrastructure resources and a CAPI Machine so that CAPI provisions the physical node and joins it to the workload cluster.

flowchart TD t1["1. Operator applies a ScheduledMachine CR declaring schedule, bootstrap spec, infrastructure spec, and cluster name."] t2["2. Controller watch observes the CR and the provider object named by spec.schedule; the provider's status.active (e.g. TimeBasedSpotSchedule evaluating daysOfWeek/hoursOfDay in its timezone) reports the machine should be active (ADR 0009)."] t3["3. Controller creates K0sWorkerConfig (bootstrap), RemoteMachine (infra), and a CAPI Machine referencing both; phase -> Scheduled."] t4["4. CAPI core reconciles the Machine and hands off to bootstrap + infra providers."] t5["5. Infrastructure provider SSHes to the physical node, installs k0s with bootstrap data, and joins the workload cluster; phase -> Active."] t1 --> t2 --> t3 --> t4 --> t5

_{Source: flow flow-schedule-activation in architecture.json.}

Schedule Deactivation (machine exits window or kill switch)

When the window closes (or killSwitch is set), the controller cordons the node, evicts pods within nodeDrainTimeout, then deletes the CAPI Machine so the infra provider tears it down.

flowchart TD t1["1. Scheduler tick detects out-of-window (or killSwitch=true); phase -> Removing; gracefulShutdownTimeout starts."] t2["2. Controller cordons the node (patch spec.unschedulable=true) and creates pod eviction requests, respecting nodeDrainTimeout."] t3["3. Controller deletes the CAPI Machine (and associated bootstrap/infra objects if owned); phase -> Inactive / UnScheduled."] t4["4. CAPI core propagates deletion to providers; physical node is removed from the workload cluster."] t1 --> t2 --> t3 --> t4

_{Source: flow flow-schedule-deactivation in architecture.json.}

Emergency Reclaim (process-match eject)

A matched process on a workload node triggers an ASAP eject that skips graceful drain. The 7-step ordering contract is load-bearing: spec.enabled=false (step 5, ADR 0009) must be written BEFORE the annotation clear (step 6) so a crash between them is replay-safe. Full narrative in docs/src/concepts/emergency-reclaim.md.

flowchart TD t1["1. Agent detects a process matching spec.killIfCommands via /proc scan and PATCHes the reclaim annotation triple (5spot.finos.org/reclaim-requested="true", /reclaim-reason, /reclaim-requested-at) onto its own Node via the kubelet node-scoped token. Field manager: 5spot-reclaim-agent."] t2["2. Controller Node watch fires. check_emergency_reclaim reads the annotation, transitions status.phase = EmergencyRemove, and emits Event Reason: EmergencyReclaim with the annotation's reason string."] t3["3. Best-effort kubectl drain --grace-period=0 --force --disable-eviction against the workload cluster, bounded by EMERGENCY_DRAIN_TIMEOUT_SECS=60. Failure is log-and-continue — the eject has already been committed to."] t4["4. Controller deletes the CAPI Machine on the management cluster — no PDB respect, no graceful shutdown. CAPI propagates deletion to providers and the physical node leaves the workload cluster."] t5["5. Controller PATCHes ScheduledMachine spec.enabled=false (the loop-breaker: without this, the provider's next active window re-adds the node and the agent re-fires forever) and emits Event Reason: EmergencyReclaimDisabledSchedule. spec.enabled is SM-scoped (ADR 0009) — the provider object is never patched, since it may be shared by other SMs. Load-bearing: must run BEFORE step 6."] t6["6. Controller PATCHes the Node to null all three reclaim annotations. Best-effort: a crash between steps 5 and 6 is replay-safe because the agent re-annotates on the next poll cycle if the matched process is still running."] t7["7. Controller transitions ScheduledMachine status.phase = Disabled. Re-enable is a manual operator action (kubectl patch ... spec.enabled=true)."] t1 --> t2 --> t3 --> t4 --> t5 --> t6 --> t7

_{Source: flow flow-emergency-reclaim in architecture.json.}

Kata Config Delivery (per-node drop-in + k0s restart)

A ScheduledMachine with spec.kata set drives delivery of a Kata containerd drop-in onto its backing node and a one-shot host k0s restart so containerd reloads it. The applied-hash annotation (step 4, written BEFORE the restart in step 5) is the load-bearing restart-loop guard: it ensures the agent restarts at most once per distinct config change even though the restart SIGKILLs the agent. Decisions: ADR 0002 (contract + projection) and ADR 0003 (host write + nsenter restart).

flowchart TD t1["1. Controller (Active phase, best-effort, workload-cluster-scoped) reads (get) the source Secret/ConfigMap named in spec.kata in spec.kata.namespace (default 5spot-system) on the workload cluster. If present, it stamps 5spot.finos.org/kata-config=enabled plus the 5spot.finos.org/kata-config-ref annotation on the backing Node; if absent, it clears the opt-in and reports a fail-fast status reason (SourceNotFound / TargetNamespaceMissing)."] t2["2. The kata config agent DaemonSet (scheduled onto the node by the opt-in label) watches its per-node ConfigMap and reads the projected drop-in content at /etc/5spot/kata-config/<key>."] t3["3. Agent compares the SHA-256 of the source content against the fixed host drop-in path (/etc/k0s/containerd.d/kata.toml under the /host mount) and the kata-config-applied annotation. On mismatch it atomically writes the drop-in (temp-file + rename, mode 0644, root:root) to that fixed path. If the source key is absent it unlinks the host file (GitOps delete)."] t4["4. Agent records the applied hash (bare content SHA-256, or 'absent' after tear-down) in the 5spot.finos.org/kata-config-applied annotation on its own Node via the kubelet node-scoped token. Load-bearing: written BEFORE the restart so a crash/SIGKILL during restart does not re-trigger on the next loop."] t5["5. Agent runs `nsenter -t 1 -m -u -i -n -p -- systemctl restart <restartService>` to restart the host k0s service; containerd reloads the new drop-in. The restart SIGKILLs the agent pod; systemd still completes the D-Bus job. Kubelet restarts the agent; on the next loop hashes match the applied annotation, so it is a no-op. Single-cycle convergence."] t1 --> t2 --> t3 --> t4 --> t5

_{Source: flow flow-kata-config-delivery in architecture.json.}

Spot Schedule Activation (provider-driven active/inactive)

Every ScheduledMachine delegates its active/inactive decision to the single provider named by spec.schedule (ADR 0009) via that provider's duck-typed status.active — the default being the first-party TimeBasedSpotSchedule. The controller watches the provider CR event-driven; the provider verdict IS the should-be-active decision (no inline schedule to AND with after ADR 0009; killSwitch always wins). An unresolved/unready provider never flaps machines: the controller surfaces SpotScheduleResolved=False and holds last-known state (fail-inactive when never resolved). ADR 0006 §4, ADR 0009.

flowchart TD t1["1. External provider controller evaluates its own spec (e.g. exchange calendar/holidays/early-closes) and PATCHes status.active (and the Ready condition) onto its CR. 5-Spot never reads the provider spec — only this status."] t2["2. Controller's dynamic watch on the provider GVK fires; the reverse index maps the event to the referencing ScheduledMachine(s) and enqueues them. The resolver reads status.active and the Ready condition defensively (missing CRD/object/status or Ready=False ⇒ Unresolved, surfaced as SpotScheduleResolved=False, not inactive)."] t3["3. Controller composes the provider verdict with evaluate_schedule (AND when spec.schedule is also set; killSwitch overrides). If the composite verdict is active and the machine is not yet up, it proceeds into flow-schedule-activation (create bootstrap/infra + CAPI Machine)."] t4["4. On a provider transition to status.active=false (composite verdict inactive), the controller proceeds into flow-schedule-deactivation (cordon, drain within nodeDrainTimeout, delete the CAPI Machine). Provider Unresolved does NOT trigger this — last-known state is held."] t1 --> t2 --> t3 --> t4

_{Source: flow flow-spot-schedule-activation in architecture.json.}