High Availability & Worker Recovery

STRATUM provides distributed workload execution across physical servers, private and public clouds, geographically separated datacenters, and remote edge systems. Compute resources are organized into a Continuum that retains centralized scheduling, management, and operational visibility across participating workers.

The architecture accommodates large deployments with thousands of workers operating across different hardware, network conditions, and geographic locations.

A STRATUM VM is placed onto an execution worker for CPU, memory, and the physical capabilities required to run it. Its network identity and durable storage path are represented by the larger STRATUM fabric. That separation allows the Execution Fabric to place a movable workload on another compatible worker without redesigning the datacenter around the destination host.

When STRATUM evaluates another execution host, it checks the capabilities the workload actually requires, including architecture such as x86 / ARM / s390x, available CPU and memory, virtual-machine-manager support, GPU requirements, TPM requirements, site policy, and other placement constraints.

A STRATUM virtual switch port belongs to the STRATUM network model. Through STRATUM Continuum, the distributed network fabric can realize that attachment on participating workers instead of tying the topology to the physical server where a VM happened to start.

When a movable VM is restarted on another eligible worker, STRATUM can recreate its existing network attachment at the new execution location. The topology stays the topology while execution moves underneath it.

This same architecture allows STRATUM networking to extend across datacenters, clouds, labs, and edge environments that participate in the Continuum.

The VM disk remains associated with its STRATUM Storage owner and target while the execution location can change.

The execution worker does not need to own a local copy of the VM disk or mount a shared filesystem containing it.

The destination worker reconnects to the assigned storage path through STRATUM's storage protocol and the Continuum Storage Lane. Compute can move while the durable storage relationship remains stable, making execution workers substantially more interchangeable.

After deployment, STRATUM workers execute their assigned workloads locally. Virtual machines and virtual switches can continue running through controller restarts, controller software upgrades, or temporary controller connectivity failures.

Worker execution agents retain local allocation information. When the controller returns, workers report their active workloads. STRATUM reconciles those reports with persisted execution records, retaining original job identities and worker assignments. Existing running VMs remain on their workers instead of being restarted simply because the controller was unavailable.

Actual workload availability still depends on the underlying network and storage services. STRATUM Network Fabric and STRATUMStore can operate independently of controller management connectivity as long as their required infrastructure remains operational.

STRATUM monitors worker connectivity and execution status. If a worker becomes unreachable, its jobs transition to OFFLINE. Their allocation records, last known state, and assigned worker are preserved in Compute Fabric.

An OFFLINE job may still be running normally on the disconnected worker. This is particularly important for intermittent WAN links, isolated edge sites, and temporary inter-datacenter routing failures.

When communications return, the worker reports its active allocations. STRATUM reconciles the original job identities and restores RUNNING status only when ongoing execution is confirmed.

Virtual machines can be configured with High Availability enabled. STRATUM uses this setting to determine whether a workload should be automatically resubmitted following a confirmed execution failure or a completed worker recovery operation.

For a permanently failed worker, recovery begins only after an administrator confirms that the original worker has been powered off or otherwise safely fenced. A disconnected worker cannot receive controller commands; marking it blocked in the UI is not a substitute for physical or infrastructure-level fencing.

1. Confirm the failure. Examine the worker and its OFFLINE allocations in Compute Fabric. Distinguish an unreachable management connection from a worker that is actually stopped.
2. Fence the original worker. Confirm that it has been powered off or otherwise prevented from continuing execution and using the original writable storage.
3. Block the worker. In Compute Fabric, Block the failed worker to prevent it from reconnecting and accepting assignments during recovery.
4. Release its OFFLINE workloads. After fencing is confirmed, Release the affected allocations so their former execution ownership can be cleared.
5. Resubmit HA-enabled VMs. STRATUM places HA-enabled virtual machines back into the execution queue. Each replacement receives a new execution attempt and VM-state ownership epoch. VMs without HA enabled remain available for manual restart.
6. Rebuild on an eligible worker. The scheduler selects compatible capacity with access to the required networking and persistent storage. The VM reconnects to its existing storage target and virtual network topology, then starts on its replacement worker.
7. Verify execution and ownership. Confirm the replacement is running and that any previously unavailable worker is reconciled against current ownership records before it can resume associated workloads.

This is restart-based recovery, not a claim that in-memory VM execution state or a running guest is live-migrated across a failed host. Availability depends on the remaining worker capacity, placement rules, networking, and storage.

Virtual switches may also require local bridges, physical interfaces, or site-specific connectivity. Recover or place those workloads only where their dependencies are available.

Compute Fabric provides separate controls for worker eligibility, enrollment, and workload execution ownership:

• Disable: Prevents new workload assignments while existing workloads continue running.
• Block: Prevents a worker from reconnecting and accepting assignments. Use it when quarantining or retiring a worker. It does not remotely stop a disconnected worker.
• Remove: Deletes a worker's enrollment after the required cleanup.
• Destroy: Terminates a reachable workload and clears its active execution binding.
• Release: Clears ownership of an OFFLINE workload after an administrator confirms that its previous worker is safely stopped or fenced.

Destroy and Release do not delete the underlying VM configuration or persistent disks. They manage execution bindings, not the durable workload definition.

STRATUMStore provides persistent storage through iSCSI exports backed by VM disk overlays. A VM's writable storage can retain a stable export identity across execution attempts, allowing a replacement instance to use existing disk data when the storage configuration permits.

Creating a new execution attempt does not automatically invalidate the previous worker's storage access. Confirm that the original worker is fenced before releasing an OFFLINE VM for recovery on another worker. This prevents concurrent access by competing execution instances to the same writable storage.

When a previously unavailable worker reconnects, STRATUM reconciles its local allocations with the controller's current ownership records. Stale execution attempts must be rejected before that worker is allowed to resume their associated workloads.

Some workload state deliberately creates locality. Active snapshots are stored on the current execution host, so a VM with an active snapshot chain remains pinned there and HA placement is unavailable until those snapshots are removed or consolidated.

Direct physical-device passthrough can also constrain placement because a destination must provide the required physical capability. Storage still needs a resilient underlying failure domain; multiple logical storage roles on one physical server remain one physical failure domain.

STRATUM exposes these constraints to placement instead of assuming every workload can run everywhere.

Virtual switches may depend on physical NICs, local bridges, or site-specific connectivity. Those dependencies must be evaluated before restarting a switch on a different worker; not all networking resources are portable across sites.

STRATUM uses worker-level reporting and persistent allocation records to coordinate many concurrent workloads. Workers report active allocations in batches; the controller does not depend on per-VM heartbeat requests to keep those guests executing.

This architecture supports large fleets of conventional datacenter servers, cloud-hosted workers, GPU compute systems, and intermittently connected edge sites. Controller outages are handled through reconciliation. Worker connectivity interruptions preserve assignments. Confirmed failures can use the HA scheduler after the necessary fencing and Release operations.

These recovery paths minimize unnecessary interruptions while allowing operators to retain explicit control over workload and storage ownership across the Continuum.

Compute workers, storage workers, distributed networking, remote accelerators, and workload state participate in one operating model. The execution location can change without forcing the mission topology to change with it.

A datacenter can be in a rack, in a cloud, at the edge, on a remote platform, or distributed across geographic locations. STRATUM treats those resources as part of the same Continuum and uses capability and policy to decide where work runs.

Remote and intermittently connected sites may deliberately pin workloads to local compute and storage instead of failing them off-site when communications disappear. Application-level and guest-level HA can still operate within that site where required.