Service 04
Failover Testing & Emergency Runbooks
Tests before go-live and during operations, each with a written log, plus documented emergency plans for scenarios beyond automatic failover.
Failover tests with a written logSolution · High availability
In brief
A high availability cluster (HA cluster) is a group of servers with no single point of failure: every component whose failure would stop your application is redundant. HA clusters from managed-cluster.net have at least three nodes. If one fails, the Proxmox HA Manager automatically restarts its virtual machines on another node. The availability target is set in your operations contract.
Architecture
A single point of failure is any component whose failure alone takes the service down. High availability does not come from better hardware but from an architecture that eliminates such points. That is why our HA clusters have redundancy at every layer beneath your application, from the data center and network to load balancers, nodes and storage.
| Layer | What is redundant | What happens on failure | Who is responsible |
|---|---|---|---|
| Data center FRA01 | According to the operator, Maincubes: redundant, uninterruptible power supply (2N power), N+1 cooling | According to the operator, the power supply is redundant and uninterruptible. Failed components are replaced around the clock via remote hands. | Operator building, power, cooling; Us hardware |
| Network | Our own network, AS47263, with BGP routing, redundant core routers, redundant switches and uplinks, bonding/LACP on every node | If a core router, a switch or an uplink fails, traffic continues over the remaining path. | Us |
| Load balancers | Redundant load balancers with a floating IP and health checks | The load balancer removes a failed node from rotation within seconds; existing connections are re-established. If a load balancer itself fails, a redundant load balancer takes over the floating IP. | Us |
| Compute (nodes) | At least three nodes with N+1 redundancy: there is enough capacity even with one node missing. | Corosync detects the failure, the node is fenced, and the Proxmox HA Manager restarts its virtual machines on the remaining nodes. For those VMs, this means a short interruption. | Us |
| Storage | Ceph with three replicas of every block on different nodes, dedicated storage network | If a disk or an entire node fails, no data is lost. Ceph restores the target redundancy on its own as soon as enough disks and nodes are available for it. Recovery increases I/O load. | Us |
| Application | Depends on your architecture, for example several instances behind the load balancer or database replication for PostgreSQL or MySQL/MariaDB | Instances on the remaining nodes keep serving requests. Health checks from the user's perspective verify that your application responds. | You application; we plan load balancing and replication with you in the design |
Data center details: third-party information from the operator, Maincubes, according to its FRA01 site page and FRA01 fact sheet (PDF), retrieved on October 9, 2026 · As of October 2026
Our HA cluster hosting follows a clear division of responsibility: you take care of your application, and we take care of everything beneath it.
High availability for servers and applications comes from these layers working together. Details: power and cooling at FRA01, our own network, AS47263, replicated Ceph storage and the platform underneath, a managed Proxmox cluster.
When things go wrong
If a node in a high availability cluster fails, the Proxmox HA Manager automatically restarts the affected virtual machines on another node. Before that, Corosync detects the failure and fencing isolates the node. Services running on several nodes behind the load balancer remain available. At the same time, our monitoring triggers an on-call alert.
The load balancer removes a failed node from rotation within seconds. The node's virtual machines restart on another node after fencing; how long that takes depends on fencing and on the guest system's boot time.
Fencing makes sure that a failed node, or one cut off from the cluster, has really stopped working before its services start anywhere else. Otherwise two nodes could write to the same virtual disk at once; according to the Proxmox documentation, this can destroy all of a VM's data.
According to Proxmox, Proxmox VE uses a watchdog timer for this, which every node keeps resetting during normal operation. A node without quorum can no longer reset it; if HA services are active on it, the node reboots itself once the timer expires. Only after fencing has succeeded does the cluster start the services on the remaining nodes. More on this: quorum, fencing and split brain explained.
Requirements according to Proxmox
For high availability, the Proxmox documentation calls for at least three cluster nodes to get reliable quorum, shared storage and redundant hardware. That is why our clusters have at least three nodes; the Essentials and Business packages use Ceph as shared storage.
Availability
We do not promise blanket nines without an architecture to back them up. We agree on the availability target with you and write it into the operations contract, along with response times by priority level, service hours, escalation paths and recovery targets. Critical alerts reach on-call duty immediately and are handled right away, around the clock.
For context, and not as a commitment: 99.9% availability allows just under nine hours of downtime per year, 99.99% less than one hour. Which target your application needs is a business question: what does an hour of downtime cost you in revenue, follow-up costs and staff time? Our guide lets you calculate downtime for 99.9% or 99.99%.
A target is only achievable if your application is built for it: if it runs in a single VM, the cluster bridges a node failure by restarting that VM; if it runs as several instances behind the load balancer, it stays available. The monthly operations report shows how available your cluster actually was, along with incidents and utilization. The Business package adds SLA reports.
For context, not a commitment. Your availability target is set in the operations contract.
Limits
A high availability cluster protects against component failures, not against errors in your data. Whatever your application or a person writes incorrectly, the cluster replicates as faithfully as any other change. Separate backups with regular restore tests and documented emergency plans protect against that.
Backups run on infrastructure separate from the cluster, with defined retention periods and regular restore tests. In the Business package they are stored, and restore-tested, at the second site NBG01. The full picture: why high availability does not replace backups.
An HA cluster is not a backup.
Replication protects against hardware failure but copies every change as well: an accidentally deleted record is gone from all three replicas at once.
Two sites
A cluster in one data center absorbs failures of individual servers, disks, switches or load balancers. Protection against losing an entire site requires a second one. The Business package runs six nodes across FRA01 and NBG01, with Ceph HA storage replicated across both sites, backups with restore tests at the second site and a documented, regularly tested switchover plan.
NBG01 in Nuremberg holds backup copies outside the main site and supports multi-site deployments. For Enterprise environments, we design a multi-site architecture to your requirements. Scope and price: Business package with six nodes across two sites.
Testing
Only a test shows whether a high availability cluster fails over as designed. We run a full failover test before handing the cluster over to you and repeat tests regularly during operations, on a schedule agreed with you. Every test is recorded in a written log.
Service 04
Tests before go-live and during operations, each with a written log, plus documented emergency plans for scenarios beyond automatic failover.
Failover tests with a written logFAQ
The cluster fails over automatically: the remaining nodes keep quorum, the failed node is fenced, and the Proxmox HA Manager restarts its virtual machines on another node. At the same time, monitoring triggers an on-call alert. After root cause analysis and hardware replacement, you receive an incident report.
That depends on how your application is distributed. Services running on several nodes behind the load balancer remain available; existing connections are re-established. Virtual machines that ran only on the failed node restart on another node and are unavailable until they are back up. How much your users notice is something we work out in the design, based on your application.
High availability keeps operations running automatically when individual components fail; disaster recovery restores them after major damage. Disaster recovery comes into play when data has been corrupted or deleted, or when damage goes beyond what the cluster absorbs automatically, and relies on separate backups, restore tests and documented emergency plans. How we handle it: backup and disaster recovery.
No blanket figure. We agree on the availability target with you, matched to your application and the level of redundancy, and write it into the operations contract. The contract also sets response times by priority level and the recovery targets RTO (recovery time objective) and RPO (recovery point objective). The monthly operations report shows the availability actually achieved.
Not always, though three is the norm: for reliable quorum, the Proxmox documentation calls for at least three nodes. Two-node setups are possible with an external quorum device; for business-critical applications, we recommend three or more nodes. Our guide on split brain, quorum and fencing explains why.
Next step
Tell us about your application, its load profile and the maximum downtime you can tolerate. From that, we derive the level of redundancy, the cluster size and the recovery targets, and set them out in a written design with an architecture diagram and a fixed price.
Discuss your availability targetFurther reading