Scenario
Which node fails, when and in what way.
Service 04 · Failover testing
In brief
Failover testing checks whether a high availability cluster handles a node failure as designed. At managed-cluster.net, we run a full test before go-live, then repeat it regularly on a schedule agreed with you, each time with a written log. Documented emergency runbooks cover events beyond automatic failover. Both are part of an operations contract at a fixed monthly price.
Before go-live
Before your cluster goes into production, it goes through a full failover test: we deliberately take a node down and check that automatic failover works as designed. The test is part of the build, alongside setting up monitoring and backups. Migration and handover follow only after it, and you receive the test log.
Failover testing is one of four areas of managed infrastructure operations. Your production systems move over only once the test is done, and you choose the migration date. Our design and setup page shows where the pre-handover test fits into the project plan.
A failover cluster test causes exactly the failure the cluster is designed to absorb. The test runs through the same chain of events as a real node failure. The remaining nodes keep quorum, meaning the majority of votes in the cluster, and the load balancer removes the failed node from rotation. Fencing isolates the node so that no virtual machine runs twice. The Proxmox HA Manager then restarts the affected virtual machines on another node.
Our high availability cluster page shows how automatic failover works step by step; why this requires quorum is explained in our guide to quorum, fencing and split brain.
During operations
After handover, we repeat the failover test regularly while your systems are in production. We set the schedule together with you so the tests fit your operational routines. Every test is logged, so each log shows whether failover works with the current state of your cluster.
A cluster keeps changing: we apply rolling updates node by node, new virtual machines are added, and capacity planning during operations leads to expansion recommendations. Recurring tests show whether quorum, fencing and restart still work together as your cluster stands today.
Test log
For every failover test, you receive a written log. It records which failure was triggered, how the cluster responded and whether the result matched the designed behavior. That gives IT management, information security and internal audit evidence that failover has actually been tested.
Which node fails, when and in what way.
How quorum, fencing, restart and load balancer responded.
Whether the cluster behaved as designed, and any deviations.
How long a failover takes depends on the layer. The load balancer removes a failed node from rotation within seconds. The node's virtual machines restart on another node after fencing; how long that takes depends on fencing and on the guest system's boot time. The test makes this behavior visible for your own guest systems.
Our availability calculator shows how much downtime per year your availability target allows. For the evidence you can ask any hosting provider to supply, see our managed hosting provider checklist.
Emergency runbooks
Automatic failover handles the failure of a node. For incidents that go beyond that, we create documented emergency runbooks. An emergency runbook is an IT emergency plan that sets out, for a specific scenario, which steps restore operations and in which order. It is part of the written documentation of your cluster and belongs to you.
Replication protects against hardware failure, yet it also copies mistakes. Restarting on another node therefore does not help with:
Nor can a cluster at a single site absorb the loss of an entire data center.
A runbook is measured against your recovery targets: the recovery time objective (RTO), the maximum tolerable time until operations resume, and the recovery point objective (RPO), the maximum tolerable data loss, measured as a period of time before the failure. We capture both during the initial assessment and set them jointly with you in the operations contract. Our section on disaster recovery planning explains how runbooks, backups and restore tests fit together.
You can use runbooks and test logs for internal reviews and audits of your information security. The overview of documents for your vendor assessment lists everything you receive from us in writing.
A high availability cluster is no substitute for backups.
What protects against deleted or encrypted data is a separate backup whose restores are tested regularly: backup and disaster recovery.
Two sites
A cluster in a single data center absorbs failures within that data center, such as a server, a disk or a switch. The managed-cluster.net Business package spreads six nodes across FRA01 and NBG01 and adds a documented switchover plan for moving between the two sites. That plan is tested regularly, not just written down.
The package also includes Ceph HA storage replicated across both sites and backups with restore tests at the second site, NBG01. Enterprise environments with several sites are designed to your requirements, with individual RPO/RTO and SLA targets. Scope and price are listed under the Business package with two sites; our page on the data centers FRA01 and NBG01 describes the role of each site.
Scope
The three terms are often confused. Automatic failover is how the cluster responds to a failure. A failover test checks that response on purpose and with a written log. A restore test checks something else entirely: whether a backup can actually be restored. Together they cover failure, evidence and recovery.
| Procedure | What happens | When | Outcome |
|---|---|---|---|
| Automatic failover | The cluster responds to a node failure without manual intervention: the HA Manager restarts the affected virtual machines on another node. See high availability cluster. | when a failure occurs | post-incident report with cause and measures |
| Failover test | We deliberately take a node down and check whether automatic failover works as designed. | full test before go-live, then on the agreed schedule | written test log |
| Restore test | A backup is restored to check that the data can actually be recovered. See restore testing in detail. | regularly | verified recoverability of the backup |
FAQ
Once in full before go-live, then regularly during operations. We set the schedule together with you, and every test is logged.
Yes. For every failover test, you receive a written log covering scenario, sequence and result, as evidence for IT management, information security or internal audit.
The steps that restore operations in a scenario that automatic failover cannot handle. It is measured against your recovery targets, RTO and RPO. Runbooks are part of the written documentation of architecture, configuration and access, and they belong to you.
Yes, in the Business package: the documented switchover plan for moving between FRA01 and NBG01 is tested regularly. Enterprise environments spanning several sites are designed to your requirements.
Next step
Describe your environment, your availability target and your recovery targets. We will discuss with you which failure scenarios we test and which ones are documented in runbooks.
Discuss failover testing