Solution · Storage

Managed Ceph clusters: distributed storage for Proxmox

In brief

Ceph is distributed storage that keeps multiple copies of data on different nodes and restores redundancy on its own after failures. In a managed Ceph cluster from managed-cluster.net, every data block exists three times as standard; we build Ceph into your Proxmox VE cluster and run it 24/7, from €449.99 net per month with the Essentials package. Replication does not replace backups.

How it works

How a managed Ceph cluster works in Proxmox VE

Ceph spreads your virtual machines' data across the disks of every node; there is no central storage system. Each data block is stored three times on different nodes, and a write is only acknowledged once the copies on the other nodes are in place. That is why any virtual machine can start on any node.

Diagram 03 Ceph in a three-node Proxmox cluster
  • Ceph replication over the storage network
  • Node active
  • A, B, C: data blocks; the three copies of block A are highlighted
Diagram as text
  1. Three nodes running Proxmox VE, each hosting virtual machines.
  2. Each disk in a node is managed by a Ceph service (OSD).
  3. Data blocks A, B and C exist once on every node, so three times on different nodes.
  4. Replication runs over a dedicated storage network.
  5. If a node fails, every data block still exists in two copies.
Every data block exists as three copies on different nodes, replicated over a separate storage network. Schematic view; we define the number of disks per node in the design.

Hyperconverged Proxmox and Ceph: compute and storage on the same nodes

In a managed Proxmox cluster, virtual machines and Ceph storage run on the same nodes; Proxmox calls this a hyper-converged setup. There is no separate storage system such as a SAN (storage area network), and with it goes a central component whose failure would hit every node at once.

OSDs, CRUSH and min_size

Each disk is managed by its own Ceph service, the OSD (object storage daemon). Ceph computes where a block lives with the CRUSH algorithm, without a central lookup table. By default, Proxmox VE creates three copies (size 3); at least two must be reachable for reads and writes to continue (min_size 2).

A dedicated network for replication

Replication runs over a dedicated storage network, separate from management and public traffic. Proxmox highly recommends this separation because recovery traffic could otherwise interfere with cluster communication over Corosync. If a node fails, the Proxmox HA Manager restarts its virtual machines on another node; see how high availability in the cluster fits together.

Sources: Proxmox VE Administration Guide, “Deploy Hyper-Converged Ceph Cluster” (version 9.2); Ceph documentation, “Architecture” (Tentacle release); retrieved October 9, 2026 · As of October 2026

Failure behavior

What happens when a disk or a node fails

If a disk or an entire node fails, no data is lost: Ceph keeps every block in three copies on different nodes and restores the target redundancy on its own as soon as enough disks and nodes are available for it. Recovery increases I/O load; with several simultaneous failures, some data may be temporarily unavailable until recovery completes.

Failure scenarios Ceph storage in Proxmox with three copies per data block
ScenarioYour dataWhat Ceph and the cluster doWhat we do
One disk fails Available No data loss; reads and writes continue. Ceph rewrites the missing copies to the remaining disks (self-healing); I/O load rises until it finishes. On-call duty is alerted, we replace the disk, you get a post-incident report.
One node fails Available No data loss; every block still exists twice. The HA Manager restarts the node's virtual machines on another node. Because each copy must sit on a different node, a three-node cluster only regains its third copy once the node is repaired. On-call duty is alerted, we analyze the fault and replace the hardware, you get a post-incident report.
Two disks fail at the same time Partial No data loss; some data temporarily unavailable. Same node: every block remains reachable twice. Different nodes: some data has only one copy left, and I/O to it pauses until Ceph has restored a second copy. On-call duty is alerted, we replace both disks, you get a post-incident report.

Assumption: Proxmox default of size 3 and min_size 2. According to the Ceph documentation, a replicated pool can lose one OSD fewer than it has copies, all at the same time, without losing data. Sources: Proxmox VE Administration Guide; Ceph documentation, “Pools” (Tentacle release), retrieved October 9, 2026 · As of October 2026

At the compute layer, the virtual machines of a failed node restart on another node. Services running on several nodes behind the load balancer remain available. Our high availability cluster page walks through the step-by-step failover.

Decision guide

Ceph vs ZFS: which storage fits your cluster

The Essentials and Business packages use Ceph. We use ZFS replication in individually designed, more compact setups. The key difference is timing: Ceph writes synchronously to three nodes, while ZFS replicates at intervals. That interval determines how much data can be lost when hardware fails.

Comparison Ceph and ZFS replication in a Proxmox cluster
CriterionCephZFS replication
Minimum size three nodes more compact setups, including two nodes with an external quorum device
Replication synchronous, three copies on different nodes at intervals, configurable from every minute to once a week according to Proxmox
Impact on the RPO (recovery point objective) after a hardware failure Acknowledged writes already exist on several nodes; losing a disk or a node loses no data. Changes since the last sync can be lost; the interval sets the RPO.
VM restart after a node failure on any remaining node on a node holding a replica, with the data of the last sync
In the packages Yes Essentials and Business No individually designed setups
Replaces backups No replicates errors and deletions too No replicates errors and deletions too

Source on ZFS replication: Proxmox VE Administration Guide, “Storage Replication” (version 9.2), retrieved October 9, 2026. More on RPO and replication methods · As of October 2026

Which method fits depends on data volume, I/O profile and RPO target; we work this out during design and sizing.

Ceph is standard in the cluster packages: Ceph HA from the Essentials package at €449.99 net per month, with three nodes and full operations.

All prices are net, plus VAT.

Capacity

How much usable capacity three copies leave you

With three-way replication, roughly a third of the raw capacity is available for data. You should not fill all of it: Ceph needs free space to rewrite missing copies after a disk failure, and by default it warns once a disk is 85 percent full.

Example calculation, not a quote Three nodes with four disks of equal size each, values as a percentage of raw capacity
StepCalculationResult
Raw capacityAll disks of the three nodes combined100%
Usable with three copies100% ÷ 3about 33%
Up to the “nearfull” warning33% × 0.85 (Ceph default)about 28%
With headroom for one disk failure per nodeA failed disk's data spreads across the node's three remaining disks: 0.85 × ¾ ≈ 64% utilization beforehand, so 33% × 0.64about 21%

Simplified, assuming even distribution and excluding metadata. Ceph defaults: “nearfull” 0.85, “full” 0.95. Source: Ceph documentation, “Monitor Config Reference” (Tentacle release), retrieved October 9, 2026 · As of October 2026

The last row follows from placement: because each copy must sit on a different node, a failed disk's data in a three-node cluster lands on the remaining disks of the same node. If a disk reaches the “full” threshold, Ceph blocks I/O to prevent data loss, according to its documentation.

Larger disks increase storage density, but after a failure Ceph then has to recover more data at once (per Proxmox). We discuss the hardware of the cluster nodes (CPU, RAM, storage, network) individually and define it in the design proposal, together with headroom for disk failures, load peaks and growth.

Operations

Managed Ceph storage, operated 24/7: what we take on

We monitor your Ceph storage around the clock as part of your cluster: disks, storage network and Ceph services. Critical alerts reach on-call duty immediately and are handled right away, at any hour. We replace failed disks, and every incident ends with a post-incident report on cause and remediation.

  1. A DISK FAILS. Ceph keeps working with the remaining copies.
  2. Ceph writes the missing copies to the remaining disks (self-healing).
  3. Monitoring alerts on-call duty, including nights and weekends.
  4. We replace the disk, and it rejoins the cluster.
  5. You receive a post-incident report on cause and remediation.

In the FRA01 data center in the Rhine-Main region, failed components are replaced around the clock via remote hands.

  • Monitoring at every layer: hardware, virtualization, operating system, services and network.
  • Capacity planning: utilization trends and expansion recommendations before headroom runs low.
  • Node-by-node updates while your systems keep running. Exceptions such as a major database upgrade are announced in advance and scheduled in a window you choose.
  • Monthly operations report covering availability, incidents and utilization; more on 24/7 monitoring and on-call duty.

Scope

Why Ceph replication does not replace backups

Ceph protects against hardware failure, not against errors in your data. It writes every change synchronously to all copies, including an accidental deletion or encryption by ransomware. To recover, you need an earlier, undamaged copy of your data outside the cluster.

A high availability cluster is not a backup.

Backups run on infrastructure separate from the cluster, with defined retention periods and regular restore tests. In the Business package they are stored, and restore-tested, at the second site NBG01. Background and recovery targets: why replication is not a backup.

FAQ

Common questions about managed Ceph clusters

Ceph or ZFS: which fits my cluster?

From three nodes, usually Ceph; for more compact setups, ZFS replication. Ceph replicates synchronously and lets any virtual machine start on any node; the Essentials and Business packages use it. ZFS syncs data at intervals. We make the call during the design phase, based on data volume, I/O profile and RPO target.

How many nodes does Ceph need?

At least three. Ceph places the three copies of a block on three different nodes, and Proxmox also requires at least three servers for a hyper-converged Ceph cluster. Smaller environments can use ZFS replication; see Ceph vs ZFS compared.

What happens when a disk fails?

No data is lost. Ceph rewrites the missing copies to the remaining disks on its own, which temporarily increases I/O load. Our on-call duty is alerted, we replace the disk and document the incident in a post-incident report.

How much usable storage is left with three-way replication?

About a third of the raw capacity, before headroom. Because Ceph needs free space for recovery and warns at 85 percent utilization, plan with about 21 percent of the raw capacity in the example of three nodes with four disks each. The capacity example shows the full calculation.

Does Ceph replication replace backups?

No. Replication also carries deletions and errors to every copy. Our backup and disaster recovery page explains how separate backups with restore tests cover this.

Next step

How much storage does your application need?

Tell us your data volume, I/O profile and RPO target. You will receive a written design with an architecture diagram, a component list and a fixed price for setup and operations.

Discuss storage requirements

Related