|
| 1 | +--- |
| 2 | +title: "How LINSTOR and DRBD Keep Your Data Alive When a Server Dies" |
| 3 | +slug: linstor-drbd-replicated-storage |
| 4 | +date: 2026-07-22 |
| 5 | +author: "Timur Tukaev" |
| 6 | +description: "How Cozystack uses LINSTOR and DRBD for synchronous block-level replication on bare metal: setting up storage pools, the replicated StorageClass, and what happens when a node fails." |
| 7 | +images: |
| 8 | + - "linstor-drbd-replicated-storage.png" |
| 9 | +article_types: |
| 10 | + - tech-article |
| 11 | + - storage |
| 12 | +topics: |
| 13 | + - storage |
| 14 | + - linstor |
| 15 | + - drbd |
| 16 | + - kubernetes |
| 17 | + - platform |
| 18 | + |
| 19 | +--- |
| 20 | + |
| 21 | + |
| 22 | + |
| 23 | +Every bare-metal operator's nightmare: a disk fails at 3 AM. If your storage is node-local (a `local-path` or `hostPath` volume pinned to one node), you're restoring from backup and praying it's recent. Ceph gives you replication but demands 3+ dedicated storage nodes, a PhD in CRUSH maps, and constant tuning. Most teams either over-invest in storage infrastructure or under-invest and pay the price in outages. |
| 24 | + |
| 25 | +Cozystack uses LINSTOR + DRBD for replicated block storage. DRBD replicates at the block device level — below the filesystem, below the database, below everything. It's been battle-tested in Linux HA setups for over two decades. LINSTOR adds the Kubernetes-native orchestration layer on top. |
| 26 | + |
| 27 | +## How it works |
| 28 | + |
| 29 | +1. **DRBD** (Distributed Replicated Block Device) synchronously mirrors block devices between nodes. Every write to Node A is immediately written to Node B before the application gets an acknowledgment. |
| 30 | +2. **LINSTOR** manages DRBD resources as Kubernetes StorageClasses. When a PVC requests storage, LINSTOR creates a DRBD volume, selects replica nodes, and handles the lifecycle. |
| 31 | +3. **CSI driver** exposes LINSTOR volumes as standard Kubernetes PersistentVolumes. |
| 32 | + |
| 33 | +Result: any pod or VM that uses a `replicated` StorageClass gets synchronous block-level replication across nodes — transparently. |
| 34 | + |
| 35 | +## Setting up storage |
| 36 | + |
| 37 | +**Step 1 — Prepare disks:** |
| 38 | + |
| 39 | +```bash |
| 40 | +# Set up LINSTOR CLI alias |
| 41 | +alias linstor='kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor' |
| 42 | + |
| 43 | +# Check what LINSTOR sees |
| 44 | +linstor node list |
| 45 | +linstor physical-storage list |
| 46 | +``` |
| 47 | + |
| 48 | +If disks don't appear, they likely have leftover metadata. Clean them: |
| 49 | + |
| 50 | +```bash |
| 51 | +# WARNING: never wipe the OS install disk (machine.install.disk) — only data disks. |
| 52 | +talm -f nodes/node1.yaml wipe disk nvme0n1 nvme1n1 |
| 53 | +``` |
| 54 | + |
| 55 | +**Step 2 — Create storage pools (ZFS):** |
| 56 | + |
| 57 | +```bash |
| 58 | +linstor physical-storage create-device-pool \ |
| 59 | + zfs node1 \ |
| 60 | + /dev/nvme0n1 /dev/nvme1n1 \ |
| 61 | + --pool-name data \ |
| 62 | + --storage-pool data |
| 63 | +``` |
| 64 | + |
| 65 | +Repeat for each node. |
| 66 | + |
| 67 | +**Step 3 — Verify:** |
| 68 | + |
| 69 | +```bash |
| 70 | +linstor storage-pool list |
| 71 | +``` |
| 72 | + |
| 73 | +You should see a `data` pool on each node with available capacity. |
| 74 | + |
| 75 | +**Step 4 — Use it:** |
| 76 | + |
| 77 | +Any application that specifies `storageClass: replicated` now gets DRBD-replicated volumes (three replicas by default). PostgreSQL, MongoDB, VM disks — all of them. |
| 78 | + |
| 79 | +```yaml |
| 80 | +# In any application config: |
| 81 | +values: |
| 82 | + storageClass: replicated |
| 83 | + size: 50Gi |
| 84 | +``` |
| 85 | +
|
| 86 | +## What happens when a node fails |
| 87 | +
|
| 88 | +1. DRBD detects the node is gone. |
| 89 | +2. The volume remains available on the surviving replica node. |
| 90 | +3. Kubernetes reschedules the pod to a node that has the replica. |
| 91 | +4. When the failed node comes back, DRBD automatically resyncs. |
| 92 | +
|
| 93 | +No manual intervention. No restore from backup. No data loss. |
| 94 | +
|
| 95 | +## Documentation |
| 96 | +
|
| 97 | +- [Storage overview](https://cozystack.io/docs/v1/storage/) |
| 98 | +- [Disk Preparation](https://cozystack.io/docs/v1/storage/disk-preparation/) |
| 99 | +- [Disk Encryption](https://cozystack.io/docs/v1/storage/disk-encryption/) |
| 100 | +- [DRBD Tuning](https://cozystack.io/docs/v1/storage/drbd-tuning/) |
| 101 | +- [NFS (RWX)](https://cozystack.io/docs/v1/storage/nfs/) |
| 102 | +- [LINSTOR GUI](https://cozystack.io/docs/v1/storage/linstor-gui/) |
| 103 | +- [Dedicated Storage Network](https://cozystack.io/docs/v1/storage/dedicated-network/) |
| 104 | +
|
| 105 | +## Join the community |
| 106 | +
|
| 107 | +- [Cozystack on GitHub](https://github.com/cozystack/cozystack) |
| 108 | +- Telegram [group](https://t.me/cozystack) |
| 109 | +- Slack [group](https://kubernetes.slack.com/archives/C06L3CPRVN1) (Get invite at [https://slack.kubernetes.io](https://slack.kubernetes.io)) |
| 110 | +- [Community Meeting Calendar](https://calendar.google.com/calendar?cid=ZTQzZDIxZTVjOWI0NWE5NWYyOGM1ZDY0OWMyY2IxZTFmNDMzZTJlNjUzYjU2ZGJiZGE3NGNhMzA2ZjBkMGY2OEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t) |
0 commit comments