I got 4 20TB drives from Amazon around Black Friday that I want to get setup for network storage. I’ve got 3 descent Ryzen 5000 series desktops that I was thinking about setting up so that I could build my own mini-Kubernetes cluster, but I don’t know if I have enough motivation. I’m pretty OCD so small projects often turn into big projects.
I don’t have an ECC motherboard though, so I want to get some input if BTRFS, ZFS, TrueNAS, or some other solution should be relatively safe without it? I guess it is a risk-factor but I haven’t had any issues yet (fingers crossed). I’ve been out of the CNCF space for a while but Rook used to be the way to go for Ceph on Kubernetes. Has there been any new projects worth checking out or should I just do RAID and get it over with? Does Ceph offer the same level of redundancy or performance? The boards have a single M.2 slot so I could add in some SSD caching.
If I go with RAID, should I do RAID 5 or 6? I’m also a bit worried because the drives are all the same so if there is an issue it could hit multiple drives at once, but I plan to try to have an online backup somewhere and if I order more drives I’ll balance it out with a different manufacturer.
If you want to have a single node hosting your storage, ZFS either directly or through TrueNAS is the way to go. ECC is not really necessary.
If you do want to have a higher availability and as you have 3 potential nodes already I can recommend Ceph. It’s pretty much THE open source network storage provider and has been battle tested a lot. With Ceph you can define how you want redundancy configured, but I haven’t gone into that enough to tell you what to do.
As long as 2/3 nodes are up you won’t have problems, but less nodes will lead to either a broken cluster or split brain scenario if you’re not careful.
Not sure how Ceph handles nodes with mismatched drives, I only tested using the same number and size of drives. Maybe you’d have to get two more drives for your other nodes.
Re RAID: At the moment RAID 6 does not offer any advantage over RAID 10, so you could go with that instead. RAID 10 would provide higher performance, but future upgrades need to be done in pairs of 2 disks.
RAID 5 is okish, but with 20TB you’re looking at very long restore times and the likelyhood of failure is very high during the restore and it’s very likely you’ll have a half finished restore when a second drive fails.
See this article for an explaination. (The same issue also applies to RAID 6, but the drives can be bigger as you have 2 redundant drives. Regardless, RAID 10 doesn’t have this problem due to being so simple)