36TB Pi Storage Server

After seeing Jeff Geerling compare his home-made storage server with a commercial one I thought I would expand on my OpenZFS post. You wouldn't mine Chia on a Pi but you can use it as a cheap storage server after creating your storage plots.

Parts
1 x Raspberry Pi4 and case
1 x Rpi power supply
1 x Micro SD card
1 x Powered USB 3.0 hub
5 x External HDD (powered)
2 x Power boards for all the power adapters

I'm not sponsored by anyone so I had to pay for all the parts myself. The power boards are old ones I had left over (I swap out old power boards every few years as the cheap ones tend to fail). The Pi4 was reallocated from number crunching duties. I picked 10TB drives based on the price per GB. If I wanted lots of disk space and price wasn't an issue I would have used 16TB drives. Its best to keep the drives the same size, otherwise you have to use multiple pools.

I am using a 4GB Pi4. The USB hub is a TP-Link UH700 which gives 7 ports and is rated for USB 3.0 speeds. Most USB hubs larger than 4 ports seem to come with a power adapter. The disks are Seagate Expansion 10TB drives.

 

I had to use two 6 outlet power boards because the spacing between the sockets means I can't plug the adapters in next to each other. Wiring gets messy. I would recommend zip-tying the cables to keep it tidy.

I assume that you have plugged everything in, powered it up and installed OpenZFS on your Raspberry Pi. See my earlier blog post for getting OpenZFS installed Here

Lets see what drives are attached. The lsblk (list block device) command will show them:

lsblk

NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  9.1T  0 disk
|-sda1        8:1    0  9.1T  0 part
`-sda9        8:9    0    8M  0 part
sdb           8:16   0  9.1T  0 disk
|-sdb1        8:17   0  9.1T  0 part
`-sdb9        8:25   0    8M  0 part
sdc           8:32   0  9.1T  0 disk
|-sdc1        8:33   0  9.1T  0 part
`-sdc9        8:41   0    8M  0 part
sdd           8:48   0  9.1T  0 disk
|-sdd1        8:49   0  9.1T  0 part
`-sdd9        8:57   0    8M  0 part
sde           8:64   0  9.1T  0 disk
|-sde1        8:65   0  9.1T  0 part
`-sde9        8:73   0    8M  0 part
mmcblk0     179:0    0 14.9G  0 disk
|-mmcblk0p1 179:1    0  256M  0 part /boot
`-mmcblk0p2 179:2    0 14.6G  0 part /

This shows I have 5 disks called sda through sde. Each disk is 9.1TB in capacity and has two partitions (sda1 and sda9). The mmcblk0 device is the micro SD card which the Pi is booting off.


Create a ZFS pool
Before you do this command make sure that there isn't any data you want to keep on the drives.

sudo zfs create pool1 raidz /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde -f

This says create a pool (of disks) called "pool1" its a raidz and we're using disks sda through sde. Raidz allows for a single disk failure. That is it can rebuild the missing information from data stored across the other 4 disks. Of course this means we lose space to allow for this data redundancy.

Pool attributes
To get more usable space we can turn compression on for our pool. Turning compression on means we try to compress the data before storing it on disk, however it requires some CPU time to do this.

You may have files that don't compress well such as zip files in which case there is no point turning compression on. I don't mine Chia so can't comment on storage plots being compressible or not.

The command to turn on zstd compression is:

sudo zfs set compression=zstd pool1

Note that you need OpenZFS 2 to get zstd compression. If running an older version you can use lz4 compression.

We can save some disk activity by not updating the atime attribute, that is the last time a file was accessed. The command to do this is:

sudo zfs set atime=off pool1

To check our pool the command is:

zpool status

  pool: pool1
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        pool1       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sde     ONLINE       0     0     0

errors: No known data errors


Check disk space
Lets see how much space we have after creating our pool. The command would be:

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  1.5G   13G  11% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  8.6M  1.9G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   30M  223M  12% /boot
tmpfs           385M     0  385M   0% /run/user/1000
pool1            36T  256K   36T   1% /pool1

You can see pool1 at the bottom and we have 36TB of available disk space after allowing for data redundancy. The theoretical disk capacity is 45.5TB (5 x 9.1TB).

It could be much larger by:

• Use more drives. I only have 5 drives plugged in but the USB hub can handle up to 7.

• Use a second hub with another 7 drives.

• Use larger disks. The expansion drives are available in sizes up to 16TB currently.

If you are going to have one of these running 24/7 I would recommend you keep a spare drive (or two) for when one fails. Being a spinning disk its just a matter of when, not if, a drive will fail. A raid array is not a backup so keep a backup. A UPS would be a good idea as well.

After powering off the Pi and turning it on again the disk pool disappeared. A simple reboot fixed that. I believe this is caused by the drive spin up time. You can configure a boot delay for the Pi which should give the drives enough time to spin up.

This setup should be able to hold approximately 363 Chia storage plots without compression.

 

Pricing
The pricing below is in US dollars.

1 x Pi4 4GB ($68)
1 x Pi4 case ($13)
1 x Pi4 power supply ($15)
1 x 16GB Micro SD card ($17)
1 x USB hub ($65)
5 x 10TB external drives ($215 each)
2 x Power strip ($13 each)
Total = $1,279 (USD)

Some parts (eg the Pi4) can be bought 2nd hand or found cheaper. As you can see the most expensive parts are the external hard disks. I've rounded up to the nearest dollar. Pricing taken from amazon on the 9th of May 2021.

A word of warning
SMR (Shingled Magnetic Recording) drives are not suitable for use in a RAID array as described above. They are sometimes called Archive Disks.

I watched a video where the guy bought a bunch of external hard disks, because they were cheaper than internal ones. He then removed them from the case, a process called shucking. When he looked up the drive specs it turned out half of them were SMR drives. There was no warning on the packaging.