36TB Pi Storage Server

After seeing Jeff Geerling compare his home-made storage server with a commercial one I thought I would expand on my OpenZFS post. You wouldn't mine Chia on a Pi but you can use it as a cheap storage server after creating your storage plots.

Parts
1 x Raspberry Pi4 and case
1 x Rpi power supply
1 x Micro SD card
1 x Powered USB 3.0 hub
5 x External HDD (powered)
2 x Power boards for all the power adapters

I'm not sponsored by anyone so I had to pay for all the parts myself. The power boards are old ones I had left over (I swap out old power boards every few years as the cheap ones tend to fail). The Pi4 was reallocated from number crunching duties. I picked 10TB drives based on the price per GB. If I wanted lots of disk space and price wasn't an issue I would have used 16TB drives. Its best to keep the drives the same size, otherwise you have to use multiple pools.

I am using a 4GB Pi4. The USB hub is a TP-Link UH700 which gives 7 ports and is rated for USB 3.0 speeds. Most USB hubs larger than 4 ports seem to come with a power adapter. The disks are Seagate Expansion 10TB drives.

 

I had to use two 6 outlet power boards because the spacing between the sockets means I can't plug the adapters in next to each other. Wiring gets messy. I would recommend zip-tying the cables to keep it tidy.

I assume that you have plugged everything in, powered it up and installed OpenZFS on your Raspberry Pi. See my earlier blog post for getting OpenZFS installed Here

Lets see what drives are attached. The lsblk (list block device) command will show them:

lsblk

NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  9.1T  0 disk
|-sda1        8:1    0  9.1T  0 part
`-sda9        8:9    0    8M  0 part
sdb           8:16   0  9.1T  0 disk
|-sdb1        8:17   0  9.1T  0 part
`-sdb9        8:25   0    8M  0 part
sdc           8:32   0  9.1T  0 disk
|-sdc1        8:33   0  9.1T  0 part
`-sdc9        8:41   0    8M  0 part
sdd           8:48   0  9.1T  0 disk
|-sdd1        8:49   0  9.1T  0 part
`-sdd9        8:57   0    8M  0 part
sde           8:64   0  9.1T  0 disk
|-sde1        8:65   0  9.1T  0 part
`-sde9        8:73   0    8M  0 part
mmcblk0     179:0    0 14.9G  0 disk
|-mmcblk0p1 179:1    0  256M  0 part /boot
`-mmcblk0p2 179:2    0 14.6G  0 part /

This shows I have 5 disks called sda through sde. Each disk is 9.1TB in capacity and has two partitions (sda1 and sda9). The mmcblk0 device is the micro SD card which the Pi is booting off.


Create a ZFS pool
Before you do this command make sure that there isn't any data you want to keep on the drives.

sudo zfs create pool1 raidz /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde -f

This says create a pool (of disks) called "pool1" its a raidz and we're using disks sda through sde. Raidz allows for a single disk failure. That is it can rebuild the missing information from data stored across the other 4 disks. Of course this means we lose space to allow for this data redundancy.

Pool attributes
To get more usable space we can turn compression on for our pool. Turning compression on means we try to compress the data before storing it on disk, however it requires some CPU time to do this.

You may have files that don't compress well such as zip files in which case there is no point turning compression on. I don't mine Chia so can't comment on storage plots being compressible or not.

The command to turn on zstd compression is:

sudo zfs set compression=zstd pool1

Note that you need OpenZFS 2 to get zstd compression. If running an older version you can use lz4 compression.

We can save some disk activity by not updating the atime attribute, that is the last time a file was accessed. The command to do this is:

sudo zfs set atime=off pool1

To check our pool the command is:

zpool status

  pool: pool1
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        pool1       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sde     ONLINE       0     0     0

errors: No known data errors


Check disk space
Lets see how much space we have after creating our pool. The command would be:

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  1.5G   13G  11% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  8.6M  1.9G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   30M  223M  12% /boot
tmpfs           385M     0  385M   0% /run/user/1000
pool1            36T  256K   36T   1% /pool1

You can see pool1 at the bottom and we have 36TB of available disk space after allowing for data redundancy. The theoretical disk capacity is 45.5TB (5 x 9.1TB).

It could be much larger by:

• Use more drives. I only have 5 drives plugged in but the USB hub can handle up to 7.

• Use a second hub with another 7 drives.

• Use larger disks. The expansion drives are available in sizes up to 16TB currently.

If you are going to have one of these running 24/7 I would recommend you keep a spare drive (or two) for when one fails. Being a spinning disk its just a matter of when, not if, a drive will fail. A raid array is not a backup so keep a backup. A UPS would be a good idea as well.

After powering off the Pi and turning it on again the disk pool disappeared. A simple reboot fixed that. I believe this is caused by the drive spin up time. You can configure a boot delay for the Pi which should give the drives enough time to spin up.

This setup should be able to hold approximately 363 Chia storage plots without compression.

 

Pricing
The pricing below is in US dollars.

1 x Pi4 4GB ($68)
1 x Pi4 case ($13)
1 x Pi4 power supply ($15)
1 x 16GB Micro SD card ($17)
1 x USB hub ($65)
5 x 10TB external drives ($215 each)
2 x Power strip ($13 each)
Total = $1,279 (USD)

Some parts (eg the Pi4) can be bought 2nd hand or found cheaper. As you can see the most expensive parts are the external hard disks. I've rounded up to the nearest dollar. Pricing taken from amazon on the 9th of May 2021.

A word of warning
SMR (Shingled Magnetic Recording) drives are not suitable for use in a RAID array as described above. They are sometimes called Archive Disks.

I watched a video where the guy bought a bunch of external hard disks, because they were cheaper than internal ones. He then removed them from the case, a process called shucking. When he looked up the drive specs it turned out half of them were SMR drives. There was no warning on the packaging.

OpenZFS on a Pi

I used a Pi3 as a guinea pig and tried installing OpenZFS on it using a pair of Seagate Expansion 2TB drives connected via the USB ports. Initially I tried with Raspbian because that is what the SD card in the Pi had installed. During the install it complained it was a 32 bit operating system and it wasn't going to work.

I went to image the SD card with Raspberry Pi OS 64 bit (Lite) and Win32 Disk Imager was having trouble (speeds of 0.45MB/sec) writing to the SD card. I replaced it with a new one. After booting up I added the buster-backports repo to the /etc/apt/sources.list and got the operating system up to date. Start by changing the password via raspi-config.

sudo raspi-config

sudo nano /etc/apt/sources.list and add:

deb http://deb.debian.org/debian buster-backports main contrib non-free

sudo apt update

sudo apt upgrade -y

sudo rm /etc/apt/sources.list.d/vscode.list

sudo rm /etc/apt/trusted.gpg.d/microsoft.gpg

sudo reboot

After it restarted I then told it to install OpenZFS from buster-backports which picked up OpenZFS 2.0.2.

sudo apt install -t buster-backports zfs-dkms raspberrypi-kernel-headers -y

It pops up a warning about licenses not being compatible which you have to acknowledge before it proceeds. It takes a while to build. Once its done reboot again and you should now have OpenZFS installed on a Raspberry Pi. I created a mirror with two external HDD (devices sda and sdb)

sudo zpool create pool1 mirror /dev/sda /dev/sdb -f

I then copied some files off my NFS server. It was quite slow which I attribute to using a Pi3 (slow USB ports, slow processor and limited memory). To make sure it all worked, reboot and it should automatically mount the drives in the pool when it starts up. You can check by doing a scrub:

sudo zpool scrub pool1

zpool status

 

Where to next?

At this point its booting off the SD card and the pool is using both disks. What would be nice is to have a bootable ZFS pool rather than using the SD card as a boot device.

If you were after more speed you could use SSD's instead of hard disks but you pay more for the capacity. Something like the Samsung T2 would be ideal. Just remember to do a zpool trim command regularly.

I expect its possible to use a USB hub to get more ports so you can have more drives but think the performance wouldn't be as good as directly plugged devices.

The ZFS forums recommend 1GB of memory for each TB of storage so you would get better performance with a 4GB or 8GB Pi4. 

Update 5 Apr 2021

Added the kernel-headers to the install command and clarified what to add to /etc/apt/sources.list.

I used a Pi4 8GB and two external USB3 HDD this time and it was somewhat quicker at copying files. Buster-backports has OpenZFS 2.0.3 as of writing.