OpenZFS on a Pi

I used a Pi3 as a guinea pig and tried installing OpenZFS on it using a pair of Seagate Expansion 2TB drives connected via the USB ports. Initially I tried with Raspbian because that is what the SD card in the Pi had installed. During the install it complained it was a 32 bit operating system and it wasn't going to work.

I went to image the SD card with Raspberry Pi OS 64 bit (Lite) and Win32 Disk Imager was having trouble (speeds of 0.45MB/sec) writing to the SD card. I replaced it with a new one. After booting up I added the buster-backports repo to the /etc/apt/sources.list and got the operating system up to date. Start by changing the password via raspi-config.

sudo raspi-config

sudo nano /etc/apt/sources.list and add:

deb http://deb.debian.org/debian buster-backports main contrib non-free

sudo apt update

sudo apt upgrade -y

sudo rm /etc/apt/sources.list.d/vscode.list

sudo rm /etc/apt/trusted.gpg.d/microsoft.gpg

sudo reboot

After it restarted I then told it to install OpenZFS from buster-backports which picked up OpenZFS 2.0.2.

sudo apt install -t buster-backports zfs-dkms raspberrypi-kernel-headers -y

It pops up a warning about licenses not being compatible which you have to acknowledge before it proceeds. It takes a while to build. Once its done reboot again and you should now have OpenZFS installed on a Raspberry Pi. I created a mirror with two external HDD (devices sda and sdb)

sudo zpool create pool1 mirror /dev/sda /dev/sdb -f

I then copied some files off my NFS server. It was quite slow which I attribute to using a Pi3 (slow USB ports, slow processor and limited memory). To make sure it all worked, reboot and it should automatically mount the drives in the pool when it starts up. You can check by doing a scrub:

sudo zpool scrub pool1

zpool status

 

Where to next?

At this point its booting off the SD card and the pool is using both disks. What would be nice is to have a bootable ZFS pool rather than using the SD card as a boot device.

If you were after more speed you could use SSD's instead of hard disks but you pay more for the capacity. Something like the Samsung T2 would be ideal. Just remember to do a zpool trim command regularly.

I expect its possible to use a USB hub to get more ports so you can have more drives but think the performance wouldn't be as good as directly plugged devices.

The ZFS forums recommend 1GB of memory for each TB of storage so you would get better performance with a 4GB or 8GB Pi4. 

Update 5 Apr 2021

Added the kernel-headers to the install command and clarified what to add to /etc/apt/sources.list.

I used a Pi4 8GB and two external USB3 HDD this time and it was somewhat quicker at copying files. Buster-backports has OpenZFS 2.0.3 as of writing.

A little controversial

The Raspberry Pi foundation recently added a Microsoft repository to Raspberry Pi OS which some may find controversial. Jeff Geerling made a short video where he goes into the details:

Is Microsoft spying on your Raspberry Pi 

I understand the foundation might want to make it easier to install some Microsoft apps but my Pi's are headless number crunchers. The apps are for a desktop installation not a headless machine. There is no reason for me to be checking their repo every time I do an update. Its just a waste of the Pi's time and my internet bandwidth. I feel it should have been something people opt-into rather than rolling it out to the masses or at least skip those Pi without a desktop. Jeff gives instructions on how to remove it, which are:

sudo rm /etc/apt/sources.list.d/vscode.list

sudo rm /etc/apt/trusted.gpg.d/microsoft.gpg

sudo apt update

Update: 10 March 2021

The foundation pushed through an update today that changes the vscode.list to a comment and removes the microsoft.gpg. You don't need to disable it any more.

7th of February

The cluster is continuing to run 24/7. The Pi3's are doing Einstein BRP4 work and the Pi4's are doing a mix of Einstein and Rosetta work.

Previously I had the file server running the GPS as well as serving files but decided for performance reasons to put the GPS onto its own Pi. The cluster now has 3 support nodes. One is an internet proxy server, two is a file server, and three is the time server with a GPS attached. The reason for having a GPS is to have correct time even if there is no internet connectivity.

Debian did a 10.8 point release today so there were a number of updates for that. In addition the Raspberry Pi foundation updated to the 5.10.11 Linux kernel. That meant I spent a bit of time updating all the Pis in the cluster.