27th of March

Marks Rpi Cluster continues computing 24/7. The Pi3's are running the Einstein BRP4 work and the Pi4's are running Einstein and Rosetta work in a 50/50 split.

 

Pi^4 case

I finally managed to get my Pi^4 case and one of the power supplies cases back from the 3D printing studio. They printed off a sample top piece for the Pi^4 case but its not right. They also printed off a cover for the Mean Well power supply. Unfortunately they pulled the earth wire in the power cable off its spade lug so I will have to get it re-crimped before I can use it. In the mean time I have swapped the power cable with another one.

I'll be trying out the power supply shortly and if it works out will see if I can get 4 of the Pi4's going. I've got 9 sitting in their boxes ready to go along with fans. Its taken almost 6 months just to get the design and case back. I will have to find another 3D printing company to adjust the design, get another sample and if that is okay run off a couple of cases. It has been extremely frustrating.

11th of March

Marks Rpi Cluster continues running 24/7. It currently consists of:

3 x Pi4 2GB support nodes

2 x Pi4 4GB compute nodes

3 x Pi4 8GB compute nodes

6 x Pi3 compute nodes

The Pi3 compute nodes are running Einstein BRP4 work. The Pi4 compute nodes are running Einstein BRP4 and Rosetta work in a 50/50 project split.

 

Other news

Still no contact with the 3D printing studio. I think I'll talk to the manager to see if I can get things moving. They don't answer the phone or reply to emails. I've not been pushing the issue as I need them to finish the design and print the parts for the Pi^4 cases and the power supplies. I need these to get the cluster up to 12 x Pi4 8GB and retire the Pi3's. I have cabling, power supplies and Pi4's ready to go if only I can get the cases sorted out.

The Raspberry Pi foundation did an about-face in the vscode repository. See my post "A little controversial".

 

OpenZFS on a Pi

I used a Pi3 as a guinea pig and tried installing OpenZFS on it using a pair of Seagate Expansion 2TB drives connected via the USB ports. Initially I tried with Raspbian because that is what the SD card in the Pi had installed. During the install it complained it was a 32 bit operating system and it wasn't going to work.

I went to image the SD card with Raspberry Pi OS 64 bit (Lite) and Win32 Disk Imager was having trouble (speeds of 0.45MB/sec) writing to the SD card. I replaced it with a new one. After booting up I added the buster-backports repo to the /etc/apt/sources.list and got the operating system up to date. Start by changing the password via raspi-config.

sudo raspi-config

sudo nano /etc/apt/sources.list and add:

deb http://deb.debian.org/debian buster-backports main contrib non-free

sudo apt update

sudo apt upgrade -y

sudo rm /etc/apt/sources.list.d/vscode.list

sudo rm /etc/apt/trusted.gpg.d/microsoft.gpg

sudo reboot

After it restarted I then told it to install OpenZFS from buster-backports which picked up OpenZFS 2.0.2.

sudo apt install -t buster-backports zfs-dkms raspberrypi-kernel-headers -y

It pops up a warning about licenses not being compatible which you have to acknowledge before it proceeds. It takes a while to build. Once its done reboot again and you should now have OpenZFS installed on a Raspberry Pi. I created a mirror with two external HDD (devices sda and sdb)

sudo zpool create pool1 mirror /dev/sda /dev/sdb -f

I then copied some files off my NFS server. It was quite slow which I attribute to using a Pi3 (slow USB ports, slow processor and limited memory). To make sure it all worked, reboot and it should automatically mount the drives in the pool when it starts up. You can check by doing a scrub:

sudo zpool scrub pool1

zpool status

 

Where to next?

At this point its booting off the SD card and the pool is using both disks. What would be nice is to have a bootable ZFS pool rather than using the SD card as a boot device.

If you were after more speed you could use SSD's instead of hard disks but you pay more for the capacity. Something like the Samsung T2 would be ideal. Just remember to do a zpool trim command regularly.

I expect its possible to use a USB hub to get more ports so you can have more drives but think the performance wouldn't be as good as directly plugged devices.

The ZFS forums recommend 1GB of memory for each TB of storage so you would get better performance with a 4GB or 8GB Pi4. 

Update 5 Apr 2021

Added the kernel-headers to the install command and clarified what to add to /etc/apt/sources.list.

I used a Pi4 8GB and two external USB3 HDD this time and it was somewhat quicker at copying files. Buster-backports has OpenZFS 2.0.3 as of writing.

A little controversial

The Raspberry Pi foundation recently added a Microsoft repository to Raspberry Pi OS which some may find controversial. Jeff Geerling made a short video where he goes into the details:

Is Microsoft spying on your Raspberry Pi 

I understand the foundation might want to make it easier to install some Microsoft apps but my Pi's are headless number crunchers. The apps are for a desktop installation not a headless machine. There is no reason for me to be checking their repo every time I do an update. Its just a waste of the Pi's time and my internet bandwidth. I feel it should have been something people opt-into rather than rolling it out to the masses or at least skip those Pi without a desktop. Jeff gives instructions on how to remove it, which are:

sudo rm /etc/apt/sources.list.d/vscode.list

sudo rm /etc/apt/trusted.gpg.d/microsoft.gpg

sudo apt update

Update: 10 March 2021

The foundation pushed through an update today that changes the vscode.list to a comment and removes the microsoft.gpg. You don't need to disable it any more.