Sunday, October 23, 2022

23rd of October

Marks Rpi Cluster continues to run 24/7. The cluster has 19.4M credits accumulated for Einstein@home and has a RAC (Recent Average Credit) of 40k. If output continues at the current pace it should hit 20M credits in a bit over a fortnight.

I would like to make the cluster a bit more resilient by duplicating a couple of the support nodes. However I can't seem to find the Pi4 2GB model for sale unless I resort to the overpriced ones on eBay. They make approximately 500k units a month according to Eben Upton but they still seem impossible to find. Their advice was to keep checking rpilocator.com to see where stock is available.


Sunday, October 9, 2022

9th of October

Marks Rpi Cluster continues to run 24/7. As mentioned in previous posts its concentrating on running Einstein@home BRP4 work. That is 24 Raspberry Pi4's looking through data captured from the Arecibo radio telescope (before its demise), searching for Binary Radio Pulsars.

Most of the compute nodes have managed to get their RAC (Recent Average Credit) up to 1600, with a couple even getting up to 1700.

I was asked last week what the up-time on the compute nodes was. These days it is usually one to two weeks due to security fixes and other updates, but I have gone for a few months before rebooting them.


Universe@home
I had a brief look at running Universe@home on Marks Rpi Cluster. They are an astronomy project currently doing analysis on Black Holes.

Their Raspberry Pi app is for the ARMv6 (original Raspberry Pi). The Pi2 was ARMv7 and the Pi3 and Pi4 are both ARMv8. In other words the app is not able to take advantage of the newer Pis hardware. Its also a 32 bit app so won't take advantage of a 64 bit operating system and one has to install support libraries to run them under Raspberry Pi OS 64 bit.

Needless to say I gave up on the idea of running it.

Saturday, September 3, 2022

Progress as of 3rd of September

This is where Marks Rpi Cluster is as of the 3rd of September for Einstein@home. Total credit is 17,387,442 and recent average is 31,909 credits.

 

Einstein@home had a power outage earlier in the week and apparently the Atlas Cluster that they use had UPS failures and they haven't managed to revive enough of the cluster. Its used to pre-process the BRP4 work so they have run out. In addition Rosetta@home doesn't have any work, but they haven't for months.

The dip on the graph just before the word Today is when they they went down earlier in the week and the spike hitting the word Today was Marks Rpi Cluster returning completed work after the initial outage. Normally I keep 0.3 days of work cached, but given the recent instability of the project I will be increasing it.


Cluster make up
As of the 3rd of September the cluster is made up of:

3 x Pi4 (2GB) as support nodes
24 x Pi4 (8GB) as compute nodes

For those that like numbers that is 96 ARMv8 CPU cores, 192GB of memory and 768GB of SD card storage for the compute nodes.

The support nodes are a proxy server, a storage server and a time server with a GPS.

Saturday, August 27, 2022

BTRFS on the Raspberry Pi

Given all the interest in BTRFS as a file system I thought I would see what works on the Pi and what doesn't. In x64 Linux its a simple matter of installing an extra package or two and support is baked into the kernel. With the Raspberry Pi, well its not so good.


My experimental rigs

I tried with a couple of setups. One was a Pi3 with a PiDrive and the other was a Pi4 4GB with a Seagate Expansion 2TB. In both cases I am booting off the SD card with Raspberry Pi OS (64 bit) installed and the Pi's are headless (no screen or keyboard).


Moving the root partition

I wrote about how to move the root partition to external storage in 2016. Those instructions still work today if the external storage is not bootable. A link to that is HERE

Unfortunately the PiDrive nor my Seagate Expansion would boot after putting an image on them, so its back to the old method of booting off the SD card and moving the root partition onto the external storage. This works for ext3 and ext4 formatted drives, but NOT for btrfs formatted drives.


Why would you want to?

"Btrfs is a copy on write filesystem for Linux aimed at implementing advanced features while focusing on fault tolerance, repair and easy administration" to quote kernel.org.

Btrfs offers a number of advantages over the ext4 file system which is most commonly used. It supports RAID configurations, scrubbing, deduplication, defragmentation, snapshots and can detect bit rot. Basically a similar feature set to OpenZFS. However unlike OpenZFS its natively supported in Linux.


Btrfs on external storage

It is possible to partition the external storage as Linux and format it as btrfs on the Pi without issue. I first got the Pi booting off its SD card and then installed the btrfs-progs package with the "sudo apt install btrfs-progs" command. I then partitioned the external drive as above. To format it you use the command "sudo mkfs.btrfs /dev/sda1" assuming your external disk was sda and you only had 1 partition on it. The "sudo mke2fs -f btrfs" command doesn't work.

You can even have the Pi mount the external drive in btrfs format by adding it to /etc/fstab. They recommend you use the UUID or PARTUUID in fstab these day so the easy way to find that out is to do a "lsblk -o name,size,type,partuuid" command. I then cut the uuid and paste it into fstab. You can still use the old way of specifying /dev/sda1 in fstab if you prefer but if you have more than one external drive it can lead to issues.

If you change /boot/config.txt to specify a btrfs partition it will hang when you boot. I have read that it leads to a kernel panic but because I didn't have a screen plugged into the Pi's while testing can't confirm what it does, other than it definitely hangs.

There are various hacks to have the root partition in btrfs format, which fail when there is a kernel update so at the moment the nearest you can reliably get is a drive formatted for btrfs and mounted - just not the root partition.

Btrfs support has been requested a number of times of the Raspberry Pi foundation but I believe the issue is around the customized kernel the Pi runs not having btrfs support when it boots, possibly due to the memory needed. If enough people request it I am sure the foundation would address the issue(s), especially now we're 64 bit and have more memory on the Pi4 and Pi5, but they haven't so far.


Update: Feb 2024

For some reason the Pi5 refuses to mount a btrfs volume using the "sudo mount /dev/sda /mnt/exthdd" format. I managed to get it to work using a UUID and it also works on the Pi4. The command would be "sudo mount UUID=xxx /mnt/exthdd" and you can also use this format in /etc/fstab. To get the UUID if you missed it when doing the mkfs.btrfs command above you can do a "sudo btrfs filesystem show" command and it will list the btrfs disks and their UUID. In my case I had two drives in what btrfs calls single (ie one big volume) so I mount the disk rather than a partition.