Saturday, August 27, 2022

BTRFS on the Raspberry Pi

Given all the interest in BTRFS as a file system I thought I would see what works on the Pi and what doesn't. In x64 Linux its a simple matter of installing an extra package or two and support is baked into the kernel. With the Raspberry Pi, well its not so good.


My experimental rigs

I tried with a couple of setups. One was a Pi3 with a PiDrive and the other was a Pi4 4GB with a Seagate Expansion 2TB. In both cases I am booting off the SD card with Raspberry Pi OS (64 bit) installed and the Pi's are headless (no screen or keyboard).


Moving the root partition

I wrote about how to move the root partition to external storage in 2016. Those instructions still work today if the external storage is not bootable. A link to that is HERE

Unfortunately the PiDrive nor my Seagate Expansion would boot after putting an image on them, so its back to the old method of booting off the SD card and moving the root partition onto the external storage. This works for ext3 and ext4 formatted drives, but NOT for btrfs formatted drives.


Why would you want to?

"Btrfs is a copy on write filesystem for Linux aimed at implementing advanced features while focusing on fault tolerance, repair and easy administration" to quote kernel.org.

Btrfs offers a number of advantages over the ext4 file system which is most commonly used. It supports RAID configurations, scrubbing, deduplication, defragmentation, snapshots and can detect bit rot. Basically a similar feature set to OpenZFS. However unlike OpenZFS its natively supported in Linux.


Btrfs on external storage

It is possible to partition the external storage as Linux and format it as btrfs on the Pi without issue. I first got the Pi booting off its SD card and then installed the btrfs-progs package with the "sudo apt install btrfs-progs" command. I then partitioned the external drive as above. To format it you use the command "sudo mkfs.btrfs /dev/sda1" assuming your external disk was sda and you only had 1 partition on it. The "sudo mke2fs -f btrfs" command doesn't work.

You can even have the Pi mount the external drive in btrfs format by adding it to /etc/fstab. They recommend you use the UUID or PARTUUID in fstab these day so the easy way to find that out is to do a "lsblk -o name,size,type,partuuid" command. I then cut the uuid and paste it into fstab. You can still use the old way of specifying /dev/sda1 in fstab if you prefer but if you have more than one external drive it can lead to issues.

If you change /boot/config.txt to specify a btrfs partition it will hang when you boot. I have read that it leads to a kernel panic but because I didn't have a screen plugged into the Pi's while testing can't confirm what it does, other than it definitely hangs.

There are various hacks to have the root partition in btrfs format, which fail when there is a kernel update so at the moment the nearest you can reliably get is a drive formatted for btrfs and mounted - just not the root partition.

Btrfs support has been requested a number of times of the Raspberry Pi foundation but I believe the issue is around the customized kernel the Pi runs not having btrfs support when it boots, possibly due to the memory needed. If enough people request it I am sure the foundation would address the issue(s), especially now we're 64 bit and have more memory on the Pi4 and Pi5, but they haven't so far.


Update: Feb 2024

For some reason the Pi5 refuses to mount a btrfs volume using the "sudo mount /dev/sda /mnt/exthdd" format. I managed to get it to work using a UUID and it also works on the Pi4. The command would be "sudo mount UUID=xxx /mnt/exthdd" and you can also use this format in /etc/fstab. To get the UUID if you missed it when doing the mkfs.btrfs command above you can do a "sudo btrfs filesystem show" command and it will list the btrfs disks and their UUID. In my case I had two drives in what btrfs calls single (ie one big volume) so I mount the disk rather than a partition.

Saturday, August 20, 2022

EC12 update

I picked up another BitScope Edge Cluster 12 along with 12 x Pi4's so I need to assemble it and flash a bunch of micro SD cards. I'll post about it once I have done it.

While at the BitScope office I dropped off the faulty top plate and they gave me a replacement for my existing EC12. They also gave me replacement nylon stand-offs that were missing from one corner of a cluster plate which I had already worked around. Once home I corrected my assembly of the existing EC12 so it now all fits together correctly. I have the front cover on, the power connector is at the back and its been working fine. This is what it looks like from the front now.

 


I had one of the Pi4's getting rather a high number of invalid tasks compared to the other Pi's in the cluster so I flashed a new micro SD card for it. When I first powered it on it wouldn't let me login so I had to power everything down and redo the SD card, which worked on the second attempt. Hopefully that will resolve its issue.

I had to buy a bunch of micro SD cards for the new EC12. When I was in the shop they didn't have my usual Sandisk Ultra 32GB cards so they gave me some Sandisk High Endurance 32GB cards. About all I can find online is their rated speed drops from 120MB/sec to 100MB/sec on reads and writes are also slower. The only endurance figures I can find is on the packaging than mentions "Up to 2500 hours of continuous video recording".


Update: 24 August 2022

Here are both EC12's side by side. The EC12 on the right has a 16 port switch on top and all those light blue network cables running off to each Pi. The EC12 on the left is connected to another 16 port switch.



Saturday, August 6, 2022

7th of August

Marks Rpi Cluster continues to run 24/7. The three newer Pi4's have built up their Recent Average Credit for einstein@home to 1000 and some of the existing Pi4's are up to 1300 credits. It takes about a month to stabilise so pretty good for two weeks of running them in the EC12.


Validation failure rate
I looked at the validation failures I was getting. That is where I compute a work unit and it doesn't agree with other computers that have also done the same work unit.

I've been running an ARM64 optimized version of the BRP4 app written by a volunteer with the handle N30DG. I ran the project supplied 1.61 app for a few days on a single Pi4. I checked the average validation failure rate for all computers. It was around 10 to 11% from einstein@home's server status page. It varies a bit from day to day. My validation failures are around 5%, so better than most.

The optimized BRP4 app is taking slightly over 3 hours. The project supplied 1.61 app takes around 4 and a half hours.


Another EC12
I ordered another Edge Cluster 12. BitScope had stock of the Pi4 8GB at the same price as my usual suppliers so I ordered 12 to go in it. I should get them next week. Hopefully the Pi4's will be the 1.8GHz version.

I received another couple of Meanwell power supplies that I ordered online from an electrical distributor. The Meanwell power supplies are designed to drive LED lights, but I happen to use them to power the EC12 and before that the Blade Rack 20.

I need more 32GB MicroSD cards. I have a few spares as they tend to wear out, but I don't have 12 of them. I should be fine for network cables and switches.