BTRFS on the Raspberry Pi

Given all the interest in BTRFS as a file system I thought I would see what works on the Pi and what doesn't. In x64 Linux its a simple matter of installing an extra package or two and support is baked into the kernel. With the Raspberry Pi, well its not so good.

My experimental rigs

I tried with a couple of setups. One was a Pi3 with a PiDrive and the other was a Pi4 4GB with a Seagate Expansion 2TB. In both cases I am booting off the SD card with Raspberry Pi OS (64 bit) installed and the Pi's are headless (no screen or keyboard).

Moving the root partition

I wrote about how to move the root partition to external storage in 2016. Those instructions still work today if the external storage is not bootable. A link to that is HERE

Unfortunately the PiDrive nor my Seagate Expansion would boot after putting an image on them, so its back to the old method of booting off the SD card and moving the root partition onto the external storage. This works for ext3 and ext4 formatted drives, but NOT for btrfs formatted drives.

Why would you want to?

"Btrfs is a copy on write filesystem for Linux aimed at implementing advanced features while focusing on fault tolerance, repair and easy administration" to quote kernel.org.

Btrfs offers a number of advantages over the ext4 file system which is most commonly used. It supports RAID configurations, scrubbing, deduplication, defragmentation, snapshots and can detect bit rot. Basically a similar feature set to OpenZFS. However unlike OpenZFS its natively supported in Linux.

Btrfs on external storage

It is possible to partition the external storage as Linux and format it as btrfs on the Pi without issue. I first got the Pi booting off its SD card and then installed the btrfs-progs package with the "sudo apt install btrfs-progs" command. I then partitioned the external drive as above. To format it you use the command "sudo mkfs.btrfs /dev/sda1" assuming your external disk was sda and you only had 1 partition on it. The "sudo mke2fs -f btrfs" command doesn't work.

You can even have the Pi mount the external drive in btrfs format by adding it to /etc/fstab. They recommend you use the UUID or PARTUUID in fstab these day so the easy way to find that out is to do a "lsblk -o name,size,type,partuuid" command. I then cut the uuid and paste it into fstab. You can still use the old way of specifying /dev/sda1 in fstab if you prefer but if you have more than one external drive it can lead to issues.

If you change /boot/config.txt to specify a btrfs partition it will hang when you boot. I have read that it leads to a kernel panic but because I didn't have a screen plugged into the Pi's while testing can't confirm what it does, other than it definitely hangs.

There are various hacks to have the root partition in btrfs format, which fail when there is a kernel update so at the moment the nearest you can reliably get is a drive formatted for btrfs and mounted - just not the root partition.

Btrfs support has been requested a number of times of the Raspberry Pi foundation but I believe the issue is around the customized kernel the Pi runs not having btrfs support when it boots, possibly due to the memory needed. If enough people request it I am sure the foundation would address the issue(s), especially now we're 64 bit and have more memory on the Pi4 and Pi5, but they haven't so far.

Update: Feb 2024

For some reason the Pi5 refuses to mount a btrfs volume using the "sudo mount /dev/sda /mnt/exthdd" format. I managed to get it to work using a UUID and it also works on the Pi4. The command would be "sudo mount UUID=xxx /mnt/exthdd" and you can also use this format in /etc/fstab. To get the UUID if you missed it when doing the mkfs.btrfs command above you can do a "sudo btrfs filesystem show" command and it will list the btrfs disks and their UUID. In my case I had two drives in what btrfs calls single (ie one big volume) so I mount the disk rather than a partition.

OpenZFS on a Pi

I used a Pi3 as a guinea pig and tried installing OpenZFS on it using a pair of Seagate Expansion 2TB drives connected via the USB ports. Initially I tried with Raspbian because that is what the SD card in the Pi had installed. During the install it complained it was a 32 bit operating system and it wasn't going to work.

I went to image the SD card with Raspberry Pi OS 64 bit (Lite) and Win32 Disk Imager was having trouble (speeds of 0.45MB/sec) writing to the SD card. I replaced it with a new one. After booting up I added the buster-backports repo to the /etc/apt/sources.list and got the operating system up to date. Start by changing the password via raspi-config.

sudo raspi-config

sudo nano /etc/apt/sources.list and add:

deb http://deb.debian.org/debian buster-backports main contrib non-free

sudo apt update

sudo apt upgrade -y

sudo rm /etc/apt/sources.list.d/vscode.list

sudo rm /etc/apt/trusted.gpg.d/microsoft.gpg

sudo reboot

After it restarted I then told it to install OpenZFS from buster-backports which picked up OpenZFS 2.0.2.

sudo apt install -t buster-backports zfs-dkms raspberrypi-kernel-headers -y

It pops up a warning about licenses not being compatible which you have to acknowledge before it proceeds. It takes a while to build. Once its done reboot again and you should now have OpenZFS installed on a Raspberry Pi. I created a mirror with two external HDD (devices sda and sdb)

sudo zpool create pool1 mirror /dev/sda /dev/sdb -f

I then copied some files off my NFS server. It was quite slow which I attribute to using a Pi3 (slow USB ports, slow processor and limited memory). To make sure it all worked, reboot and it should automatically mount the drives in the pool when it starts up. You can check by doing a scrub:

sudo zpool scrub pool1

zpool status

 

Where to next?

At this point its booting off the SD card and the pool is using both disks. What would be nice is to have a bootable ZFS pool rather than using the SD card as a boot device.

If you were after more speed you could use SSD's instead of hard disks but you pay more for the capacity. Something like the Samsung T2 would be ideal. Just remember to do a zpool trim command regularly.

I expect its possible to use a USB hub to get more ports so you can have more drives but think the performance wouldn't be as good as directly plugged devices.

The ZFS forums recommend 1GB of memory for each TB of storage so you would get better performance with a 4GB or 8GB Pi4. 

Update 5 Apr 2021

Added the kernel-headers to the install command and clarified what to add to /etc/apt/sources.list.

I used a Pi4 8GB and two external USB3 HDD this time and it was somewhat quicker at copying files. Buster-backports has OpenZFS 2.0.3 as of writing.

Do something useful with your Pi4

You can now run Rosetta@home on the Pi4. They released a new app called "Rosetta for Portable devices" that will run under Raspbian but its for the ARM64 architecture.

Link to Rosetta@home: http://boinc.bakerlab.org/rosetta/

For those that don't know Rosetta is searching for proteins in relation to the Covid-19 virus. Its a BOINC-based project so you'll need to install that first. There are a few catches though.

• You'll have to put the kernel into 64 bit mode.

• You have to tell BOINC its an aarch64 platform (ARM64).

• Some tasks can use up to 1.3GB of memory each so you need to limit how many run at the same time.

Most BOINC projects don't have aarch64 apps, particularly Einstein which gave me an incompatible app and then all the work units failed. You can go back to armhf (32 bit) mode later by reversing these steps.

Before somebody asks, yes you could theoretically do this on a Pi3B or a Pi3B+ but they only have 1GB of memory so its not worthwhile.

Assuming you still want to proceed this is the process.


1. Install BOINC
If you already have it installed then skip this step.If not install it and sign up to any projects you want to run. I won't go into detail here but if there is interest I can cover it in another blog post.

Set it to No New Tasks for all projects. If it has any work in progress let it finish and report it before proceeding.


2. Put the kernel into 64 bit mode
This tells the kernel to use 64 bit mode also known as ARM64 or aarch64. By default the kernel runs in 32 bit mode for compatibility with the older Pi software. You'll need the 4.19.75 or later kernel. To find out what you have currently type the following in a terminal or ssh session:
uname -a

If you have an older one then you'd better upgrade it to the latest packages and then come back to this step.

Make a copy of the config.txt first. Type:
cd /boot
sudo cp config.txt config.old

Edit the config.txt. Type:
sudo nano config.txt

Once in nano (its a text editor) scroll to the very end and add "arm_64bit=1" (without the quotes) on a new line.

Press Control-O to save the file.

Press Control-X to exit from nano.

Now reboot it by typing:
sudo reboot


3. Tell BOINC to use aarch64
Back in a terminal or ssh session type:
uname -a

You should see it says "aarch64" on the end of the line. If not then you need to fix that up first. If it does we now need to edit the cc_config.xml file that BOINC uses. Type:
cd /etc/boinc-client
sudo nano cc_config.xml

Blank out the contents using the delete or backspace keys and then paste or type this in:

<cc_config>
  <log_flags>
    <task>1</task>
    <file_xfer>1</file_xfer>
    <sched_ops>1</sched_ops>
  </log_flags>
  <options>
    <alt_platform>aarch64-unknown-linux-gnu</alt_platform>
  </options>
</cc_config>

Press Control-O to save the file.

Press Control-X to exit from nano.

We need to restart BOINC to pick up this change so type the following command:
sudo systemctl restart boinc-client

Now you can allow BOINC to accept tasks.


Why do we need to this?
When you install Raspbian on your Pi its all 32 bit, all the add-on programs and so on. Its known as armhf (ARM hard float). The Raspberry Pi Foundation made kernel changes to allow it to run in 64 bit mode. That's the kernel taken care of but the apps are still 32 bit.

When you install BOINC you get the armhf version. That is because the Foundation don't have aarch64 repositories. We then need to tell BOINC to use aarch64 as an alternative to armhf in order get 64 bit apps.

Add buster-backports to a Raspberry Pi

Lets assume you wanted some app from buster-backports, usually because its more up to date than the version in buster. The Raspberry Pi foundation doesn't have a backports repo so you have to use the Debian one instead. This is how to add it.

Update: March 2021

The Raspberry Pi OS 64 bit already has the necessary signing keys. I haven't used the 32 bit one for a while but it may also have the necessary signing keys. You only need to do point 2 if you get an error telling you you don't have the GPG keys.

As mentioned in the comments the GPG keyservers disappear. The kfwebs.net listed below is no longer available but there are others listed in the comments that can be used instead.

 

1. Add to /etc/apt/sources.list
This adds the Debian buster-backports repo to our list of repositories to use. To edit the list of repositories:

sudo nano /etc/apt/sources.list

Add the following line to the end

deb http://deb.debian.org/debian buster-backports main


2. Add Debian Buster keys
This adds the signing keys so we can trust the Debian repos.

gpg --keyserver hkp://keys2.kfwebs.net:80 --recv-keys 04EE7237B7D453EC
gpg --keyserver hkp://keys2.kfwebs.net:80 --recv-keys 648ACFD622F3D138

gpg --export 04EE7237B7D453EC | sudo apt-key add -
gpg --export 648ACFD622F3D138 | sudo apt-key add -


3. Tell apt to update
Tell apt to get the latest list of packages.

sudo apt update


4. See if there is an updated package
If you want to see if there is an updated package called xxx use the following command:

sudo apt list xxx -a


5. Get a package from the backports repo
By default apt won't install packages from the backports repo, you need to tell it. If we want to install package xxx from buster-backports use the following command:

sudo apt install -t buster-backports xxx

ARM64 on Raspberry Pi 4

It doesn't work. Well bits of it do. If you have got an up to date kernel, I am on 4.19.75, you can simply put it into 64 bit mode by adding "arm_64bit=1" (without the quotes) to the end of /boot/config.txt and reboot it. The kernel will now be an aarch64 kernel.

The Raspberry Pi foundation doesn’t currently have an arm64 repository so all of the programs are 32 bit (armhf) apart from the kernel. I had BOINC installed from the foundation repo (ie armhf version) and then switched the kernel to 64 bit mode and it continued to work. Not everything will. I wanted to get the arm64 version of BOINC. It kept choosing the foundation repo. I tried specifying I wanted the 64 bit version and it complained about dependencies. I commented out the foundation repos so it only knew about Debian and then I removed the armhf version of BOINC and tried installing the 64 bit version - it still complained. In the end I gave up.

My next step will be to try a 64 bit Ubuntu on it. Until then I have put it back to 32 bit mode and reinstalled the armhf apps.