Saturday, May 8, 2021

36TB Pi Storage Server

After seeing Jeff Geerling compare his home-made storage server with a commercial one I thought I would expand on my OpenZFS post. You wouldn't mine Chia on a Pi but you can use it as a cheap storage server after creating your storage plots.

Parts
1 x Raspberry Pi4 and case
1 x Rpi power supply
1 x Micro SD card
1 x Powered USB 3.0 hub
5 x External HDD (powered)
2 x Power boards for all the power adapters

I'm not sponsored by anyone so I had to pay for all the parts myself. The power boards are old ones I had left over (I swap out old power boards every few years as the cheap ones tend to fail). The Pi4 was reallocated from number crunching duties. I picked 10TB drives based on the price per GB. If I wanted lots of disk space and price wasn't an issue I would have used 16TB drives. Its best to keep the drives the same size, otherwise you have to use multiple pools.

I am using a 4GB Pi4. The USB hub is a TP-Link UH700 which gives 7 ports and is rated for USB 3.0 speeds. Most USB hubs larger than 4 ports seem to come with a power adapter. The disks are Seagate Expansion 10TB drives.

 

I had to use two 6 outlet power boards because the spacing between the sockets means I can't plug the adapters in next to each other. Wiring gets messy. I would recommend zip-tying the cables to keep it tidy.



I assume that you have plugged everything in, powered it up and installed OpenZFS on your Raspberry Pi. See my earlier blog post for getting OpenZFS installed Here


Lets see what drives are attached. The lsblk (list block device) command will show them:

lsblk

NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  9.1T  0 disk
|-sda1        8:1    0  9.1T  0 part
`-sda9        8:9    0    8M  0 part
sdb           8:16   0  9.1T  0 disk
|-sdb1        8:17   0  9.1T  0 part
`-sdb9        8:25   0    8M  0 part
sdc           8:32   0  9.1T  0 disk
|-sdc1        8:33   0  9.1T  0 part
`-sdc9        8:41   0    8M  0 part
sdd           8:48   0  9.1T  0 disk
|-sdd1        8:49   0  9.1T  0 part
`-sdd9        8:57   0    8M  0 part
sde           8:64   0  9.1T  0 disk
|-sde1        8:65   0  9.1T  0 part
`-sde9        8:73   0    8M  0 part
mmcblk0     179:0    0 14.9G  0 disk
|-mmcblk0p1 179:1    0  256M  0 part /boot
`-mmcblk0p2 179:2    0 14.6G  0 part /


This shows I have 5 disks called sda through sde. Each disk is 9.1TB in capacity and has two partitions (sda1 and sda9). The mmcblk0 device is the micro SD card which the Pi is booting off.


Create a ZFS pool
Before you do this command make sure that there isn't any data you want to keep on the drives.

sudo zfs create pool1 raidz /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde -f

This says create a pool (of disks) called "pool1" its a raidz and we're using disks sda through sde. Raidz allows for a single disk failure. That is it can rebuild the missing information from data stored across the other 4 disks. Of course this means we lose space to allow for this data redundancy.



Pool attributes
To get more usable space we can turn compression on for our pool. Turning compression on means we try to compress the data before storing it on disk, however it requires some CPU time to do this.

You may have files that don't compress well such as zip files in which case there is no point turning compression on. I don't mine Chia so can't comment on storage plots being compressible or not.

The command to turn on zstd compression is:

sudo zfs set compression=zstd pool1

Note that you need OpenZFS 2 to get zstd compression. If running an older version you can use lz4 compression.

We can save some disk activity by not updating the atime attribute, that is the last time a file was accessed. The command to do this is:

sudo zfs set atime=off pool1

To check our pool the command is:

zpool status

  pool: pool1
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        pool1       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sda     ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0
            sdd     ONLINE       0     0     0
            sde     ONLINE       0     0     0

errors: No known data errors



Check disk space
Lets see how much space we have after creating our pool. The command would be:

df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  1.5G   13G  11% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G  8.6M  1.9G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   30M  223M  12% /boot
tmpfs           385M     0  385M   0% /run/user/1000
pool1            36T  256K   36T   1% /pool1


You can see pool1 at the bottom and we have 36TB of available disk space after allowing for data redundancy. The theoretical disk capacity is 45.5TB (5 x 9.1TB).

It could be much larger by:

  • Use more drives. I only have 5 drives plugged in but the USB hub can handle up to 7.
  • Use a second hub with another 7 drives.
  • Use larger disks. The expansion drives are available in sizes up to 16TB currently.


If you are going to have one of these running 24/7 I would recommend you keep a spare drive (or two) for when one fails. Being a spinning disk its just a matter of when, not if, a drive will fail. A raid array is not a backup so keep a backup. A UPS would be a good idea as well.

After powering off the Pi and turning it on again the disk pool disappeared. A simple reboot fixed that. I believe this is caused by the drive spin up time. You can configure a boot delay for the Pi which should give the drives enough time to spin up.

This setup should be able to hold approximately 363 Chia storage plots without compression.

 

Pricing
The pricing below is in US dollars.

1 x Pi4 4GB ($68)
1 x Pi4 case ($13)
1 x Pi4 power supply ($15)
1 x 16GB Micro SD card ($17)
1 x USB hub ($65)
5 x 10TB external drives ($215 each)
2 x Power strip ($13 each)
Total = $1,279 (USD)

Some parts (eg the Pi4) can be bought 2nd hand or found cheaper. I've rounded up to the nearest dollar. Pricing taken from amazon on the 9th of May 2021.


Sunday, May 2, 2021

2nd of May

Marks Rpi Cluster continues running 24/7. After purchasing a couple more 32GB micro SD cards last week the cluster now consists of:

4 x Pi3 compute nodes

8 x Pi4 8GB compute nodes

3 x Pi4 2GB support nodes


Pi-Hole

I installed Pi-Hole on a Pi4. Its a caching DNS server that can use blocklists to prevent access to websites. It comes with a list of 80,000 advertising websites that it can block and has a webpage to show statistics. I also use a caching proxy server (squid) to speed up access by caching frequently used web traffic. I had a small list of about 30 advertising websites that I block with it.

Having installed Pi-Hole I pointed my proxy server at it as a DNS server and commented out blocking of advertising websites in the proxy server. The two worked fine together. Pi-Hole has a much more complete list of advertising sites than I ever had.

The next step was to install Pi-Hole on the same Pi as the proxy server, that way they don't need to communicate across the network. I installed it on the proxy server Pi4 and adjusted the proxy server to use it instead. Its doing its job of suppressing the adverts on most websites. It doesn't get all of them but its certainly getting most of them and internet access seems seem slightly faster (due to not downloading ads).

You can get Pi-Hole from https://pi-hole.net/ 

They have versions for Raspberry Pi OS, Ubuntu (ARM and x64), Debian (ARM, x64 and i386), Fedora (ARM and x64) and CentOS (x64 only).


Saturday, April 24, 2021

25th of April

I purchased some more 32GB Micro SD cards and swapped out the two Pi4 4GB with Pi4 8GB. That means the cluster now consists of:

4 x Pi3 compute nodes

8 x Pi4 8GB compute nodes

3 x Pi4 2GB support nodes


I've have been using one of the Pi4 4GB to run Pi-Hole and its been working fine on its own Pi. I will probably combine it with the proxy server and have them both run on the same Pi4.

I have plans to use the other Pi4 4GB for a Pi NAS, not that I need lots of disk space, but that will be for another blog post.

Friday, April 16, 2021

17th of April

10 million 

The cluster passed 10 million credits for Einstein@home about two weeks ago, but I didn't notice at the time. Its had a separate account since April 2016, which means it has taken around 5 years to reach this point.

 


 

Rearranging

Two of the Pi3 nodes were retired and two Pi4 (8GB) nodes replaced them. That increases the flexibility of the cluster as the Pi4's are faster and have more memory. The cluster now consists of:

3 x Pi4 (2GB) as support nodes

4 x Pi3 compute nodes

2 x Pi4 (4GB) compute nodes

6 x Pi4 (8GB) compute nodes

The recent changes at Roesetta@home have meant the Pi4 4GB are no longer able to run their work so I will swap them for Pi4 8GB, after I buy some more micro SD cards.