I love backing up my data. I love not crying when my disks break. I'm sure I'd love to have my data safe even if my house burnt down while I were on vacation. That is why I keep my beloved data backed up on-site for easy access in case of hardware failure, and an off-site in case my primary site is compromised.

The thing is, not all people and organisations love alien servers in their networks. More often than not, sensitive data flow within these networks. Said networks are often NATed, firewalled, port filtered or otherwise unaccessible from the outside. So it's difficult to find a good off-site location!

One solution is to put a tiny backup-client in a remote guest network. As long as you get connectivity, access to a power outlet and a safe physical location for your device, you're set. For this job, a Raspberrey Pi Zero W fits perfect! It's cheap. It can be configured to fit any need. It's got wireless networking. It can run headless in a drawer if you want it to. And you can connect external USB disks to it. But can a 10$ piece of hardware handle the job of backing up your data? The other day I decided to give it a try.

I had an ancient Raspberry Pi Zero W 1.1 laying around unused in my awesome garage. I also found an unopened 32 GB Micro SD card for the Raspian OS and a "USB-A to Micro USB" adapter to connect a USB Raid enclosure. With these parts I made an automated backup client. I tested it with some load, and it backed up 41 GB (9000 files) in a little less than 9 hours. I limited the maximum bandwith usage to 3,75 MB/s in order to avoid congesting my home network, and achieved a transfer rate of 1,25 MB/s with my 10$ hardware. Not great, but more than acceptable for my needs!

So - how did I do it? Here comes the recipe to make a backup system using one of the cheapest of Raspberries.

Get the Raspberry Pi running and on-line

  • Download the Raspian Lite zip image (I used the Buster version, but use whichever)
  • Use the Balena etcher to burn the image on a Micro SD card

After Balena has done it's magic, your SD-card is ready to be placed in a Raspberry. First, however, you need to edit a couple of files on the SD-card so that you can get connectivity and be able to SSH in to your 10$ device.

  • The volume you need to make changes to is named "boot" by Balena. In my case, the volume was assigned drive letter D, but your mileage may vary.
  • Create an empty file named ssh in the root folder of the "boot" volume (activate SSH access)
  • Create a file named wpa_supplicant.conf in the root folder of the "boot" volume (set up WiFi connectivity)
  • Edit the wpa_supplicant.conf so it looks more or less like the following:
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=NO
network={
    ssid="HomeNetwork"
    psk="p4ssw0rd"
    id_str="Home"
}
network={
    ssid="RemoteNetwork"
    psk="p4ssw0rd!!"
    id_str="Away"
}

Enter your home network SSID and password in order to acces your device from home to set it up. Also enter remote network SSID and password. This way, when you move your Raspberry to your remote location, it'll autoamtically pick up WiFi when turned on.

NOTE! When you boot the Raspberry, it will pick up the two files you created and use them to set up the OS, and delete the files!

Connect to the Raspberry and secure it

The default hostname of the raspberry is "raspberrypi". This means you can normaly just ssh to raspberrypi. If you have a list over DHCP-clients on your network, you may also find it there when it's successfully connected to your wireless network.

When you are prompted for a username, enter: pi When you are prompted for a password, enter: raspberry

Obviously, you need to change the default hostname and the default password in order to secure your device and avoid the next raspberry to be configured to have a duplicate hostname on the same network.

To change the password, simply enter the following command in the raspberry shell after logging in:

  • passwd

To change the hostname, you need yo change the contents in two files:

  • sudo vi /etc/hostname
  • sudo vi /etc/hosts Change "raspberrypi" to a name of your liking.

Reboot (type: sudo reboot) and after a couple of minutes, SSH in with the new hostname and log in with pi and your new password.

It is NOT recommended to try to disable the pi-user.

Set up e-email notifications

It's pretty useful to get notified when backups fail og succeed. It's super easy if you've got a Google account. If you don't, set up a dummy Google account now, just for relaying emails for your servers. Whatever Google account you use, set up an application password for your servers e-mailing needs: application passwords

After this, you're ready to set up Postfix to send e-mails via your Google account. I've seen a lot of tutorials for ssmtp, but never made it work. Postfix on the other hand, works like a breeze. Follow this tutorial: set up Postfix to send via g-mail

After this, please secure your e-mail server. It should not be exposing port 25, so follow this tutorial to bind Postfix to localhost only: secure your Postfix

Connect a USB-disk

32 GB on a Micro SD card isn't much of a backup capacity, so you need an external storage device. I opted for a USB Raid enclosure. As long as it's USB connected you can choose any form of storage and follow this tutorial.

First, conenct your USB device. You'll most probably need a USB-A to Micro USB adapter for this.

Then, run the following command to find the UUID of your USB-device:

  • sudo blkid

When you found your device, make a mount point for it. You can name it anything, i named it "backupraid":

  • sudo mkdir /media/backupraid

Then, set up the device to automount when your Raspberry boots:

  • sudo vi /etc/fstab
  • Add the following line:
    • UUID=fb35a51a-3430-3b05-ba12-f1ab877f10d9 /media/backupraid ext4 defaults,nofail 0 0
    • Replace the string of characters after UUID= with your own UUID, replace "backupraid" with your own mount point
  • Make sure your USB device is connected and powered on and run:
    • sudo reboot

NOTE: This setup doesn't make your USB device hot swappable. The USB device needs to be powered on and ready when your Raspberry boots.

Write your backup routine This is where the magic happens. You need a script that runs on your Raspberry that will connect to your server and fetch the files that need backing up. I use rsync over SSH for this.

The script goes something like:

#!/bin/bash

#Identify the server in emails
HOST=remote-raid-backup

#Who should receive logs
RECIPIENT=recipient@example.com

#Where to log results on disk
LOG_FILE=/var/log/rsync.log

#Remote server
REMOTE_SERVER=remote.server.example.com

#Remote port
REMOTE_PORT=22

#Remote server user
REMOTE_USER=backup-user

#Private key for authentication - remember to have your public key in remote server user's authorized_keys file
IDENTITY_FILE=/home/pi/.ssh/id_rsa

#Bandwith limit in KB/s to avoid congesting your router
BWLIMIT=3750

SECONDS=0
REMOTE_FILES="/remote/dir/photos/"
LOCAL_FILES="/media/backupraid/photos"

RESULT=`/usr/bin/rsync --log-file=${LOG_FILE} --bwlimit=${BWLIMIT} -az --stats -h -e "/usr/bin/ssh -i ${IDENTITY_FILE} -p ${REMOTE_PORT}" --delete --backup --backup-dir="/media/backupraid/photos_rolled_out_\`date '+\%F_\%H-\%M'\`" "${REMOTE_USER}@${REMOTE_SERVER}:$REMOTE_FILES" ${LOCAL_FILES} 2>&1`
FAILED=$?

ELAPSED="$(($SECONDS / 3600))h $((($SECONDS / 60) % 60))m $(($SECONDS % 60))s"

if [ $FAILED != "0" ]
then
  FAILURE=1
  MESSAGE="${MESSAGE}\n\n${DIR} - failed rsyncing PHOTOS after ${ELAPSED}.\n${RESULT}"
else
  STATS=`echo "${RESULT}" | awk '/Number of regular files transferred|Total transferred file size/'`
  MESSAGE="${MESSAGE}\n\n${DIR} - succeeded rsyncing PHOTOS in ${ELAPSED}.\n${STATS}"
fi

if [ $FAILURE != "0" ]
  then
    echo -e "${MESSAGE}" | mail -s "${HOST} - FAILURE in rsync backup" ${RECIPIENT}
  else
    echo -e "${MESSAGE}" | mail -s "${HOST} - rsync backup succeeded" ${RECIPIENT}
fi

The script detects and sends a backup status by e-mail. Files that are deleted or changed will be "rolled out" of the main backup directory and placed in a separate directory with the date the files were made obsolete.

I run the script in root cron like this:

  • sudo crontab -e
  • And add the following line:
    • 30 4 * /path/to/rsync_backup.sh

Add a reverse SSH tunnel

Your backup client is ready to go now, but it's also nice to have a back door in. So I set up a reverse SSH tunnel to a secure server at home. This way, I can easily log in to my Raspberry and check status, as long as it's online.

#!/bin/bash

while true; do
    ssh -i /home/pi/.ssh/id_rsa -N -o ServerAliveInterval=240 -R 2225:localhost:22 -p2222 user@home.server.example.com
    # To connect from the home server type: ssh pi@localhost -p 2225
    sleep 5
done

Add the following line to your root crontab:

  • @reboot /path/to/reverse_ssh.sh

Make sure you establish the SSH-connection from your Raspberry once before you rely on it working from cron, because you need to manually confirm the remote (home) server identity before cron can automatically connect.

Previous Post Next Post