I love backing up my data. I love not crying when my disks break. I'm sure I'd love to have my data safe even if my house burnt down while I were on vacation. That is why I keep my beloved data backed up on-site for easy access in case of hardware failure, and an off-site in case my primary site is compromised.
The thing is, not all people and organisations love alien servers in their networks. More often than not, sensitive data flow within these networks. Said networks are often NATed, firewalled, port filtered or otherwise unaccessible from the outside. So it's difficult to find a good off-site location!
One solution is to put a tiny backup-client in a remote guest network. As long as you get connectivity, access to a power outlet and a safe physical location for your device, you're set. For this job, a Raspberrey Pi Zero W fits perfect! It's cheap. It can be configured to fit any need. It's got wireless networking. It can run headless in a drawer if you want it to. And you can connect external USB disks to it. But can a 10$ piece of hardware handle the job of backing up your data? The other day I decided to give it a try.
I had an ancient Raspberry Pi Zero W 1.1 laying around unused in my awesome garage. I also found an unopened 32 GB Micro SD card for the Raspian OS and a "USB-A to Micro USB" adapter to connect a USB Raid enclosure. With these parts I made an automated backup client. I tested it with some load, and it backed up 41 GB (9000 files) in a little less than 9 hours. I limited the maximum bandwith usage to 3,75 MB/s in order to avoid congesting my home network, and achieved a transfer rate of 1,25 MB/s with my 10$ hardware. Not great, but more than acceptable for my needs!
So - how did I do it? Here comes the recipe to make a backup system using one of the cheapest of Raspberries.
Get the Raspberry Pi running and on-line
After Balena has done it's magic, your SD-card is ready to be placed in a Raspberry. First, however, you need to edit a couple of files on the SD-card so that you can get connectivity and be able to SSH in to your 10$ device.
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=NO
network={
ssid="HomeNetwork"
psk="p4ssw0rd"
id_str="Home"
}
network={
ssid="RemoteNetwork"
psk="p4ssw0rd!!"
id_str="Away"
}Enter your home network SSID and password in order to acces your device from home to set it up. Also enter remote network SSID and password. This way, when you move your Raspberry to your remote location, it'll autoamtically pick up WiFi when turned on.
NOTE! When you boot the Raspberry, it will pick up the two files you created and use them to set up the OS, and delete the files!
Connect to the Raspberry and secure it
The default hostname of the raspberry is "raspberrypi". This means you can normaly just ssh to raspberrypi. If you have a list over DHCP-clients on your network, you may also find it there when it's successfully connected to your wireless network.
When you are prompted for a username, enter: pi When you are prompted for a password, enter: raspberry
Obviously, you need to change the default hostname and the default password in order to secure your device and avoid the next raspberry to be configured to have a duplicate hostname on the same network.
To change the password, simply enter the following command in the raspberry shell after logging in:
To change the hostname, you need yo change the contents in two files:
Reboot (type: sudo reboot) and after a couple of minutes, SSH in with the new hostname and log in with pi and your new password.
It is NOT recommended to try to disable the pi-user.
Set up e-email notifications
It's pretty useful to get notified when backups fail og succeed. It's super easy if you've got a Google account. If you don't, set up a dummy Google account now, just for relaying emails for your servers. Whatever Google account you use, set up an application password for your servers e-mailing needs: application passwords
After this, you're ready to set up Postfix to send e-mails via your Google account. I've seen a lot of tutorials for ssmtp, but never made it work. Postfix on the other hand, works like a breeze. Follow this tutorial: set up Postfix to send via g-mail
After this, please secure your e-mail server. It should not be exposing port 25, so follow this tutorial to bind Postfix to localhost only: secure your Postfix
Connect a USB-disk
32 GB on a Micro SD card isn't much of a backup capacity, so you need an external storage device. I opted for a USB Raid enclosure. As long as it's USB connected you can choose any form of storage and follow this tutorial.
First, connect your USB device. You'll most probably need a USB-A to Micro USB adapter for this.
Then, run the following command to find the UUID of your USB-device:
When you found your device, make a mount point for it. You can name it anything, i named it "backupraid":
Then, set up the device to automount when your Raspberry boots:
NOTE: This setup doesn't make your USB device hot swappable. The USB device needs to be powered on and ready when your Raspberry boots.
NOTE 2: USB disks are prone to be lost and stolen - especially if you bring them to open-plan offices and the likes. If you'd like to secure your data physically from theft, check out my disk encryption tutorial.
Write your backup routine This is where the magic happens. You need a script that runs on your Raspberry that will connect to your server and fetch the files that need backing up. I use rsync over SSH for this.
The script goes something like:
#!/bin/bash
#Identify the server in emails
HOST=remote-raid-backup
#Who should receive logs
[email protected]
#Where to log results on disk
LOG_FILE=/var/log/rsync.log
#Remote server
REMOTE_SERVER=remote.server.example.com
#Remote port
REMOTE_PORT=22
#Remote server user
REMOTE_USER=backup-user
#Private key for authentication - remember to have your public key in remote server user's authorized_keys file
IDENTITY_FILE=/home/pi/.ssh/id_rsa
#Bandwith limit in KB/s to avoid congesting your router
BWLIMIT=3750
SECONDS=0
REMOTE_FILES="/remote/dir/photos/"
LOCAL_FILES="/media/backupraid/photos"
RESULT=`/usr/bin/rsync --log-file=${LOG_FILE} --bwlimit=${BWLIMIT} -az --stats -h -e "/usr/bin/ssh -i ${IDENTITY_FILE} -p ${REMOTE_PORT}" --delete --backup --backup-dir="/media/backupraid/photos_rolled_out_\`date '+\%F_\%H-\%M'\`" "${REMOTE_USER}@${REMOTE_SERVER}:$REMOTE_FILES" ${LOCAL_FILES} 2>&1`
FAILED=$?
ELAPSED="$(($SECONDS / 3600))h $((($SECONDS / 60) % 60))m $(($SECONDS % 60))s"
if [ $FAILED != "0" ]
then
FAILURE=1
MESSAGE="${MESSAGE}\n\n${DIR} - failed rsyncing PHOTOS after ${ELAPSED}.\n${RESULT}"
else
STATS=`echo "${RESULT}" | awk '/Number of regular files transferred|Total transferred file size/'`
MESSAGE="${MESSAGE}\n\n${DIR} - succeeded rsyncing PHOTOS in ${ELAPSED}.\n${STATS}"
fi
if [ $FAILURE != "0" ]
then
echo -e "${MESSAGE}" | mail -s "${HOST} - FAILURE in rsync backup" ${RECIPIENT}
else
echo -e "${MESSAGE}" | mail -s "${HOST} - rsync backup succeeded" ${RECIPIENT}
fiThe script detects and sends a backup status by e-mail. Files that are deleted or changed will be "rolled out" of the main backup directory and placed in a separate directory with the date the files were made obsolete.
I run the script in root cron like this:
Add a reverse SSH tunnel
Your backup client is ready to go now, but it's also nice to have a back door in. So I set up a reverse SSH tunnel to a secure server at home. This way, I can easily log in to my Raspberry and check status, as long as it's online.
#!/bin/bash
while true; do
ssh -i /home/pi/.ssh/id_rsa -N -o ServerAliveInterval=240 -R 2225:localhost:22 -p2222 [email protected]
# To connect from the home server type: ssh pi@localhost -p 2225
sleep 5
doneAdd the following line to your root crontab:
Make sure you establish the SSH-connection from your Raspberry once before you rely on it working from cron, because you need to manually confirm the remote (home) server identity before cron can automatically connect.