Jock in a Frock
Posts: 72
Joined: Thu Jan 12, 2012 6:39 pm

The Raspberry Pi Backup Thread

Sat Jul 21, 2012 9:16 pm

I’m starting this thread to get a discussion going about backing up the OS.

I don’t need to know how to put my SD card in a Windows box and create an image. I’m more interested in creating a mechanism where the OS will automatically backup to either a USB stick, USB drive, or a network share.

I want to develop a script which will control this mechanism, allowing users to determine the schedule, backup destination, compression / encryption options etc.

I also want to discuss the best methodology for backing up the OS.

So far, I have been able to set up a CRON job to backup the entire SD card to a USB stick, and also to a network share on my Windows box. I haven’t tried restoring these yet (since I’ve been busy updating to Raspbian Wheezy), but I expect they would work fine. The backups were done with the following commands:

Code: Select all

sudo dd bs=1M if=/dev/mmcblk0p2 of=/mnt/usb/RPI_Backup.img
I have tried piping the dd input file through gzip before writing it to the HDD, and although it compressed down from 7.9Gb to about 2Gb, it did increase the backup time from about 1 hour to 2 hours. Not sure if compression is a good idea in my case.

Now I have installed a 120Gb USB HDD on my RPi, and plan to leave that connected full-time. Eventually I’d like to shift the OS onto that to improve performance & capacity (but that’s a whole other thread!), so in the mean time, this will be for backing up the OS.

Personally I’m aiming for a schedule as follows:

A full image backup on a Sunday night.
Incremental backups on subsequent nights through the week.


So, lets kick off this discussion and try to come up with a comprehensive guide to backups.

Jock in a Frock
Posts: 72
Joined: Thu Jan 12, 2012 6:39 pm

Re: The Raspberry Pi Backup Thread

Sun Jul 29, 2012 5:59 am

Progress so far - I have a script to automate a full backup to the USB HDD (mounted at /mnt/usb/ on my system):

Code: Select all

#!/bin/bash
# Backup OS to the USB Hard Disk Drive

# Create a filename with datestamp for our current backup (without .img suffix)
ofile="/mnt/usb/backup_$(date +%d-%b-%y_%T)"

# Create final filename, with suffix
ofilefinal=$ofile.img

# Begin the backup process, should take about 1 hour from 8Gb SD card to HDD
sudo dd if="/dev/mmcblk0" of=$ofile bs=1M

# Collect result of backup procedure
result=$?

# If command has completed successfully, delete previous backups and exit
if [ $result=0 ]; then rm -f /mnt/usb/backup_*.img; mv $ofile $ofilefinal; exit 0;fi

#If command has failed, then delete partial backup file
if [ $result=1 ]; then rm -f $ofile; exit 1;fi
This will run a backup to the HDD, and if successful, will append a .img suffix, then delete the previous images

If the backup fails (HDD is full for example) the script will not delete previous backups, but will delete the attempted backup.

I have this automated to run at 4am every morning with the following line in my /etc/crontab file:

Code: Select all

01 4    * * *   root    /home/jock/scripts/backup_usbhdd.sh
This is a first draft, but I think it should work well, and I've tried to make it pretty self explanatory. So now it's over to the community to comment, advise, ridicule etc. :)

khh
Posts: 49
Joined: Thu Jul 26, 2012 12:16 am

Re: The Raspberry Pi Backup Thread

Sun Jul 29, 2012 8:46 am

After a quick glance at the dd documentation, I can't see it guaranteed it'll fail with an exit code of 1. I'd change the last to

Code: Select all

# If command has completed successfully, delete previous backups and exit
if [ $result=0 ]; then 
	rm -f /mnt/usb/backup_*.img;
	mv $ofile $ofilefinal;
	exit 0;
# Else remove attempted backup file
else 
	rm -f $ofile;
	exit 1;
fi
Also, the backup directory should be configurable.

Jock in a Frock
Posts: 72
Joined: Thu Jan 12, 2012 6:39 pm

Re: The Raspberry Pi Backup Thread

Sun Jul 29, 2012 9:50 am

Thanks, khh, you spotted my deliberate mistake. ;)

I have amended the script, and also added a few echo lines to indicate success / failure etc.

Code: Select all

#!/bin/bash
# Backup OS to the USB Hard Disk Drive

# Create a filename with datestamp for our current backup (without .img suffix)
ofile="/mnt/usb/backup_$(date +%d-%b-%y_%T)"

# Create final filename, with suffix
ofilefinal=$ofile.img

# Begin the backup process, should take about 1 hour from 8Gb SD card to HDD
echo "Backing up SD card to USB HDD as "$ofilefinal
echo "This will take around 1 hour. Please wait..."
sudo dd if="/dev/mmcblk0" of=$ofile bs=1M

# Collect result of backup procedure
result=$?

# If command has completed successfully, delete previous backups and exit
if [ $result = 0 ]; then
   echo "Successful backup, previous backup files will be deleted."
   rm -f /mnt/usb/backup_*.img
   mv $ofile $ofilefinal
   exit 0
# Else remove attempted backup file
else
   echo "Backup failed! Previous backup files untouched."
   echo "Please check there is sufficient space on the HDD."
   rm -f $ofile
   exit 1
fi
I have tested this and proved the fail condition with sudo kill -USR <pid>. That kills the dd command with return code 143, and triggers the else section to delete the incomplete backup.

As for selecting a backup directory, that will be in Version 2.

User avatar
AndrewS
Posts: 3625
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
Contact: Website

Re: The Raspberry Pi Backup Thread

Sun Jul 29, 2012 11:04 am

Probably a good idea to put a

Code: Select all

sync; sync
before the dd command, to make sure that any pending-writes actually get flushed to disk before dd starts reading the disk.

And theoretically some other background process could still be writing to disk during the dd process, which would give you a corrupt backup image (since dd operates at a level 'below' the filesystem). So if you've got any 'servers' running on your Pi, it'd probably be a good idea to stop them at the start of your script and re-enable them at the end of your script.

Holy One
Posts: 29
Joined: Mon Jul 02, 2012 5:44 pm
Contact: Website

Re: The Raspberry Pi Backup Thread

Sun Jul 29, 2012 1:08 pm

yeah, I thought you had to unmount all paritions before you tried to make the image because active processes might still try and write data during the process? will it not screw up the image if you dont?

User avatar
AndrewS
Posts: 3625
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
Contact: Website

Re: The Raspberry Pi Backup Thread

Sun Jul 29, 2012 1:13 pm

Holy One wrote:will it not screw up the image if you dont?
Depends if you're lucky or not ;-)
Has Jock tested any of his backup images yet :?:

Jock in a Frock
Posts: 72
Joined: Thu Jan 12, 2012 6:39 pm

Re: The Raspberry Pi Backup Thread

Mon Jul 30, 2012 8:11 am

AndrewS wrote:
Holy One wrote:Has Jock tested any of his backup images yet :?:
Yes, I have tested two backup images, without any problems.

Currently I'm not running any servers, but I will be using Apache or nginx, with MySQL. So I'm conscious of the limitations and problems of using dd for backing up. I hope to look at a more robust solution when I can. Any suggestions?

Will it always be necessary to halt all services? Anyone know a backup package which is designed for backing up production servers?

User avatar
AndrewS
Posts: 3625
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
Contact: Website

Re: The Raspberry Pi Backup Thread

Mon Jul 30, 2012 6:25 pm

Jock in a Frock wrote:Currently I'm not running any servers, but I will be using Apache or nginx, with MySQL. So I'm conscious of the limitations and problems of using dd for backing up. I hope to look at a more robust solution when I can. Any suggestions?
The most robust solution will obviously be when your filesystem is mounted read only e.g. you could use http://www.raspberrypi.org/phpBB3/viewt ... 29&t=10543

I guess if you wanted to do this automatically on the RPi, one clever way to do it would be to set up a second (minimal) root partition on the same SD card, and then have a cron-job on your main rootfs which switched settings in /boot/cmdline.txt to boot the minimal rootfs and rebooted, then when your minimal rootfs boots up just have it automatically backup the main rootfs using partclone (or dd if you really want) and when the backup has finished, twiddle cmdline.txt again and reboot back into your main rootfs :) :ugeek:
Will it always be necessary to halt all services? Anyone know a backup package which is designed for backing up production servers?
More than you can shake a stick at ;) https://www.google.co.uk/search?q=linux+backup+software

The problem specifically with dd is that it works at such a low level beneath the filesystem (which makes it simple & super-reliable when running from a read-only partition!) that it's easy for things to go wrong if the raw-partition is being read while the filesystem is being written to. But other backup tools (I have no recommendations for system backups, I just tend to use rsync to backup my homedir) do work at the filesystem layer, so there's less chance of things going wrong and the backup-software can use more "intelligence" (incremental backups, etc.).

Jock in a Frock
Posts: 72
Joined: Thu Jan 12, 2012 6:39 pm

Re: The Raspberry Pi Backup Thread

Mon Jul 30, 2012 6:55 pm

Thanks AndrewS, some good info there. I'm intrigued by the "reboot / backup / reboot" method, since I expect I could script that fairly easily. I think I'll try that next.

I do understand the difference between filesystem backups and the way dd does it (which is more of a disk cloning function). I also want to look at incremental backups, primarily due to the ability to save time. Do you know of any packages that offer that? If not I'll keep scouring. Thanks for the link though. :D

Jaac
Posts: 1
Joined: Thu Jan 26, 2012 1:23 pm
Contact: Website

Re: The Raspberry Pi Backup Thread

Sat Sep 15, 2012 11:46 am

Hi there chaps, first i say thanks for the work done on the script! It's built-up pretty solid, but i just had to add a few things. :lol:
  • My script runs as root, so i removed sudo before several commands.
  • Added a progress indicator using the 'pv' package, and made sure it is installed before running the command
  • Added a check to see if backup directory exists, if not, create it.
  • Shutdown several services before creating the backup and start them again after it has finished
  • Added tarring of the image file after dd is done (not directly on the dd process)
  • Changed the DATE stamp on the img file, so that the filename doesn't break the tar command.
  • Changed and added some more echo's to the console.

Code: Select all

#!/bin/bash

# Setting up directories
SUBDIR=raspberrypi_backups
DIR=/hdd/$SUBDIR

echo "Starting RaspberryPI backup process!"

# First check if pv package is installed, if not, install it first
PACKAGESTATUS=`dpkg -s pv | grep Status`;

if [[ $PACKAGESTATUS == S* ]]
   then
      echo "Package 'pv' is installed."
   else
      echo "Package 'pv' is NOT installed."
      echo "Installing package 'pv'. Please wait..."
      apt-get -y install pv
fi

# Check if backup directory exists
if [ ! -d "$DIR" ];
   then
      echo "Backup directory $DIR doesn't exist, creating it now!"
      mkdir $DIR
fi

# Create a filename with datestamp for our current backup (without .img suffix)
OFILE="$DIR/backup_$(date +%Y%m%d_%H%M%S)"

# Create final filename, with suffix
OFILEFINAL=$OFILE.img

# First sync disks
sync; sync

# Shut down some services before starting backup process
echo "Stopping some services before backup."
service apache2 stop
service mysql stop
service cron stop

# Begin the backup process, should take about 1 hour from 8Gb SD card to HDD
echo "Backing up SD card to USB HDD."
echo "This will take some time depending on your SD card size and read performance. Please wait..."
SDSIZE=`blockdev --getsize64 /dev/mmcblk0`;
pv -tpreb /dev/mmcblk0 -s $SDSIZE | dd of=$OFILE bs=1M conv=sync,noerror iflag=fullblock

# Wait for DD to finish and catch result
RESULT=$?

# Start services again that where shutdown before backup process
echo "Start the stopped services again."
service apache2 start
service mysql start
service cron start

# If command has completed successfully, delete previous backups and exit
if [ $RESULT = 0 ];
   then
      echo "Successful backup, previous backup files will be deleted."
      rm -f $DIR/backup_*.tar.gz
      mv $OFILE $OFILEFINAL
      echo "Backup is being tarred. Please wait..."
      tar zcf $OFILEFINAL.tar.gz $OFILEFINAL
      rm -rf $OFILEFINAL
      echo "RaspberryPI backup process completed! FILE: $OFILEFINAL.tar.gz"
      exit 0
# Else remove attempted backup file
   else
      echo "Backup failed! Previous backup files untouched."
      echo "Please check there is sufficient space on the HDD."
      rm -f $OFILE
      echo "RaspberryPI backup process failed!"
      exit 1
fi
Maybe someone else finds a use for the changes i made, i'm aware of the problems that come with creating backups with dd while the filesystem is active.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Wed Oct 24, 2012 2:01 pm

Very interesting topic.
I'm newbie to linux world but the reboot-backup-reboot looks a very "professional" solution.
I'm subscribing to this topic to be aware of future updates.
Tar the img file is a nice feature.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Wed Oct 24, 2012 3:42 pm

Regarding the last script posted could be interesting to retrieve all the services running, stop all of them and after restart all the services.
Why not?

gmc
Posts: 121
Joined: Fri Mar 09, 2012 11:31 am
Location: Cheshire, UK
Contact: Website

Re: The Raspberry Pi Backup Thread

Fri Feb 15, 2013 5:17 pm

I had to mod the script and add in sudo commands at various places to get it working.

TravelinMax
Posts: 25
Joined: Mon Nov 26, 2012 3:46 am
Location: Michigan, USA

Re: The Raspberry Pi Backup Thread

Sat Feb 16, 2013 5:54 am

After editing this script several times I found out I do not need to add sudo anywhere, but I have to run the script with sudo. ie

Code: Select all

$ sudo ./backup.sh
I added the script to chrontab as demonstrated above only I chose to run at 4am sunday mornings. I will report back after then if it ran correctly. I also chose to disable ssh and networking and disabled tarring the image. I added appropriate warnings 2 minutes before services shutdown (seeing as I almost exclusively access the Pi via ssh).

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Mon Feb 18, 2013 11:16 am

So the Jaac script version is the "last" one or someone added any improvements?
I'm going to read infos about "live backup" and apply them in the week end.

dr_d_gee
Posts: 84
Joined: Fri Jan 04, 2013 1:30 pm

Re: The Raspberry Pi Backup Thread

Mon Feb 18, 2013 1:33 pm

Just some suggestions:

It might be worth backing up each of the partitions separately--the FAT one is the more likely to be corrupted if power is lost.

Also, why not use a more conventional tool to backup files to the hard drive? Something like rsync or the various graphical interfaces like grsync or Back In Time? You'd still need a backup of the entire image but it would only be needed if the card was completely corrupted. Then you could restore the card from the image and then restore the latest backup of the files?

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Mon Feb 18, 2013 3:47 pm

I'm looking for something that can provide me a complete SD image: I don't want to make a backup manually each time I do an edit on the rasp system. I'm looking for something that can clone the SD storing the image on the USB hdd each night, so I have a "last" working image automatically.

1HzCoder
Posts: 31
Joined: Thu Jul 12, 2012 1:15 am
Location: Lower Alabama

Re: The Raspberry Pi Backup Thread

Mon Feb 18, 2013 7:44 pm

It's already been done, check out this thread
http://www.raspberrypi.org/phpBB3/viewt ... 29&t=21342
Works great and if you mess up your present SD card, just swap and re-rerun.
Just did it last night as a matter of fact.
As far as automatic, you might ask the author which lines to change to avoid the "OK to proceed" prompt.
Einstein once said you don't really understand anything until you can explain it to your Grandmother

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Tue Feb 19, 2013 10:05 am

Great, I'm going to check your suggestion!
But this thread is not related to this topic?

TravelinMax
Posts: 25
Joined: Mon Nov 26, 2012 3:46 am
Location: Michigan, USA

Re: The Raspberry Pi Backup Thread

Thu Feb 21, 2013 5:22 am

So I ran my version of the script manually and it worked just fine. Going through my backups now it looks like the auto backup on Sunday night did not run correctly and my entire drive is wiped... except the backup folder. sldkfjsdlkjfslkjflksdjflsdkjflsdfjlsdkjflsdjflsdkjlkfj

I will not be posting my updated version of the code.

paddy10tellys
Posts: 2
Joined: Mon Mar 11, 2013 10:08 pm

Re: The Raspberry Pi Backup Thread

Fri Mar 22, 2013 9:52 pm

I have come to the conclusion that controlling a loop with dpkg, pv and grep does not work - Jaac's implementation always does 'else' rather than 'then' and using S* means that the condition that controls the loop will be true just if $PACKAGESTATUS starts with 'S' - sorry, Jaac... Me too though, wasted hours++ on it...

This doesn't work!!!

Code: Select all

# First check if pv package is installed, if not, install it first
PACKAGESTATUS=`dpkg -s pv | grep Status`;

if [[ $PACKAGESTATUS == S* ]]
   then
      echo "Package 'pv' is installed."
   else
      echo "Package 'pv' is NOT installed."
      echo "Installing package 'pv'. Please wait..."
      apt-get -y install pv
fi
nb - this does though ... Instead of capturing the output of the dpkg | grep pipeline, just check its exit status:

Code: Select all

if dpkg -s pv | grep -q Status; then
   then
      echo "Package 'pv' is installed."
   else
      echo "Package 'pv' is NOT installed."
      echo "Installing package 'pv'. Please wait..."
      apt-get -y install pv
fi
The -q option to grep is used to suppress the output of any matched lines, since you don't need to see them.

Many thanks to, "Chepner" who helped me out with this - see http://stackoverflow.com/questions/1556 ... 5#15570745

flyinghappy
Posts: 115
Joined: Mon Nov 05, 2012 5:31 pm

Re: The Raspberry Pi Backup Thread

Fri Sep 06, 2013 10:21 pm

Really like this script. I think I'm going to go ahead and through it on my Pi this weekend. The only change I'm going to make is to use xz for compression instead of gz. It should compress it a lot further so it doesn't take up so much space on my external HDD (really need something larger for my movies than my old 500GB!)
Pi with ArchLinux running a minidlna/samba home server

tiritchi
Posts: 1
Joined: Wed Oct 02, 2013 5:03 pm

Re: The Raspberry Pi Backup Thread

Wed Oct 02, 2013 5:07 pm

Hi everyone, I really enjoy this topic but one question still in my mind :

what is the procedure if we want to restore the backup in .img.tar.gz done by the script ?

I put the script in /etc/cron.weekly/ and modified the destination of backup directory but if my system crash I really don't know how to restore it :/

thanks a lot nevertheless for this good job.

CumpsD
Posts: 1
Joined: Sat Oct 26, 2013 7:23 am

Re: The Raspberry Pi Backup Thread

Sat Oct 26, 2013 7:24 am

Maybe a stupid question, but how to restore the final image made by the script?

Return to “General discussion”

Who is online

Users browsing this forum: rpiMike and 66 guests