Jamesking56
Posts: 13
Joined: Sat Oct 26, 2013 10:37 am
Location: Redditch, UK
Contact: Website

Re: The Raspberry Pi Backup Thread

Mon Oct 28, 2013 4:54 pm

Yeah I would like to know how we restore it, do we have to connect the SD card into another computer to restore it?

xXAzazelXx
Posts: 130
Joined: Tue Sep 18, 2012 8:32 am
Location: Australia
Contact: Website

Re: The Raspberry Pi Backup Thread

Mon Oct 28, 2013 8:19 pm

CumpsD wrote:Maybe a stupid question, but how to restore the final image made by the script?
Unzip it and use win32diskimager

michaelkrtikos
Posts: 43
Joined: Mon Dec 16, 2013 9:48 pm

Re: The Raspberry Pi Backup Thread

Fri Dec 27, 2013 11:48 am

Hello Everyone,

This might be an old post but i would like to ask a question:

I am using the following script:

Code: Select all

#!/bin/bash
# Backup OS to the USB Hard Disk Drive

# Create a filename with datestamp for our current backup (without .img suffix)
ofile="/media/USBHDD2/PIBackup/backup_$(date +%d-%b-%y_%T).img"

# Create final filename, with suffix
ofilefinal=$ofile.img

# Begin the backup process, should take about 3 hours from 16Gb SD card to HDD
echo "Backing up SD card to USB HDD as "$ofilefinal
echo "This will take around 1 hour. Please wait..."
sudo dd if="/dev/mmcblk0" of=$ofile bs=10M

# Collect result of backup procedure
result=$?

# If command has completed successfully, delete previous backups and exit
if [ $result = 0 ]; then
   echo "Successful backup, previous backup files will be deleted."
   rm -f /media/USBHDD2/PIBackup/backup_*.img
   mv $ofile $ofilefinal
   exit 0
# Else remove attempted backup file
else
   echo "Backup failed! Previous backup files untouched."
   echo "Please check there is sufficient space on the HDD."
   rm -f $ofile
   exit 1
fi
I have edited my crontab file to start automatically and everything seems to work fine. There is only one thing. The backup process is starting, it takes about 2 hours to complete (meanwhile i am checking the size of the file to know that it still works) and when it finishes it probably cannot catch the results and it fails deleting the file.

The size before failure matches the SD card so i believe that the file is correct. For some reason it returns failed results and it triggers the deletion.

Could you please have a look on the above script and let me know what can be wrong?
http://www.theraspberrypiblog.com

michaelkrtikos
Posts: 43
Joined: Mon Dec 16, 2013 9:48 pm

Re: The Raspberry Pi Backup Thread

Fri Dec 27, 2013 5:08 pm

michaelkrtikos wrote:Hello Everyone,

This might be an old post but i would like to ask a question:

I am using the following script:

Code: Select all

#!/bin/bash
# Backup OS to the USB Hard Disk Drive

# Create a filename with datestamp for our current backup (without .img suffix)
ofile="/media/USBHDD2/PIBackup/backup_$(date +%d-%b-%y_%T).img"

# Create final filename, with suffix
ofilefinal=$ofile.img

# Begin the backup process, should take about 3 hours from 16Gb SD card to HDD
echo "Backing up SD card to USB HDD as "$ofilefinal
echo "This will take around 1 hour. Please wait..."
sudo dd if="/dev/mmcblk0" of=$ofile bs=10M

# Collect result of backup procedure
result=$?

# If command has completed successfully, delete previous backups and exit
if [ $result = 0 ]; then
   echo "Successful backup, previous backup files will be deleted."
   rm -f /media/USBHDD2/PIBackup/backup_*.img
   mv $ofile $ofilefinal
   exit 0
# Else remove attempted backup file
else
   echo "Backup failed! Previous backup files untouched."
   echo "Please check there is sufficient space on the HDD."
   rm -f $ofile
   exit 1
fi
I have edited my crontab file to start automatically and everything seems to work fine. There is only one thing. The backup process is starting, it takes about 2 hours to complete (meanwhile i am checking the size of the file to know that it still works) and when it finishes it probably cannot catch the results and it fails deleting the file.

The size before failure matches the SD card so i believe that the file is correct. For some reason it returns failed results and it triggers the deletion.

Here is what is the reason code in terminal:

Code: Select all

[email protected]:/home/pi/scripts# sh dailybackup.sh
Backing up SD card to USB HDD as /media/USBHDD2/PIBackup/backup_27-Dec-13_16:11:09.img
This will take around 2-3 hours to complete. Please wait...
1536+0 records in
1536+0 records out
16106127360 bytes (16 GB) copied, 5935.03 s, 2.7 MB/s
dailybackup.sh: 19: [: =: unexpected operator
Backup failed! Previous backup files untouched.
Please check there is sufficient space on the HDD.
Could you please have a look on the above script and let me know what can be wrong?
http://www.theraspberrypiblog.com

User avatar
DougieLawson
Posts: 34166
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: The Raspberry Pi Backup Thread

Fri Dec 27, 2013 10:23 pm

I'd probably simplify

Code: Select all

# Collect result of backup procedure
result=$?

# If command has completed successfully, delete previous backups and exit
if [ $result = 0 ]; then
To

Code: Select all

# If command has completed successfully, delete previous backups and exit
if [ $? -eq 0 ]; then
You don't appear to need that intermediate variable anywhere further on in the script.
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

2012-18: 1B*5, 2B*2, B+, A+, Z, ZW, 3Bs*3, 3B+

Any DMs sent on Twitter will be answered next month.

User avatar
Richard-TX
Posts: 1549
Joined: Tue May 28, 2013 3:24 pm
Location: North Texas

Re: The Raspberry Pi Backup Thread

Fri Dec 27, 2013 11:25 pm

I happen to like rsync. It copies the files so that I can restore individual files as needed.

On the plus side, incrementals only updates those files that have changed.

Creating a tar archive would work too but tar ignores special files like devices.

The program cpio does save device files.
Richard
Doing Unix since 1985.
The 9-25-2013 image of Wheezy can be found at:
http://downloads.raspberrypi.org/raspbian/images/raspbian-2013-09-27/2013-09-25-wheezy-raspbian.zip

edmun
Posts: 39
Joined: Thu Jan 09, 2014 11:37 am

Re: The Raspberry Pi Backup Thread

Thu Jan 09, 2014 12:06 pm

I would like to say a really big thank you to everyone that helped me manage mine backup.
Just to not be a person that only downloads script and do nothing, I though that maybe I will share mine script with comments.
Of course mine code is only an alteration of michaelkrtikos & Jaac works which I thanks the most, but maybe someone will find this alteration useful.

Soon I will also try to add some progress because I will need to do some more things with it:

Things changed (see next section for more details) :
* added sudo for commands so normal user does not need to run it as root
* added colors and breaklines so when script will be manually started, it's much easier to read
* removed compressing the *.img backup file to archive
* removed the deleting of all backup files
* added deleting backup *.img files that are more than 5 days old
* added installation of pv dialog for future purpose
* if backup fails, will print list of the last backups stored on HDD
* added creating a log file in cron


Things to add (currently under development):
* Trying to change progress bar to be more graphical one using 'pv dialog'
* Checking if tar can compress img file faster. Backup script creates img file in around 45 minutes from 8GB SDHC Card to HDD but compressing takes around 3-4 hours depending on RaspberryPI load. That's why I've disabled compressing img file in mine script. Also I don't really see a reason of compressing the img file. When there will be any issue with RaspberryPI, there will be a need of decompress the img file to use it with win32disk imager which will slow down the process of bringing RaspberryPI back to life. Still thinking about it, unless I will find how to compress and decompress it much faster
* I will try to add some notification by e-mail (cron by default does it) but need to learn some more about postfix

Code: Select all


#!/bin/bash

# Setting up backup directories
SUBDIR=RaspberryPi_backups
DIR=/media/4-Seagate/$SUBDIR

# Setting up echo fonts
red='\e[0;31m'
green='\e[0;32m'
cyan='\e[0;36m'
yellow='\e[1;33m'
purple='\e[0;35m'
NC='\e[0m' #No Color
bold=`tput bold`
normal=`tput sgr0`

#Screen clear
clear

echo -e "${green}${bold}Starting RaspberryPI backup process!${NC}${normal}"
echo ""
# First check if pv package is installed, if not, install it first
PACKAGESTATUS=`dpkg -s pv | grep Status`;

if [[ $PACKAGESTATUS == S* ]]
   then
      echo -e "${cyan}${bold}Package 'pv' is installed${NC}${normal}"
      echo ""
   else
      echo -e "${yellow}${bold}Package 'pv' is NOT installed${NC}${normal}"
      echo -e "${yellow}${bold}Installing package 'pv' + 'pv dialog'. Please wait...${NC}${normal}"
      echo ""
      sudo apt-get -y install pv && sudo apt-get -y install pv dialog
fi



# Check if backup directory exists
if [ ! -d "$DIR" ];
   then
      echo -e "${yellow}${bold}Backup directory $DIR doesn't exist, creating it now!${NC}${normal}"
      sudo mkdir $DIR

fi

# Create a filename with datestamp for our current backup (without .img suffix)
OFILE="$DIR/backup_$(date +%Y%m%d_%H%M%S)"

# Create final filename, with suffix
OFILEFINAL=$OFILE.img

# First sync disks
sync; sync

# Shut down some services before starting backup process
echo ""
echo -e "${purple}${bold}Stopping services before backup${NC}${normal}"
sudo pkill deluged
sudo pkill deluge-web
sudo service deluge-daemon stop
sudo service noip stop
sudo service cron stop
sudo service proftpd stop
sudo service webmin stop
sudo service xrdp stop


# Begin the backup process, should take about 45 minutes hour from 8Gb SD card to HDD
echo ""
echo -e "${green}${bold}Backing up SD card to img file on HDD${NC}${normal}"
SDSIZE=`sudo blockdev --getsize64 /dev/mmcblk0`;
sudo pv -tpreb /dev/mmcblk0 -s $SDSIZE | dd of=$OFILE bs=1M conv=sync,noerror iflag=fullblock

# Wait for DD to finish and catch result
RESULT=$?

# Start services again that where shutdown before backup process
echo ""
echo -e "${purple}${bold}Starting the stopped services${NC}${normal}"
sudo service deluge-daemon start
sudo deluged
sudo deluge-web
sudo service noip start
sudo service cron start
sudo service proftpd start
sudo service webmin start
sudo service xrdp start


# If command has completed successfully, if not, delete created files
if [ $RESULT = 0 ];
   then
     sudo mv $OFILE $OFILEFINAL
      echo ""
      echo -e "${green}${bold}RaspberryPI backup process completed! FILE: $OFILEFINAL${NC}${normal}"
      echo -e "${yellow}Removing backups older than 5 days${NC}"
     sudo find $DIR -maxdepth 1 -name "*.img" -mtime +5 -exec rm {} \;
      echo -e "${cyan}If any backups older than 5 days were found, they were deleted${NC}"
      exit 0
# Else remove attempted backup file
   else
      echo ""
      echo -e "${red}${bold}Backup failed!${NC}${normal}"
     sudo rm -f $OFILE
      echo ""
      echo -e "${purple}Last backups on HDD:${NC}"
     sudo find $DIR -maxdepth 1 -name "*.img" -exec ls {} \;
      echo ""
      echo -e "${red}${bold}RaspberryPI backup process failed!${NC}${normal}"
      exit 1
fi

Of course keep in mind to change the paths and services in this script.

Things that you might want to change before use:
Line 4: Backups directory name
Line 5: Backups location on HDD
Line 58 to 65: Check which services and tasks you want to disable
Line 71: Check if yours SD card is mounted with the same name
Line 72: Check if yours SD card is mounted with the same name
Line 80 to 87: Check which services are needed to be enabled after backup will be finished

also for mine purposes I have set crontab scheduler to run backup task twice per week (Tuesday at Friday) at 4 am and save all progress into the log file in the same folder and make the script executable

Code: Select all

chmod +x /media/4-Seagate/TaskForBackup.sh

Code: Select all

crontab -e
And added couple of lines at the end of the crontab file:

Code: Select all

* 4 * * 2 /media/4-Seagate/TaskForBackup.sh > /media/4-Seagate/RaspberryPi_backups/cron.log 2>&1
* 4 * * 5 /media/4-Seagate/TaskForBackup.sh > /media/4-Seagate/RasbperryPi_backups/cron.log 2>&1

Feel free to comment and let me know if we would like to do something more with it
Last edited by edmun on Wed Feb 04, 2015 4:13 pm, edited 2 times in total.

xXAzazelXx
Posts: 130
Joined: Tue Sep 18, 2012 8:32 am
Location: Australia
Contact: Website

Re: The Raspberry Pi Backup Thread

Thu Jan 09, 2014 11:08 pm

On the topic of backup. I found this script to work amazing when you are just running Raspberry Pi of a SD card. But I've changed my setup to only boot from SD, and the rest is pulled from a USB stick. Would this still work in the same way? Say my setup gets corrupted tomorrow, where would I restore the image? The USB, the SD, both? Or would I need to make a separate image of each using something like dd?

SunboX
Posts: 1
Joined: Mon Jan 27, 2014 8:38 am

Re: The Raspberry Pi Backup Thread

Mon Jan 27, 2014 8:43 am

Really great script! Thanks @ all!

I've added auto update after backup, this is what it looks like:

Code: Select all

# ...

    # Shut down some services before starting backup process
    echo ""
    echo -e "${purple}${bold}Stopping services before backup${NC}${normal}"
    sudo service ddclient stop
    sudo service cron stop
    sudo service apache2 stop
    sudo service samba stop
    sudo service avahi-daemon stop
    sudo service netatalk stop
    sudo service sendmail stop
    sudo /var/ossec/bin/ossec-control stop
    sudo service ssh stop

# ...

# If command has completed successfully, if not, delete created files
    if [ $RESULT = 0 ];
       then
         sudo mv $OFILE $OFILEFINAL
         echo ""
         echo -e "${green}${bold}RaspberryPI backup process completed! FILE: $OFILEFINAL${NC}${normal}"
         echo -e "${yellow}Removing backups older than 30 days${NC}"
         sudo find $DIR -maxdepth 1 -name "*.img" -mtime +30 -exec rm {} \;
         echo -e "${cyan}If any backups older than 30 days were found, they were deleted${NC}"

         # Update Packages and Kernel
         echo -e "${yellow}Update Packages and Kernel${NC}${normal}"
         sudo apt-get update
         sudo apt-get upgrade -y
         sudo apt-get autoclean

         # Update Raspberry Pi Firmware
         echo -e "${yellow}Update Raspberry Pi Firmware${NC}${normal}"
         sudo rpi-update
         sudo ldconfig

         # Reboot now
         echo -e "${yellow}Reboot now ...${NC}${normal}"
         sudo reboot

    # Else remove attempted backup file
       else
         echo ""
         echo -e "${red}${bold}Backup failed!${NC}${normal}"
         sudo rm -f $OFILE
         echo ""
         echo -e "${purple}Last backups on HDD:${NC}"
         sudo find $DIR -maxdepth 1 -name "*.img" -exec ls {} \;
         echo ""
         echo -e "${red}${bold}RaspberryPI backup process failed!${NC}${normal}"

         # Update Packages and Kernel
         echo -e "${yellow}Update Packages and Kernel${NC}${normal}"
         sudo apt-get update
         sudo apt-get upgrade -y
         sudo apt-get autoclean

         # Update Raspberry Pi Firmware
         echo -e "${yellow}Update Raspberry Pi Firmware${NC}${normal}"
         sudo rpi-update
         sudo ldconfig

          # Reboot now
         echo -e "${yellow}Reboot now ...${NC}${normal}"
         sudo reboot

    fi

primeform
Posts: 11
Joined: Wed Jan 29, 2014 6:46 am

Re: The Raspberry Pi Backup Thread

Sat Feb 15, 2014 10:26 am

xXAzazelXx wrote:On the topic of backup. I found this script to work amazing when you are just running Raspberry Pi of a SD card. But I've changed my setup to only boot from SD, and the rest is pulled from a USB stick. Would this still work in the same way? Say my setup gets corrupted tomorrow, where would I restore the image? The USB, the SD, both? Or would I need to make a separate image of each using something like dd?
Can someone make this happen? I have my system running its filesystem on my flash disk /sda1. Could someone modify the script to pull from there instead of the SD card? The boot partition is still on the SD card obviously.

edmun
Posts: 39
Joined: Thu Jan 09, 2014 11:37 am

Re: The Raspberry Pi Backup Thread

Sat Feb 15, 2014 5:53 pm

primeform are you serious with this question? why can't you just change values in code? you can even schedule two bash files, and one of them will be creating a backup of boot and the second will create a backup of flash disk. is it really so hard to check what are the partition names or maybe I don't get it and I am now so completely confused ?! ... just to help you a little, try with "sudo fdisk -l"

xXAzazelXx
Posts: 130
Joined: Tue Sep 18, 2012 8:32 am
Location: Australia
Contact: Website

Re: The Raspberry Pi Backup Thread

Sun Feb 16, 2014 7:29 am

edmun wrote:primeform are you serious with this question? why can't you just change values in code? you can even schedule two bash files, and one of them will be creating a backup of boot and the second will create a backup of flash disk. is it really so hard to check what are the partition names or maybe I don't get it and I am now so completely confused ?! ... just to help you a little, try with "sudo fdisk -l"
way to be a dick about.
But he is right. I've made two files. Went to fstab and changed the the location from /dev/mmcblk0 to /dev/sda1 and /dev/mmcblk0p1.

/dev/sda1 / ext4 defaults,noatime 0
/dev/mmcblk0p1 /boot vfat defaults 0 2

edmun
Posts: 39
Joined: Thu Jan 09, 2014 11:37 am

Re: The Raspberry Pi Backup Thread

Sun Feb 16, 2014 10:05 am

I wasn't trying to be rude. not yet.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Sun Feb 16, 2014 10:14 am

Hi, I was away from this topic for ages but now I'm coming back.
I have my rasp running from sd with an external hdd plugged.
Is edmun's script fine to backup "live/online" raspberry?

edmun
Posts: 39
Joined: Thu Jan 09, 2014 11:37 am

Re: The Raspberry Pi Backup Thread

Thu Sep 04, 2014 3:55 pm

I am not happy that this thread just died :(
I was hoping someone will take it over and expand this code even more :(

jinx
Posts: 10
Joined: Wed Nov 05, 2014 11:44 am

Re: The Raspberry Pi Backup Thread

Wed Nov 05, 2014 12:02 pm

Here's an alternative method of backing up. Instead of doing a binary copy of the complete SD Card, including unused sectors, this method creates an empty image replica of the SD Card, mounts this as a "filesystem in a file" and uses rsync to dynamically mirror the contents of the SD Card to the image.

Code: Select all

#!/bin/bash

DIR=/media/USBHDD1/backup/images
NAME=$(uname -n)-$(date +%Y-%W)
IMAGE=$DIR/${NAME}.img
LOG=$DIR/${NAME}-$(date +%Y%m%d%H%M).log

SDCARD=/dev/mmcblk0

LOOPBACK=/dev/loop0
LOOPBACK1=${LOOPBACK}p1
LOOPBACK2=${LOOPBACK}p2

MOUNTDIR=/mnt/$NAME

if [ $(id -u) -ne 0 ]
then
    echo "Please run as root. Try sudo."
    exit 1
fi

trace () {
    echo -e "$(tput setaf 3)${1}$(tput sgr 0)"
}

trace "Starting image backup process"

if [ -s $IMAGE ]
then
    trace "Attaching $IMAGE to $LOOPBACK"
    losetup $LOOPBACK $IMAGE
    partx --add $LOOPBACK
else
    trace "Creating new $IMAGE, the size of $SDCARD"
    dd if=/dev/zero of=$IMAGE bs=$(blockdev --getss $SDCARD) count=0 seek=$(blockdev --getsz $SDCARD)

    trace "Attaching $IMAGE to $LOOPBACK"
    losetup $LOOPBACK $IMAGE

    trace "Copying partition table from $SDCARD to $LOOPBACK"
    parted -s $LOOPBACK mklabel msdos
    sfdisk --dump $SDCARD | sfdisk --force $LOOPBACK

    trace "Formatting partitions"
    partx --add $LOOPBACK
    mkfs.vfat -I $LOOPBACK1
    mkfs.ext4 $LOOPBACK2
fi

trace "Mounting $LOOPBACK1 and $LOOPBACK2 to $MOUNTDIR"
mkdir $MOUNTDIR
mount $LOOPBACK2 $MOUNTDIR
mkdir -p $MOUNTDIR/boot
mount $LOOPBACK1 $MOUNTDIR/boot

trace "Backing up / and /boot/ to $MOUNTDIR"
rsync -aEvx --del --stats --log-file $LOG /boot/ $MOUNTDIR/boot/
rsync -aEvx --del --stats --log-file $LOG / $MOUNTDIR/

trace "Flushing to disk"
sync; sync

trace "Unmounting $LOOPBACK1 and $LOOPBACK2 from $MOUNTDIR"
umount $MOUNTDIR/boot
umount $MOUNTDIR
rmdir $MOUNTDIR

trace "Detaching $IMAGE from $LOOPBACK"
partx --delete $LOOPBACK
losetup -d $LOOPBACK

trace "Image backup process completed."
trace "See log in $LOG"
This method should be much quicker than performing a binary replica but with the same end result and the added benefit of being able to create incremental backups in a matter of seconds.

The current configuration creates a new image file every week, but by modifying $NAME, a new image can be created every day "$(uname -n)-$(date +%Y-%m-%d)", every month "$(uname -n)-$(date +%Y-%m)", or you can create a static image which you can archive when you want "$(uname -n)".

Please modify $DIR to suit your setup (I have a 1 TB USB HDD attached to an external powered hub) and double check that $SDCARD is correct.

Update

There was an issue with the initial post, where the /boot/ partition was incorrectly copied into the root. This has now been corrected with two calls to rsync.
Last edited by jinx on Wed Nov 05, 2014 11:05 pm, edited 1 time in total.

edmun
Posts: 39
Joined: Thu Jan 09, 2014 11:37 am

Re: The Raspberry Pi Backup Thread

Wed Nov 05, 2014 6:42 pm

for sure I am going to test this one up.

jinx
Posts: 10
Joined: Wed Nov 05, 2014 11:44 am

Re: The Raspberry Pi Backup Thread

Wed Nov 05, 2014 11:13 pm

edmun wrote:for sure I am going to test this one up.
Please see my error correction above about /boot/ not being properly backed up before you run this, if it's not too late.
I'd been running the script with different parameters and did not spot it until after the post.

edmun
Posts: 39
Joined: Thu Jan 09, 2014 11:37 am

Re: The Raspberry Pi Backup Thread

Thu Nov 06, 2014 9:49 am

It's not too late. I haven't test it yet. Thanks for the update

kimba
Posts: 3
Joined: Fri Nov 07, 2014 2:05 pm

Re: The Raspberry Pi Backup Thread

Fri Nov 07, 2014 2:23 pm

jinx wrote:Here's an alternative method of backing up. Instead of doing a binary copy of the complete SD Card, including unused sectors, this method creates an empty image replica of the SD Card, mounts this as a "filesystem in a file" and uses rsync to dynamically mirror the contents of the SD Card to the image.

Code: Select all

#!/bin/bash

DIR=/media/USBHDD1/backup/images
NAME=$(uname -n)-$(date +%Y-%W)
IMAGE=$DIR/${NAME}.img
LOG=$DIR/${NAME}-$(date +%Y%m%d%H%M).log

SDCARD=/dev/mmcblk0

LOOPBACK=/dev/loop0
LOOPBACK1=${LOOPBACK}p1
LOOPBACK2=${LOOPBACK}p2

MOUNTDIR=/mnt/$NAME

if [ $(id -u) -ne 0 ]
then
    echo "Please run as root. Try sudo."
    exit 1
fi

trace () {
    echo -e "$(tput setaf 3)${1}$(tput sgr 0)"
}

trace "Starting image backup process"

if [ -s $IMAGE ]
then
    trace "Attaching $IMAGE to $LOOPBACK"
    losetup $LOOPBACK $IMAGE
    partx --add $LOOPBACK
else
    trace "Creating new $IMAGE, the size of $SDCARD"
    dd if=/dev/zero of=$IMAGE bs=$(blockdev --getss $SDCARD) count=0 seek=$(blockdev --getsz $SDCARD)

    trace "Attaching $IMAGE to $LOOPBACK"
    losetup $LOOPBACK $IMAGE

    trace "Copying partition table from $SDCARD to $LOOPBACK"
    parted -s $LOOPBACK mklabel msdos
    sfdisk --dump $SDCARD | sfdisk --force $LOOPBACK

    trace "Formatting partitions"
    partx --add $LOOPBACK
    mkfs.vfat -I $LOOPBACK1
    mkfs.ext4 $LOOPBACK2
fi

trace "Mounting $LOOPBACK1 and $LOOPBACK2 to $MOUNTDIR"
mkdir $MOUNTDIR
mount $LOOPBACK2 $MOUNTDIR
mkdir -p $MOUNTDIR/boot
mount $LOOPBACK1 $MOUNTDIR/boot

trace "Backing up / and /boot/ to $MOUNTDIR"
rsync -aEvx --del --stats --log-file $LOG /boot/ $MOUNTDIR/boot/
rsync -aEvx --del --stats --log-file $LOG / $MOUNTDIR/

trace "Flushing to disk"
sync; sync

trace "Unmounting $LOOPBACK1 and $LOOPBACK2 from $MOUNTDIR"
umount $MOUNTDIR/boot
umount $MOUNTDIR
rmdir $MOUNTDIR

trace "Detaching $IMAGE from $LOOPBACK"
partx --delete $LOOPBACK
losetup -d $LOOPBACK

trace "Image backup process completed."
trace "See log in $LOG"
This method should be much quicker than performing a binary replica but with the same end result and the added benefit of being able to create incremental backups in a matter of seconds.

The current configuration creates a new image file every week, but by modifying $NAME, a new image can be created every day "$(uname -n)-$(date +%Y-%m-%d)", every month "$(uname -n)-$(date +%Y-%m)", or you can create a static image which you can archive when you want "$(uname -n)".

Please modify $DIR to suit your setup (I have a 1 TB USB HDD attached to an external powered hub) and double check that $SDCARD is correct.

Update

There was an issue with the initial post, where the /boot/ partition was incorrectly copied into the root. This has now been corrected with two calls to rsync.
hi jinx, i am new to the world of raspberry pi, so pardon me if my question is silly. my sd card size is 16gb and a full backup will take too much time. i like your idea of using rsync to backup. however, instead of an external hard drive attached to the raspberrypi, i would like to back up directly to my NAS. i have added 2 lines to mount and unmount my nas and changed the DIR to point to the mount:

Code: Select all

mount -t cifs -o username=myusername,password=mypassword //mynas/raspberrybackup /home/pi/backups

DIR=/home/pi/backups
NAME=$(uname -n)-$(date +%Y-%W)
IMAGE=$DIR/${NAME}.img
LOG=$DIR/${NAME}-$(date +%Y%m%d%H%M).log
.
.
.
trace "Image backup process completed."
trace "See log in $LOG"

umount /home/pi/backups
i ran the script and it generated an image file on my NAS. but when the script ended, my entire storage on the raspberrypi was used up. i checked and found that the directory on /mnt/raspberrypi-2014-44 was taking up all the space. i have removed that directory since.

the end of my log file is as follows:

Code: Select all

*** Skipping any contents from this failed directory ***
rsync: connection unexpectedly closed (7602676 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
Flushing to disk
Unmounting /dev/loop0p1 and /dev/loop0p2 from /mnt/raspberrypi-2014-44
umount: /mnt/raspberrypi-2014-44: not mounted
rmdir: failed to remove `/mnt/raspberrypi-2014-44': Directory not empty
Detaching /home/pi/backups/images/raspberrypi-2014-44.img from /dev/loop0
partx: /dev/loop0: error deleting partition 4
Image backup process completed.
i am definitely missing something here. appreciate if you or anyone else is kind enough to shed some light on how to perform the backup to a network share.

jinx
Posts: 10
Joined: Wed Nov 05, 2014 11:44 am

Re: The Raspberry Pi Backup Thread

Fri Nov 07, 2014 3:43 pm

The mounting of the SD Image failed, causing you to write into the mount point instead of the image. This is probably caused by an incorrectly created image file by "dd"

First. Check that you copied the file correctly, with no unexpected line feeds. I had a similar issue, where the "dd" command was split up, and the "seek" part was never included, causing a zero sized file and a failed mount.

Second. If you run the commands manually, one by one, by copying them from the script, you will see where it fails.

I have never tried mounting "filesystem in a file" which resides over the network, so I don't know how this affects the script.

jinx
Posts: 10
Joined: Wed Nov 05, 2014 11:44 am

Re: The Raspberry Pi Backup Thread

Fri Nov 07, 2014 4:08 pm

Actually, a good idea would probably be to extend the script to make sure /mnt/$MOUNTDIR is a mount point before starting rsync, to avoid that your local filesystem gets flooded:

Code: Select all

if mountpoint -q $MOUNTDIR; then
    trace "Backing up / and /boot/ to $MOUNTDIR"
    rsync -aEvx --del --stats --log-file $LOG /boot/ $MOUNTDIR/boot/
    rsync -aEvx --del --stats --log-file $LOG / $MOUNTDIR/
else
    trace "Skipping rsync since $MOUNTDIR is not a mount point"
fi

jinx
Posts: 10
Joined: Wed Nov 05, 2014 11:44 am

Re: The Raspberry Pi Backup Thread

Mon Nov 10, 2014 12:05 pm

I have now attempted to create a utility script which can be used to perform a standard backup (using rsync) with more error handling than before.
I have also added the possibility to optionally compress the image after backup, as well as abilities to mount and unmount the image (and in theory any Raspberry Pi dist image) so that data can be viewed, restored or edited, and a Ctrl-C interrupt trap, which should allow you to stop lengthy processes as rsync or gzip while still trying to exit gracefully.

From "bkup_rpimage.sh --help":
bkup_rpimage.sh v1.0 by jinx

Usage:

bkup_rpimage.sh start [-clzdf] [-L logfile] [-i sdcard] sdimage
bkup_rpimage.sh mount [-c] sdimage [mountdir]
bkup_rpimage.sh umount sdimage [mountdir]
bkup_rpimage.sh gzip [-df] sdimage

Commands:

start starts complete backup of RPi's SD Card to 'sdimage'
mount mounts the 'sdimage' to 'mountdir' (default: /mnt/'sdimage'/)
umount unmounts the 'sdimage' from 'mountdir'
gzip compresses the 'sdimage' to 'sdimage'.gz

Options:

-c creates the SD Image if it does not exist
-l writes rsync log to 'sdimage'-YYYYmmddHHMMSS.log
-z compresses the SD Image (after backup) to 'sdimage'.gz
-d deletes the SD Image after successful compression
-f forces overwrite of 'sdimage'.gz if it exists
-L logfile writes rsync log to 'logfile'
-i sdcard specifies the SD Card location (default: /dev/mmcblk0)

Examples:

bkup_rpimage.sh rsync -c /path/to/rpi_backup.img
starts backup to 'rpi_backup.img', creating it if it does not exist

bkup_rpimage.sh rsync /path/to/$(uname -n).img
uses the RPi's hostname as the SD Image filename

bkup_rpimage.sh rsync -cz /path/to/$(uname -n)-$(date +%Y-%m-%d).img
uses the RPi's hostname and today's date as the SD Image filename,
creating it if it does not exist, and compressing it after backup

bkup_rpimage.sh mount /path/to/$(uname -n).img /mnt/rpi_image
mounts the RPi's SD Image in /mnt/rpi_image

bkup_rpimage.sh umount /path/to/raspi-2014-11-10.img
unmounts the SD Image from default mountdir (/mnt/raspi-2014-11-10.img/)
Here's the script (also as attachment):

Code: Select all

#!/bin/bash
# bkup_rpimage.sh by jinx
#
# Utility script to backup Raspberry Pi's SD Card to a sparse image file
# mounted as a filesystem in a file, allowing for efficient incremental
# backups using rsync

VERSION=v1.0
SDCARD=/dev/mmcblk0

# Echos traces with yellow text to distinguish from other output
trace () {
    echo -e "$(tput setaf 3)${1}$(tput sgr 0)"
}

# Echos en error string in red text and exit
error () {
    echo -e "$(tput setaf 1)${1}$(tput sgr 0)" >&2
    exit 1
}

# Creates a sparse $IMAGE clone of $SDCARD and attaches to $LOOPBACK
do_create () {
    trace "Creating sparse $IMAGE, the apparent size of $SDCARD"
    dd if=/dev/zero of=$IMAGE bs=$(blockdev --getss $SDCARD) count=0 seek=$(blockdev --getsz $SDCARD)

    if [ -s $IMAGE ]; then
        trace "Attaching $IMAGE to $LOOPBACK"
        losetup $LOOPBACK $IMAGE
    else
        error "$IMAGE was not created or has zero size"
    fi

    trace "Copying partition table from $SDCARD to $LOOPBACK"
    parted -s $LOOPBACK mklabel msdos
    sfdisk --dump $SDCARD | sfdisk --force $LOOPBACK

    trace "Formatting partitions"
    partx --add $LOOPBACK
    mkfs.vfat -I $LOOPBACK1
    mkfs.ext4 $LOOPBACK2
}

# Mounts the $IMAGE to $LOOPBACK (if needed) and $MOUNTDIR
do_mount () {
    # Check if do_create already attached the SD Image
    if [ $(losetup -f) = $LOOPBACK ]; then
        trace "Attaching $IMAGE to $LOOPBACK"
        losetup $LOOPBACK $IMAGE
        partx --add $LOOPBACK
    fi

    trace "Mounting $LOOPBACK1 and $LOOPBACK2 to $MOUNTDIR"
    if [ ! -n "$opt_mountdir" ]; then
        mkdir $MOUNTDIR
    fi
    mount $LOOPBACK2 $MOUNTDIR
    mkdir -p $MOUNTDIR/boot
    mount $LOOPBACK1 $MOUNTDIR/boot
}

# Rsyncs content of $SDCARD to $IMAGE if properly mounted
do_backup () {
    if mountpoint -q $MOUNTDIR; then
        trace "Starting rsync backup of / and /boot/ to $MOUNTDIR"
        if [ -n "$opt_log" ]; then
            rsync -aEvx --del --stats --log-file $LOG /boot/ $MOUNTDIR/boot/
            rsync -aEvx --del --stats --log-file $LOG / $MOUNTDIR/
        else
            rsync -aEvx --del --stats /boot/ $MOUNTDIR/boot/
            rsync -aEvx --del --stats / $MOUNTDIR/
        fi
    else
        trace "Skipping rsync since $MOUNTDIR is not a mount point"
    fi
}

# Unmounts the $IMAGE from $MOUNTDIR and $LOOPBACK
do_umount () {
    trace "Flushing to disk"
    sync; sync

    trace "Unmounting $LOOPBACK1 and $LOOPBACK2 from $MOUNTDIR"
    umount $MOUNTDIR/boot
    umount $MOUNTDIR
    if [ ! -n "$opt_mountdir" ]; then
        rmdir $MOUNTDIR
    fi

    trace "Detaching $IMAGE from $LOOPBACK"
    partx --delete $LOOPBACK
    losetup -d $LOOPBACK
}

# Compresses $IMAGE to $IMAGE.gz using a temp file during compression
do_compress () {
    trace "Compressing $IMAGE to ${IMAGE}.gz"
    pv -tpreb $IMAGE | gzip > ${IMAGE}.gz.tmp
    if [ -s ${IMAGE}.gz.tmp ]; then
        mv -f ${IMAGE}.gz.tmp ${IMAGE}.gz
        if [ -n "$opt_delete" ]; then
            rm -f $IMAGE
        fi
    fi
}

# Tries to cleanup after Ctrl-C interrupt
ctrl_c () {
    trace "Ctrl-C detected."

    if [ -s ${IMAGE}.gz.tmp ]; then
        rm ${IMAGE}.gz.tmp
    else
        do_umount
    fi

    if [ -n "$opt_log" ]; then
        trace "See rsync log in $LOG"
    fi

    error "SD Image backup process interrupted"
}

# Prints usage information
usage () {
    echo -e ""
    echo -e "$(basename $0) $VERSION by jinx"
    echo -e ""
    echo -e "Usage:"
    echo -e ""
    echo -e "    $(basename $0) $(tput bold)start$(tput sgr 0) [-clzdf] [-L logfile] [-i sdcard] sdimage"
    echo -e "    $(basename $0) $(tput bold)mount$(tput sgr 0) [-c] sdimage [mountdir]"
    echo -e "    $(basename $0) $(tput bold)umount$(tput sgr 0) sdimage [mountdir]"
    echo -e "    $(basename $0) $(tput bold)gzip$(tput sgr 0) [-df] sdimage"
    echo -e ""
    echo -e "    Commands:"
    echo -e ""
    echo -e "        $(tput bold)start$(tput sgr 0)  starts complete backup of RPi's SD Card to 'sdimage'"
    echo -e "        $(tput bold)mount$(tput sgr 0)  mounts the 'sdimage' to 'mountdir' (default: /mnt/'sdimage'/)"
    echo -e "        $(tput bold)umount$(tput sgr 0) unmounts the 'sdimage' from 'mountdir'"
    echo -e "        $(tput bold)gzip$(tput sgr 0)   compresses the 'sdimage' to 'sdimage'.gz"
    echo -e ""
    echo -e "    Options:"
    echo -e ""
    echo -e "        $(tput bold)-c$(tput sgr 0)         creates the SD Image if it does not exist"
    echo -e "        $(tput bold)-l$(tput sgr 0)         writes rsync log to 'sdimage'-YYYYmmddHHMMSS.log"
    echo -e "        $(tput bold)-z$(tput sgr 0)         compresses the SD Image (after backup) to 'sdimage'.gz"
    echo -e "        $(tput bold)-d$(tput sgr 0)         deletes the SD Image after successful compression"
    echo -e "        $(tput bold)-f$(tput sgr 0)         forces overwrite of 'sdimage'.gz if it exists"
    echo -e "        $(tput bold)-L logfile$(tput sgr 0) writes rsync log to 'logfile'"
    echo -e "        $(tput bold)-i sdcard$(tput sgr 0)  specifies the SD Card location (default: $SDCARD)"
    echo -e ""
    echo -e "Examples:"
    echo -e ""
    echo -e "    $(basename $0) rsync -c /path/to/rpi_backup.img"
    echo -e "        starts backup to 'rpi_backup.img', creating it if it does not exist"
    echo -e ""
    echo -e "    $(basename $0) rsync /path/to/\$(uname -n).img"
    echo -e "        uses the RPi's hostname as the SD Image filename"
    echo -e ""
    echo -e "    $(basename $0) rsync -cz /path/to/\$(uname -n)-\$(date +%Y-%m-%d).img"
    echo -e "        uses the RPi's hostname and today's date as the SD Image filename,"
    echo -e "        creating it if it does not exist, and compressing it after backup"
    echo -e ""
    echo -e "    $(basename $0) mount /path/to/\$(uname -n).img /mnt/rpi_image"
    echo -e "        mounts the RPi's SD Image in /mnt/rpi_image"
    echo -e ""
    echo -e "    $(basename $0) umount /path/to/raspi-$(date +%Y-%m-%d).img"
    echo -e "        unmounts the SD Image from default mountdir (/mnt/raspi-$(date +%Y-%m-%d).img/)"
    echo -e ""
}

# Read the command from command line
case $1 in
    start|mount|umount|gzip) 
        opt_command=$1
        ;;
    -h|--help)
        usage
        exit 0
        ;;
    --version)
        trace "$(basename $0) $VERSION by jinx"
        exit 0
        ;;
    *)
        error "Invalid command or option: $1\nSee '$(basename $0) --help' for usage";;
esac
shift 1

# Make sure we have root rights
if [ $(id -u) -ne 0 ]; then
    error "Please run as root. Try sudo."
fi

# Read the options from command line
while getopts ":czdflL:i:" opt; do
    case $opt in
        c)  opt_create=1;;
        z)  opt_compress=1;;
        d)  opt_delete=1;;
        f)  opt_force=1;;
        l)  opt_log=1;;
        L)  opt_log=1
            LOG=$OPTARG
            ;;
        i)  SDCARD=$OPTARG;;
        \?) error "Invalid option: -$OPTARG\nSee '$(basename $0) --help' for usage";;
        :)  error "Option -$OPTARG requires an argument\nSee '$(basename $0) --help' for usage";;
    esac
done
shift $((OPTIND-1))

# Read the sdimage path from command line
IMAGE=$1
if [ -z $IMAGE ]; then
    error "No sdimage specified"
fi

# Check if sdimage exists
if [ $opt_command = umount ] || [ $opt_command = gzip ]; then
    if [ ! -f $IMAGE ]; then
        error "$IMAGE does not exist"
    fi
else
    if [ ! -f $IMAGE ] && [ ! -n "$opt_create" ]; then
        error "$IMAGE does not exist\nUse -c to allow creation"
    fi
fi

# Check if we should compress and sdimage.gz exists
if [ -n "$opt_compress" ] || [ $opt_command = gzip ]; then
    if [ -s ${IMAGE}.gz ] && [ ! -n "$opt_force" ]; then
        error "${IMAGE}.gz already exists\nUse -f to force overwriting"
    fi
fi

# Define default rsync logfile if not defined
if [ -z $LOG ]; then
    LOG=${IMAGE}-$(date +%Y%m%d%H%M%S).log
fi

# Identify which loopback device to use
LOOPBACK=$(losetup -j $IMAGE | grep -o ^[^:]*)
if [ $opt_command = umount ]; then
    if [ -z $LOOPBACK ]; then
        error "No /dev/loop<X> attached to $IMAGE"
    fi
elif [ ! -z $LOOPBACK ]; then
    error "$IMAGE already attached to $LOOPBACK mounted on $(grep ${LOOPBACK}p2 /etc/mtab | cut -d ' ' -f 2)/"
else
    LOOPBACK=$(losetup -f)
fi
LOOPBACK1=${LOOPBACK}p1
LOOPBACK2=${LOOPBACK}p2

# Read the optional mountdir from command line
MOUNTDIR=$2
if [ -z $MOUNTDIR ]; then
    MOUNTDIR=/mnt/$(basename $IMAGE)/
else
    opt_mountdir=1
    if [ ! -d $MOUNTDIR ]; then
        error "Mount point $MOUNTDIR does not exist"
    fi
fi

# Check if default mount point exists
if [ $opt_command = umount ]; then
    if [ ! -d $MOUNTDIR ]; then
        error "Default mount point $MOUNTDIR does not exist"
    fi
else
    if [ ! -n "$opt_mountdir" ] && [ -d $MOUNTDIR ]; then
        error "Default mount point $MOUNTDIR already exists"
    fi
fi

# Trap keyboard interrupt (ctrl-c)
trap ctrl_c SIGINT

# Check for dependencies
for c in dd losetup parted sfdisk partx mkfs.vfat mkfs.ext4 mountpoint rsync; do
    command -v $c >/dev/null 2>&1 || error "Required program $c is not installed"
done
if [ -n "$opt_compress" ] || [ $opt_command = gzip ]; then
    for c in pv gzip; do
        command -v $c >/dev/null 2>&1 || error "Required program $c is not installed"
    done
fi

# Do the requested functionality
case $opt_command in
    start)
            trace "Starting SD Image backup process"
            if [ ! -f $IMAGE ] && [ -n "$opt_create" ]; then
                do_create
            fi
            do_mount
            do_backup
            do_umount
            if [ -n "$opt_compress" ]; then
                do_compress
            fi
            trace "SD Image backup process completed."
            if [ -n "$opt_log" ]; then
                trace "See rsync log in $LOG"
            fi
            ;;
    mount)
            if [ ! -f $IMAGE ] && [ -n "$opt_create" ]; then
                do_create
            fi
            do_mount
            trace "SD Image has been mounted and can be accessed at:\n    $MOUNTDIR"
            ;;
    umount)
            do_umount
            ;;
    gzip)
            do_compress
            ;;
    *)
            error "Unknown command: $opt_command"
            ;;
esac

exit 0
bkup_rpimage.sh.gz
Backup script as attachment
(2.85 KiB) Downloaded 338 times
The script creates a sparse image which only has the apparent size of the SD Card, but where empty sectors do not use disk space. For an empty, sparse image clone of a 4GB SD Card, the partition and file system overhead is only 66 MB, as can been seen by:

Code: Select all

# bkup_rpimage mount -c /path/to/sdempty.img
# bkup_rpimage umount /path/to/sdempty.img
# du -h --apparent-size /path/to/sdempty.img
3.8G    /path/to/sdempty.img
# du -h /path/to/sdempty.img*
66M     /path/to/sdempty.img
3.8M    /path/to/sdempty.img.gz
On my Raspberry Pi, the sparse image only contains my files, no empty sectors, as can be seen by:

Code: Select all

# df -h
Filesystem              Size  Used Avail Use% Mounted on
rootfs                  3.7G  1.3G  2.2G  37% /
...
# du -h --apparent-size /path/to/sdimage.img
3.8G    /path/to/sdimage.img
# du -h /path/to/sdimage.img*
1.4G    /path/to/sdimage.img
565M    /path/to/sdimage.img.gz
Performing the initial "bkup_rpimage.sh start -c /path/to/sdimage.img" (35'112 files / 1'231'978'449 bytes) took 14m44s
Performing an incremental "bkup_rpimage.sh start /path/to/sdimage.img" (11 files / 71'411'770 bytes) took 1m06s
Compressing "bkup_rpimage.sh gzip /path/to/sdimage.img" took 36m07s

Please test this and let me know if you run into any issues.

kimba
Posts: 3
Joined: Fri Nov 07, 2014 2:05 pm

Re: The Raspberry Pi Backup Thread

Mon Nov 10, 2014 1:48 pm

thanks jinx!
I will test it out over the weekends and let you know if it works.

kimba
Posts: 3
Joined: Fri Nov 07, 2014 2:05 pm

Re: The Raspberry Pi Backup Thread

Wed Nov 12, 2014 7:49 am

jinx wrote:I have now attempted to create a utility script which can be used to perform a standard backup (using rsync) with more error handling than before.
I have also added the possibility to optionally compress the image after backup, as well as abilities to mount and unmount the image (and in theory any Raspberry Pi dist image) so that data can be viewed, restored or edited, and a Ctrl-C interrupt trap, which should allow you to stop lengthy processes as rsync or gzip while still trying to exit gracefully.

From "bkup_rpimage.sh --help":
bkup_rpimage.sh v1.0 by jinx

Usage:

bkup_rpimage.sh start [-clzdf] [-L logfile] [-i sdcard] sdimage
bkup_rpimage.sh mount [-c] sdimage [mountdir]
bkup_rpimage.sh umount sdimage [mountdir]
bkup_rpimage.sh gzip [-df] sdimage

Commands:

start starts complete backup of RPi's SD Card to 'sdimage'
mount mounts the 'sdimage' to 'mountdir' (default: /mnt/'sdimage'/)
umount unmounts the 'sdimage' from 'mountdir'
gzip compresses the 'sdimage' to 'sdimage'.gz

Options:

-c creates the SD Image if it does not exist
-l writes rsync log to 'sdimage'-YYYYmmddHHMMSS.log
-z compresses the SD Image (after backup) to 'sdimage'.gz
-d deletes the SD Image after successful compression
-f forces overwrite of 'sdimage'.gz if it exists
-L logfile writes rsync log to 'logfile'
-i sdcard specifies the SD Card location (default: /dev/mmcblk0)

Examples:

bkup_rpimage.sh rsync -c /path/to/rpi_backup.img
starts backup to 'rpi_backup.img', creating it if it does not exist

bkup_rpimage.sh rsync /path/to/$(uname -n).img
uses the RPi's hostname as the SD Image filename

bkup_rpimage.sh rsync -cz /path/to/$(uname -n)-$(date +%Y-%m-%d).img
uses the RPi's hostname and today's date as the SD Image filename,
creating it if it does not exist, and compressing it after backup

bkup_rpimage.sh mount /path/to/$(uname -n).img /mnt/rpi_image
mounts the RPi's SD Image in /mnt/rpi_image

bkup_rpimage.sh umount /path/to/raspi-2014-11-10.img
unmounts the SD Image from default mountdir (/mnt/raspi-2014-11-10.img/)
Here's the script (also as attachment):

Code: Select all

#!/bin/bash
# bkup_rpimage.sh by jinx
#
# Utility script to backup Raspberry Pi's SD Card to a sparse image file
# mounted as a filesystem in a file, allowing for efficient incremental
# backups using rsync

VERSION=v1.0
SDCARD=/dev/mmcblk0

# Echos traces with yellow text to distinguish from other output
trace () {
    echo -e "$(tput setaf 3)${1}$(tput sgr 0)"
}

# Echos en error string in red text and exit
error () {
    echo -e "$(tput setaf 1)${1}$(tput sgr 0)" >&2
    exit 1
}

# Creates a sparse $IMAGE clone of $SDCARD and attaches to $LOOPBACK
do_create () {
    trace "Creating sparse $IMAGE, the apparent size of $SDCARD"
    dd if=/dev/zero of=$IMAGE bs=$(blockdev --getss $SDCARD) count=0 seek=$(blockdev --getsz $SDCARD)

    if [ -s $IMAGE ]; then
        trace "Attaching $IMAGE to $LOOPBACK"
        losetup $LOOPBACK $IMAGE
    else
        error "$IMAGE was not created or has zero size"
    fi

    trace "Copying partition table from $SDCARD to $LOOPBACK"
    parted -s $LOOPBACK mklabel msdos
    sfdisk --dump $SDCARD | sfdisk --force $LOOPBACK

    trace "Formatting partitions"
    partx --add $LOOPBACK
    mkfs.vfat -I $LOOPBACK1
    mkfs.ext4 $LOOPBACK2
}

# Mounts the $IMAGE to $LOOPBACK (if needed) and $MOUNTDIR
do_mount () {
    # Check if do_create already attached the SD Image
    if [ $(losetup -f) = $LOOPBACK ]; then
        trace "Attaching $IMAGE to $LOOPBACK"
        losetup $LOOPBACK $IMAGE
        partx --add $LOOPBACK
    fi

    trace "Mounting $LOOPBACK1 and $LOOPBACK2 to $MOUNTDIR"
    if [ ! -n "$opt_mountdir" ]; then
        mkdir $MOUNTDIR
    fi
    mount $LOOPBACK2 $MOUNTDIR
    mkdir -p $MOUNTDIR/boot
    mount $LOOPBACK1 $MOUNTDIR/boot
}

# Rsyncs content of $SDCARD to $IMAGE if properly mounted
do_backup () {
    if mountpoint -q $MOUNTDIR; then
        trace "Starting rsync backup of / and /boot/ to $MOUNTDIR"
        if [ -n "$opt_log" ]; then
            rsync -aEvx --del --stats --log-file $LOG /boot/ $MOUNTDIR/boot/
            rsync -aEvx --del --stats --log-file $LOG / $MOUNTDIR/
        else
            rsync -aEvx --del --stats /boot/ $MOUNTDIR/boot/
            rsync -aEvx --del --stats / $MOUNTDIR/
        fi
    else
        trace "Skipping rsync since $MOUNTDIR is not a mount point"
    fi
}

# Unmounts the $IMAGE from $MOUNTDIR and $LOOPBACK
do_umount () {
    trace "Flushing to disk"
    sync; sync

    trace "Unmounting $LOOPBACK1 and $LOOPBACK2 from $MOUNTDIR"
    umount $MOUNTDIR/boot
    umount $MOUNTDIR
    if [ ! -n "$opt_mountdir" ]; then
        rmdir $MOUNTDIR
    fi

    trace "Detaching $IMAGE from $LOOPBACK"
    partx --delete $LOOPBACK
    losetup -d $LOOPBACK
}

# Compresses $IMAGE to $IMAGE.gz using a temp file during compression
do_compress () {
    trace "Compressing $IMAGE to ${IMAGE}.gz"
    pv -tpreb $IMAGE | gzip > ${IMAGE}.gz.tmp
    if [ -s ${IMAGE}.gz.tmp ]; then
        mv -f ${IMAGE}.gz.tmp ${IMAGE}.gz
        if [ -n "$opt_delete" ]; then
            rm -f $IMAGE
        fi
    fi
}

# Tries to cleanup after Ctrl-C interrupt
ctrl_c () {
    trace "Ctrl-C detected."

    if [ -s ${IMAGE}.gz.tmp ]; then
        rm ${IMAGE}.gz.tmp
    else
        do_umount
    fi

    if [ -n "$opt_log" ]; then
        trace "See rsync log in $LOG"
    fi

    error "SD Image backup process interrupted"
}

# Prints usage information
usage () {
    echo -e ""
    echo -e "$(basename $0) $VERSION by jinx"
    echo -e ""
    echo -e "Usage:"
    echo -e ""
    echo -e "    $(basename $0) $(tput bold)start$(tput sgr 0) [-clzdf] [-L logfile] [-i sdcard] sdimage"
    echo -e "    $(basename $0) $(tput bold)mount$(tput sgr 0) [-c] sdimage [mountdir]"
    echo -e "    $(basename $0) $(tput bold)umount$(tput sgr 0) sdimage [mountdir]"
    echo -e "    $(basename $0) $(tput bold)gzip$(tput sgr 0) [-df] sdimage"
    echo -e ""
    echo -e "    Commands:"
    echo -e ""
    echo -e "        $(tput bold)start$(tput sgr 0)  starts complete backup of RPi's SD Card to 'sdimage'"
    echo -e "        $(tput bold)mount$(tput sgr 0)  mounts the 'sdimage' to 'mountdir' (default: /mnt/'sdimage'/)"
    echo -e "        $(tput bold)umount$(tput sgr 0) unmounts the 'sdimage' from 'mountdir'"
    echo -e "        $(tput bold)gzip$(tput sgr 0)   compresses the 'sdimage' to 'sdimage'.gz"
    echo -e ""
    echo -e "    Options:"
    echo -e ""
    echo -e "        $(tput bold)-c$(tput sgr 0)         creates the SD Image if it does not exist"
    echo -e "        $(tput bold)-l$(tput sgr 0)         writes rsync log to 'sdimage'-YYYYmmddHHMMSS.log"
    echo -e "        $(tput bold)-z$(tput sgr 0)         compresses the SD Image (after backup) to 'sdimage'.gz"
    echo -e "        $(tput bold)-d$(tput sgr 0)         deletes the SD Image after successful compression"
    echo -e "        $(tput bold)-f$(tput sgr 0)         forces overwrite of 'sdimage'.gz if it exists"
    echo -e "        $(tput bold)-L logfile$(tput sgr 0) writes rsync log to 'logfile'"
    echo -e "        $(tput bold)-i sdcard$(tput sgr 0)  specifies the SD Card location (default: $SDCARD)"
    echo -e ""
    echo -e "Examples:"
    echo -e ""
    echo -e "    $(basename $0) rsync -c /path/to/rpi_backup.img"
    echo -e "        starts backup to 'rpi_backup.img', creating it if it does not exist"
    echo -e ""
    echo -e "    $(basename $0) rsync /path/to/\$(uname -n).img"
    echo -e "        uses the RPi's hostname as the SD Image filename"
    echo -e ""
    echo -e "    $(basename $0) rsync -cz /path/to/\$(uname -n)-\$(date +%Y-%m-%d).img"
    echo -e "        uses the RPi's hostname and today's date as the SD Image filename,"
    echo -e "        creating it if it does not exist, and compressing it after backup"
    echo -e ""
    echo -e "    $(basename $0) mount /path/to/\$(uname -n).img /mnt/rpi_image"
    echo -e "        mounts the RPi's SD Image in /mnt/rpi_image"
    echo -e ""
    echo -e "    $(basename $0) umount /path/to/raspi-$(date +%Y-%m-%d).img"
    echo -e "        unmounts the SD Image from default mountdir (/mnt/raspi-$(date +%Y-%m-%d).img/)"
    echo -e ""
}

# Read the command from command line
case $1 in
    start|mount|umount|gzip) 
        opt_command=$1
        ;;
    -h|--help)
        usage
        exit 0
        ;;
    --version)
        trace "$(basename $0) $VERSION by jinx"
        exit 0
        ;;
    *)
        error "Invalid command or option: $1\nSee '$(basename $0) --help' for usage";;
esac
shift 1

# Make sure we have root rights
if [ $(id -u) -ne 0 ]; then
    error "Please run as root. Try sudo."
fi

# Read the options from command line
while getopts ":czdflL:i:" opt; do
    case $opt in
        c)  opt_create=1;;
        z)  opt_compress=1;;
        d)  opt_delete=1;;
        f)  opt_force=1;;
        l)  opt_log=1;;
        L)  opt_log=1
            LOG=$OPTARG
            ;;
        i)  SDCARD=$OPTARG;;
        \?) error "Invalid option: -$OPTARG\nSee '$(basename $0) --help' for usage";;
        :)  error "Option -$OPTARG requires an argument\nSee '$(basename $0) --help' for usage";;
    esac
done
shift $((OPTIND-1))

# Read the sdimage path from command line
IMAGE=$1
if [ -z $IMAGE ]; then
    error "No sdimage specified"
fi

# Check if sdimage exists
if [ $opt_command = umount ] || [ $opt_command = gzip ]; then
    if [ ! -f $IMAGE ]; then
        error "$IMAGE does not exist"
    fi
else
    if [ ! -f $IMAGE ] && [ ! -n "$opt_create" ]; then
        error "$IMAGE does not exist\nUse -c to allow creation"
    fi
fi

# Check if we should compress and sdimage.gz exists
if [ -n "$opt_compress" ] || [ $opt_command = gzip ]; then
    if [ -s ${IMAGE}.gz ] && [ ! -n "$opt_force" ]; then
        error "${IMAGE}.gz already exists\nUse -f to force overwriting"
    fi
fi

# Define default rsync logfile if not defined
if [ -z $LOG ]; then
    LOG=${IMAGE}-$(date +%Y%m%d%H%M%S).log
fi

# Identify which loopback device to use
LOOPBACK=$(losetup -j $IMAGE | grep -o ^[^:]*)
if [ $opt_command = umount ]; then
    if [ -z $LOOPBACK ]; then
        error "No /dev/loop<X> attached to $IMAGE"
    fi
elif [ ! -z $LOOPBACK ]; then
    error "$IMAGE already attached to $LOOPBACK mounted on $(grep ${LOOPBACK}p2 /etc/mtab | cut -d ' ' -f 2)/"
else
    LOOPBACK=$(losetup -f)
fi
LOOPBACK1=${LOOPBACK}p1
LOOPBACK2=${LOOPBACK}p2

# Read the optional mountdir from command line
MOUNTDIR=$2
if [ -z $MOUNTDIR ]; then
    MOUNTDIR=/mnt/$(basename $IMAGE)/
else
    opt_mountdir=1
    if [ ! -d $MOUNTDIR ]; then
        error "Mount point $MOUNTDIR does not exist"
    fi
fi

# Check if default mount point exists
if [ $opt_command = umount ]; then
    if [ ! -d $MOUNTDIR ]; then
        error "Default mount point $MOUNTDIR does not exist"
    fi
else
    if [ ! -n "$opt_mountdir" ] && [ -d $MOUNTDIR ]; then
        error "Default mount point $MOUNTDIR already exists"
    fi
fi

# Trap keyboard interrupt (ctrl-c)
trap ctrl_c SIGINT

# Check for dependencies
for c in dd losetup parted sfdisk partx mkfs.vfat mkfs.ext4 mountpoint rsync; do
    command -v $c >/dev/null 2>&1 || error "Required program $c is not installed"
done
if [ -n "$opt_compress" ] || [ $opt_command = gzip ]; then
    for c in pv gzip; do
        command -v $c >/dev/null 2>&1 || error "Required program $c is not installed"
    done
fi

# Do the requested functionality
case $opt_command in
    start)
            trace "Starting SD Image backup process"
            if [ ! -f $IMAGE ] && [ -n "$opt_create" ]; then
                do_create
            fi
            do_mount
            do_backup
            do_umount
            if [ -n "$opt_compress" ]; then
                do_compress
            fi
            trace "SD Image backup process completed."
            if [ -n "$opt_log" ]; then
                trace "See rsync log in $LOG"
            fi
            ;;
    mount)
            if [ ! -f $IMAGE ] && [ -n "$opt_create" ]; then
                do_create
            fi
            do_mount
            trace "SD Image has been mounted and can be accessed at:\n    $MOUNTDIR"
            ;;
    umount)
            do_umount
            ;;
    gzip)
            do_compress
            ;;
    *)
            error "Unknown command: $opt_command"
            ;;
esac

exit 0
bkup_rpimage.sh.gz
The script creates a sparse image which only has the apparent size of the SD Card, but where empty sectors do not use disk space. For an empty, sparse image clone of a 4GB SD Card, the partition and file system overhead is only 66 MB, as can been seen by:

Code: Select all

# bkup_rpimage mount -c /path/to/sdempty.img
# bkup_rpimage umount /path/to/sdempty.img
# du -h --apparent-size /path/to/sdempty.img
3.8G    /path/to/sdempty.img
# du -h /path/to/sdempty.img*
66M     /path/to/sdempty.img
3.8M    /path/to/sdempty.img.gz
On my Raspberry Pi, the sparse image only contains my files, no empty sectors, as can be seen by:

Code: Select all

# df -h
Filesystem              Size  Used Avail Use% Mounted on
rootfs                  3.7G  1.3G  2.2G  37% /
...
# du -h --apparent-size /path/to/sdimage.img
3.8G    /path/to/sdimage.img
# du -h /path/to/sdimage.img*
1.4G    /path/to/sdimage.img
565M    /path/to/sdimage.img.gz
Performing the initial "bkup_rpimage.sh start -c /path/to/sdimage.img" (35'112 files / 1'231'978'449 bytes) took 14m44s
Performing an incremental "bkup_rpimage.sh start /path/to/sdimage.img" (11 files / 71'411'770 bytes) took 1m06s
Compressing "bkup_rpimage.sh gzip /path/to/sdimage.img" took 36m07s

Please test this and let me know if you run into any issues.
Hi jinx, i have just tried out your script.
I started by mounting my NAS share to /home/pi/backups/images.
I then executed your script via sudo bash bkup_rpimage.sh start -cl /home/pi/backups/images/$(uname -n)-$(date +%Y-%m-%d).img
the script ran for a few seconds, created the image file and then ended without backing up anything.
this is what the script returned:

Code: Select all

/home/pi/backups/images/Raspberrypi-2014-11-12.img: No such file or directory
Starting SD Image backup process
Creating sparse /home/pi/backups/images/Raspberrypi-2014-11-12.img, the apparent size of /dev/mmcblk0
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00122599 s, 0.0 kB/s
Attaching /home/pi/backups/images/Raspberrypi-2014-11-12.img to /dev/loop0
Copying partition table from /dev/mmcblk0 to /dev/loop0
Checking that no-one is using this disk right now ...
BLKRRPART: Invalid argument
OK
Disk /dev/loop0: cannot get geometry

Disk /dev/loop0: 1911 cylinders, 255 heads, 63 sectors/track
Old situation:
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/loop0p1          0       -       0          0    0  Empty
/dev/loop0p2          0       -       0          0    0  Empty
/dev/loop0p3          0       -       0          0    0  Empty
/dev/loop0p4          0       -       0          0    0  Empty
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/loop0p1          8192   1609375    1601184   e  W95 FAT16 (LBA)
/dev/loop0p2       1613824  30638079   29024256  85  Linux extended
/dev/loop0p3      30638080  30703615      65536  83  Linux
/dev/loop0p4             0         -          0   0  Empty
/dev/loop0p5       1622016   1744895     122880   c  W95 FAT32 (LBA)
/dev/loop0p6       1753088  30638079   28884992  83  Linux
Warning: partition 1 does not end at a cylinder boundary
Warning: partition 2 does not start at a cylinder boundary
Warning: partition 2 does not end at a cylinder boundary
Warning: partition 3 does not start at a cylinder boundary
Warning: partition 3 does not end at a cylinder boundary
Warning: partition 5 does not end at a cylinder boundary
Warning: partition [6] does not start at a cylinder boundary
Warning: partition [6] does not end at a cylinder boundary
Warning: partition 6 does not end at a cylinder boundary
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table

Re-reading the partition table ...
BLKRRPART: Invalid argument

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
Formatting partitions
mkfs.vfat 3.0.13 (30 Jun 2012)
unable to get drive geometry, using default 255/63
mke2fs 1.42.5 (29-Jul-2012)
mkfs.ext4: inode_size (128) * inodes_count (0) too big for a
        filesystem with 0 blocks, specify higher inode_ratio (-i)
        or lower inode count (-N).

Mounting /dev/loop0p1 and /dev/loop0p2 to /mnt/Raspberrypi-2014-11-12.img/
mount: /dev/loop0p2: can't read superblock
Skipping rsync since /mnt/Raspberrypi-2014-11-12.img/ is not a mount point
Flushing to disk
Unmounting /dev/loop0p1 and /dev/loop0p2 from /mnt/Raspberrypi-2014-11-12.img/
umount: /mnt/Raspberrypi-2014-11-12.img/: not mounted
rmdir: failed to remove `/mnt/Raspberrypi-2014-11-12.img/': Directory not empty
Detaching /home/pi/backups/images/Raspberrypi-2014-11-12.img from /dev/loop0
partx: /dev/loop0: error deleting partition 4
SD Image backup process completed.
See rsync log in /home/pi/backups/images/Raspberrypi-2014-11-12.img-20141112153314.log
any idea what went wrong this time?

PS: there's no rsync argument as shown in your examples. you might have forgotten to update that part.

Return to “General discussion”