stedew
Posts: 4
Joined: Sun Oct 21, 2012 9:37 am

Re: The Raspberry Pi Backup Thread

Sun Nov 23, 2014 8:35 pm

Hello jinx,

I tested this amazing script and it works excellent with some hickups... I run first time on one sd (original) and dumped this on /media/usbhdd/test/sdimage.img no problem so far....
sudo ~/backup/bkup_rpimage/./bkup_rpimage.sh start -c /media/usbhdd/test/sdimage.img
Then unmount the HDD and switched over to my redmount system ;) .
Mount the drive with the (excellent) tool Extfs from Paragon... and copied the image from the disk to another SD card..
With win32DiskImager again succeed. so far mission accomplished.
When i want to run again with same param. the system won't work
>> Default mount point /mnt/sdimage.img/ already exists
without the extension (.img) no problem and the rsync does its job (re-imaging in 20secs )
Probably it is totally my fault... Thanks for this great script!!!

>>edit: Re-inserted the original sdCard works like a charm.... :oops:

jinx
Posts: 10
Joined: Wed Nov 05, 2014 11:44 am

Re: The Raspberry Pi Backup Thread

Mon Nov 24, 2014 12:09 pm

kimba wrote: Hi jinx, i have just tried out your script.
I started by mounting my NAS share to /home/pi/backups/images.
I then executed your script via sudo bash bkup_rpimage.sh start -cl /home/pi/backups/images/$(uname -n)-$(date +%Y-%m-%d).img
the script ran for a few seconds, created the image file and then ended without backing up anything.
this is what the script returned:

Code: Select all

/home/pi/backups/images/Raspberrypi-2014-11-12.img: No such file or directory
Starting SD Image backup process
Creating sparse /home/pi/backups/images/Raspberrypi-2014-11-12.img, the apparent size of /dev/mmcblk0
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00122599 s, 0.0 kB/s
Attaching /home/pi/backups/images/Raspberrypi-2014-11-12.img to /dev/loop0
Copying partition table from /dev/mmcblk0 to /dev/loop0
Checking that no-one is using this disk right now ...
BLKRRPART: Invalid argument
OK
Disk /dev/loop0: cannot get geometry

Disk /dev/loop0: 1911 cylinders, 255 heads, 63 sectors/track
Old situation:
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/loop0p1          0       -       0          0    0  Empty
/dev/loop0p2          0       -       0          0    0  Empty
/dev/loop0p3          0       -       0          0    0  Empty
/dev/loop0p4          0       -       0          0    0  Empty
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/loop0p1          8192   1609375    1601184   e  W95 FAT16 (LBA)
/dev/loop0p2       1613824  30638079   29024256  85  Linux extended
/dev/loop0p3      30638080  30703615      65536  83  Linux
/dev/loop0p4             0         -          0   0  Empty
/dev/loop0p5       1622016   1744895     122880   c  W95 FAT32 (LBA)
/dev/loop0p6       1753088  30638079   28884992  83  Linux
Warning: partition 1 does not end at a cylinder boundary
Warning: partition 2 does not start at a cylinder boundary
Warning: partition 2 does not end at a cylinder boundary
Warning: partition 3 does not start at a cylinder boundary
Warning: partition 3 does not end at a cylinder boundary
Warning: partition 5 does not end at a cylinder boundary
Warning: partition [6] does not start at a cylinder boundary
Warning: partition [6] does not end at a cylinder boundary
Warning: partition 6 does not end at a cylinder boundary
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table

Re-reading the partition table ...
BLKRRPART: Invalid argument

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
Formatting partitions
mkfs.vfat 3.0.13 (30 Jun 2012)
unable to get drive geometry, using default 255/63
mke2fs 1.42.5 (29-Jul-2012)
mkfs.ext4: inode_size (128) * inodes_count (0) too big for a
        filesystem with 0 blocks, specify higher inode_ratio (-i)
        or lower inode count (-N).

Mounting /dev/loop0p1 and /dev/loop0p2 to /mnt/Raspberrypi-2014-11-12.img/
mount: /dev/loop0p2: can't read superblock
Skipping rsync since /mnt/Raspberrypi-2014-11-12.img/ is not a mount point
Flushing to disk
Unmounting /dev/loop0p1 and /dev/loop0p2 from /mnt/Raspberrypi-2014-11-12.img/
umount: /mnt/Raspberrypi-2014-11-12.img/: not mounted
rmdir: failed to remove `/mnt/Raspberrypi-2014-11-12.img/': Directory not empty
Detaching /home/pi/backups/images/Raspberrypi-2014-11-12.img from /dev/loop0
partx: /dev/loop0: error deleting partition 4
SD Image backup process completed.
See rsync log in /home/pi/backups/images/Raspberrypi-2014-11-12.img-20141112153314.log
any idea what went wrong this time?

PS: there's no rsync argument as shown in your examples. you might have forgotten to update that part.
kimba,

I did not know that there were SD Card distributions which did not use the standard 2 partition model, where the first is the /boot and the second the / (root). I have tried the script on the Raspian distribution http://www.raspbian.org/RaspbianImages and Arch Linux http://archlinuxarm.org/platforms/armv6/raspberry-pi

Your SD Card has 6 partitions, and somewhere while trying to clone the SD Card (assuming that there are only two partition) the formatting of the second partition fails.

Regarding the "bkup_rpimage.sh rsync ..." command, I missed updating the examples before posting the script. This is the same as "bkup_rpimage.sh start ..."

wintergriller
Posts: 1
Joined: Wed Feb 11, 2015 8:23 pm

Re: The Raspberry Pi Backup Thread

Wed Feb 11, 2015 8:51 pm

jinx wrote:
kimba wrote: Hi jinx, i have just tried out your script.
I started by mounting my NAS share to /home/pi/backups/images.
I then executed your script via sudo bash bkup_rpimage.sh start -cl /home/pi/backups/images/$(uname -n)-$(date +%Y-%m-%d).img
the script ran for a few seconds, created the image file and then ended without backing up anything.
this is what the script returned:

kimba,

I did not know that there were SD Card distributions which did not use the standard 2 partition model, where the first is the /boot and the second the / (root). I have tried the script on the Raspian distribution http://www.raspbian.org/RaspbianImages and Arch Linux http://archlinuxarm.org/platforms/armv6/raspberry-pi

Your SD Card has 6 partitions, and somewhere while trying to clone the SD Card (assuming that there are only two partition) the formatting of the second partition fails.

Regarding the "bkup_rpimage.sh rsync ..." command, I missed updating the examples before posting the script. This is the same as "bkup_rpimage.sh start ..."
Hi there,

First thanks jinx for the amazing script (no kidding). Exactly what I was looking for.
I had similar issues as kimba trying to rsync to an image located on a cifs mount on my home router.
In my case, the issue came from the sparse file, which seems to be not handled correctly if located on a cifs mount.
So I've added an option to the script to create a "real" file instead of a sparse file (which of course takes much more time).

I just did a last test with an image created with your script from scratch and updated several times by a cron job. Wrote the image to a new SD card using my Windows 7 PC and Win32DiskImager. Works like a charm.

Here is the simple change I made (not showing option handling, usage, ... )

Code: Select all

do_create () {
        if [ -n "$opt_nosparse" ]; then
                trace "Creating $IMAGE, the apparent size of $SDCARD"
                dd if=/dev/zero of=$IMAGE bs=$(blockdev --getss $SDCARD) count=$(blockdev --getsz $SDCARD)
        else
                trace "Creating sparse $IMAGE, the apparent size of $SDCARD"
                dd if=/dev/zero of=$IMAGE bs=$(blockdev --getss $SDCARD) count=0 seek=$(blockdev --getsz $SDCARD)
        fi
Maybe somebody is interested...

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Thu Feb 19, 2015 9:33 pm

jinx wrote:I have now attempted to create a utility script which can be used to perform a standard backup (using rsync) with more error handling than before.
I have also added the possibility to optionally compress the image after backup, as well as abilities to mount and unmount the image (and in theory any Raspberry Pi dist image) so that data can be viewed, restored or edited, and a Ctrl-C interrupt trap, which should allow you to stop lengthy processes as rsync or gzip while still trying to exit gracefully.

From "bkup_rpimage.sh --help":
bkup_rpimage.sh v1.0 by jinx

Usage:

bkup_rpimage.sh start [-clzdf] [-L logfile] [-i sdcard] sdimage
bkup_rpimage.sh mount [-c] sdimage [mountdir]
bkup_rpimage.sh umount sdimage [mountdir]
bkup_rpimage.sh gzip [-df] sdimage

Commands:

start starts complete backup of RPi's SD Card to 'sdimage'
mount mounts the 'sdimage' to 'mountdir' (default: /mnt/'sdimage'/)
umount unmounts the 'sdimage' from 'mountdir'
gzip compresses the 'sdimage' to 'sdimage'.gz

Options:

-c creates the SD Image if it does not exist
-l writes rsync log to 'sdimage'-YYYYmmddHHMMSS.log
-z compresses the SD Image (after backup) to 'sdimage'.gz
-d deletes the SD Image after successful compression
-f forces overwrite of 'sdimage'.gz if it exists
-L logfile writes rsync log to 'logfile'
-i sdcard specifies the SD Card location (default: /dev/mmcblk0)

Examples:

bkup_rpimage.sh rsync -c /path/to/rpi_backup.img
starts backup to 'rpi_backup.img', creating it if it does not exist

bkup_rpimage.sh rsync /path/to/$(uname -n).img
uses the RPi's hostname as the SD Image filename

bkup_rpimage.sh rsync -cz /path/to/$(uname -n)-$(date +%Y-%m-%d).img
uses the RPi's hostname and today's date as the SD Image filename,
creating it if it does not exist, and compressing it after backup

bkup_rpimage.sh mount /path/to/$(uname -n).img /mnt/rpi_image
mounts the RPi's SD Image in /mnt/rpi_image

bkup_rpimage.sh umount /path/to/raspi-2014-11-10.img
unmounts the SD Image from default mountdir (/mnt/raspi-2014-11-10.img/)
Here's the script (also as attachment):

Code: Select all

#!/bin/bash
# bkup_rpimage.sh by jinx
#
# Utility script to backup Raspberry Pi's SD Card to a sparse image file
# mounted as a filesystem in a file, allowing for efficient incremental
# backups using rsync

VERSION=v1.0
SDCARD=/dev/mmcblk0

# Echos traces with yellow text to distinguish from other output
trace () {
    echo -e "$(tput setaf 3)${1}$(tput sgr 0)"
}

# Echos en error string in red text and exit
error () {
    echo -e "$(tput setaf 1)${1}$(tput sgr 0)" >&2
    exit 1
}

# Creates a sparse $IMAGE clone of $SDCARD and attaches to $LOOPBACK
do_create () {
    trace "Creating sparse $IMAGE, the apparent size of $SDCARD"
    dd if=/dev/zero of=$IMAGE bs=$(blockdev --getss $SDCARD) count=0 seek=$(blockdev --getsz $SDCARD)

    if [ -s $IMAGE ]; then
        trace "Attaching $IMAGE to $LOOPBACK"
        losetup $LOOPBACK $IMAGE
    else
        error "$IMAGE was not created or has zero size"
    fi

    trace "Copying partition table from $SDCARD to $LOOPBACK"
    parted -s $LOOPBACK mklabel msdos
    sfdisk --dump $SDCARD | sfdisk --force $LOOPBACK

    trace "Formatting partitions"
    partx --add $LOOPBACK
    mkfs.vfat -I $LOOPBACK1
    mkfs.ext4 $LOOPBACK2
}

# Mounts the $IMAGE to $LOOPBACK (if needed) and $MOUNTDIR
do_mount () {
    # Check if do_create already attached the SD Image
    if [ $(losetup -f) = $LOOPBACK ]; then
        trace "Attaching $IMAGE to $LOOPBACK"
        losetup $LOOPBACK $IMAGE
        partx --add $LOOPBACK
    fi

    trace "Mounting $LOOPBACK1 and $LOOPBACK2 to $MOUNTDIR"
    if [ ! -n "$opt_mountdir" ]; then
        mkdir $MOUNTDIR
    fi
    mount $LOOPBACK2 $MOUNTDIR
    mkdir -p $MOUNTDIR/boot
    mount $LOOPBACK1 $MOUNTDIR/boot
}

# Rsyncs content of $SDCARD to $IMAGE if properly mounted
do_backup () {
    if mountpoint -q $MOUNTDIR; then
        trace "Starting rsync backup of / and /boot/ to $MOUNTDIR"
        if [ -n "$opt_log" ]; then
            rsync -aEvx --del --stats --log-file $LOG /boot/ $MOUNTDIR/boot/
            rsync -aEvx --del --stats --log-file $LOG / $MOUNTDIR/
        else
            rsync -aEvx --del --stats /boot/ $MOUNTDIR/boot/
            rsync -aEvx --del --stats / $MOUNTDIR/
        fi
    else
        trace "Skipping rsync since $MOUNTDIR is not a mount point"
    fi
}

# Unmounts the $IMAGE from $MOUNTDIR and $LOOPBACK
do_umount () {
    trace "Flushing to disk"
    sync; sync

    trace "Unmounting $LOOPBACK1 and $LOOPBACK2 from $MOUNTDIR"
    umount $MOUNTDIR/boot
    umount $MOUNTDIR
    if [ ! -n "$opt_mountdir" ]; then
        rmdir $MOUNTDIR
    fi

    trace "Detaching $IMAGE from $LOOPBACK"
    partx --delete $LOOPBACK
    losetup -d $LOOPBACK
}

# Compresses $IMAGE to $IMAGE.gz using a temp file during compression
do_compress () {
    trace "Compressing $IMAGE to ${IMAGE}.gz"
    pv -tpreb $IMAGE | gzip > ${IMAGE}.gz.tmp
    if [ -s ${IMAGE}.gz.tmp ]; then
        mv -f ${IMAGE}.gz.tmp ${IMAGE}.gz
        if [ -n "$opt_delete" ]; then
            rm -f $IMAGE
        fi
    fi
}

# Tries to cleanup after Ctrl-C interrupt
ctrl_c () {
    trace "Ctrl-C detected."

    if [ -s ${IMAGE}.gz.tmp ]; then
        rm ${IMAGE}.gz.tmp
    else
        do_umount
    fi

    if [ -n "$opt_log" ]; then
        trace "See rsync log in $LOG"
    fi

    error "SD Image backup process interrupted"
}

# Prints usage information
usage () {
    echo -e ""
    echo -e "$(basename $0) $VERSION by jinx"
    echo -e ""
    echo -e "Usage:"
    echo -e ""
    echo -e "    $(basename $0) $(tput bold)start$(tput sgr 0) [-clzdf] [-L logfile] [-i sdcard] sdimage"
    echo -e "    $(basename $0) $(tput bold)mount$(tput sgr 0) [-c] sdimage [mountdir]"
    echo -e "    $(basename $0) $(tput bold)umount$(tput sgr 0) sdimage [mountdir]"
    echo -e "    $(basename $0) $(tput bold)gzip$(tput sgr 0) [-df] sdimage"
    echo -e ""
    echo -e "    Commands:"
    echo -e ""
    echo -e "        $(tput bold)start$(tput sgr 0)  starts complete backup of RPi's SD Card to 'sdimage'"
    echo -e "        $(tput bold)mount$(tput sgr 0)  mounts the 'sdimage' to 'mountdir' (default: /mnt/'sdimage'/)"
    echo -e "        $(tput bold)umount$(tput sgr 0) unmounts the 'sdimage' from 'mountdir'"
    echo -e "        $(tput bold)gzip$(tput sgr 0)   compresses the 'sdimage' to 'sdimage'.gz"
    echo -e ""
    echo -e "    Options:"
    echo -e ""
    echo -e "        $(tput bold)-c$(tput sgr 0)         creates the SD Image if it does not exist"
    echo -e "        $(tput bold)-l$(tput sgr 0)         writes rsync log to 'sdimage'-YYYYmmddHHMMSS.log"
    echo -e "        $(tput bold)-z$(tput sgr 0)         compresses the SD Image (after backup) to 'sdimage'.gz"
    echo -e "        $(tput bold)-d$(tput sgr 0)         deletes the SD Image after successful compression"
    echo -e "        $(tput bold)-f$(tput sgr 0)         forces overwrite of 'sdimage'.gz if it exists"
    echo -e "        $(tput bold)-L logfile$(tput sgr 0) writes rsync log to 'logfile'"
    echo -e "        $(tput bold)-i sdcard$(tput sgr 0)  specifies the SD Card location (default: $SDCARD)"
    echo -e ""
    echo -e "Examples:"
    echo -e ""
    echo -e "    $(basename $0) rsync -c /path/to/rpi_backup.img"
    echo -e "        starts backup to 'rpi_backup.img', creating it if it does not exist"
    echo -e ""
    echo -e "    $(basename $0) rsync /path/to/\$(uname -n).img"
    echo -e "        uses the RPi's hostname as the SD Image filename"
    echo -e ""
    echo -e "    $(basename $0) rsync -cz /path/to/\$(uname -n)-\$(date +%Y-%m-%d).img"
    echo -e "        uses the RPi's hostname and today's date as the SD Image filename,"
    echo -e "        creating it if it does not exist, and compressing it after backup"
    echo -e ""
    echo -e "    $(basename $0) mount /path/to/\$(uname -n).img /mnt/rpi_image"
    echo -e "        mounts the RPi's SD Image in /mnt/rpi_image"
    echo -e ""
    echo -e "    $(basename $0) umount /path/to/raspi-$(date +%Y-%m-%d).img"
    echo -e "        unmounts the SD Image from default mountdir (/mnt/raspi-$(date +%Y-%m-%d).img/)"
    echo -e ""
}

# Read the command from command line
case $1 in
    start|mount|umount|gzip) 
        opt_command=$1
        ;;
    -h|--help)
        usage
        exit 0
        ;;
    --version)
        trace "$(basename $0) $VERSION by jinx"
        exit 0
        ;;
    *)
        error "Invalid command or option: $1\nSee '$(basename $0) --help' for usage";;
esac
shift 1

# Make sure we have root rights
if [ $(id -u) -ne 0 ]; then
    error "Please run as root. Try sudo."
fi

# Read the options from command line
while getopts ":czdflL:i:" opt; do
    case $opt in
        c)  opt_create=1;;
        z)  opt_compress=1;;
        d)  opt_delete=1;;
        f)  opt_force=1;;
        l)  opt_log=1;;
        L)  opt_log=1
            LOG=$OPTARG
            ;;
        i)  SDCARD=$OPTARG;;
        \?) error "Invalid option: -$OPTARG\nSee '$(basename $0) --help' for usage";;
        :)  error "Option -$OPTARG requires an argument\nSee '$(basename $0) --help' for usage";;
    esac
done
shift $((OPTIND-1))

# Read the sdimage path from command line
IMAGE=$1
if [ -z $IMAGE ]; then
    error "No sdimage specified"
fi

# Check if sdimage exists
if [ $opt_command = umount ] || [ $opt_command = gzip ]; then
    if [ ! -f $IMAGE ]; then
        error "$IMAGE does not exist"
    fi
else
    if [ ! -f $IMAGE ] && [ ! -n "$opt_create" ]; then
        error "$IMAGE does not exist\nUse -c to allow creation"
    fi
fi

# Check if we should compress and sdimage.gz exists
if [ -n "$opt_compress" ] || [ $opt_command = gzip ]; then
    if [ -s ${IMAGE}.gz ] && [ ! -n "$opt_force" ]; then
        error "${IMAGE}.gz already exists\nUse -f to force overwriting"
    fi
fi

# Define default rsync logfile if not defined
if [ -z $LOG ]; then
    LOG=${IMAGE}-$(date +%Y%m%d%H%M%S).log
fi

# Identify which loopback device to use
LOOPBACK=$(losetup -j $IMAGE | grep -o ^[^:]*)
if [ $opt_command = umount ]; then
    if [ -z $LOOPBACK ]; then
        error "No /dev/loop<X> attached to $IMAGE"
    fi
elif [ ! -z $LOOPBACK ]; then
    error "$IMAGE already attached to $LOOPBACK mounted on $(grep ${LOOPBACK}p2 /etc/mtab | cut -d ' ' -f 2)/"
else
    LOOPBACK=$(losetup -f)
fi
LOOPBACK1=${LOOPBACK}p1
LOOPBACK2=${LOOPBACK}p2

# Read the optional mountdir from command line
MOUNTDIR=$2
if [ -z $MOUNTDIR ]; then
    MOUNTDIR=/mnt/$(basename $IMAGE)/
else
    opt_mountdir=1
    if [ ! -d $MOUNTDIR ]; then
        error "Mount point $MOUNTDIR does not exist"
    fi
fi

# Check if default mount point exists
if [ $opt_command = umount ]; then
    if [ ! -d $MOUNTDIR ]; then
        error "Default mount point $MOUNTDIR does not exist"
    fi
else
    if [ ! -n "$opt_mountdir" ] && [ -d $MOUNTDIR ]; then
        error "Default mount point $MOUNTDIR already exists"
    fi
fi

# Trap keyboard interrupt (ctrl-c)
trap ctrl_c SIGINT

# Check for dependencies
for c in dd losetup parted sfdisk partx mkfs.vfat mkfs.ext4 mountpoint rsync; do
    command -v $c >/dev/null 2>&1 || error "Required program $c is not installed"
done
if [ -n "$opt_compress" ] || [ $opt_command = gzip ]; then
    for c in pv gzip; do
        command -v $c >/dev/null 2>&1 || error "Required program $c is not installed"
    done
fi

# Do the requested functionality
case $opt_command in
    start)
            trace "Starting SD Image backup process"
            if [ ! -f $IMAGE ] && [ -n "$opt_create" ]; then
                do_create
            fi
            do_mount
            do_backup
            do_umount
            if [ -n "$opt_compress" ]; then
                do_compress
            fi
            trace "SD Image backup process completed."
            if [ -n "$opt_log" ]; then
                trace "See rsync log in $LOG"
            fi
            ;;
    mount)
            if [ ! -f $IMAGE ] && [ -n "$opt_create" ]; then
                do_create
            fi
            do_mount
            trace "SD Image has been mounted and can be accessed at:\n    $MOUNTDIR"
            ;;
    umount)
            do_umount
            ;;
    gzip)
            do_compress
            ;;
    *)
            error "Unknown command: $opt_command"
            ;;
esac

exit 0
bkup_rpimage.sh.gz
The script creates a sparse image which only has the apparent size of the SD Card, but where empty sectors do not use disk space. For an empty, sparse image clone of a 4GB SD Card, the partition and file system overhead is only 66 MB, as can been seen by:

Code: Select all

# bkup_rpimage mount -c /path/to/sdempty.img
# bkup_rpimage umount /path/to/sdempty.img
# du -h --apparent-size /path/to/sdempty.img
3.8G    /path/to/sdempty.img
# du -h /path/to/sdempty.img*
66M     /path/to/sdempty.img
3.8M    /path/to/sdempty.img.gz
On my Raspberry Pi, the sparse image only contains my files, no empty sectors, as can be seen by:

Code: Select all

# df -h
Filesystem              Size  Used Avail Use% Mounted on
rootfs                  3.7G  1.3G  2.2G  37% /
...
# du -h --apparent-size /path/to/sdimage.img
3.8G    /path/to/sdimage.img
# du -h /path/to/sdimage.img*
1.4G    /path/to/sdimage.img
565M    /path/to/sdimage.img.gz
Performing the initial "bkup_rpimage.sh start -c /path/to/sdimage.img" (35'112 files / 1'231'978'449 bytes) took 14m44s
Performing an incremental "bkup_rpimage.sh start /path/to/sdimage.img" (11 files / 71'411'770 bytes) took 1m06s
Compressing "bkup_rpimage.sh gzip /path/to/sdimage.img" took 36m07s

Please test this and let me know if you run into any issues.

Hi jinx, great job, thanks!
I'm thinking about try your script but I'm really a newbie, how can I keep only the last 2 SD Image backup file?

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Mon Mar 23, 2015 10:39 pm

Great script.
When I run

Code: Select all

sh bkup_rpimage.sh rsync -cz /media/screenplay/Rpi_backup/$(uname -n)-$(date +%Y-%m-%d).img
I receive thi result:

Code: Select all

-e Invalid command or option: rsync
See 'bkup_rpimage.sh --help' for usage
And how can I keep only last 3 backup o something that clean old backups?

User avatar
davidcoton
Posts: 3163
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: The Raspberry Pi Backup Thread

Mon Mar 23, 2015 11:00 pm

mcgyver83 wrote:And how can I keep only last 3 backup o something that clean old backups?
Manually delete old ones, and only after checking that the newer backups are there, and at least about the right size. NEVER trust a system to delete backups automatically. Under certain fault conditions, old backups can be deleted without making a valid new one. So you end up with none, and only notice when you need it. :x :( :o :shock: etc.
"Thanks for saving my life." See https://www.raspberrypi.org/forums/viewtopic.php?p=1327656#p1327656
“Raspberry Pi is a trademark of the Raspberry Pi Foundation”

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Mon Mar 23, 2015 11:11 pm

I'm almost agree with you, but I'm talking about a not a system that I require to survive :)
So I can monitor the system for a smart period of time to check that all is working fine and after I can rely on the script.
If one day I will wake up and find that the script erased all the backup because there was a one each million days event I can survive :)

User avatar
bucker
Posts: 99
Joined: Thu May 23, 2013 4:33 pm
Location: Uberlandia, Brazil

Re: The Raspberry Pi Backup Thread

Mon Mar 23, 2015 11:22 pm

Is there any way to backup a USB install in a safe way?
Winning is for losers.

Pi Zero W (inside a Pikachu case ;) ) with DietPi running Sonarr,Couchpotato,Pi-hole,Pi-VPN
Pi B running Home Assistant
Pi 2B & HiFiBerry DAC+ running Volumio OS
Pi 3B (inside a genesis case 8-) ) running RetroPie
--

User avatar
kfonda
Posts: 6
Joined: Sat Mar 21, 2015 6:14 pm
Location: Manasquan, NJ, USA

Re: The Raspberry Pi Backup Thread

Tue Mar 24, 2015 6:54 pm

Can anybody help me with this error?

I used this command:

Code: Select all

    sudo ./bkup_rpimage.sh start -cl /home/pi/backup/$(uname -n).img
and got this error:

Code: Select all

    .
    .
    home/pi/.kodi/userdata/Thumbnails/8/864b040b.jpg
    home/pi/.kodi/userdata/Thumbnails/8/864f47ee.jpg
    home/pi/.kodi/userdata/Thumbnails/8/8653da11.jpg
    rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
    rsync: write failed on "/mnt/raspberrypi.img/home/pi/.kodi/userdata/Thumbnails/8/864f47ee.jpg": No space left on device (28)
    rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9]
    rsync: connection unexpectedly closed (265141 bytes received so far) [sender]
    rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
    Flushing to disk
    Unmounting /dev/loop0p1 and /dev/loop0p2 from /mnt/raspberrypi.img/
    Detaching /home/pi/backup/raspberrypi.img from /dev/loop0
    SD Image backup process completed.
    See rsync log in /home/pi/backup/raspberrypi.img-20150324142939.log
relevant info:

Code: Select all

    [email protected] ~/backup $ df -h
      Filesystem      Size  Used Avail Use% Mounted on
      rootfs           29G  5.3G   22G  20% /
      /dev/root        29G  5.3G   22G  20% /
      devtmpfs        365M     0  365M   0% /dev
      tmpfs            74M  396K   74M   1% /run
      tmpfs           5.0M     0  5.0M   0% /run/lock
      tmpfs           148M     0  148M   0% /run/shm
      /dev/mmcblk0p1   56M   17M   40M  30% /boot

    [email protected] ~/backup $ du -h raspberrypi.img
      980M    raspberrypi.img

    [email protected] ~/backup $ du -h --apparent-size raspberrypi.img
      30G     raspberrypi.img

    [email protected] ~/backup $ ls -hl raspberrypi.img*
      -rw-r--r-- 1 root root  30G Mar 24 14:36 raspberrypi.img
      -rw-r--r-- 1 root root 1.7M Mar 24 14:36 raspberrypi.img-20150324142939.log

jinx
Posts: 10
Joined: Wed Nov 05, 2014 11:44 am

Re: The Raspberry Pi Backup Thread

Wed Mar 25, 2015 5:15 am

You are backing up the image to your home folder, and are running out of space. You should use an external drive.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Wed Mar 25, 2015 11:32 am

mcgyver83 wrote:Great script.
When I run

Code: Select all

sh bkup_rpimage.sh rsync -cz /media/screenplay/Rpi_backup/$(uname -n)-$(date +%Y-%m-%d).img
I receive thi result:

Code: Select all

-e Invalid command or option: rsync
See 'bkup_rpimage.sh --help' for usage
Any idea about that?

jinx
Posts: 10
Joined: Wed Nov 05, 2014 11:44 am

Re: The Raspberry Pi Backup Thread

Wed Mar 25, 2015 2:42 pm

Sorry, that's a typo in the example commands lines.
The correct command should be

Code: Select all

bkup_rpimage.sh start -cz /media/screenplay/Rpi_backup/$(uname -n)-$(date +%Y-%m-%d).img

User avatar
kfonda
Posts: 6
Joined: Sat Mar 21, 2015 6:14 pm
Location: Manasquan, NJ, USA

Re: The Raspberry Pi Backup Thread

Wed Mar 25, 2015 7:01 pm

@jinx

I tried mounting an SMB share and backing up to that and got the same rsync error in just about the same place;

Code: Select all

sudo mount -t cifs -o username=sysadmin,password=EZ2guess //SERVER2012/Users/SysAdmin/RaspberryPiBackups /mnt/backup/
Then I tried changing the sparse setting like someone suggested above and tried to back up to the SMB share and still got the same error in just about the same place, never at the exact same file but after just about the same amount of data.

Do you have any other ideas I can try?

P.S. - Thanks for posting the script, It looks like it will be very helpful when I get it working.

jinx
Posts: 10
Joined: Wed Nov 05, 2014 11:44 am

Re: The Raspberry Pi Backup Thread

Thu Mar 26, 2015 10:12 am

I do not know enough about smb shares to be of any help, unless the reason is that the target file system does not support files the size of your image, such as for example FAT32 which cannot handle files larger than 4 GB

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Wed Apr 01, 2015 9:53 pm

jinx wrote:Sorry, that's a typo in the example commands lines.
The correct command should be

Code: Select all

bkup_rpimage.sh start -cz /media/screenplay/Rpi_backup/$(uname -n)-$(date +%Y-%m-%d).img
Excellent, is working now :)
About how to keep only last 3 backups?

jinx
Posts: 10
Joined: Wed Nov 05, 2014 11:44 am

Re: The Raspberry Pi Backup Thread

Thu Apr 02, 2015 6:56 am

Rotating backups is not the intent of the script and would add a lot of unnecessary complexity. This should be solved by other means such as for example using logrotate:
http://ramenlabs.com/2007/12/08/easy-ro ... logrotate/

User avatar
framp
Posts: 18
Joined: Sun Jan 27, 2013 7:03 pm
Location: Stuttgart, Germany
Contact: Website

Re: The Raspberry Pi Backup Thread

Mon Apr 06, 2015 9:37 pm

Just stumbled upon this quite old thread and the question how to restore the backup.
I have multiple raspberries backing them up on their own with a backup script I wrote. It also can be used to restore the backup. It's used by a lot of people already. Just give it a try.
"Really, I'm not out to destroy Microsoft. That will just be a completely unintentional side effect." Linus Torvalds, 28.9.2003

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: The Raspberry Pi Backup Thread

Wed Apr 15, 2015 6:57 pm

Sorry, probably I missed a step; I'm running this command:

Code: Select all

./bkup_rpimage.sh start -cz /media/screenplay/Rpi_backup/$(uname -n)-$(date +%Y-%m-%d).img
How can I restore an .img file made by the script?
Is it possibile "on the fly" restoring the raspberry I'm currently connected to (ssh)?
Via another computer I know how :)

Also the script leave the *.img file together with the gz one :)

yangxuan8282
Posts: 7
Joined: Tue Jun 16, 2015 4:43 pm

Re: The Raspberry Pi Backup Thread

Wed Jul 15, 2015 6:34 am

sorry for dig old posts
@jinx
thx,you did great work,and ur backup shell work like a charm here on my RPI
but I am wonder,can I use that bkup_rpimage.sh backup raspbian on a f2fs file system
I did have tried it,with directly replace ext4 with f2fs in your bkup_rpimage.sh
but no lucky,it doesn't work as expected
every time it reports that no space left on device,actually there are still 11G free on my card

User avatar
renierD
Posts: 3
Joined: Sat Sep 19, 2015 11:15 am
Location: South Africa
Contact: Website

Re: The Raspberry Pi Backup Thread

Fri Nov 13, 2015 1:05 pm

xXAzazelXx wrote:
CumpsD wrote:Maybe a stupid question, but how to restore the final image made by the script?
Unzip it and use win32diskimager
Does anyone know if you can copy the .img file to a larger SD card or does it have to be the same size as the one being backed up?
Pi is not only for guys.

jaynew
Posts: 3
Joined: Wed Feb 10, 2016 6:08 pm

Re: The Raspberry Pi Backup Thread

Wed Feb 10, 2016 6:19 pm

Jinx,

Not sure if you still keep up with this thread, but THANK YOU. Wow. Found your script and it does *everything* I was looking for.

Just a note for everyone who finds this thread. Rather than compressing the image file created, I've started using the Pearl script by user DeanC, found here:

viewtopic.php?f=91&t=58069

What that script does is reduce the size of a Raspbian .img file.

So while my Pi takes ~30 minutes to compress a 1GB .img file, it can instead have the size reduced with this second script in less than 5 minutes.

All you need to do is expand the file system using raspi-config after you load the image.

-Jay

GeoBePi
Posts: 3
Joined: Fri Mar 18, 2016 8:52 pm

Re: The Raspberry Pi Backup Thread

Mon Apr 25, 2016 5:50 am

I've been using rpi-clone every day for a few weeks to clone my running RPi3 running Raspian Jessie:
https://github.com/billw2/rpi-clone
It takes about 10 minutes to clone to a second Micro SD card. I then switch off, take out both cards, store the original and boot using the clone. Hence I know that the backup works. I rotate 3 cards in this way, so that I always have a safe card. It works for me ;-)
RPi3 Raspian Jessy, UPS Pico, Domoticz (C++, mySensors), Jasper (PocketSphinx, Python), Bluetooth (audio via A2DP profile), ESPduino (Arduino), nRF24L01+ (radio). Intelligent Home (Automation & Security). Aspirations ;-)

redbeardbeer
Posts: 4
Joined: Sat Jul 02, 2016 1:19 am

Re: The Raspberry Pi Backup Thread

Sat Jul 02, 2016 1:35 am

Hi @jinx

Thanks for your script - works well except for one thing - it seems to go around in loops where it copies files to the image, and then finds files in that image and copies them to itself, over and over again.

For me it also appears to be creating the image on the SD card so it ends up filling itself up really quick.

What am I doing wrong?
Below is a df:

Code: Select all

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       15183540 4638524   9873536  32% /
devtmpfs          372100       0    372100   0% /dev
tmpfs             376436       0    376436   0% /dev/shm
tmpfs             376436    5292    371144   2% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             376436       0    376436   0% /sys/fs/cgroup
/dev/mmcblk0p1     64456   21016     43440  33% /boot
tmpfs              75288       4     75284   1% /run/user/1001
/dev/sda2       14573600   36780  13773480   1% /media/hunt/4add620b-5476-4bbd-a0f2-d54dab4efb66
The I have in the script at the start:

Code: Select all

DIR=/media/hunt/4add620b-5476-4bbd-a0f2-d54dab4efb66/
NAME=$(uname -n)-$(date +%Y-%W)
IMAGE=$DIR/${NAME}.img
LOG=$DIR/${NAME}-$(date +%Y%m%d%H%M).log

SDCARD=/dev/mmcblk0p1

LOOPBACK=/dev/loop0
LOOPBACK1=${LOOPBACK}p1
LOOPBACK2=${LOOPBACK}p2

MOUNTDIR=/mnt/$NAME
Any help would be greatly appreciated!

Thanks
Hunt

redbeardbeer
Posts: 4
Joined: Sat Jul 02, 2016 1:19 am

Re: The Raspberry Pi Backup Thread

Sun Jul 03, 2016 12:35 am

OK - sorry - I stand corrected.

I ran the script that you attached in a later post viewtopic.php?p=638345#p638345

And it worked perfect.

Thank you thank you!

redbeardbeer
Posts: 4
Joined: Sat Jul 02, 2016 1:19 am

Re: The Raspberry Pi Backup Thread

Sat Aug 27, 2016 2:03 pm

@jinx

Hey

So your script seems to back up the pi as needed and create the image, but how do I restore to it?

I tried using win32 disk imager - but it doesn't seem to write anything to the boot partition and my pi doesn't boot.
I've tried re-formating with sd formatter too

Advice?

Cheers
Hunt

Return to “General discussion”

Who is online

Users browsing this forum: asavah and 52 guests