ctrl
Posts: 9
Joined: Sat Jul 07, 2012 6:21 pm

Faster sdcard backup&restore

Sat Jul 07, 2012 6:25 pm

I was wondering... rather than using gzip and dd to backup and restore my 16 gb SD, should it be possible to use clonezilla tools like partimage and partclone and so on to just clone partition table and used partition space?

ctrl
Posts: 9
Joined: Sat Jul 07, 2012 6:21 pm

Re: Faster sdcard backup&restore

Sun Jul 08, 2012 9:15 am

I did some tests, the following procedure works on Natty (partclone was built from source with fat and extfs support). I have my card reader listed as /dev/sde, you have to change it accordingly.

To backup:
  • Unmount everything
  • Backup mbr
  • Backup boot partition
  • Backup system partition

Code: Select all

umount /dev/sde?

dd bs=512 count=1 if=/dev/sde of=/path/whatever/mbr.img

partclone.fat   --clone --source /dev/sde1 --output /path/whatever/sde1.img
partclone.extfs --clone --source /dev/sde2 --output /path/whatever/sde2.img
To restore:
  • Unmount everything
  • Restore mbr
  • Restore boot partition
  • Restore system partition
  • Set up swap partition

Code: Select all

umount /dev/sde?

dd bs=512 count=1 of=/dev/sde if=/path/whatever/mbr.img
sync
partprobe

partclone.restore --output /dev/sde1 --source /path/whatever/sde1.img
partclone.restore --output /dev/sde2 --source /path/whatever/sde2.img

mkswap /dev/sde3
Unplug and it should work :)

You can of course add gzip in the middle to compress the image on the fly.

ctrl
Posts: 9
Joined: Sat Jul 07, 2012 6:21 pm

Re: Faster sdcard backup&restore

Sun Jul 08, 2012 1:25 pm

Here are two scripts for backup and restore of a "standard" raspberry sd card (I only tested the stable debian, other distros might have a different partition layout), with optional compression. I tested and they work, but I take no responsibility :)

Usage is:
  • rpi-backup [-c] [-a] -i /dev/sdX -o /path/to/image/folder
    rpi-restore [-a] -i /path/to/image/folder -o /dev/sdX

    -c option activates compression
    -a options disables confirmation (careful)
When creating a backup, output folder must not exist.
Attachments
rpi-backup-restore.tar.gz
(1.46 KiB) Downloaded 1306 times

User avatar
AndrewS
Posts: 3625
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
Contact: Website

Re: Faster sdcard backup&restore

Thu Jul 12, 2012 12:09 am

Nice, this is something I had on my "todo list" but hadn't got round to yet http://www.raspberrypi.org/phpBB3/viewt ... one#p86421

This would make a great guide on the wiki :) http://elinux.org/RPi_Guides
Could you do some benchmarks showing how much faster it is than a regular 'dd'?

TarjeiB
Posts: 157
Joined: Thu Jul 12, 2012 3:33 pm

Re: Faster sdcard backup&restore

Thu Jul 12, 2012 3:36 pm

Thanks for the rpi-backup script, that works really well!

I was thinking though, for improving speed and ease - wouldn't it be possible to run this script ON the RPi itself after remounting read-only somehow? It could dump output to anything elsewhere - rcp comes to mind, or even a cifs share.
Afterwards you could reboot, or maybe even just remount read-write for resumed operation.

Anyone know why this would be a really bad idea before I start fiddling with it?

User avatar
AndrewS
Posts: 3625
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
Contact: Website

Re: Faster sdcard backup&restore

Thu Jul 12, 2012 3:59 pm

TarjeiB wrote:Anyone know why this would be a really bad idea before I start fiddling with it?
Without lots of jiggery-pokery, it's difficult to remount an in-use root filesystem as readonly. AFAIK ;)

ctrl
Posts: 9
Joined: Sat Jul 07, 2012 6:21 pm

Re: Faster sdcard backup&restore

Thu Jul 12, 2012 9:40 pm

Here are my benchmarks, with 16 GB Samsung class 6 micro sd (MB-MSAGA), Intel Q9450, Transcend TS-RDP8K usb card reader (so many useless info... :) )

The sd card contains the stable debian image (1.8 GB I think), but the system partition has been extended to 16 gb (shouldn't change anything)

dd without compression: 13 mins and 29 secs
dd with compression: 16 mins and 39 secs
script without compression: 1 min and 28 secs
script with compression: 4 min and 48 secs

With dd I forgot to use bzip2 and used gzip, that is generally faster.
For restore, timing should be proportional to read/write ratio of your sd card.


And about remounting in read only the root partition, I did a quick try via ssh and got disconnected when I tried to kill all the processing locking / :roll: ..... will retry with a kybd attached :)

User avatar
AndrewS
Posts: 3625
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
Contact: Website

Re: Faster sdcard backup&restore

Thu Jul 12, 2012 11:51 pm

Thanks, quite impressive benchmark figures :)

ctrl
Posts: 9
Joined: Sat Jul 07, 2012 6:21 pm

Re: Faster sdcard backup&restore

Fri Jul 13, 2012 11:35 pm

I played a bit with the hot backup: dd works, partclone doesn't seem to... I get many ugly errors... it complains about beign unable to write a log file (because / is read-only), and then that /boot is mounted (even if it's read only) and so on and then segfaults. Whatever... maybe wheezy package works better, but I'm stuck with stable.

This is what I did to dump with dd via ssh on a remote server... I won't go into details, I suppose that if you are reading this you're able to understand or use google :)

Login once from rpi on target machine via ssh to avoid being asked about the known hosts afterwards, when you are in read-only mode on / and cannot save known hosts file.

From rpi console (doesn't work with ssh) get root access and set (if you never did before) a password for root account, then go to single user mode (you'll be asked root pw) and remount root fs as read only

Code: Select all

sudo -s
passwd root
init 1
mount -o ro,remount /
Now you can safely dump you sd card, that should be /dev/mmcblk0, and pipe it to ssh with this syntax:

Code: Select all

dd bs=1M if=/dev/mmcblk0 | ssh [email protected] 'dd of=/remote/path/to/sdcard.img'
You'll be asked for user password and dd on the other side will write everything to sdcard.img. You can insert gzip or bzip2 in the middle, I'd suggest on remote side, because rpi cpu is slower:

Code: Select all

dd bs=1M if=/dev/mmcblk0 | ssh [email protected] 'gzip -9 > /remote/path/to/sdcard.img.gz'
This way I get a 2 MB/s speed, and the bottleneck seems to be rpi card reader or memory. It's 10 times slower than dd with the usb card reader.

Remember that is you want to interrupt dd, you have to kill it (or gzip) on the remote machine, ctrl-c on rpi won't work.

User avatar
AndrewS
Posts: 3625
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
Contact: Website

Re: Faster sdcard backup&restore

Mon Jul 16, 2012 1:17 am

There's also a bit of discussion about using dd for live backups in this topic: http://www.raspberrypi.org/phpBB3/viewt ... 63&t=10366
From what's being said in that thread, sounds like dd works without needing to make the filesystem readonly :?: (but I guess it could lead to inconsistent results?)

ctrl
Posts: 9
Joined: Sat Jul 07, 2012 6:21 pm

Re: Faster sdcard backup&restore

Mon Jul 16, 2012 3:54 am

Yep it works, I once did a dd image of my debian server while it was running, to recover a deleted file later by mounting the image as loop device, because I did't want to shut it down. But if a file is deleted or grows, for instance, and it's relocated to reduce fragmentation, you can have unpredictable results.

AndrewJohnson
Posts: 21
Joined: Mon Jul 30, 2012 7:34 am
Location: Derbs UK

Re: Faster sdcard backup&restore

Mon Jul 30, 2012 8:11 am

Wow - these posts reminded me of how powerful the command line tools are....

If you just want to back up and restore your card directly, rather than over a network, try this:

After hours of trying different methods, I worked out the best way ( I think) of backing up then expanding your Boot SD-Card root partition. However, you need the raspbian wheezy or later image.

1) Download the win32diskimager http://www.softpedia.com/get/CD-DVD-Too ... ager.shtml (you may have already used this to create your bootable Pi SD-Card if you're like me - or use Mac equivalent)

2) Put your Pi SD-Card back into your PC read.

3) Run Win32DiskImager and select your memory card and set your file name to store your card image in (use the little folder icon to bring up a file selection dialogue)

4) Select "Read file" - and leave for about 10 mins or more...

5) You now have a snapshot of your card!

6) If you are using the debian-wheezy image, then either open a terminal window, or SSH in from your PC and type

Code: Select all

sudo raspi-config
Then just select

expand_rootfs Expand root partition to fill SD card

and it does all that "clever stuff" (that I could never get to work myself) for you!

I hope this saves someone the hours that I spent on this!

ctrl
Posts: 9
Joined: Sat Jul 07, 2012 6:21 pm

Re: Faster sdcard backup&restore

Mon Jul 30, 2012 4:52 pm

Yyyyes, but what if you now want to backup it again, after you expanded? That will take hours to image with dd or similar and will produce a 16 gb file in my case, instead of 1.8gb uncompressed... here comes partclone.

TarjeiB
Posts: 157
Joined: Thu Jul 12, 2012 3:33 pm

Re: Faster sdcard backup&restore

Tue Jul 31, 2012 12:03 pm

ctrl wrote:I played a bit with the hot backup: dd works, partclone doesn't seem to... I get many ugly errors... it complains about beign unable to write a log file (because / is read-only), and then that /boot is mounted (even if it's read only) and so on and then segfaults. Whatever... maybe wheezy package works better, but I'm stuck with stable.
/boot can be unmounted while the RPi is running, so that's fine.
The init 1 way is brilliant! Just wish we could get partclone to work with it. Maybe partimage?

I can't quite toss away the idea of an effective hot-backup - I mess around with the contents of my RPi so much it'd be nice to have a frequent return point.
A couple of ideas:

How about a partition table backup, /boot backup, then just a contents backup? (tar). Presents a potential restore problem though.

"Dual boot"? A script that will first unmount and backup /boot, then change to another kernel/os/setup that only uses /boot (or a small USB stick), then backup the second partition, then reboot back to RPi.

Some automated way of backup/restore without actually physically accessing the RPi would be awesome. Will play some more with it for sure. Oh, using netcat instead of ssh seems to be giving me a bit better speeds too.
Last edited by TarjeiB on Tue Jul 31, 2012 12:12 pm, edited 1 time in total.

User avatar
AndrewS
Posts: 3625
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
Contact: Website

Re: Faster sdcard backup&restore

Tue Jul 31, 2012 12:11 pm

TarjeiB wrote:"Dual boot"? A script that will first unmount and backup /boot, then change to another kernel/os/setup that only uses /boot (or a small USB stick), then backup the second partition, then reboot back to RPi.
Haha, great minds think alike :-D http://www.raspberrypi.org/phpBB3/viewt ... 12#p136912

AndrewJohnson
Posts: 21
Joined: Mon Jul 30, 2012 7:34 am
Location: Derbs UK

Re: Faster sdcard backup&restore

Wed Aug 01, 2012 6:27 pm

Aha...
/boot can be unmounted while the RPi is running, so that's fine.
I think it's the /boot reference I needed... I tried various commands shown in tutorials - but I couldn't get them to work.

Also, when trying to re-partition using Gparted, I probably have old versions of Gparted which don't 100% work with USB cards or something....

AndrewJohnson
Posts: 21
Joined: Mon Jul 30, 2012 7:34 am
Location: Derbs UK

Re: Faster sdcard backup&restore

Wed Aug 01, 2012 10:56 pm

ctrl wrote:Yyyyes, but what if you now want to backup it again, after you expanded? That will take hours to image with dd or similar and will produce a 16 gb file in my case, instead of 1.8gb uncompressed... here comes partclone.
You might think so... but I just re-saved my 8 Gig card and it took about 10-15 mins with Win32 DiskImager - perfectly acceptable to me - try it yourself...

(it was showing a read speed of 11MBs/sec so maybe it doesn't read the empty space, I don't know - as it was it had about 2.1 GBs of space used, perhaps).

ctrl
Posts: 9
Joined: Sat Jul 07, 2012 6:21 pm

Re: Faster sdcard backup&restore

Sun Aug 05, 2012 9:20 am

Yes Andrew that's the point, I don't find acceptable 13 mins for my 16 gb card, I like 1.5 mins much more :)

POPEYE
Posts: 20
Joined: Fri Jul 27, 2012 2:31 pm

Re: Faster sdcard backup&restore

Fri Aug 17, 2012 4:36 pm

I've shrinked the partition on 16GB SD Card
Can I now make image using dd with count parameter to make an image of only boot and data partitions?

Can I use dd to backup SD on raspberry itself, or i need to put SD to another system?
Can I save image to the third partition of SD?

ctrl
Posts: 9
Joined: Sat Jul 07, 2012 6:21 pm

Re: Faster sdcard backup&restore

Sun Aug 19, 2012 8:29 pm

POPEYE wrote:I've shrinked the partition on 16GB SD Card
Can I now make image using dd with count parameter to make an image of only boot and data partitions?

Can I use dd to backup SD on raspberry itself, or i need to put SD to another system?
Can I save image to the third partition of SD?
IMHO you can't, because dd is going to image everything on the sd, also the "blank" space after your shrinked partition. You could play with partition size and dd only the useful part, but since partimage was written for that, I don't see why one shouldn't use it.

POPEYE
Posts: 20
Joined: Fri Jul 27, 2012 2:31 pm

Re: Faster sdcard backup&restore

Mon Aug 20, 2012 11:37 am

IMHO you can't, because dd is going to image everything on the sd, also the "blank" space after your shrinked partition. You could play with partition size and dd only the useful part, but since partimage was written for that, I don't see why one shouldn't use it.
I've told that I'm planning to use count parameter to prevent it from copying "blank" space

I would like to copy everything from the beginning of the disk (including mbr and so on) till the total end of mmcblk0p2 and It would be great if I could save that to the mmcblk0p3 (I've just thought that there would be some problems with partition table when I write it back from image as there would be information about mmcblk0p3 too).
Attachments
Gparted-img.png
Gparted-img.png (37.79 KiB) Viewed 15505 times

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: Faster sdcard backup&restore

Wed Oct 24, 2012 3:12 pm

I'm interested in this topic, I'm also subscribed to this thread http://www.raspberrypi.org/phpBB3/viewt ... 21#p201421 because I'm interested in live backup.
From my understanding using dd will create an image big as the real partition (take off compresssion) while partimage and partclone will create an image big as the real allocated space.
We still have the problem of "live backup", isn't it?

Der_Gute
Posts: 36
Joined: Wed Aug 15, 2012 6:14 am

Re: Faster sdcard backup&restore

Mon Dec 03, 2012 2:20 pm

Hi,
i didnt like the method to hot backup so i developed an own logic to solve that issue :D.
i have a monitoringserver that can turn the power of the raspberry on and off.
When the raspberry boots (not because someone typed "reboot") the raspberry changes its cmdline.txt to a nfs root filesystem and on next boot (when not rebooted manually) the raspberry boots from nfsshare.
So if the monitor recognizes that raspberry does not react to ping or ssh or whatever, the monitoringserver turns the raspberry off and on again.
This time the raspberry boots from nfs because noone did a reboot manually.
The nfs rootsystem has 2 functions: run the services that run with the raspberry rootsystem on sdcard to provide availability and flash an definitely working image on the sdcard of raspberry.
so i run a crontab with nfs rootsystem which flashes the sdcard, checks the checksum of sdcard with img and then reboots the system to sdcard root filesystem.

It works pretty fine!

Kind regards
Der_Gute

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: Faster sdcard backup&restore

Mon Dec 03, 2012 2:36 pm

so the core of your system is the monitoringserver, the modified cmdline.txt and the crontab script; please can you post them in the forum?
When the process starts? when the raspberry wasn't rebooted manually,but how?
This approach doesn't solve the "img size" problem but made a perfect sd backup, isn't it?

Der_Gute
Posts: 36
Joined: Wed Aug 15, 2012 6:14 am

Re: Faster sdcard backup&restore

Tue Dec 04, 2012 2:35 pm

I am using a Monitorscript, which isnt finalized yet, so i wont post it. It checks in a loop whether different ips are reachable, or such services(if you want). If not it sends an email and performs actions.
i turn off the power of the backuppi with the following solution:
http://www.raspberrypi.org/phpBB3/viewt ... p?p=157647
I turn off/on the raspberry via sockets that are controlled via a transmitter plugged on the monitoring server.
When i perform a sudo reboot, the server performs init 6 and so some services are shut down.
I added some code to a service that will be shut down to mod the cmdline.txt to boot from sdcard.
When my Monitoringserver turns the power off and on, the service wont be stopped and the cmdline.txt wont be modified -> next boot from nfsshare rootfs.
Because of the fact that the root fs on nfsshare is just the filesystem of the filesystem on sdcard the services will run normally while restoring backup on rpi. To submit changes to the nfsshare rootfilesystem i use a crontab which daily copies the maybe changed configurationfiles from the sdcard rootfs to the nfsshare rootfs. You can enhance this procedure if you want.
Okay here you go:
This are the files of monitorserver:
This is a init.d script which will be performed when the nfsshare root filesystem on monitoring server(please take care because i use a rpi for it , so dont wonder about /home/pi=)) boots:

Code: Select all

#! /bin/sh
### BEGIN INIT INFO
# Provides:          changeboot
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Changes the Source from boot system
# Description:       Changes the Source from boot system
### END INIT INFO

sudo rm /boot/cmdline.txt
sudo echo "dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait" > /boot/cmdline.txt
then the crontab, which can be set by sudo crontab -e. It will be performed every 15 Minutes:

Code: Select all

15 * * * * sudo /home/pi/restorescript.sh
Then the restorescript which will be performed by crontab. it will only be performed when a specific file was not created(it will be created before backing up so dd cant run twice). Take care, all these operations will be performed on the backup raspberry which has booted nfs share of the monitoring pi:

Code: Select all

#check if  this script already runs. If it does this script will not be performed again
#check if specific file exists, that will be only created when the script is running
if [ ! -f /home/pi/scriptrunning ]
#nothing happens when scriptrunning already exists
then
sudo echo "hallo">/home/pi/scriptrunning
# unmount the /boot partition so the image can be flashed
sudo umount /dev/mmcblk0p1
sudo dd bs=4M if=/image.img of=/dev/mmcblk0
sudo rm /home/pi/scriptrunning
# the img has set boot into sdcard so if a reboot will be performed the nfs share wont be used anymore until next failsave
sudo reboot

fi
Add this do your /home/pi/.bashrc to symbolize you boot in failsave mode:

Code: Select all

echo "###############################################################"
echo "#################RPI IS IN FAILOVER MODE#######################"
echo "###############################################################"
When you boot on nfs share and log into ssh this will be displayed.

Now the Backuppi:
the init.d script which will be performed on each boot:

Code: Select all

#! /bin/sh
### BEGIN INIT INFO
# Provides:          changeboot
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Changes the Source from boot system
# Description:       Changes the Source from boot system
### END INIT INFO

sudo rm /boot/cmdline.txt
sudo echo "dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 ip=192.168.2.109:192.168.2.107:192.168.2.1:255.255.255.0:rpi:eth0:off root=/dev/nfs nfsroot=192.168.2.107:/mnt/usbstick/,vers=3 rw rootwait"> /boot/cmdline.txt
Add these lines to a service of your choice that will be stopped on init6. Take care: the service has to be performed BEFORE the local Filesystem is gonna be umounted:

Code: Select all

echo "because of the fact that the pi was restarted manually the pi wont go in failovermode"
        sudo rm -f /boot/cmdline.txt
        sudo echo "dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait" > /boot/cmdline.txt
Ask if you like to :).

Kind regards

Return to “Advanced users”

Who is online

Users browsing this forum: No registered users and 5 guests