SD Card Backup and Custom Image...

[thongsia]

The cloning script looks straightforward. But what would be the restore process?

[chorlton2080]

There is no restore. It generates a usable and bootable SD Card

[billw]

I see your comment about being able to back up a booted RPI. That's what I was always worried about with using "dd". It sounds like you're saying this script is safe to run on a booted/mounted drive. Is that right? I'll try to answer that question for myself too. Thanks!

Justin

I'm not sure I would want to do a dd for backup on a booted "active" file system and rpi-clone doesn't do that. It does use dd to get the first vfat boot partition where the boot files are, but that file system is very unlikely to have open inodes, etc. Then the root partition is backed up using rsync which simply captures the state of a file when it is copied from the booted file system to the backup SD card and it is not disk imaging the root partition. That's why rpi-clone can back up to different sized SD cards. I would try not to be doing anything else on my system when I run rpi-clone, but if a log file or so gets updated while it is running, it only means the file backed up may immediately be out of date. I also frequently run rpi-clone in a xfce4 terminal after startx so I don't think you have to be careful about that. I have a similar clone script I wrote a long time ago to back up my main linux box from hard drive to USB backup drives and have never had a problem.

If you want to be paranoid, I would say just have more than one backup SD card

[davecollinsjr]

Thanks for this utility!

One issue that plagued me was invalid line endings. Tried twice but I kept getting "command not found \r". This solved it:

sudo apt-get install dos2unix
dos2unix rpi-clone.sh

Maybe that will help someone else!

[Shoka]

Another happy customer .

Very nice tool

[molecules]

Hi, thanks for the script. After backing up to an external drive should I just use dd to write the image to another SD card if I need to restore?

[molecules]

oops, I see. Just read the readme and realized that this creates an image directly on an SD card. Just bought a usb SD card cable

[socialdefect]

I notice a lot of misconceptions on the forums on how to clone SD cards and creating backups.

First of all there is a big difference between cloning a disk and creating a backup.

Cloning a disk is done by using a bitcopy tool like dd or cat. With these tools you can clone a partition or a complete disk incl. the partition table to a second disk or raw image file. this can only be executed on an inactive device, so none of the partitions on the disk may be mounted. dd will not start the cloning process if the device is mounted so a disk cannot be corrupted by dd if it's mounted for dd will give an error message and will exit if the device is in use.
The dangerous part about dd is typo's. If you want to clone to sda1 for example and you type sda then you'll have a problem for it will erase your entire partition table on /dev/sda instead of cloning into partition sda1.
You can prevent most stupid mistakes by using a GUI tool like Win32DiskImager or PiWriter.

Using cp or rsync to create a backup of an operating system can be done but it's not the recommended way, specially not when cloning the OS on which you are running the sync tool. Also you need to do a lot of debugging, excluding files and re-creating files, links and permissions in order to create a proper OS backup using rsync. When using a bitcopy tool you won't have to worry about any of those things.

So if you want to create a backup of your personal files or configuration rsync is the way to go. If you want a proper clone of your disk better trust in dd. BUT; when using the dd command check your syntax at least 3 times or use a GUI tool to help you prevent typo's. Also always compare the disk checksum with the checksum of the clone like this:

Code: Select all

md5sum /dev/sda
md5sum /dev/sdb

If both commands give the exact same output hash you've created an exact clone.

[RaTTuS]

I notice a lot of misconceptions on the forums on how to clone SD cards and creating backups.

First of all there is a big difference between cloning a disk and creating a backup.

Cloning a disk is done by using a bitcopy tool like dd or cat. With these tools you can clone a partition or a complete disk incl. the partition table to a second disk or raw image file. this can only be executed on an inactive device, so none of the partitions on the disk may be mounted. dd will not start the cloning process if the device is mounted so a disk cannot be corrupted by dd if it's mounted for dd will give an error message and will exit if the device is in use.
The dangerous part about dd is typo's. If you want to clone to sda1 for example and you type sda then you'll have a problem for it will erase your entire partition table on /dev/sda instead of cloning into partition sda1.
You can prevent most stupid mistakes by using a GUI tool like Win32DiskImager or PiWriter.

Using cp or rsync to create a backup of an operating system can be done but it's not the recommended way, specially not when cloning the OS on which you are running the sync tool. Also you need to do a lot of debugging, excluding files and re-creating files, links and permissions in order to create a proper OS backup using rsync. When using a bitcopy tool you won't have to worry about any of those things.

So if you want to create a backup of your personal files or configuration rsync is the way to go. If you want a proper clone of your disk better trust in dd. BUT; when using the dd command check your syntax at least 3 times or use a GUI tool to help you prevent typo's. Also always compare the disk checksum with the checksum of the clone like this:

Code: Select all

md5sum /dev/sda
md5sum /dev/sdb

If both commands give the exact same output hash you've created an exact clone.

dd can and will work on mounted devices,,,
it is possible to dd a live SD card to a remote location or mounted SD or hd or usb stick
and it is possible you will get a good working image [YMMV]
dd in the wrong direction will make a mess of your day [but so will rysnc or many other tools]

dd may not make an image that you can easily fit onto another device - not all similar sized SD cards are the same size [exactly]
useful options on dd are :-
conv=sync,noerror
so things will appear in the right locations after the fact if there are errors

bs=64M
so not as to default to the very small defaul movement size - 64M is probably overkill size wise and 8M may be just as good.

rsync is my preferred way - once you have it setup and know what you are doing - and have tested it it is great for complete images,
however keeping data and live os images separate is a good practice and you should start now. - this way your data can be restored to a completely different os system very easily
windows , mac , other versions of linux

[Nocturnal_Nick]

Hi all,

Just like to thank OP for what looks to be an incredibly useful script, one which I'm sure I'll use very often as I toy around with rbPi.

I've managed to get it to create the disk, it gets all the way to be unmounted, which I hit enter for. I then power off my Pi and put the cloned SD card into my main SD card slot, but the Pi won't boot!

I gett an error very similar to this:

Code: Select all

VFS: Cannot open root device "mmcblk0p2" or unknown-block (179,2)
Please append a correct "root=" boot option; here are the available partitions:
b300 7846912 mmcblk0 driver: mmcblk
b301 7839688 mmcblk0p1 00000000-0000-0000-0000-000000000000
Kernel Panic -not syncing: VFS : Unable to mount root fs on unknown-block(179,2)

though not identical, I've just not been able to see the error for very long (borrowed an HDMI display from work and copied down the basics while I had it)

What am I doing wrong? I've run the script twice, and all I can see that I might be doing any differently is the part that allows you to label a partition (I believe). First I typed SDBackup as the label, then the second time trying I typed nothing (just hit enter) but encounter the same issue :-S.

It probably goes without saying that I'm relatively new to Pi, so any help I can get would be great!

[martinmarty]

Help! How do I know what to specify for "sdN"? I have my card reader attached and a spare SD card inserted but I don't want to make a mistake and trash my running system. I was also thinking of using the -f option so it would set up the destination card as needed because I have junked up a bunch of cards trying to clone using other tools.

When I do ls /dev, among other things, I have:
sda
sdb
sdc
sdc1
sdd
sde

How do I know what to use for the destination?

THANKS,
-Marty

[DougieLawson]

fdisk -l
will list the partitions

mount
will tell you which are mounted

dmesg
will tell you which device appeared when a new hardware is attached

Use all three and it should be obvious which devices are the SDcard.

[martinmarty]

Thanks, Dougie!

fdisk -l put me right onto sdC, and I've already made and tested one full backup.

This script is wonderful. I've been fighting for numerous sessions over the past couple weeks of how to make backups of a full working Pi application and having no luck because all the imaging methods require SDcards of exactly the same size.

This method solves my problem and it's fast, like 12-15min for a backup as opposed to 45 for a 32GB card with win32diskimager.

Thanks, billw!

[jkw]

Hi, I've used this very very nice backup script quite a while on my raspberrys.
Right now I've added a banana to my fruit collection and used it there as well.

Running Bananian will not automatically mount the boot partition so I've modified the script to mount it.
After starting my modification I've created a fork and added some more functionality. My problem was that I don't have enough SD cards to have a "running" and a "backup" card for each pi, but I do have lots of unused usb flash drives.

So my idea was to use a flash drive instead of a SD card. The problem is obviously that, once I've messed up my OS, I can't boot from the USB flash drive, so I needed a kind of "restore mode". That's why I've added an additional mode to the script.

So calling it "pi-clone sda" will copy everything to my USB-flash drive (/dev/sda). After destroying my system I just remove my SD card go to another pi/pc/whatever, insert the backup USB-flash drive and the messed up SD-Card (via cardreader) and run "pi-clone sda sdb" to restore everything. (Just ensure that the first drive (in the case "sda") will be source and the second destination for your files).

In addition to that I'd like to run my backup every night, just to store my changes. To do that if added a switch "-y / --yes-to-all" to answer every upcoming question with "yes" instead of waiting on a user input. This made it compatible to my cronjob.

@Bill: I've tried to write a message to you, actually I've just created this user account here to do that, but the forum won't allow me to send private messages while having 0 posts ... therefor this post.
Is it ok for you, if i "publish" my script and share the link? Its of cause 95% your work, therefore I ask
So please take a look at https://github.com/KoljaWindeler/pi-clone and if you don't like that, I'll remove this post

This script is still compatible to your syntax, calls, etc. I've just renamed it (was kind of odd to use a "rpi" script on my "bpi" )
JKW

[NoToy]

Can this script be used for an out of the box noobs - PI?
I can see from /dev there is no sdN.
What I am looking for is a method to do a backup, which if I have a failure, then I can setup a new system with NOOBS, either on that card or a new one, and then restore the backup on top of it.
The backup/restore ideally coming off network drive like /media/blah....
I did try the following
Making a backup:
sudo dd bs=4M if=/dev/root of=/media/pie/root.img
Restoring the backup:
sudo dd bs=4M if=/media/pie/root.img of=/dev/root

Not very cutting edge, and unfortunately after the restore the file system had some problems, so obviously restoring to /dev/root whilst mounted is not great

Simple question, would the script work, and if so, do I have to attach an sdcard to do backup onto/restore from?
Thanks
Tony

[martinmarty]

p.p.p.s. ALL MY FAULT. After I made my first backup today, I was getting ready to make a second and I plugged in my USB 3.0 card reader to RPi while it was booting on backup #1. RPi doesn't take kindly to this USB 3.0 device and immediately crashes and coldstarts. I did not test XWindows (incorrectly assuming everything was fine)and proceeded to make backup #2, an rpi-clone of backup #1 which had apparently been corrupted by my self-induced crash. So rpi-clone caused none of the problems and made a perfect broken copy of my broken copy.

I'm leaving this post instead of deleting it just in case anyone gets the same error.
==========
Problem: When I made backups on 9-27 using rpi-clone, they worked great.

Backups I made today have a problem. When I boot Raspbian, it presents a login prompt and I login as pi (or root, doesn't matter) and then do startx. It doesn't work:

/usr/bin/X: error while loading shared libraries: /usr/lib/arm-Linux-gnueabihf.so.0: invalid ELF header
xinit: giving up
xinit: unable to connect to Xserver: Bad file descriptor
xinit: server error

Startx used to work. It works on the SDHC card that was the source of the copy.

I've googled a little but I'm not making much headway.

All help appreciated.

Thanks.

p.s. Could this be cause by my answers to rpi-clone's questions about unmounting things? I do not recall it asking those questions the first time I used it. When prompted, I told it yes, unmount. Wrong answer?

p.p.s. I could be mistaken. I might have caused this rather than something weird with rpi-clone. Will post more later after I can test the theory.

[pksm11]

Hi,

I'm having problems unmounting the cloned sd card. The script works great until the "hit enter when ready to unmount the...".
Should I wait for it?



Thanks.

[andig2]

When booting the cloned image I'm receiving various Input/Output errors and the image does not come up. fsck complains about unexpected inconsistency.

Any ideas what I might be doing wrong?

[Jeppedy]

So I have a card where one of my partitions seems to have some structural issues. It boots and runs just fine, but when backing up with rpi-clone, it complains about the SuperBlock. Also, booting from the new card does not work, complaining of the card and file system.

The partition with the issue is: /dev/mmcblk0p02

What I'd like to do is this:

1. Create a new, fresh Noobs card
2. Configure fresh for Raspbian
3. Remove the New card and boot from my current working card
4. Copy all OS and user files to the New card, but ignoring the partition structures

Seems to me, that will give me a clean and valid partition structure and file system, but with my current content and OS configuration.

Feels to me like rpi-clone is the closest to what I need to do, but I don't know how to stop it from making a whole new partition structure on the new card.

Here's the partition information on the existing card:
Model: SD SDU1 (sd/mmc)
Disk /dev/mmcblk0: 15.8GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 4194kB 662MB 658MB primary fat32 lba
2 663MB 8015MB 7353MB extended
5 667MB 730MB 62.9MB logical fat32 lba
6 734MB 8015MB 7281MB logical ext4
3 8015MB 8049MB 33.6MB primary ext4

Can anyone help me in my quest?

[RaTTuS]

forget noobs "who needs it"

take a new SD card and partition it up how you like
e.g
sudo bash
#assuming your sdcard is found as /dev/sda

fdisk /dev/sda
create 2 partitions
one of type c [fast32 lba] - size approx 100M
one of type 83 [linux] - the rest of the SDcard
write the changes
then make the file systems
mkfs.vfat /dev/sda1
mkfs.ext4 /dev/sda2 #this can be ext2

#assuming this is being done on a raspberry then
#copy filesytems

mount /dev/sda1 /mnt
cp -ax /boot /mnt
sync
umount /mnt
mount /dev/sda2 /mnt
cp -ax / /mnt
sync
umount /mnt

exit
...


now if you are happy with the image as a backup then great what you can do in the future is to update the backup to a new image by

sudo sh
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot
rsync -ax /boot/ /mnt/boot
rsync -ax / /mnt
sync
umount /mnt/boot /mnt

[Jeppedy]

THANKS for all the help!

forget noobs "who needs it"

I have no problem with that. It's just how I started and was stuck with it. I'm new to Linux, so I just need to confirm a few things so I understand...

• I'm new to the bootloader issues. Will this "cp" approach be an issue with the previous NOOBS boot process?
• I've got more experience with Windows system work than Linux. Is the "cp" command you show sufficient to move absolutely everything to the new card?
• In your commands, you don't change the source location for the second "cp". I assume that's just an oops and I should change to the root of the Root filesystem before the second "cp". Yes?
• If desired, could I just use the rsync from the very beginning, or is there some reason I should be using the cp command initially?

Thanks again for both confirming my hunch about just copying the needed partitions as well as specifics on how to partition the new SD card.

[electronicsguy]

Isn't it true that using dd to copy an active filesystem can lead to data corruption? I've read this in several places and it seems to make sense as some of the files will be open at the time (and possibly being written to). This tool looks great but I want to make sure I'm backing up my Pi in a way that won't cause data corruption.

Yup. I have experienced this. IMHO, you must unmount before making a clone or restoring to a partition.

I personally use a mini usb drive to store my root partition and all data, as I dislike the low speeds and unreliability of sd-cards. I just clone my data (including root fs) from the USB drive to my laptop whenever I want to. a simple 'dd' after unmounting suffices. Same for restore. just use the standard 'dd' to restore the backup to another sd card/usb drive partition.

[davehiggins2170]

I've been using HDD Raw Copy to backup my Pi images but it's started failing with error 8 and a message about lock handle. I can't remember whether I've used it since I went to windows 10 as I've not had time. to do much on my pi since then. Could it be w10 that's causing it to fail? I will try rpi-clone when I get time but I really like HDD raw copy.

Thanks for any help Dave

[jellis]

I just want to say THANK YOU to billw for tackling this horrendous task and giving the community a simple tool to fix a desperately aggravating problem. I wish billw had a donation page; that is how grateful I am!

All I wanted to do was replace the card in a pi that runs my house because I'm in a hot part of France and the SD card was failing in the heat, so I bought an industrial SD card of "the same size" and you all know how the rest of the story goes!!

Anyway, billw, you are a star; thank you so much. The world of Pi is a richer place for your thorough endeavours.