Backup SD

[xalixo]

Imagine this example:

You have a 32GB SD card.
You are using 1.8GB of space on it.
You want to back it up into an image that is 1.8GB big, not 32GB big.
Then you want to be able to put it on an SD card that is 2,4,8 or 16GB big.

If you do a normal image backup, you'll end up with an image that is 32GB big. As a backup that's a lot of wasted of space. If you just want to store it as a backup you could gzip it, which would remove the wasted space, but you still won't be able to put it on a smaller SD card in a bootable format.

If you do a backup by copying the files, you'll end up with an SD card that won't boot unless you create different partitions that are in the correct filesystem format and make sure each file is in the correct partition. It's possible you could miss some hidden files and end up changing file permissions which you'd have to chmod back, so this way wouldn't work without a lot of messing about.

Thanks to help from "milhouse" on the forum, I now know how to make an image of the SD card that's the same size as the actual space being used. This cannot be done on the pi itself, you'll need a seperate linux machine.

Using Gparted (or parted), reduce the size of your main partition on the SD card to remove most of the empty space (You'll probably want to leave a little free space, maybe a few megabytes), but do not reduce the size of the other partitions. Then use Gparted to make sure each partition runs consecutively from the start without any unallocated space in between them. Take note of the total size all the partitions are using up and the location of your SD card (dev/mmcblk0 for example).

Next we're going to use dd. You need to be very careful when using dd, because you could end up wiping your hard drive if you mistype something.

Now run the command:(See below on how to adapt)

Code: Select all

dd if=/dev/mmcblk0 of=/home/YOUR_USERNAME/Desktop/backup.img bs=1M count=2048

"if=/dev/mmcblk0" (mmcblk0 is the name of your SD card, it could be different on your system.)

"of=/home/YOUR_USERNAME/Desktop/backup.img" (This is where you want to send your backup. In this example it will go to your desktop, but you can send it anywhere you want.)

"bs=1M" (This specifies block size. Here we've said copy over blocks of one megabyte (1024x1024 bytes) at a time. Read the man page for dd if you want to change this value.)

"count=2048" (This is how many blocks to copy over. In this example we've said copy over only 2048 blocks and then stop. Because we set the block size as 1 megabyte, we will get an image that is 2048 times 1 megabyte, which is 2 gigabytes.)

The count can be adapted to any size, for example, if you wanted a 1.8 gigabyte image, the count should be set to around 1850: 1.8 gigabytes(1.8 x 1024 x 1024 x 1024) divided by megabytes(1024 x 1024) = 1843.2 (the count must be an integer number, so the smallest count could be set to is 1844)

It is very important you adapt the bs and count values to the size of all your paritions on the SD card combined. Storage device capacities are normally given in powers of 1000, not 1024. So what your operating system sees as a 2 gigabyte(2 x 1024 x 1024 x 1024 bytes) image probably won't fit onto a 2 gigabyte(2 x 1000 x 1000 x 1000 bytes) SD card. dd does give the option of specifying blocks in powers of 1024 or 1000, so read the man page.

Because Gparted doesn't show the exact size of each partition in bytes, you'll generally want to make the image a little larger than the total size of all the partitions, just incase you don't make it large enough. If it's crucial the image is the smallest possible size it could be, you can find out exactly how many bytes each partition is by running the command:

Code: Select all

df -B1

If your sd card is "mmcblk0" Look for "/dev/mmcblk0p1" and "/dev/mmcblk0p2". This will not show unmounted partitions however (like the swap), so make sure you include the size of the swap partition in your final calculation.


I hope this helps and thanks to milhouse for this method.

[Dweeber]

That is a lot of work to make a backup and requires that you have another Linux box or LiveCD available.

That is why I created an script to use instead of the standard raspi-config that expands the FS to the size of the image. Instead, it expands it but backs off just a bit so that the same image will fit on any SDcard of the same size.

It does create images that are as large as the SDcard, so if you are going to play with 32gb cards you will have close to 32gb size... but I stick with 16GB which when stored on a TB drive is not that big of a deal and only takes about 15 mins to make a backup.

They key difference is that it is easy to do, which means there is a better chance you will actually make a backup when needed. I then simply use Win32diskimage to make and create copies. No extra computer needed.

You do need to start off by expanding using the utility though... if you have already fully expanded you would have to go through the hassle of using gparted etc...

Creating a backup is then as simple as inserting the SDcard into the winbox, using Win32diskimage to make a copy and you are done. Little chance of making a mistake or doing a bunch of extra steps.

ref:
http://rpi.tnet.com/project/scripts/rpi-wiggle

[xalixo]

Once you understand how to do this it's easy. I might have written a very long explanation, but there really are only two simple steps:

1) Resize the main partition with gparted and add up (in your head) the size of the partitions combined.
2) Run the dd command specifying how big the image should be.

We all know how to do a full image backup. In linux you'd just run the command:

Code: Select all

dd if=/dev/sdcard of=/anywhere/backup.img

The reason I posted the example, was so that people could now have the option to put the same image on any sized SD card. I couldn't find an example of how to do that anywhere.

Your script looks like a good way to make sure people can transfer a setup to an SD card of the same size and I agree it is a safer way to do that, but if people want to transfer a setup from a large card to a smaller card, I do not know of any other way. Maybe creating a full backup image and then using truncate to remove the empty end of the image would work, but that would mean the swap partition would be gone, so you'd have to replace it.

If anyone has any ideas on how to improve this method, please let us know.

[billw]

I solved this problem by writing a script that backs up the Pi partitions
to larger or smaller SD cards using a combination of dd and rsync.
It handles the two standard Pi partitions /dev/mmcblk0p1 and
/dev/mmcblk0p2 but probably could be modified to handle SD cards that
have a third swap partition.

The backup is done entirely on the Pi and copies the booted Pi SD card
to a SD card plugged into a Pi USB port via a USB card reader.

So if you have a USB card reader to plug your SD cards into, you can give
it a try. The script is rpi-clone and I have it on pastebin:

http://pastebin.com/48fr9BAS

[Dweeber]

...So if you have a USB card reader to plug your SD cards into, you can give
it a try. The script is rpi-clone and I have it on pastebin:

I gave that a shot back when you first posted it in September and had Permission issues. It would be a decent method since all of my SDcards are actually MicroSDHC cards using SDcard adapters.

I need to go back to my notes to see exactly what I was seeing though or at least give it another shot. I can give it a try tonight on one of my dev units.

[Wheel_nut]

Hi Billw, nice piece of code! I have yet to try it but I love the concept of creating the clone on the RPi itself.

If you are taking requests, it would be nice to have the ability to save the clone as an image file that could be stored as a data file on one USB drive and restored to an SD Card mounted as another USB drive.

Well done

[Collie714]

What you could do is us gparted and rsync too.

Use Gparted or any partition maker, create the partitions the similar to the original.

sudo rsync -av /media/pathtoSDcardPartition1/* /media/pathtonewSDcardPartition1/

I just gave it a bash there on a 3.8 gb sd card down to a 3.7 gb sd card and it works fine.

[agocs]

If you are taking requests, it would be nice to have the ability to save the clone as an image file that could be stored as a data file on one USB drive and restored to an SD Card mounted as another USB drive.

Wheel_nut, I'm just looking for the same solution, with no luck so far, but I had an idea.
I'm trying to modify billw's awesome script to make it capable of making a backup to an .img file, without the need of an empty USB stick or an SD card. My idea was a loop device as a destination device, but handling partitions on a loop device is really a nightmare... I'm not sure i can make it at the end...
So far I've made an .img file filled with zeros with dd:

Code: Select all

dd if=/dev/zero of=/mnt/ext/backup-tmp.img bs=1024 count=3072000 

Then attached it to the next free loop device:

Code: Select all

losetup -f /mnt/ext/backup-temp.img

I modified the script here and there because a loop device, and any partitions in it are mounted with extra parameters (-o loop,offset=...). Billw's script now can handle the loop device until the point where it creates the root partition on the dest. device. But mkfs.ext4 can't make the filesystem on a loop dev. partition, because standard partition devices (/dev/loop1p1,/dev/loop1p2) are not created in case of using a loop device. (Like /dev/sda1,/dev/sda2 when using a normal disk device /dev/sda) However the partitions are created on the device.
Fortunately, parted can make filesystems itself, without the need of a device path,

Code: Select all

parted /dev/loop1 mkfs y 2 ext2

but ext4 is not implemented in it.
So right now I'm stuck, because I could only create the root partition on the loop device with ext2, and cannot convert it to ext4 So far I'm stuck here:
My Pi's SD card:

Code: Select all

root@raspberrypi:/home/pi# parted /dev/mmcblk0 print
Model: SD 00000 (sd/mmc)
Disk /dev/mmcblk0: 31,4GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  62,9MB  58,7MB  primary  fat16        lba
 2      62,9MB  31,4GB  31,3GB  primary  ext4

And the almost perfectly prepared destination device:

Code: Select all

root@raspberrypi:/home/pi# parted /dev/loop1 print
Model:  (file)
Disk /dev/loop0: 3146MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  62,9MB  58,7MB  primary  fat16        lba
 2      62,9MB  3146MB  3083MB  primary  ext2

Partition 2 should be converted with tune2fs to ext4, but tune2fs would need a device path to work with. Which we don't have because of the loop device.

I'm not a big linux guru, so maybe this last missing step can be done with some clever hack, but for now I ran out of ideas.
Anyone any idea?

EDIT:
Ok, since this post was waiting for moderation I found the solution
kpartx does the mapping of the partitions to /dev/mapped/loopNpM devices:

Code: Select all

root@raspberrypi:/home/pi# kpartx -av /dev/loop1
add map loop1p1 (253:0): 0 114688 linear /dev/loop1 8192
add map loop1p2 (253:1): 0 5248840 linear /dev/loop1 122880

EDIT2:
Well, it seems I've succeeded. But I need to borrow an SD card to test whether the image is truly bootable and works properly. I'm quite excited

EDIT3:
Works like a charm! (At least for me.) If anyone would like to give it a try, just let me know!

[wtaniguchi]

Hi agocs, mind sharing your modified script?

I was wondering if, aside from maybe taking a bit longer, it would work creating the image file on a mounted network share. From what I could gather from billw's script and your post I can't see any problems.

Thanks!

If you are taking requests, it would be nice to have the ability to save the clone as an image file that could be stored as a data file on one USB drive and restored to an SD Card mounted as another USB drive.

Wheel_nut, I'm just looking for the same solution, with no luck so far, but I had an idea.
I'm trying to modify billw's awesome script to make it capable of making a backup to an .img file, without the need of an empty USB stick or an SD card. My idea was a loop device as a destination device, but handling partitions on a loop device is really a nightmare... I'm not sure i can make it at the end...
So far I've made an .img file filled with zeros with dd:

Code: Select all

dd if=/dev/zero of=/mnt/ext/backup-tmp.img bs=1024 count=3072000 

Then attached it to the next free loop device:

Code: Select all

losetup -f /mnt/ext/backup-temp.img

I modified the script here and there because a loop device, and any partitions in it are mounted with extra parameters (-o loop,offset=...). Billw's script now can handle the loop device until the point where it creates the root partition on the dest. device. But mkfs.ext4 can't make the filesystem on a loop dev. partition, because standard partition devices (/dev/loop1p1,/dev/loop1p2) are not created in case of using a loop device. (Like /dev/sda1,/dev/sda2 when using a normal disk device /dev/sda) However the partitions are created on the device.
Fortunately, parted can make filesystems itself, without the need of a device path,

Code: Select all

parted /dev/loop1 mkfs y 2 ext2

but ext4 is not implemented in it.
So right now I'm stuck, because I could only create the root partition on the loop device with ext2, and cannot convert it to ext4 So far I'm stuck here:
My Pi's SD card:

Code: Select all

root@raspberrypi:/home/pi# parted /dev/mmcblk0 print
Model: SD 00000 (sd/mmc)
Disk /dev/mmcblk0: 31,4GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  62,9MB  58,7MB  primary  fat16        lba
 2      62,9MB  31,4GB  31,3GB  primary  ext4

And the almost perfectly prepared destination device:

Code: Select all

root@raspberrypi:/home/pi# parted /dev/loop1 print
Model:  (file)
Disk /dev/loop0: 3146MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      4194kB  62,9MB  58,7MB  primary  fat16        lba
 2      62,9MB  3146MB  3083MB  primary  ext2

Partition 2 should be converted with tune2fs to ext4, but tune2fs would need a device path to work with. Which we don't have because of the loop device.

I'm not a big linux guru, so maybe this last missing step can be done with some clever hack, but for now I ran out of ideas.
Anyone any idea?

EDIT:
Ok, since this post was waiting for moderation I found the solution
kpartx does the mapping of the partitions to /dev/mapped/loopNpM devices:

Code: Select all

root@raspberrypi:/home/pi# kpartx -av /dev/loop1
add map loop1p1 (253:0): 0 114688 linear /dev/loop1 8192
add map loop1p2 (253:1): 0 5248840 linear /dev/loop1 122880

EDIT2:
Well, it seems I've succeeded. But I need to borrow an SD card to test whether the image is truly bootable and works properly. I'm quite excited

EDIT3:
Works like a charm! (At least for me.) If anyone would like to give it a try, just let me know!

[agocs]

[quote="wtaniguchi"]Hi agocs, mind sharing your modified script?

I was wondering if, aside from maybe taking a bit longer, it would work creating the image file on a mounted network share. From what I could gather from billw's script and your post I can't see any problems.

Thanks!

Hi wtaniguchi,

Sure, you can download it from here: http://pastebin.com/JUSawD2m
Please let me know if you have any trouble with it, because I'm the only one who have tested it so far.

[wtaniguchi]

Hi agocs! Sorry for the delay, I somehow didn't receive a notification when you replied.

I will check it out and will let you know!

Thanks!

[guacia]

Thanks for the info. It's been of great help.

[mulepic]

Thanks to agocs' script at http://pastebin.com/JUSawD2m it saved me a huge hassle dealing w/ variant SD card sizes, thanks agocs! I mounted a network share and ran it w/ the -l option. I had an issue and I couldn't get it to run by passing in my mount path so I had to hard code it in the if block

if [ "$LOOP" != "" ]; then
DST_FILE=/mnt/winmnt/output


After that it works like a charm!

[loafbread]

Thanks for the great script. I had to modify a couple lines to make it work like the original rpi-clone script. I think it might have been a typo that left off the suffixes on these variables.

Change
DST_ROOT_PARTITION=/dev/${DST_DISK}
DST_BOOT_PARTITION=/dev/${DST_DISK}
to
DST_ROOT_PARTITION=/dev/${DST_DISK}2
DST_BOOT_PARTITION=/dev/${DST_DISK}1

[agocs]

Thanks for the great script. I had to modify a couple lines to make it work like the original rpi-clone script. I think it might have been a typo that left off the suffixes on these variables.

Change
DST_ROOT_PARTITION=/dev/${DST_DISK}
DST_BOOT_PARTITION=/dev/${DST_DISK}
to
DST_ROOT_PARTITION=/dev/${DST_DISK}2
DST_BOOT_PARTITION=/dev/${DST_DISK}1

True! I also found it since I uploaded the script, I just forgot to update it here. Thanks for the fix!

[gwh0923]

All this is nice but I still have a 16G sd card that i used gparted to reduce to less than 4G. When I Win32Diskimager on my pc I get a 16G image.

I don't have a linux machince, is there a way I can reduce the image file size on the PC so I can copy that image onto a smaller (say 8G sd card).

[DougieLawson]

I don't have a linux machince ...

You've got a Raspberry Pi, so you DO have a linux machine.

Try using a USB SDCard reader to work on your backup card.

[MrEngman]

All this is nice but I still have a 16G sd card that i used gparted to reduce to less than 4G. When I Win32Diskimager on my pc I get a 16G image.

I don't have a linux machince, is there a way I can reduce the image file size on the PC so I can copy that image onto a smaller (say 8G sd card).

Regardless of what is on your SD card Win32DiskImager will copy the whole of the card contents to the image file, including all unused space.

I think you could use dd on your Windows machine to copy your SD card to an image file. You can then specify the amount of data to save to the image. You could probably also use dd to copy your 16G image to an 8G image by copying the file with a limit on the amount it copies rather than copying the SD card again. These loads of stuff on the internet about using dd.


MrEngman

[lostgdi]

All this is nice but I still have a 16G sd card that i used gparted to reduce to less than 4G. When I Win32Diskimager on my pc I get a 16G image.

I don't have a linux machince, is there a way I can reduce the image file size on the PC so I can copy that image onto a smaller (say 8G sd card).

Hi. I don't have a linuxe machine either. But what I had got is my raspberry pi. here is my solution for backup , familiar with other:
1. use Gparted downsize my own SD card.(only change the used and largest partition) to about 3.2G (less than that will cause error, I was wondering)
2. then replug the SD card to pi and turn on.
3. do a mount directory to my LAN share folder which is windows computer.
4. dd the amount of content from mmcblk0(pi running SD card) to remote share folder

Is it a way to backup from a running system ?

[javier-alba]

I cannot execute the script. The script is in my Pi home.
When I use this command:

./rpi-clone2.sh

I get this output:

-bash: ./rpi-clone2.sh: /bin/bash^M: bad interpreter: No such file or directory

If I use this one:

sh ./rpi-clone2.sh

I get this:

: not found2.sh: 5: ./rpi-clone2.sh:
: not found2.sh: 7: ./rpi-clone2.sh:
: not found2.sh: 9: ./rpi-clone2.sh:
: not found2.sh: 12: ./rpi-clone2.sh:
: not found2.sh: 15: ./rpi-clone2.sh:
: not found2.sh: 17: ./rpi-clone2.sh:
: not found2.sh: 19: ./rpi-clone2.sh:
: not found2.sh: 22: ./rpi-clone2.sh:
./rpi-clone2.sh: 43: ./rpi-clone2.sh: Syntax error: word unexpected (expecting "then")

Am I missing a parameter, permission or something else?

[DougieLawson]

You've written it in Windows format (\r\n at the ends of lines).

sudo apt-get install dos2unix
dos2unix ./rpi-clone2.sh

should fix it.

[javier-alba]

Thanks!!

It works now. Now I don't know how to use with the parameters . I'll see the script more deeply.

[megasoft]

Thaks, Gparted worked great for me moving a 8GB to 16GB

[cchisholm]

I use the Windows Win32DiskImager program to read everything off my SD card and store it in a .img file. I can then write this to a different SD card, if I ever need to

Have you ever actually tested this? So far, I (and several others) have deen unable to restore these images.

[haonc]

Imagine this example:

You have a 32GB SD card.
You are using 1.8GB of space on it.
You want to back it up into an image that is 1.8GB big, not 32GB big.
Then you want to be able to put it on an SD card that is 2,4,8 or 16GB big.

If you do a normal image backup, you'll end up with an image that is 32GB big. As a backup that's a lot of wasted of space. If you just want to store it as a backup you could gzip it, which would remove the wasted space, but you still won't be able to put it on a smaller SD card in a bootable format.

If you do a backup by copying the files, you'll end up with an SD card that won't boot unless you create different partitions that are in the correct filesystem format and make sure each file is in the correct partition. It's possible you could miss some hidden files and end up changing file permissions which you'd have to chmod back, so this way wouldn't work without a lot of messing about.

Thanks to help from "milhouse" on the forum, I now know how to make an image of the SD card that's the same size as the actual space being used. This cannot be done on the pi itself, you'll need a seperate linux machine.

Using Gparted (or parted), reduce the size of your main partition on the SD card to remove most of the empty space (You'll probably want to leave a little free space, maybe a few megabytes), but do not reduce the size of the other partitions. Then use Gparted to make sure each partition runs consecutively from the start without any unallocated space in between them. Take note of the total size all the partitions are using up and the location of your SD card (dev/mmcblk0 for example).

Next we're going to use dd. You need to be very careful when using dd, because you could end up wiping your hard drive if you mistype something.

Now run the command:(See below on how to adapt)

Code: Select all

dd if=/dev/mmcblk0 of=/home/YOUR_USERNAME/Desktop/backup.img bs=1M count=2048

"if=/dev/mmcblk0" (mmcblk0 is the name of your SD card, it could be different on your system.)

"of=/home/YOUR_USERNAME/Desktop/backup.img" (This is where you want to send your backup. In this example it will go to your desktop, but you can send it anywhere you want.)

"bs=1M" (This specifies block size. Here we've said copy over blocks of one megabyte (1024x1024 bytes) at a time. Read the man page for dd if you want to change this value.)

"count=2048" (This is how many blocks to copy over. In this example we've said copy over only 2048 blocks and then stop. Because we set the block size as 1 megabyte, we will get an image that is 2048 times 1 megabyte, which is 2 gigabytes.)

The count can be adapted to any size, for example, if you wanted a 1.8 gigabyte image, the count should be set to around 1850: 1.8 gigabytes(1.8 x 1024 x 1024 x 1024) divided by megabytes(1024 x 1024) = 1843.2 (the count must be an integer number, so the smallest count could be set to is 1844)

It is very important you adapt the bs and count values to the size of all your paritions on the SD card combined. Storage device capacities are normally given in powers of 1000, not 1024. So what your operating system sees as a 2 gigabyte(2 x 1024 x 1024 x 1024 bytes) image probably won't fit onto a 2 gigabyte(2 x 1000 x 1000 x 1000 bytes) SD card. dd does give the option of specifying blocks in powers of 1024 or 1000, so read the man page.

Because Gparted doesn't show the exact size of each partition in bytes, you'll generally want to make the image a little larger than the total size of all the partitions, just incase you don't make it large enough. If it's crucial the image is the smallest possible size it could be, you can find out exactly how many bytes each partition is by running the command:

Code: Select all

df -B1

If your sd card is "mmcblk0" Look for "/dev/mmcblk0p1" and "/dev/mmcblk0p2". This will not show unmounted partitions however (like the swap), so make sure you include the size of the swap partition in your final calculation.


I hope this helps and thanks to milhouse for this method.

Hi xalixo,
Thank you for your answers. It's exactly what i need. But i have a questions about this problem.
I've backed up SD Card Image with 4GB storage. When i write this image into 8GB SD Card, how much storage of SD Card will be used, 8GB or 4GB?
Thank you. Sorry for my English. Hope that you'll understand my question.