SD Card Image Exclude Unallocated Space


16 posts
by Wanderlei » Fri Nov 02, 2012 7:39 am
I have been using win32diskimager and it works great. Problem is I change between SD cards 2GB and 4GB in size. Only about 1GB is actually partitioned, the rest is unallocated space.

Is there a way to take SD card images and restore them so that they exclude unallocated disk space?

Thank you
Posts: 79
Joined: Tue Aug 21, 2012 2:01 pm
by sjfaustino » Fri Nov 02, 2012 9:34 am
Wanderlei wrote:I have been using win32diskimager and it works great. Problem is I change between SD cards 2GB and 4GB in size. Only about 1GB is actually partitioned, the rest is unallocated space.

Is there a way to take SD card images and restore them so that they exclude unallocated disk space?

Thank you

I believe win32diskimager will always read and write the whole device, with no way to choose just part of it.
You can use dd (get it for windows here: http://www.chrysocome.net/dd ) to choose just the partition that you want to backup or restore.
Remember that just the partition means exactly that, therefore no boot sector! That has to be done in another operation.
Posts: 87
Joined: Tue Jun 12, 2012 5:21 pm
by Wanderlei » Fri Nov 02, 2012 12:10 pm
sjfaustino wrote:
Wanderlei wrote:Remember that just the partition means exactly that, therefore no boot sector! That has to be done in another operation.


Cheers the response.

Ok I see what you are saying, there will boot sector information that resides outside of the partitions I see. (It lives in the 'unallocated space'). Bugger.

Maybe, is there a way to trim down a image file once you have it? Or is that going in to area harder than restoring the missing boot sector?
Posts: 79
Joined: Tue Aug 21, 2012 2:01 pm
by sjfaustino » Fri Nov 02, 2012 7:31 pm
Wanderlei wrote:Maybe, is there a way to trim down a image file once you have it? Or is that going in to area harder than restoring the missing boot sector?

If this is a image that will be backed up and restored only in case of problems and not a constant day to day backup, you can shrink the partition on the bigger card to the size that you want with Partition Magic/Paragon Partition Manager in Windows or GParted in Linux, do the backup and then expand it back again.

Another option in Linux is to use FSArchiver to backup the bigger partition (it actually copies only the files and their attributes, not the whole partition) to a file, create the file system and partitions on the smaller device as normal and then just restore the backup file to the smaller device.
Posts: 87
Joined: Tue Jun 12, 2012 5:21 pm
by Dilligaf » Fri Nov 02, 2012 7:59 pm
As far as I can tell the Pi doesn't use a boot sector. I can copy the files from the vfat (boot) partition to another card and it will boot, I think it just looks for specific file names. You should be able to backup the files in your various partitions and then copy them to any card after recreating the needed partitions on it as long as the permissions on the files remain unchanged.
Posts: 283
Joined: Wed May 23, 2012 6:48 pm
by milhouse » Fri Nov 02, 2012 11:17 pm
sjfaustino wrote:You can use dd (get it for windows here: http://www.chrysocome.net/dd ) to choose just the partition that you want to backup or restore.
Remember that just the partition means exactly that, therefore no boot sector! That has to be done in another operation.


Or you can calculate the used size of all the partitions, and instruct dd to read only that amount. For example, if the combined partitions amount to 1GB used on a 4GB card, then "dd if=/dev/sda of=./backup.img bs=1M count=1050" will backup just the space used by all of the partitions, plus a tiny bit extra to be safe (adjust count to suit individual requirements, etc.)
Posts: 552
Joined: Mon Jan 16, 2012 12:59 pm
by Dilligaf » Sat Nov 03, 2012 12:09 am
milhouse wrote:
sjfaustino wrote:You can use dd (get it for windows here: http://www.chrysocome.net/dd ) to choose just the partition that you want to backup or restore.
Remember that just the partition means exactly that, therefore no boot sector! That has to be done in another operation.


Or you can calculate the used size of all the partitions, and instruct dd to read only that amount. For example, if the combined partitions amount to 1GB used on a 4GB card, then "dd if=/dev/sda of=./backup.img bs=1M count=1050" will backup just the space used by all of the partitions, plus a tiny bit extra to be safe (adjust count to suit individual requirements, etc.)


Interesting idea but isn't that assuming that all the data is at the beginning of the partition? what if files have been deleted putting spaces in the data? Or is dd "smart" enough to copy data before it copies blanks, I thought that it just copies sectors in the order they appear.
Posts: 283
Joined: Wed May 23, 2012 6:48 pm
by milhouse » Sat Nov 03, 2012 12:29 am
Dilligaf wrote:Interesting idea but isn't that assuming that all the data is at the beginning of the partition?


It's assuming all of the data (partitions) is at the beginning of the disk/card. Fortunately, this is usually the case.

Dilligaf wrote:what if files have been deleted putting spaces in the data?


If the space is within a partition, it needs to be backed up, so not really sure why this should be a concern or problem.

Dilligaf wrote:Or is dd "smart" enough to copy data before it copies blanks, I thought that it just copies sectors in the order they appear.


Yep, that's all it does (it's not smart at all!) :)

My understanding was that the OP has a relatively small partition (1GB) on a large 4GB card and only wanted to back up the used data (ie. the small, 1GB, part) and not the entire card, which is mostly empty. For this, dd is ideal, particularly when the amount of data to be backed up is specified so that the entire card is not read.

If the size of the on-disk image is an issue, pipe it through gz to compress it as the image is being created, though Windows Disk Imager is sadly too deficient to understand gz or any other form of compression (surely there must be a better tool for Windows users, one that understands compressed images, and might even include options to limit the size when reading or writing the image?)

If, on the other hand, the OP wants to back up only the "active" areas of the disk, ignoring any inactive space within a partition, including deleted files etc., then he'll need a backup tool that understands all of the filesystems on the disk. Rolling your own would probably be easier, ie. tar up the entire contents of each partition individually, create empty partitions, then simply untar to restore, but this is likely to be fraught with problems (particularly with some filesystems such as /proc or /dev), slower and IMHO, more trouble than it's worth! :)
Posts: 552
Joined: Mon Jan 16, 2012 12:59 pm
by Dilligaf » Sat Nov 03, 2012 12:45 am
OK, I read it wrong, I thought they had a 4GB partition and only wanted to backup the 1GB of data off the partition. It all makes sense now. Usbit (usb image tool) will make img.gz in Windows up to 4GB, larger partitions you need to just make a .img and then gzip it with 7zip or whatever. It will write the resulting .img.gz back to a usb device.
Posts: 283
Joined: Wed May 23, 2012 6:48 pm
by milhouse » Sat Nov 03, 2012 12:55 am
Dilligaf wrote:Usbit (usb image tool) will make img.gz in Windows up to 4GB, larger partitions you need to just make a .img and then gzip it with 7zip or whatever. It will write the resulting .img.gz back to a usb device.


Is there any reason why the Foundation don't promote Usbit over Windows Disk Imager, which is really quite basic. Even the images the Foundation ship have to be unzipped before they can be used, whereas with Usbit it sounds like this wouldn't be necessary. Or maybe the very basic nature of Windows Disk Imager is part of its appeal as far as the RPF are concerned (can't really fault them there, I guess - wouldn't want to be blamed for trashing someone's hard disk because the imaging tool was more complex than absolutely necessary).
Posts: 552
Joined: Mon Jan 16, 2012 12:59 pm
by Dilligaf » Sat Nov 03, 2012 1:04 am
http://www.alexpage.de/ is the link to usbit. It's free but I don't think it's open source, that's probably why the foundation doesn't go with it. Maybe it's as simple as they found Windows Disk Imager and didn't look any further.
Posts: 283
Joined: Wed May 23, 2012 6:48 pm
by milhouse » Sat Nov 03, 2012 1:15 am
Dilligaf wrote:http://www.alexpage.de/ is the link to usbit. It's free but I don't think it's open source, that's probably why the foundation doesn't go with it. Maybe it's as simple as they found Windows Disk Imager and didn't look any further.


Yeah, just tried it. Seems to work OK, though lacks any option to create truncated images (ie. 1GB image from a 4GB card), and the minimal progress indicator is a bit annoying (ie. no indication of read/write speed and thus actual activity, though admittedly I did test it by backing up a 32GB card which even at 16MB/s takes a while to get to 1%!) So in some ways Usbit is not even as good as Windows Disk Imager, and the support for compressed images is it's main advantage.
Posts: 552
Joined: Mon Jan 16, 2012 12:59 pm
by Dilligaf » Sat Nov 03, 2012 1:22 am
I use it to make card backups, a 8GB image compresses to under 1GB (I know it depends on the data on the card) So the ability for it to use compressed images is the deciding factor for me.
Posts: 283
Joined: Wed May 23, 2012 6:48 pm
by Wanderlei » Sat Nov 03, 2012 5:32 am
milhouse wrote:My understanding was that the OP has a relatively small partition (1GB) on a large 4GB card and only wanted to back up the used data (ie. the small, 1GB, part) and not the entire card, which is mostly empty. For this, dd is ideal, particularly when the amount of data to be backed up is specified so that the entire card is not read.


Yeah this is exactly what I was meaning, sorry if I was unclear.

Only thing that put me off DD is that is cmd line only and very powerful, bit scary:) I recently had a accident via carelessness with a GUI imager and wrote to an important disk, luckily recovered. But it looks the tool to use for what I want to accomplish.

My SD card H:
124MB FAT Partition
741MB Ext4 Partition
2.8GB Unallocated

I run this command
dd if=\\.\h: of=d:\disk1.img bs=1M count=1250

What I seem to get is a 128,000KB disk image, changing the 'count' parameter, has no effect. I think it is just getting the FAT partition because thats all windows recognises?

I am a windows button monkey that is completely bamboozled by linux file structure, where's my C: at yo. Not sure its wise for me too mess with a tool that I can wipe disks with if its my only option?

Checked out usbit too, only seems to get FAT partition too.

(Thanks again for help, I am closer than I was and learned a thing or too)
Posts: 79
Joined: Tue Aug 21, 2012 2:01 pm
by Dilligaf » Sat Nov 03, 2012 5:52 am
Use usbit in "device mode" in the top left corner is a dropdown, and then make an img.gz of your card. when writing the image back to a smaller card go to options in usbit and check the second one, truncate oversized images. Since your partitions are at the beginning of the sd they will be at the beginning of the image. When you write the image back your partitions should be written and then blanks to the end of the sd where the rest of the image file will be cut off.

Hope this makes sense, would recommend trying it first to make sure it works.
Posts: 283
Joined: Wed May 23, 2012 6:48 pm
by Wanderlei » Sat Nov 03, 2012 6:43 am
Dilligaf wrote:Use usbit in "device mode" in the top left corner is a dropdown


Excellent, cheers mate.

It was greyed out, I just needed to run with administrator privledges in windows 7.

Tested out a 4GB image taken with usbit and win32diskimager and wrote them to a 2GB SD card with truncate , both worked fine.

My SD card juggling life just become a tonne easier. Feels good kicking a 1st world problem in the arse:)
Posts: 79
Joined: Tue Aug 21, 2012 2:01 pm