Solved: SD image too big for another SD Card

[Saran]

Driven by the problem I stumbled upon and by inability to find a complete solution online (even in the thread in this forum), I did some digging and testing and decided to share my How-To.

This is how I fixed the problem of copying 8GB card to another, somewhat smaller, 8GB card
(Cards could be of any size, it's important only that the target card is smaller and that actual data from the source card could fit on it)

Prerequisite: separate Linux and Windows boxes (Pi not needed).

Resize the source card
(on Linux)
Run gparted to reduce the size of the "data" partition on the source card.
I reduced it by ~500MB, but I guess 100-200MB would have sufficed. As can be seen in the code later (*), the actual difference was 120MB.
This needs to be done only one time.

Make an image of the source card
(on Windows, sorry)
Make an image of the source card with Win32DiskImager. Don't be alarmed by result -- the image will be the same size as before the resizing (because physical sectors did not change), but what's important, the end of it will now contain uninitialized partition (500MB), so it's safe if we cut that off when copying to a smaller card later.

Prepare for copying
(on Linux)
First we need to check the size of the target card with this command:

Code: Select all

sudo fdisk /dev/mmcblk0

or check the size of the image you have already made from the target card, with this command:

Code: Select all

sudo fdisk target.img

You should get something like this:
(after hitting enter on the fdisk command, press 'p' to display the partition info)

Code: Select all

ubuntu@ubuntu:/media/Other$ sudo fdisk OpenElec_2013-11-28.img 

Command (m for help): p

Disk OpenElec_2013-11-28.img: 7822 MB, 7822376960 bytes
255 heads, 63 sectors/track, 951 cylinders, total 15278080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001080e

                  Device Boot      Start         End      Blocks   Id  System
OpenElec_2013-11-28.img1   *        2048      258047      128000    c  W95 FAT32 (LBA)
OpenElec_2013-11-28.img2          258048     1830911      786432   83  Linux

From this the only important info is: 7822376960 bytes.

Next, as we will use 'dd' command for copying data, we have to calculate some parameters for it.

('bs' as in block size)
bs=1M (kilo times kilo equals mega: 1024 * 1024 = 1M)

('count' as in number of blocks to copy)
count=7460 (divide target size with block size to get number of blocks: 7822376960 / (1024 * 1024) = 7460)


Do the copy
As the shell operation last for some minutes (depending on the speed of the card, 10-20 min), we'll use 'pv' to track the progress. This is the complete command:

Code: Select all

pv source.img | sudo dd of=/dev/mmcblk0 bs=1M iflag=fullblock count=7460

or, if we just wanted to create a target-compatible image, we could use this:

Code: Select all

pv source.img | sudo dd of=target.img bs=1M iflag=fullblock count=7460

Don't forget: use YOUR card/image sizes!


Additional info
(*) To calculate the actual difference for which the target card should be resized down, we also need the size of the source card. Use the same 'fdisk' command explained above, just with the source card inserted in the reader:

Code: Select all

sudo fdisk /dev/mmcblk0

or check the size of the image you have already made from the source card, with this command:

Code: Select all

sudo fdisk source.img

My source card size was 7948206080 bytes, so:
7948206080 - 7822376960 = 125829120 / 1024 / 1024 = 120M ==> the source partition should be reduced by at least 120M to prevent data loss!

NB: you don't actually need 'fdisk' if you have both images -- just check their sizes with

Code: Select all

ls -l

For the sceptic: TDC
(as in Test Driven Copying )

Unsure will it work? Test writing on (virtual) device first!

Code: Select all

# create an empty image the size of target card
dd if=/dev/zero of=test.img bs=1M count=7460

#connect the loopback device
losetup /dev/loop1 test.img

#to simulate an error, don't limit the number of written blocks
pv source.img | sudo dd of=/dev/loop1 bs=1M iflag=fullblock
#result: "dd: writing `/dev/loop1': No space left on device"

#copy all content from the source to the target (while watching progress)
pv source.img | sudo dd of=/dev/loop1 bs=1M iflag=fullblock count=7460
#et voila! dd should finish w/o error

[kioan]

Make an image of the source card(on Windows, sorry)

Why not on Linux?
You can make the image using the dd command.

[Saran]

Make an image of the source card(on Windows, sorry)

Why not on Linux?
You can make the image using the dd command.

Yea, I was so into researching if's and why's that, when I was at that step, decided I will leave it as is because I already had the images created that way.

Can you confirm this (from the top of my head) command will work?

Code: Select all

pv /dev/mmcblk0 | sudo dd of=source.img bs=1M iflag=fullblock

[Dilligaf]

Usbit (usb image tool) will allow you to write the oversized image with a blank at the end directly to a card, just select "truncate oversize images" in settings. As long as the last partition ends before it truncates the card will be usable. If your last partition is swap you can just write the image to the card then remake the swap partition after booting (swap partition will end up smaller than the original)

[vistauser]

what a valuable hint!
Thank you Dilligaf, it works perfect.

vistauser

[Joe Schmoe]

This whole problem would go away if the raspbian maintainers would just take the hint (the hint that's been floating around for a long time now) and make their "expand rootfs" thing leave a little breathing room at the end. I think this concept has been embodied in a 3rd party tool called "rpi-wriggle" (or something like that).

P.S. Alternatively, you could use "file based" methods instead of "image based" methods. I've documented how to do it this way several times already - basically, it eliminates all this worrying about re-sizing partitions, etc.

[Saran]

Usbit (usb image tool) will allow you to write the oversized image with a blank at the end directly to a card, just select "truncate oversize images" in settings. As long as the last partition ends before it truncates the card will be usable. If your last partition is swap you can just write the image to the card then remake the swap partition after booting (swap partition will end up smaller than the original)

I stumbled upon this tool when I was researching the ways how to do it, but this info from its FAQ made me look otherway:

Truncating this area will result in data loss. Also the partition info and file system info doesn’t match the real values, if you restore it to a smaller flash drive. The result is a corrupted file system that is not safe to use anymore.

Now that I did the step that also Usbit requires for the "trunkate" mode (the step "Resize the source card", explained above), I'll give USB Image Tool a try.
Thanks for the hint, Dilligaf.

[Saran]

This whole problem would go away if the raspbian maintainers would just take the hint (the hint that's been floating around for a long time now) and make their "expand rootfs" thing leave a little breathing room at the end. I think this concept has been embodied in a 3rd party tool called "rpi-wriggle" (or something like that).

Indeed, Joe. From what I've found, lots of users have the same problem, plus the "expand rootfs" sounds too tempting not to use

"expand rootfs" should use "sane limit", or at least allow for an option in that direction.

[DeanC]

I have a perl script that will take a .img file and reduce it to it's minimum size so that it can be burned to a card. Then just run raspi-config to expand the file system once you go to use it.

http://www.raspberrypi.org/phpBB3/viewt ... p?p=444354

[Joe Schmoe]

I have a perl script that will take a .img file and reduce it to it's minimum size so that it can be burned to a card. Then just run raspi-config to expand the file system once you go to use it.

That's good. I've been kicking the idea around for quite a while now - that someone should take my various clues and tidbits and write it all up as a self-contained, goof-proof script. I'm glad you've done it. Congrats!