Reducing Raspbian File System Image Size

[DougieLawson]

I have also written another script to help in reminding to expand the filesystem.

http://www.raspberrypi.org/phpBB3/viewt ... 91&t=60918

Thank you. That's shrunk my 32GB card down to 5GB.

[tvjon]

Ok, here is a newer version. This one uses the truncate command, which eliminates the need to duplicate the image file.

To use this script, copy the code to a file called 'resizeimage.pl'. Then issue 'sudo perl ./resizeimage.pl'.
....
Please let me know if any problems arise from this script.

Hello Dean, thank you for the script.

Usually these days, I download RPi related software & scripts directly from a RPi, rather than a mac as I did originally, & doing so has highlighted some interesting things.
Attempting to execute this script in Terminal produces:

Unrecognized character \xC2; marked by <-- HERE after <-- HERE near column 1 at ./resize.pl line 21.

I fixed that, along with several other occurrences, so next:

Unrecognized character \xA0; marked by <-- HERE after <-- HERE near column 2 at ./shrink line 21.

I then reverted to dowloading the script on a mac & those characters didn't show up, so presumably filtered by Textedit, the mac app I opened the script in. I imagine most other forum readers
don't download forum scripts on a RPi.

After clearing up all the extraneous characters, the sript ran, producing:


Argument "fil" isn't numeric in division (/) at ./resize.pl line 61.
Old size - 0 MB (0.00 GB)
e2fsck: Invalid argument while trying to open /dev/loop0
Argument "Invalid argument while trying to open /dev/loop0\nCouldn..." isn't numeric in addition (+) at ./shrink line 72.
New size - 7766 MB (7.58 GB)
Image file was reduced by -7766 MB (-7.58 GB)

The original image appeared to be untouched in fact.

I'm unfamiliar with perl, so wonder if you have any ideas please?

[DeanC]

I'm unfamiliar with perl, so wonder if you have any ideas please?

Sorry, but the only Mac I have ever worked on was the Macintosh 512K, so I can't say off the top of my head. But I have had no problems with this script on Linux or Windows, so I am curious if maybe it has something to do with this:

"After clearing up all the extraneous characters, the sript ran..."

It might be possible that you removed more than just 'extraneous characters'. Btw, how was your .img file create? And is the image residing on the RPi or somewhere on your network?

I have a friend who likes to blow Apple, so I'll get them to try and run the script. What terminal or program do you use on the Mac to connect to the RPi? This way I can see if I can re-create it.

[tvjon]

512k eh? I may still have one in the loft....

Actually, the mac reference isn't that important. I was just pointing out that I didn't see those characters in a mac environment, ie., opening the script in a mac browser pointing to this sub forum, & then copied to a text editor. I imagine MS win users don't see them either, or you'd have reports from other users. I've seen other forum posts where "space" characters have appeared in a forum posted script. On the rare occasions I post a script, I compress it first so I know what people will see, irrespective of text editor. Maybe the forum introduces the characters, but I don't know.

Selecting your script from a browser on RPi & pasting it into leafpad, or scite, produced a file which wouldn't run because of the reported characters.
I removed them on the RPi using "sed", so no other characters were deleted.

I've subsequently tested an image dd'd on the mac, then transferred to a usb stick plugged into RPi, & your script (the one "edited" by sed) has reduced that file just fine. I've initialised & imaged many SD cards on mac's with no problems.

I just wondered if anything occurred to you from reading that cryptic error message from RPi's terminal.

The LAN wasn't involved in this scenario.

In fact, as the script isn't huge, those characters aren't a major problem.

I've tried some more images, & it looks like your script may be a handy quick "image integrity check", as one image I've tried reports:

Old size - 7646 MB (7.47 GB)

/dev/loop0: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
Use of uninitialized value $size in scalar chop at ./resize.pl line 71.
Use of uninitialized value $size in addition (+) at ./resize.pl line 72.
New size - 120 MB (0.12 GB)
Image file was reduced by 7526 MB (7.35 GB)

So, thinking more about it, the error messages probably arise from unexpected data being read from the image by your script.

I think it would be useful if your script had an option to write the shrunken image to a different file, preferably on a different volume.
Thanks for your help.

[DeanC]

I think it would be useful if your script had an option to write the shrunken image to a different file, preferably on a different volume.

Well, the reason the script is so straight foreword is because I wanted beginners to be able to use it with little fuss. If all you do is input the image file's location, then very little confusion can arise. If I add options to the script (ie. file output and desired image size), it lends itself to a myriad of ridiculous questions because beginning users are using features they don't understand or need.

I can, however, offer a suggestion. If you have a image file you want to shrink and put the output image in a different location, just copy the image file to the final location and run the script on the copy instead. This way you end up with the same results.

[ilovemipi]

Just tried this on a berryboot (squashfs) image. 8GB SD card image becomes 120MB becuase it is only seeing the initial berryboot partition.

I guess the way to do it would be just dd the main data on the berryboot SD card and then run DeanC's script on the partitioned image. Then add the truncated image back to (a smaller) SD card with berryboot on it.

Maybe the script can't handle berryboot squashfs images, anyone know before I try it?

[DeanC]

Maybe the script can't handle berryboot squashfs images, anyone know before I try it?

Unfortunately, this script was written for Raspbian images. I may have forgotten to mention that.

If I have some free time in the future, I will look at the other file systems, and how to reduce their images.

[DeanC]

I wonder if a moderator could change the post title to:
Reducing Raspbian FS image size

[ilovemipi]

Maybe the script can't handle berryboot squashfs images, anyone know before I try it?

Unfortunately, this script was written for Raspbian images. I may have forgotten to mention that.

If I have some free time in the future, I will look at the other file systems, and how to reduce their images.

Thanks for your prompt reply.

[DeanC]

I've tried some more images, & it looks like your script may be a handy quick "image integrity check", as one image I've tried reports:

Old size - 7646 MB (7.47 GB)

/dev/loop0: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
Use of uninitialized value $size in scalar chop at ./resize.pl line 71.
Use of uninitialized value $size in addition (+) at ./resize.pl line 72.
New size - 120 MB (0.12 GB)
Image file was reduced by 7526 MB (7.35 GB)

Ok, I have added a test in the script. It will stop the script from executing if an error is encountered while checking the file system that e2fsck cannot automatically fix. There is no reason to continue the script if the image is corrupt.

*** THIS SCRIPT ONLY WORKS ON RASPBIAN IMAGES. ***

To use this script, copy the code to a file called 'resizeimage.pl'. Then issue 'sudo perl ./resizeimage.pl'.

After burning, always remember to run 'sudo raspi-config' first thing, and expand the file system before use.

Code: Select all

#!/usr/bin/perl

use utf8;
use 5.010;
use strict;
#use autodie;
use warnings;
#use diagnostics;

my $who = `whoami`;

if ($who !~ /root/)
{

	print "This should be run as root or with the sudo command.\n";
	exit 1;

}

if (!$ARGV[0])
{
	
	print "No image file given.\n";
	exit 1;
	
}

my $image = $ARGV[0];

if ($image !~ /^\//)
{

	print "Please enter full path to image file.\n";
	exit 1;
	
}

if (! -e $image)
{

	print "$image does not exist.\n";
	exit 1;
	
}

my @name = split (/\//, $image);
print "\n$name[(scalar @name) - 1]:\n";
print "=" x (length ($name[(scalar @name) - 1]) + 1) . "\n";

my $info = `parted -m $image unit B print | grep ext4`;

(my $num, my $start, my $old, my $dummy) = split (':', $info, 4);
chop $start;
chop $old;
printf "Old size - %d MB (%1.2f GB)\n", int ($old / 1048576), ($old / 1073741824);

my $loopback = `losetup -f --show -o $start $image`;
chop $loopback;

`e2fsck -p -f $loopback`;

if ($? != 0)
{

	print "There was an error in the file system that can't be automatically fixed... aborting.\n";
	`losetup -d $loopback`;
	exit 1;

}

$info = `resize2fs -P $loopback 2>&1`;

($dummy, my $size) = split (': ', $info, 2);
chop $size;
$size = $size + 1024;

`sudo resize2fs -p $loopback $size 2>&1`;
sleep 1;
`losetup -d $loopback`;

$size = ($size * 4096) + $start;

`parted $image rm $num`;
`parted -s $image unit B mkpart primary $start $size`;

$size = $size + 58720257;
printf "New size - %d MB (%1.2f GB)\n", int ($size / 1048576), ($size / 1073741824);

`truncate -s $size $image`;

my $diff = $old - $size;
printf "Image file was reduced by %d MB (%1.2f GB)\n", int ($diff / 1048576), ($diff / 1073741824);

exit 0;

[tvjon]

Hello Dean,

just looked at the weather, & seeing what it is in Canada, I'll have to stop moaning about being cold here

Thanks for the script mod'. Typical, I've just tried it & got those pesky odd characters again.
If you get a spare moment, can you open this forum page in a browser on your RPi, NOT an external computer, please?

Then "select all" for your script code, & middle mouse button it into say Leafpad. Save the file, open a RPi terminal window, & type :

pi@raspberrypi ~ $ od -c resizeimage.pl

Here's what I see when I've done that:

0000000 # ! / u s r / b i n / p e r l \n
0000020 \n u s e u t f 8 ; \n u s e 5
0000040 . 0 1 0 ; \n u s e s t r i c t
0000060 ; \n # u s e a u t o d i e ; \n
0000100 u s e w a r n i n g s ; \n # u
0000120 s e d i a g n o s t i c s ; \n
0000140 \n m y $ w h o = ` w h o a
0000160 m i ` ; \n \n i f ( $ w h o !
0000200 ~ / r o o t / ) \n { \n \n 240 240
0000220 p r i n t " T h i s s h o u
0000240 l d b e r u n a s r o o
0000260 t o r w i t h t h e s u
0000300 d o c o m m a n d . \ n " ; \n
0000320 240 240 e x i t 1 ; \n \n } \n \n i
0000340 f ( ! $ A R G V [ 0 ] ) \n { \n
0000360 240 240 \n 240 240 p r i n t " N o
0000400 i m a g e f i l e g i v e
0000420 n . \ n " ; \n 240 240 e x i t 1
0000440 ; \n 240 240 \n } \n \n m y $ i m a
0000460 g e = $ A R G V [ 0 ] ; \n \n
0000500 i f ( $ i m a g e ! ~ / ^
0000520 \ / / ) \n { \n \n 240 240 p r i n t
0000540 " P l e a s e e n t e r f
0000560 u l l p a t h t o i m a g
0000600 e f i l e . \ n " ; \n 240 240 e
0000620 x i t 1 ; \n 240 240 \n } \n \n i f
0000640 ( ! - e $ i m a g e ) \n {
0000660 \n \n 240 240 p r i n t " $ i m a
0000700 g e d o e s n o t e x i s
0000720 t . \ n " ; \n 240 240 e x i t 1
0000740 ; \n 240 240 \n } \n \n m y @ n a m
0000760 e = s p l i t ( / \ / / ,
0001000 $ i m a g e ) ; \n p r i n t
0001020 " \ n $ n a m e [ ( s c a l a r
0001040 @ n a m e ) - 1 ] : \ n "
0001060 ; \n p r i n t " = " x ( l
0001100 e n g t h ( $ n a m e [ ( s c
0001120 a l a r @ n a m e ) - 1 ]
0001140 ) + 1 ) . " \ n " ; \n \n
0001160 m y $ i n f o = ` p a r t
0001200 e d - m $ i m a g e u n i
0001220 t B p r i n t | g r e p
0001240 e x t 4 ` ; \n \n ( m y $ n u
0001260 m , m y $ s t a r t , m y
0001300 $ o l d , m y $ d u m m y
0001320 ) = s p l i t ( ' : ' ,
0001340 $ i n f o , 4 ) ; \n c h o p
0001360 $ s t a r t ; \n c h o p $ o l
0001400 d ; \n p r i n t f " O l d s
0001420 i z e - % d M B ( % 1 .
0001440 2 f G B ) \ n " , i n t (
0001460 $ o l d / 1 0 4 8 5 7 6 ) ,
0001500 ( $ o l d / 1 0 7 3 7 4 1
0001520 8 2 4 ) ; \n \n m y $ l o o p b
0001540 a c k = ` l o s e t u p -
0001560 f - - s h o w - o $ s t a
0001600 r t $ i m a g e ` ; \n c h o p
0001620 $ l o o p b a c k ; \n \n ` e 2
0001640 f s c k - p - f $ l o o p
0001660 b a c k ` ; \n \n i f ( $ ? !
0001700 = 0 ) \n { \n \n 240 240 p r i n t
0001720 " T h e r e w a s a n e
0001740 r r o r i n t h e f i l e
0001760 s y s t e m t h a t c a n
0002000 ' t b e a u t o m a t i c a
0002020 l l y f i x e d . . . a b o
0002040 r t i n g . \ n " ; \n 240 240 ` l
0002060 o s e t u p - d $ l o o p b
0002100 a c k ` ; \n 240 240 e x i t 1 ;
0002120 \n \n } \n \n $ i n f o = ` r e
0002140 s i z e 2 f s - P $ l o o p
0002160 b a c k 2 > & 1 ` ; \n \n ( $ d
0002200 u m m y , m y $ s i z e )
0002220 = s p l i t ( ' : ' , $
0002240 i n f o , 2 ) ; \n c h o p $
0002260 s i z e ; \n $ s i z e = $ s
0002300 i z e + 1 0 2 4 ; \n \n ` s u
0002320 d o r e s i z e 2 f s - p
0002340 $ l o o p b a c k $ s i z e
0002360 2 > & 1 ` ; \n s l e e p 1 ; \n
0002400 ` l o s e t u p - d $ l o o
0002420 p b a c k ` ; \n \n $ s i z e =
0002440 ( $ s i z e * 4 0 9 6 )
0002460 + $ s t a r t ; \n \n ` p a r t
0002500 e d $ i m a g e r m $ n u
0002520 m ` ; \n ` p a r t e d - s $
0002540 i m a g e u n i t B m k p
0002560 a r t p r i m a r y $ s t a
0002600 r t $ s i z e ` ; \n \n $ s i z
0002620 e = $ s i z e + 5 8 7 2
0002640 0 2 5 7 ; \n p r i n t f " N e
0002660 w s i z e - % d M B (
0002700 % 1 . 2 f G B ) \ n " , i n
0002720 t ( $ s i z e / 1 0 4 8 5
0002740 7 6 ) , ( $ s i z e / 1 0
0002760 7 3 7 4 1 8 2 4 ) ; \n \n ` t r u
0003000 n c a t e - s $ s i z e $
0003020 i m a g e ` ; \n \n m y $ d i f
0003040 f = $ o l d - $ s i z e
0003060 ; \n p r i n t f " I m a g e
0003100 f i l e w a s r e d u c e d
0003120 b y % d M B ( % 1 . 2 f
0003140 G B ) \ n " , i n t ( $ d
0003160 i f f / 1 0 4 8 5 7 6 ) ,
0003200 ( $ d i f f / 1 0 7 3 7 4 1
0003220 8 2 4 ) ; \n \n e x i t 0 ; \n
0003237



Note the "240" characters. They're quite valid in certain contexts, but I suspect the programming tools don't like any chars above 0x7f, so complain.


Anyway, I removed them, ran the script, & :

pi@raspberrypi ~ $ sudo ./resize.pl /media/ext4/lexar16.img

lexar16.img:
============
Argument "fil" isn't numeric in division (/) at ./resize.pl line 57.
Old size - 0 MB (0.00 GB)
e2fsck: Invalid argument while trying to open /dev/loop0
Argument "Invalid argument while trying to open /dev/loop0\nCouldn..." isn't numeric in addition (+) at ./resize.pl line 68.
New size - 7820 MB (7.64 GB)
Image file was reduced by -7820 MB (-7.64 GB)

So it looks like this happens before your error trap?

SO, next I ran the SD card through fsck & both the Fat & ext4 partitions report as clear. I dd'd the image on RPi itself, but I'll try it on another computer later to compare.

Over the holiday I have used your original script & it reduced the resulting images just fine, but some SD images it does not like.....


Thank you again for your efforts on this useful tool.

[DeanC]

If you get a spare moment, can you open this forum page in a browser on your RPi, NOT an external computer, please?

I can't help you as my RPi's are learning devices, and therefore they refuse to run a GUI.

If there is a problem with copying this script from a browser on the RPi, then it's clear this isn't a script problem at all. I know that this website (raspberrypi.org) pads the 'code' tags with the 'nbsp' character. This might be giving you a problem. Also, you didn't mention which browser you are using to select the text.

[tvjon]

Yes, I agree it's not a script problem, just wondered where those characters were introduced in the chain.

I didn't mention a specific browser as they all produce the same result, so pick any.

Speaking of "learning devices", that's where I imagine the extra char's to be a problem - for folk who don't have additional traditional computers, but are learning & using just a RPi alone.

[DeanC]

Yes, I agree it's not a script problem, just wondered where those characters were introduced in the chain.

I didn't mention a specific browser as they all produce the same result, so pick any.

Speaking of "learning devices", that's where I imagine the extra char's to be a problem - for folk who don't have additional traditional computers, but are learning & using just a RPi alone.

If this is really bothersome for you, you could look at starting another post under the troubleshooting forum. http://www.raspberrypi.org/phpBB3/viewforum.php?f=28

Also, have you only used to 'select all' link, or have you tried to highlight the text yourself and copy it?

[mikerr]

I take it this should work on images created with win32diskimager ?

Just tried and get this on various images:

Code: Select all

sudo perl resizeimage.pl '/tmp/win/8gb.img'

8gb.img:
========
Use of uninitialized value $start in scalar chop at resizeimage.pl line 53.
Use of uninitialized value $old in scalar chop at resizeimage.pl line 54.
Use of uninitialized value $old in division (/) at resizeimage.pl line 55.
Use of uninitialized value $old in division (/) at resizeimage.pl line 55.
Old size - 0 MB (0.00 GB)
Use of uninitialized value $start in concatenation (.) or string at resizeimage.pl line 57.
losetup: invalid offset '/tmp/win/8gb.img' specified

[DeanC]

From the very first post:

*** THIS SCRIPT ONLY WORKS ON RASPBIAN IMAGES. ***

From my last post with the updated code:

*** THIS SCRIPT ONLY WORKS ON RASPBIAN IMAGES. ***

From the subject line of this ENTIRE post:
Reducing Raspbian File System Image Size

So after careful deliberation, I have deduced that this script probably only works with RASPBIAN image files.

[mikerr]

Yes, they are all raspbian images I tried with...
By various I meant various different backups of raspbian sd cards

I don't tend to use noobs at all.

[DeanC]

By various I meant various different backups of raspbian sd cards

So what program(s) are you using to create the .img file?

[ilovemipi]

By various I meant various different backups of raspbian sd cards

So what program(s) are you using to create the .img file?

See post 782.

[DeanC]

By various I meant various different backups of raspbian sd cards

So what program(s) are you using to create the .img file?

See post 782.

His post doesn't list a program.

[ilovemipi]

Yes it does but I can't easily see what the post number is. The 782 I referred to is his current post total. Technically it is post number 487581.
If you read what he wrote he states:

I take it this should work on images created with win32diskimager ?

I guess the most popular disk imaging software in the world today.

[DeanC]

Yes it does but I can't easily see what the post number is. The 782 I referred to is his current post total. Technically it is post number 487581.
If you read what he wrote he states:

I take it this should work on images created with win32diskimager ?

I guess the most popular disk imaging software in the world today.

Sorry, I guess I'm having an off day.

Yes... if you are using Raspbian, and you create a backup image of an SD card with Win32DiskImager then it should work. But you are saying you aren't using noobs?!? or are you?

I don't tend to use noobs at all.

[geneworld]

I've found this program and was hoping to ba able to use it but I am running into some errors and was wondering if you'd be able to help. I haven't been able to figure out the solution on my own so I've had to stop lurking and join! haha

I know you've said that this only works with Rasbian systems, does Wheezy Raspian ones count?

I created a file in by:

Code: Select all

cd home/pi
sudo nano resizeimage.pl

I pasted in the code from your first post and saved and exited.

I then tried to run

Code: Select all

sudo perl ./resizeimage.pl '/media/usb0/test.img'

And I god the following:

Code: Select all

/media/usb0/test.img does not exist.

I have usbmount installed so I didn't have to mount the external flash drive.

I then tried:

Code: Select all

cd /home/pi
mkdir image
sudo perl ./resizeimage.pl '/home/pi/image/test.img'

And I still got:

Code: Select all

/home/pi/image/test.img does not exist.

Any Idea what I am doing wrong? I am not sure if it matters but I copied the text from firefox on my windows PC and pasted it in via ssh. In the past I have had issues with the since quotes (') so I went and changed them all on the Pi and then I got the following errors:

Code: Select all

pi@raspberrypi ~ $ sudo perl ./resizeimage.pl '/home/pi/image/test.img'
Useless use of a constant (losetup -d $loopback) in void context at ./resizeimage.pl line 66.
Useless use of a constant (e2fsck -p -f $loopback) in void context at ./resizeimage.pl line 60.
Useless use of a constant (sudo resize2fs -p $loopback $size 2>&1) in void context at ./resizeimage.pl line 77.
Useless use of a constant (losetup -d $loopback) in void context at ./resizeimage.pl line 79.
Useless use of a constant (parted $image rm $num) in void context at ./resizeimage.pl line 83.
Useless use of a constant (parted -s $image unit B mkpart primary $start $size) in void context at ./resizeimage.pl line 84.
Useless use of a constant (truncate -s $size $image) in void context at ./resizeimage.pl line 89.
This should be run as root or with the sudo command.

I have run out of ideas to say the least!

[DeanC]

Can't test it out right now as I am away. But it looks like using quotes around the image file name might be causing the problem.

You only need to quote the file name if it contains spaces. Yours doesn't so you should be able to get away with:

sudo perl ./resizeimage.pl /media/usb0/test.img

[geneworld]

I actually tried that too and was getting the same errors