Reducing Raspbian File System Image Size


71 posts   Page 1 of 3   1, 2, 3
by DeanC » Sat Oct 12, 2013 7:29 pm
This script will take a Raspbian image (.img) file and reduce it to it's smallest size. Once the reduce image is copied to a new SD card, make sure you expand the file system.

I have added a test in the script. It will stop the script from executing if an error is encountered while checking the file system that e2fsck cannot automatically fix. There is no reason to continue the script if the image is corrupt.

*** THIS SCRIPT ONLY WORKS ON RASPBIAN IMAGES. ***

To use this script, copy the code to a file called 'resizeimage.pl'. Then issue 'sudo perl ./resizeimage.pl'.

After burning, always remember to run 'sudo raspi-config' first thing, and expand the file system before use.

Code: Select all
#!/usr/bin/perl

use utf8;
use 5.010;
use strict;
#use autodie;
use warnings;
#use diagnostics;

my $who = `whoami`;

if ($who !~ /root/)
{

   print "This should be run as root or with the sudo command.\n";
   exit 1;

}

if (!$ARGV[0])
{
   
   print "No image file given.\n";
   exit 1;
   
}

my $image = $ARGV[0];

if ($image !~ /^\//)
{

   print "Please enter full path to image file.\n";
   exit 1;
   
}

if (! -e $image)
{

   print "$image does not exist.\n";
   exit 1;
   
}

my @name = split (/\//, $image);
print "\n$name[(scalar @name) - 1]:\n";
print "=" x (length ($name[(scalar @name) - 1]) + 1) . "\n";

my $info = `parted -m $image unit B print | grep ext4`;

(my $num, my $start, my $old, my $dummy) = split (':', $info, 4);
chop $start;
chop $old;
printf "Old size - %d MB (%1.2f GB)\n", int ($old / 1048576), ($old / 1073741824);

my $loopback = `losetup -f --show -o $start $image`;
chop $loopback;

`e2fsck -p -f $loopback`;

if ($? != 0)
{

   print "There was an error in the file system that can't be automatically fixed... aborting.\n";
   `losetup -d $loopback`;
   exit 1;

}

$info = `resize2fs -P $loopback 2>&1`;

($dummy, my $size) = split (': ', $info, 2);
chop $size;
$size = $size + 1024;

`sudo resize2fs -p $loopback $size 2>&1`;
sleep 1;
`losetup -d $loopback`;

$size = ($size * 4096) + $start;

`parted $image rm $num`;
`parted -s $image unit B mkpart primary $start $size`;

$size = $size + 58720257;
printf "New size - %d MB (%1.2f GB)\n", int ($size / 1048576), ($size / 1073741824);

`truncate -s $size $image`;

my $diff = $old - $size;
printf "Image file was reduced by %d MB (%1.2f GB)\n", int ($diff / 1048576), ($diff / 1073741824);

exit 0;
Last edited by DeanC on Fri Jan 10, 2014 4:14 pm, edited 3 times in total.
We 'idiot proofed' the world, and now it's full of idiots!
User avatar
Posts: 136
Joined: Thu Sep 26, 2013 4:07 pm
Location: Vancouver, Canada
by DeanC » Tue Oct 15, 2013 2:59 pm
This is getting frustrating. The only way I can seem to find is to use gparted. But I am not running with a gui, only command line. When I look for how to do it with parted, all I get are results that either don't work, or people using gparted.

I have a feeling that gparted still uses parted, so there must be a way to do this from the command line.

Anybody have any idea?
We 'idiot proofed' the world, and now it's full of idiots!
User avatar
Posts: 136
Joined: Thu Sep 26, 2013 4:07 pm
Location: Vancouver, Canada
by RaTTuS » Tue Oct 15, 2013 3:04 pm
yes you can do it via parted - I'm not sure on the magic incantations myself...
you could look at using
rsync to make filesystem copies

what are you buring though images ?
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
How Auto Run a script :- http://www.raspberrypi.org/forums/viewtopic.php?f=29&t=7192
1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
User avatar
Posts: 8278
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
by duberry » Tue Oct 15, 2013 4:33 pm
depending on what you need to do with your os and
how comfortable u are with linux/terminal
you might find this seemingly unrelated topic interesting
"256 MB SD card usable for RaspberryPI?"

sadly i dont know a lot about making full file system back up
but i have a little interest / knowledge in booting a system that separates
- os
- packages
- user-files

in some *cases this can make slimmer backups as simple as
copying "user-data.tgz"
saving a list of needed/loaded packages
and if necessary a copy of kernel + initrd


*this is a link to what would be my choice for a system with small footprint / backup
it might not be for everyone
nonetheless just another option for you consideration
its linux jim but not as we know it !
lend me your arms, fast as thunderbolts, for a pillow on my journey.
If the environment was a bank, would it be too big to fail
so long; and thanks for all the pi
User avatar
Posts: 380
Joined: Mon Jan 28, 2013 10:44 pm
Location: standing on a planet that's evolving. And revolving at nine hundred miles an hour
by SirLagz » Wed Oct 16, 2013 8:04 am
Do you want to minimise the size before or after making the dd image ?
You can do it with parted, I've made a script to resize an image from whatever size down to the minimum size + 100 megabytes.
So if you have an image with 600 megs of data, it'll resize the image down to ~700 megs.
My Blog - http://www.sirlagz.net
Visit my blog for Tips, Tricks, Guides and More !
WiFi Issues ? Have a look at this post ! http://www.raspberrypi.org/phpBB3/viewtopic.php?f=28&t=44044
Posts: 1704
Joined: Mon Feb 20, 2012 8:53 am
Location: Perth, Australia
by ramstrong » Wed Oct 16, 2013 8:47 pm
Backing up is a very difficult subject. I actually spent a week learning how to do it! That said, here's what I came up with:
1. dd. Clone your SD card to another one.
2. Next time you want to back up, use rsync
3. If you want to keep running copies, use tar with -z option.

Reducing SD image is to be done as a last resort. You can use sfdisk -d option to get partition table, which you can use with sfdisk to write to the card (see man page for details).

If you want to format the partition to a specific file system, use mkfs command.

HTH
Raspberry Pi Journal: http://simpletongeek.blogspot.com/p/raspberry-pi-journal-directory_4.html
Posts: 36
Joined: Thu Aug 15, 2013 11:14 pm
by DeanC » Thu Oct 17, 2013 10:53 pm
RaTTuS wrote:what are you buring though images ?


Burning the images I either use DiskImager (win) or dd (linux). I am very comfortable with using dd, as I learn the hard way years ago, and have never forgotten.

I know I can delete a partition using fdisk, and keeping in mind the starting point of the deleted partition, build a new one that is smaller. The problem is I have no way of guaranteeing that the file system only uses the partition sequentially from the beginning (ie the data is always at the beginning). It is this missing step that gparted is doing, that I just can't figure out.

SirLagz wrote:Do you want to minimise the size before or after making the dd image ?
You can do it with parted, I've made a script to resize an image from whatever size down to the minimum size + 100 megabytes.
So if you have an image with 600 megs of data, it'll resize the image down to ~700 megs.


Doesn't matter when the resize happens to me. Would you be comfortable posting your script?
We 'idiot proofed' the world, and now it's full of idiots!
User avatar
Posts: 136
Joined: Thu Sep 26, 2013 4:07 pm
Location: Vancouver, Canada
by SirLagz » Tue Oct 22, 2013 1:39 pm
Sorry about the late reply.
Sure I'll get my script up as soon as I clean up a few more bits. I've been slack on it lately haha.
I have an older version of the script on my blog, but I don't think it works so well anymore.
However you can get an idea of how it resizes images though.

http://sirlagz.net/2013/03/10/script-au ... downsizer/
My Blog - http://www.sirlagz.net
Visit my blog for Tips, Tricks, Guides and More !
WiFi Issues ? Have a look at this post ! http://www.raspberrypi.org/phpBB3/viewtopic.php?f=28&t=44044
Posts: 1704
Joined: Mon Feb 20, 2012 8:53 am
Location: Perth, Australia
by DeanC » Thu Oct 24, 2013 3:08 am
SirLagz wrote:Sorry about the late reply.
Sure I'll get my script up as soon as I clean up a few more bits. I've been slack on it lately haha.
I have an older version of the script on my blog, but I don't think it works so well anymore.
However you can get an idea of how it resizes images though.

http://sirlagz.net/2013/03/10/script-au ... downsizer/


Ty for that. It was very insightful. Although, a little hard to read at times, as you have about 4 variable assignments that do absolutely nothing (strImgFile; partnumber; part1; part2).

I have re-written it in perl as I am more comfortable in that.

I noticed after all your steps are completed, you are still left with a file that is still the same size as the original img file (although the data is only in the first portion of the file). So I have also added one final step. The dd command at the end will take the newly shrunk image and copy it to a smaller file.

Also, this is just a quick first draft for those reading this. I am currently on a different version which I think would work better, because using resize2fs to tell you the minimum size is useless as it's almost never correct. So keep posted.

Code: Select all
Please use the updated code posted in first post.


Also, remember when you burn the new image to another card, to always run 'sudo raspi-config' first thing and expand the filesystem This script currently only give an extra 30 MB after the shrink of free space on the ext4 partition (assuming resize2fs was close about the minimum size).
Last edited by DeanC on Mon Jan 13, 2014 4:13 am, edited 3 times in total.
We 'idiot proofed' the world, and now it's full of idiots!
User avatar
Posts: 136
Joined: Thu Sep 26, 2013 4:07 pm
Location: Vancouver, Canada
by SirLagz » Thu Oct 24, 2013 6:10 am
There should have been a truncate command at the end that cuts the file down to the right size...not sure if that got missed from the script when I pasted it :oops:
That was a very messy version of the script, the new one that I'm doing now is much cleaner :D

Also I've found that resize2fs minimal size is normally correct, if the filesystem uses 4k blocks.
If it doesn't then it's quite a bit off.
My Blog - http://www.sirlagz.net
Visit my blog for Tips, Tricks, Guides and More !
WiFi Issues ? Have a look at this post ! http://www.raspberrypi.org/phpBB3/viewtopic.php?f=28&t=44044
Posts: 1704
Joined: Mon Feb 20, 2012 8:53 am
Location: Perth, Australia
by swindmiller » Thu Oct 24, 2013 12:48 pm
DeanC,

I would be very interested in what you have if you are willing to share :)

I am just starting with the PI and have been backing up and re-imaging using Win32DiskImager and am using a 32gb card which ofcourse gives me a very big backup. I believe I am only using like 3gb so would love to just image it to a 4gb card once I get my second PI.
Also keeping a 3gb backup file would be much easier than a 32gb one :)

Thanks,
Scott
Posts: 23
Joined: Mon Oct 14, 2013 1:58 pm
by DeanC » Thu Oct 24, 2013 2:34 pm
SirLagz wrote:There should have been a truncate command at the end that cuts the file down to the right size...not sure if that got missed from the script when I pasted it :oops:


I did see that truncate command and just assumed it was for parsing the result as to whether it was successful. I didn't know it actually truncated the file, I will look into it more closely. I have never used bash for scripting, so when I got near the end of your script, I guess I stopped short.

swindmiller wrote:DeanC,

I would be very interested in what you have if you are willing to share :)

I am just starting with the PI and have been backing up and re-imaging using Win32DiskImager and am using a 32gb card which ofcourse gives me a very big backup. I believe I am only using like 3gb so would love to just image it to a 4gb card once I get my second PI.
Also keeping a 3gb backup file would be much easier than a 32gb one :)

Thanks,
Scott


This is exactly why I have done this.

If you want to know roughly how small you can make your image, just issue 'df -h' and read the line containing 'rootfs'. Then add another 60 MB for the dos partition. This should be roughly the smallest you can make it.
We 'idiot proofed' the world, and now it's full of idiots!
User avatar
Posts: 136
Joined: Thu Sep 26, 2013 4:07 pm
Location: Vancouver, Canada
by swindmiller » Thu Oct 24, 2013 2:38 pm
Thanks. I am going to try the script you already posted but if you have something better and want to share it I would be grateful :D
Posts: 23
Joined: Mon Oct 14, 2013 1:58 pm
by swindmiller » Thu Oct 24, 2013 5:23 pm
This worked great!!! Brought my 32gb image down to 1.7gb :D
I booted it up and everything seems to work fine so far, and I did expand the filesystem after the first boot.

Thanks so much!!!
Posts: 23
Joined: Mon Oct 14, 2013 1:58 pm
by DeanC » Sat Oct 26, 2013 12:35 am
swindmiller wrote:Thanks so much!!!


Excellent to hear. Ty.
We 'idiot proofed' the world, and now it's full of idiots!
User avatar
Posts: 136
Joined: Thu Sep 26, 2013 4:07 pm
Location: Vancouver, Canada
by rcblackwell » Sat Oct 26, 2013 3:12 pm
DeanC wrote:.... this is just a quick first draft for those reading this. I am currently on a different version which I think would work better, because using resize2fs to tell you the minimum size is useless as it's almost never correct. So keep posted.


I'm a newbie to scripting language thus I'm having difficulty running the script as provided. I've managed to create the script file however its not clear how the script should be run or where the resulting image file will be stored. My attempt at executing the script;

Code: Select all
sudo perl resizeimage.pl


resulted in an error; "No image file given." Makes sense, the script needs a place to store the resulting image file. Problem is I can't figure out where or how to store the file. Do I use USB storage and if so, what would the command line be for starting the script and saving the resultant image file to the USB storage device?
-------------------------
Robert Blackwell
Pickering, Ontario
Canada

www.blackwellgallery.com
www.afticarr.com
Posts: 13
Joined: Thu Oct 03, 2013 7:26 pm
Location: Pickering, Ontario, Canada
by swindmiller » Sat Oct 26, 2013 3:24 pm
I copied my large image to the hard drive and just put the path to it after the script. So in your case
sudo perl resizeimage.pl /WhereverYouPutIt/large image.img and the smaller image will be in the same folder.
I ran sudo resizeimage.pl /home/swindmiller/PIimage.img

Hope that helps.
Posts: 23
Joined: Mon Oct 14, 2013 1:58 pm
by rcblackwell » Sat Oct 26, 2013 3:34 pm
swindmiller wrote:Hope that helps.


Immensely! Thanks
-------------------------
Robert Blackwell
Pickering, Ontario
Canada

www.blackwellgallery.com
www.afticarr.com
Posts: 13
Joined: Thu Oct 03, 2013 7:26 pm
Location: Pickering, Ontario, Canada
by rcblackwell » Sat Oct 26, 2013 5:22 pm
swindmiller wrote:So in your case sudo perl resizeimage.pl /WhereverYouPutIt/large image.img and the smaller image will be in the same folder.


I thought I had this beat until attempting to run the script under Cygwin on a Windows PC. Unfortunately Cygwin doesn't have a 'parted' package available for installation :o .

I can't run the script from my Raspberry Pi as I don't have a storage device large enough to hold the original image. That is unless the Pi can access the HD on my WIndows PC?

Would someone point me in a direction that would help me understand how to access an image file stored on a windows PC from the Raspberry Pi? Failing that, is there a version of this script that can be run under Windows (8)?

TIA
-------------------------
Robert Blackwell
Pickering, Ontario
Canada

www.blackwellgallery.com
www.afticarr.com
Posts: 13
Joined: Thu Oct 03, 2013 7:26 pm
Location: Pickering, Ontario, Canada
by DeanC » Sat Oct 26, 2013 6:33 pm
rcblackwell wrote:I thought I had this beat until attempting to run the script under Cygwin on a Windows PC. Unfortunately Cygwin doesn't have a 'parted' package available for installation :o .


Yes, unfortunately this script uses specific programs only available to Linux. Do you have a usb stick that the image would fit onto? Copy the image file onto it. It can be formatted for windows (just NOT NTFS).

Plug it into your Pi. Now, assuming you don't have any other storage devices connected to your Pi, the following commands should mount the usb stick:

sudo mkdir /media/usb1
sudo mount /dev/sda1 /media/usb1

run the script on the image file. Then as a final step, before removing the usb stick, issue this command:

sudo umount /media/usb1
We 'idiot proofed' the world, and now it's full of idiots!
User avatar
Posts: 136
Joined: Thu Sep 26, 2013 4:07 pm
Location: Vancouver, Canada
by DeanC » Sat Oct 26, 2013 6:38 pm
Sorry, re-read your post. You need to look at installing Samba. But I don't use it, so I am of little help.
We 'idiot proofed' the world, and now it's full of idiots!
User avatar
Posts: 136
Joined: Thu Sep 26, 2013 4:07 pm
Location: Vancouver, Canada
by DeanC » Mon Oct 28, 2013 12:19 am
*** THIS SCRIPT ONLY WORKS ON RASPBIAN IMAGES. ***

Ok, here is a newer version. This one uses the truncate command, which eliminates the need to duplicate the image file.

To use this script, copy the code to a file called 'resizeimage.pl'. Then issue 'sudo perl ./resizeimage.pl'.

After burning, always remember to run 'sudo raspi-config' first thing, and expand the file system before use.

Code: Select all
Please use the updated code posted above.


Please let me know if any problems arise from this script.
Last edited by DeanC on Mon Jan 13, 2014 4:19 am, edited 6 times in total.
We 'idiot proofed' the world, and now it's full of idiots!
User avatar
Posts: 136
Joined: Thu Sep 26, 2013 4:07 pm
Location: Vancouver, Canada
by rcblackwell » Mon Oct 28, 2013 4:04 pm
DeanC wrote:Sorry, re-read your post. You need to look at installing Samba. But I don't use it, so I am of little help.


Being somewhat familiar with SAMBA from my days of hacking about with a D-Link server, I decided to followup on the idea. I'm pleased to say I've had success and that my second image file is being re-sized as I type. Here is what I did;

Code: Select all
### Install SAMBA
sudo apt-get install samba

### Add Linux User
sudo useradd WindowsUser -m -G users

### Set Linux User Password
sudo passwd WindowsUser

### Add SAMBA User and set Passwd
sudo pdbedit -a -u WindowsUser

### Edit Samba Config
sudo nano smb.conf

### Add these lines to the Global Section
   read raw = yes
   write raw = yes
   wide links = yes
   getwd cache = yes
   strict sync = no
   use sendfile = yes
   max xmit = 131072
   min receivefile size = 16384
   aio read size = 16384
   aio write size = 16384
   write cache size = 8388608

### Find the line that starts out reading 'socket options='. Change that line to read;
   socket options = TCP_NODELAY SO_RCVBUF=262144 SO_SNDBUF=262144 IPTOS_LOWDELAY SO_KEEPALIVE

### Add these lines to the end of the smb.conf file, save and exit
   [public]
   comment = Public
   path = /mnt/share
   valid users = @users
   force group = users
   create mask = 0660
   directory mask = 0771
   read only = no

### Restart Samba Services
sudo service samba restart

### Mount Shared Windows Files
sudo mount -t cifs //192.168.x.x/users/WindowsUser -o username=WindowsUser,password='WindowsUserPasswd' /home/WindowsUser

### Run the script
sudo perl shrink-image.pl '/home/WindowsUser/Path_To_Shared_Windows_Files/Raspberry_Pi_Image_File.img'



I mapped (using mount) shared files from the Windows PC to the Linux User account (WindowsUSer) that was created on the Raspberry Pi. This made it easier to find and work with the image files.

Thank you for the assistance....
-------------------------
Robert Blackwell
Pickering, Ontario
Canada

www.blackwellgallery.com
www.afticarr.com
Posts: 13
Joined: Thu Oct 03, 2013 7:26 pm
Location: Pickering, Ontario, Canada
by rcblackwell » Mon Oct 28, 2013 11:09 pm
DeanC wrote:Ok, here is a newer version....


Thank you. Worked very well for me.
-------------------------
Robert Blackwell
Pickering, Ontario
Canada

www.blackwellgallery.com
www.afticarr.com
Posts: 13
Joined: Thu Oct 03, 2013 7:26 pm
Location: Pickering, Ontario, Canada
by DeanC » Thu Nov 14, 2013 9:34 pm
I have also written another script to help in reminding to expand the filesystem.

viewtopic.php?f=91&t=60918
We 'idiot proofed' the world, and now it's full of idiots!
User avatar
Posts: 136
Joined: Thu Sep 26, 2013 4:07 pm
Location: Vancouver, Canada