New Raspbian Image

[Joshun]

Hi,
I decided to make a custom Raspbian Image. This image:
-Has sudo enabled by default
-Has a set of custom scripts, making it easy to configure keyboard layout, video and sound output, and allows you to change localisation options (layout, locales, timezone). It also has a wizard that launches the first time xorg is started, allowing the user to choose common applications (browser, IDE etc.) These have been selected for their lightweightness for the Pi.
-Has build-essential, python2&3, and ruby included in the image
-Is fairly small - only 264mb download, 1.4gb image size.

I have uploaded more information, download links and the scripts here: https://github.com/Joshun/raspbian-joshun

Hope it will be of some use
Joshun

[mpthompson]

Joshun, I'm very happy to see other Raspbian based images being worked on. Nothing better to help push things forward as image builders experimenters trying different things and users giving feedback that will drive the creation of even better images in the future.

[ejsolutions]

I'm looking for a wee bit of assistance with creating an image. In the final stages I do the following but don't get the final image to anywhere near the right size.

Code: Select all

dd if=/dev/zero of=zero.file bs=1024
sync; sleep 60; sync
rm zero.file

Code: Select all

sudo dd bs=1M if=/dev/mmcblk0 conv=sync,noerror | xz -6 -c > newimage.img.xz

Note: xz compression kept at 6 to give a trade-off between speed/size/memory use - though should be comparable to bz2.
Thanks for any insight given.

[Joshun]

ejsolutions: To get mine to size I did the following:

1) Shrink the rootfs (ext4) partition of the sd card, leaving a little bit of space for growth.
2) On a Linux PC, 'sudo parted /dev/sdX' where X is the sd card's letter. Enter 'u b' in the gparted prompt, to change units to bytes. Then do 'p' to print the partition table.
3) Note down the number of bytes at which the last partition ends, for example:

Code: Select all

Number  Start      End         Size        Type     File system  Flags
 1      1048576B   84934655B   83886080B   primary  fat16        boot, lba
 2      84934656B  253362175B  168427520B  primary  ext4

Here, the last partition ends at 253362175 bytes.

4) Now run

Code: Select all

dd if=/dev/sdX of=imagefile.img bs='number of bytes last partition ends' count=1

You have to ensure the number of bytes is exactly correct. You should find the image file copies correctly, skipping the free space, provided you have used the correct number of bytes. Note: This will only work if your image uses an MBR partitioning scheme (it probably does already) - other partitioning schemes add extra information after partitions.

Joshun

[ejsolutions]

Many thanks. It appears that you are using a 'quirk' of dd, by making the block size the full partition size. Interesting - I'll give it a go.

A few moments later:

Code: Select all

sudo dd bs=3895459839 count=1 if=/dev/mmcblk0 of=newimage.img
dd: invalid number `3895459839'

dd (coreutils) 8.5

Hmm.

EJ

[Joshun]

Are you sure the SD Card's device is /dev/mmcblk0? On Linux distros i've used (Ubuntu, Debian), SD Cards plugged in are just detected as normal devices (/dev/sdb, sdc, and so on). /dev/mmcblk0 is usually only if you actually boot the system from the SD Card itself, with the card containing the rootfs. Also, try adding the 'bs' and 'count' parameters after if and of.

[ejsolutions]

Are you sure the SD Card's device is /dev/mmcblk0? On Linux distros i've used (Ubuntu, Debian), SD Cards plugged in are just detected as normal devices (/dev/sdb, sdc, and so on). /dev/mmcblk0 is usually only if you actually boot the system from the SD Card itself, with the card containing the rootfs.

Yup. Linux Mint 11 with an inbuilt SD card reader/writer and i've written a number of images - just can't get the size down to a sensible amount.

Code: Select all

 mmc0: SDHCI controller on PCI [0000:02:01.2] using DMA
 mmc0: new high speed SDHC card at address b368
 mmcblk0: mmc0:b368 SDC   3.72 GiB 

I'll try changing the parameter order..

[ejsolutions]

Code: Select all

sudo dd if=/dev/mmcblk0 of=newimage.img bs=1M count=2182

.. for a 2182Mb populated card appears to work - testing now. However, this is not what I was trying to achieve. A 'standard' 4Gb image should be able to be compressed down to the same size, given the compression of the unused space, as per other 'official' images. It's better than nothing, however.

[Joshun]

Did you actually shrink down the filesystem (rootfs) first? Give your output of parted /dev/<device>, after you have changed units to bytes (u b) and printed the partition table. Also, for better accuracy ensure you use bytes - MB will not always ensure that the end location is actually the end of the filesystem.

[ejsolutions]

Yes, I tried shrinking it down (implied by the 2182Mb) but the points are:
Your 'bs' option doesn't work with my version of dd. You appear to be writing out to disc in one large block.
Extracted images using your method will surely have rootfs as a smaller than normal size (compared to the usual 4Gb images). I am trying to create images that reflect the 'normal' method that Pisces and other distros have.
Thanks for your continuing input though.

EJ

[Joshun]

Here is my cmdline and output:

Code: Select all

joshua@joshua-laptop:~$ sudo parted /dev/sdb
[sudo] password for joshua: 
GNU Parted 2.3
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) u b
(parted) p                                                                
Model: Generic STORAGE DEVICE (scsi)
Disk /dev/sdb: 3965190144B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start       End          Size         Type     File system     Flags
 1      1048576B    79691775B    78643200B    primary  fat32           boot
 2      79691776B   335544319B   255852544B   primary  linux-swap(v1)
 3      335544320B  1384120319B  1048576000B  primary  ext4

(parted) q

End of partition 3: 1384120319
Check if partitions are mounted:

Code: Select all

joshua@joshua-laptop:~$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda6      193309916 47343500 136288288  26% /
udev              497088        4    497084   1% /dev
tmpfs             203028      872    202156   1% /run
none                5120        0      5120   0% /run/lock
none              507564      312    507252   1% /run/shm
/dev/sdb1          76186    34455     41731  46% /media/2B76-A22E
/dev/sdb3        1017648   910768     55692  95% /media/dda7be77-7cb0-4a16-9b7d-400834f2bb79

Unmount SD partitions

Code: Select all

umount /dev/sdb1
umount /dev/sdb3

Clone disk with dd (I'd recommend doing this outside of X, in a tty, since it is very CPU intensive)

Code: Select all

sudo dd if=/dev/sdb of=imagefile.img bs=1384120319 count=1
joshua@joshua-laptop:~$ ls -l -h imagefile.img 
-rw-r--r-- 1 root root 1.3G Jul 14 17:57 imagefile.img

===EDIT: bs=1 count=1384120319 also works and doesn't seem to lock up the CPU. Expect it to run rather slowly though.===
Be sure to get the value from the 'end' column of the last partition - in this case 1384120319 (without the 'B' when passed to dd). Before I carried out these steps, I shrunk the last (ext4) partition leaving a little bit of space:


My dd version:

Code: Select all

joshua@joshua-laptop:~$ dd --version
dd (coreutils) 8.13
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Paul Rubin, David MacKenzie and Stuart Kemp.

Joshun

[jojopi]

A 'standard' 4Gb image should be able to be compressed down to the same size, given the compression of the unused space, as per other 'official' images.

How much difference are you seeing?

If I understand you correctly then I agree that filling the unused space with zeros should be almost as good for compression purposes as shrinking the filesystem and truncating the image. (Not quite as good, because the bigger filesystem will have extra metadata that can not be zeroed.)

Here I double the size of a filesystem without adding any new data. Storage size increases 1.1% because of the extra metadata, but compressed size increases only 0.22%:

Code: Select all

$ ls -log E4FS 
-rw-rw-r--. 1 1667235840 Jul 14 15:04 E4FS
$ du -k E4FS
1628164 E4FS
$ gzip <E4FS |wc -c
473501614
$ truncate -s $((2*1590*1024*1024)) E4FS 
$ resize2fs E4FS 
resize2fs 1.41.14 (22-Dec-2010)
Resizing the filesystem on E4FS to 814080 (4k) blocks.
The filesystem on E4FS is now 814080 blocks long.
$ ls -log E4FS
-rw-rw-r--. 1 3334471680 Jul 14 15:13 E4FS
$ du -k E4FS
1645812 E4FS
$ gzip <E4FS |wc -c
474551711

Resizing down is an extremely slow process that is best avoided. I have also seen resize2fs get into infinite loops during this operation.

[ejsolutions]

In short, 3.7GB is only compressing down to 543MB.

[Joshun]

It may also depend on what compression you are using - bzip2 is better than gzip (of course!), and 7zip is supposed to be better than bzip2 (though slower). 3.7GB down to 543MB is still pretty good though.

[rrolsbe]

Here is what I would do to create an img of my SD card.

Mount both the vfat and ext4 partitions and fill both using dd /dev/zero as discussed in this thread. After that, I would dd /dev/zero the swap partition then recreate afterword. Then with both partitions mounted, cd to / and issue these commands #du -khs and/or #df -kh to see your pre-compression SD card usage. What ever this usage is the final image should compress to about half that size using unxz compression. BTW, if possible, I would do all of the above on another more powerful machine.

Hope this helps.
Regards, Ron

[ChadP]

I put the image onto a sd card when it loads I put username and password in then how do I get it to load I tried start x but nothing

[Joshun]

Hi, the initial admin (root) password is raspberry (first boot). You enter this at the prompt which says: 'Give root password for maintenance' (it is possible to disable this but not recommended, since it would make the system vulnerable). After this has run, on next boot in a tty:
Username: pi Password: raspberry
Startx will not work initially since X needs to be installed, choose 'Install Desktop System' from the administration menu to install this.

If the boot script is still causing problems for you, edit cmdline.txt and remove 'single' from the end. The system will boot straight into a tty, and you can always run

Code: Select all

sudo raspi-setup

later to configure the system.

Joshun

EDIT - oh, I saw you were just having problems starting xorg. Just run

Code: Select all

sudo raspi-setup

,choose 'Install Desktop System' and xorg will be installed for you, along with fluxbox. It can then be started with the normal startx command.

[rrolsbe]

Since I do not know how to edit a previous post, this is an addendum to my previous post. Please see my post a few back for initial information.

If your partitions and swap do not use the entire flash device, when you dd at the device level you will be compressing data for the entire device. The flash controller uses any area not containing a partition for garbage collection and wear leveling. So make sure you dd /dev/zero any area not assigned to a partition (see dd man page, I believe it is the seek option). Better yet just tar/compress separately the vfat and ext4 partitions and let the user create the partitions and un-compress/tar the information into the proper partition (but this assume some Linux expertise). If you have an 8GB SDHC card and your partitions only use 4GB, when you dd at the device level the destination device will need to be at least 8Gb in size or you will get an error in dd. I believe most of the dd images available for the Pi are small and expand to 4G or more (magic) on the 1st boot to get around the potential problem described above.

Regards, Ron

[Joshun]

Both methods would safe unused space, yours a bit more foolproof than mine. Although the advantage of skipping unused space is that the image would easily fit on a smaller SD Card (e.g. 2GB), with still leaving extra space.

[ejsolutions]

Maybe mpthompson (or another of the major distro packagers) could kindly explain how they produce their image(s).
Example: rpi_pisces_r3.zip is 447.7MB yet expands to 4Gb when dd'ed out to a card, without a reboot. I'd assume that an xz version would be a fair deal smaller - see Puppi for an example of what can be achieved.

[joe wraith]

Sounds great.

[jojopi]

Example: rpi_pisces_r3.zip is 447.7MB yet expands to 4Gb when dd'ed out to a card, without a reboot. I'd assume that an xz version would be a fair deal smaller - see Puppi for an example of what can be achieved.

How does "df /" in your image compare to images that compress better? Most likely you are just including more stuff.

Did you remember to "sudo apt-get clean" before the dd zero? Otherwise there may be unnecessary debs in /var/cache/apt/archives.

[ejsolutions]

How does "df /" in your image compare to images that compress better? Most likely you are just including more stuff.

Actually not that much more; wireless-utils and firmware-ralink, plus pre-configuring a few bits.
Though obviously the xbmc derivative that I create is considerably bigger.

Did you remember to "sudo apt-get clean" before the dd zero? Otherwise there may be unnecessary debs in /var/cache/apt/archives.

Aye, that's pretty basic stuff.

Kinda given up on creating images for now, due to it taking so long to do, even with Class 10 cards. Once I have a definitive approach, then I'll get back to it. Time to play with MATE some more, though I should really be doing 'work' (e-commerce/server stuff)!

EJ

[mpthompson]

ejsolutions,

The things I do to create an image were mostly covered above. Some of the specific commands are below:

To clean any cached Debian packages:

Code: Select all

apt-get clean

To zero a disk partition are, do the following in a directory on the partition you want to zero:

Code: Select all

dd if=/dev/zero of=zero.small.file bs=1024 count=102400
dd if=/dev/zero of=zero.file bs=1024
sync ; sleep 60 ; sync
rm zero.small.file
rm zero.file

Notes: the purpose of the small file is to reduce the time when free space is completely zero; the purpose of sync is to make sure the data is actually written.

To zero out the swap partition:

Code: Select all

swapoff -a
shred -n 0 -z /dev/mmcblk0p2
mkswap /dev/mmcblk0p2

I also use fdisk to check the total number of blocks on the file system I want to image and then use that total count with the dd command:

Code: Select all

dd if=/dev/sdc of=rpi_pisces_r3.img bs=512 count=7618560

Then I use zip on the final image. I know I can get better compression with other algorithms, but zip is a little easier for most people to deal with. Particularly those with Windows PCs.

[ejsolutions]

Muchos gracias amigo.
Yes, I spotted most of that from your bash history in the image, but missed some key points: small zeroed file and the shred.
I'm interested in why you only use the default 512 block size. When doing a a partition resize/move, gparted seems to optimise at 4M blocks, for the cards that I'm using.
Thanks again - appreciated.

EJ