New Raspbian Image
Hi,
I decided to make a custom Raspbian Image. This image:
-Has sudo enabled by default
-Has a set of custom scripts, making it easy to configure keyboard layout, video and sound output, and allows you to change localisation options (layout, locales, timezone). It also has a wizard that launches the first time xorg is started, allowing the user to choose common applications (browser, IDE etc.) These have been selected for their lightweightness for the Pi.
-Has build-essential, python2&3, and ruby included in the image
-Is fairly small - only 264mb download, 1.4gb image size.
I have uploaded more information, download links and the scripts here: https://github.com/Joshun/raspbian-joshun
Hope it will be of some use
Joshun
I decided to make a custom Raspbian Image. This image:
-Has sudo enabled by default
-Has a set of custom scripts, making it easy to configure keyboard layout, video and sound output, and allows you to change localisation options (layout, locales, timezone). It also has a wizard that launches the first time xorg is started, allowing the user to choose common applications (browser, IDE etc.) These have been selected for their lightweightness for the Pi.
-Has build-essential, python2&3, and ruby included in the image
-Is fairly small - only 264mb download, 1.4gb image size.
I have uploaded more information, download links and the scripts here: https://github.com/Joshun/raspbian-joshun
Hope it will be of some use
Joshun
- mpthompson
- Posts: 620
- Joined: Fri Feb 03, 2012 7:18 pm
- Location: San Carlos, CA
- Contact: Website
Re: New Raspbian Image
Joshun, I'm very happy to see other Raspbian based images being worked on. Nothing better to help push things forward as image builders experimenters trying different things and users giving feedback that will drive the creation of even better images in the future.
-
- Posts: 96
- Joined: Fri Jun 29, 2012 2:27 pm
- Location: In the sticks of Haggisland
Re: New Raspbian Image
I'm looking for a wee bit of assistance with creating an image. In the final stages I do the following but don't get the final image to anywhere near the right size.
Note: xz compression kept at 6 to give a trade-off between speed/size/memory use - though should be comparable to bz2.
Thanks for any insight given.
Code: Select all
dd if=/dev/zero of=zero.file bs=1024
sync; sleep 60; sync
rm zero.file
Code: Select all
sudo dd bs=1M if=/dev/mmcblk0 conv=sync,noerror | xz -6 -c > newimage.img.xz
Thanks for any insight given.
Re: New Raspbian Image
ejsolutions: To get mine to size I did the following:
1) Shrink the rootfs (ext4) partition of the sd card, leaving a little bit of space for growth.
2) On a Linux PC, 'sudo parted /dev/sdX' where X is the sd card's letter. Enter 'u b' in the gparted prompt, to change units to bytes. Then do 'p' to print the partition table.
3) Note down the number of bytes at which the last partition ends, for example:
Here, the last partition ends at 253362175 bytes.
4) Now run
You have to ensure the number of bytes is exactly correct. You should find the image file copies correctly, skipping the free space, provided you have used the correct number of bytes. Note: This will only work if your image uses an MBR partitioning scheme (it probably does already) - other partitioning schemes add extra information after partitions.
Joshun
1) Shrink the rootfs (ext4) partition of the sd card, leaving a little bit of space for growth.
2) On a Linux PC, 'sudo parted /dev/sdX' where X is the sd card's letter. Enter 'u b' in the gparted prompt, to change units to bytes. Then do 'p' to print the partition table.
3) Note down the number of bytes at which the last partition ends, for example:
Code: Select all
Number Start End Size Type File system Flags
1 1048576B 84934655B 83886080B primary fat16 boot, lba
2 84934656B 253362175B 168427520B primary ext4
4) Now run
Code: Select all
dd if=/dev/sdX of=imagefile.img bs='number of bytes last partition ends' count=1
Joshun
-
- Posts: 96
- Joined: Fri Jun 29, 2012 2:27 pm
- Location: In the sticks of Haggisland
Re: New Raspbian Image
Many thanks. It appears that you are using a 'quirk' of dd, by making the block size the full partition size. Interesting - I'll give it a go. 
A few moments later:
dd (coreutils) 8.5
Hmm.
EJ

A few moments later:
Code: Select all
sudo dd bs=3895459839 count=1 if=/dev/mmcblk0 of=newimage.img
dd: invalid number `3895459839'
Hmm.
EJ
Re: New Raspbian Image
Are you sure the SD Card's device is /dev/mmcblk0? On Linux distros i've used (Ubuntu, Debian), SD Cards plugged in are just detected as normal devices (/dev/sdb, sdc, and so on). /dev/mmcblk0 is usually only if you actually boot the system from the SD Card itself, with the card containing the rootfs. Also, try adding the 'bs' and 'count' parameters after if and of.
-
- Posts: 96
- Joined: Fri Jun 29, 2012 2:27 pm
- Location: In the sticks of Haggisland
Re: New Raspbian Image
Yup. Linux Mint 11 with an inbuilt SD card reader/writer and i've written a number of images - just can't get the size down to a sensible amount.Joshun wrote:Are you sure the SD Card's device is /dev/mmcblk0? On Linux distros i've used (Ubuntu, Debian), SD Cards plugged in are just detected as normal devices (/dev/sdb, sdc, and so on). /dev/mmcblk0 is usually only if you actually boot the system from the SD Card itself, with the card containing the rootfs.
Code: Select all
mmc0: SDHCI controller on PCI [0000:02:01.2] using DMA
mmc0: new high speed SDHC card at address b368
mmcblk0: mmc0:b368 SDC 3.72 GiB

-
- Posts: 96
- Joined: Fri Jun 29, 2012 2:27 pm
- Location: In the sticks of Haggisland
Re: New Raspbian Image
Code: Select all
sudo dd if=/dev/mmcblk0 of=newimage.img bs=1M count=2182
Re: New Raspbian Image
Did you actually shrink down the filesystem (rootfs) first? Give your output of parted /dev/<device>, after you have changed units to bytes (u b) and printed the partition table. Also, for better accuracy ensure you use bytes - MB will not always ensure that the end location is actually the end of the filesystem.
-
- Posts: 96
- Joined: Fri Jun 29, 2012 2:27 pm
- Location: In the sticks of Haggisland
Re: New Raspbian Image
Yes, I tried shrinking it down (implied by the 2182Mb) but the points are:
Your 'bs' option doesn't work with my version of dd. You appear to be writing out to disc in one large block.
Extracted images using your method will surely have rootfs as a smaller than normal size (compared to the usual 4Gb images). I am trying to create images that reflect the 'normal' method that Pisces and other distros have.
Thanks for your continuing input though.
EJ
Your 'bs' option doesn't work with my version of dd. You appear to be writing out to disc in one large block.
Extracted images using your method will surely have rootfs as a smaller than normal size (compared to the usual 4Gb images). I am trying to create images that reflect the 'normal' method that Pisces and other distros have.
Thanks for your continuing input though.
EJ
Re: New Raspbian Image
Here is my cmdline and output:
End of partition 3: 1384120319
Check if partitions are mounted:
Unmount SD partitions
Clone disk with dd (I'd recommend doing this outside of X, in a tty, since it is very CPU intensive)
===EDIT: bs=1 count=1384120319 also works and doesn't seem to lock up the CPU. Expect it to run rather slowly though.===
Be sure to get the value from the 'end' column of the last partition - in this case 1384120319 (without the 'B' when passed to dd). Before I carried out these steps, I shrunk the last (ext4) partition leaving a little bit of space:

My dd version:
Joshun
Code: Select all
joshua@joshua-laptop:~$ sudo parted /dev/sdb
[sudo] password for joshua:
GNU Parted 2.3
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) u b
(parted) p
Model: Generic STORAGE DEVICE (scsi)
Disk /dev/sdb: 3965190144B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 1048576B 79691775B 78643200B primary fat32 boot
2 79691776B 335544319B 255852544B primary linux-swap(v1)
3 335544320B 1384120319B 1048576000B primary ext4
(parted) q
Check if partitions are mounted:
Code: Select all
joshua@joshua-laptop:~$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 193309916 47343500 136288288 26% /
udev 497088 4 497084 1% /dev
tmpfs 203028 872 202156 1% /run
none 5120 0 5120 0% /run/lock
none 507564 312 507252 1% /run/shm
/dev/sdb1 76186 34455 41731 46% /media/2B76-A22E
/dev/sdb3 1017648 910768 55692 95% /media/dda7be77-7cb0-4a16-9b7d-400834f2bb79
Code: Select all
umount /dev/sdb1
umount /dev/sdb3
Code: Select all
sudo dd if=/dev/sdb of=imagefile.img bs=1384120319 count=1
joshua@joshua-laptop:~$ ls -l -h imagefile.img
-rw-r--r-- 1 root root 1.3G Jul 14 17:57 imagefile.img
Be sure to get the value from the 'end' column of the last partition - in this case 1384120319 (without the 'B' when passed to dd). Before I carried out these steps, I shrunk the last (ext4) partition leaving a little bit of space:

My dd version:
Code: Select all
joshua@joshua-laptop:~$ dd --version
dd (coreutils) 8.13
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Paul Rubin, David MacKenzie and Stuart Kemp.
Re: New Raspbian Image
How much difference are you seeing?ejsolutions wrote:A 'standard' 4Gb image should be able to be compressed down to the same size, given the compression of the unused space, as per other 'official' images.
If I understand you correctly then I agree that filling the unused space with zeros should be almost as good for compression purposes as shrinking the filesystem and truncating the image. (Not quite as good, because the bigger filesystem will have extra metadata that can not be zeroed.)
Here I double the size of a filesystem without adding any new data. Storage size increases 1.1% because of the extra metadata, but compressed size increases only 0.22%:
Code: Select all
$ ls -log E4FS
-rw-rw-r--. 1 1667235840 Jul 14 15:04 E4FS
$ du -k E4FS
1628164 E4FS
$ gzip <E4FS |wc -c
473501614
$ truncate -s $((2*1590*1024*1024)) E4FS
$ resize2fs E4FS
resize2fs 1.41.14 (22-Dec-2010)
Resizing the filesystem on E4FS to 814080 (4k) blocks.
The filesystem on E4FS is now 814080 blocks long.
$ ls -log E4FS
-rw-rw-r--. 1 3334471680 Jul 14 15:13 E4FS
$ du -k E4FS
1645812 E4FS
$ gzip <E4FS |wc -c
474551711
-
- Posts: 96
- Joined: Fri Jun 29, 2012 2:27 pm
- Location: In the sticks of Haggisland
Re: New Raspbian Image
In short, 3.7GB is only compressing down to 543MB.
Re: New Raspbian Image
It may also depend on what compression you are using - bzip2 is better than gzip (of course!), and 7zip is supposed to be better than bzip2 (though slower). 3.7GB down to 543MB is still pretty good though.
Re: New Raspbian Image
Here is what I would do to create an img of my SD card.
Mount both the vfat and ext4 partitions and fill both using dd /dev/zero as discussed in this thread. After that, I would dd /dev/zero the swap partition then recreate afterword. Then with both partitions mounted, cd to / and issue these commands #du -khs and/or #df -kh to see your pre-compression SD card usage. What ever this usage is the final image should compress to about half that size using unxz compression. BTW, if possible, I would do all of the above on another more powerful machine.
Hope this helps.
Regards, Ron
Mount both the vfat and ext4 partitions and fill both using dd /dev/zero as discussed in this thread. After that, I would dd /dev/zero the swap partition then recreate afterword. Then with both partitions mounted, cd to / and issue these commands #du -khs and/or #df -kh to see your pre-compression SD card usage. What ever this usage is the final image should compress to about half that size using unxz compression. BTW, if possible, I would do all of the above on another more powerful machine.
Hope this helps.
Regards, Ron
-
- Posts: 57
- Joined: Sat Jul 07, 2012 2:30 pm
- Location: West Virginia. United States
- Contact: Yahoo Messenger
Re: New Raspbian Image
I put the image onto a sd card when it loads I put username and password in then how do I get it to load I tried start x but nothing
Re: New Raspbian Image
Hi, the initial admin (root) password is raspberry (first boot). You enter this at the prompt which says: 'Give root password for maintenance' (it is possible to disable this but not recommended, since it would make the system vulnerable). After this has run, on next boot in a tty:
Username: pi Password: raspberry
Startx will not work initially since X needs to be installed, choose 'Install Desktop System' from the administration menu to install this.
If the boot script is still causing problems for you, edit cmdline.txt and remove 'single' from the end. The system will boot straight into a tty, and you can always run later to configure the system.
Joshun
EDIT - oh, I saw you were just having problems starting xorg. Just run,choose 'Install Desktop System' and xorg will be installed for you, along with fluxbox. It can then be started with the normal startx command.
Username: pi Password: raspberry
Startx will not work initially since X needs to be installed, choose 'Install Desktop System' from the administration menu to install this.
If the boot script is still causing problems for you, edit cmdline.txt and remove 'single' from the end. The system will boot straight into a tty, and you can always run
Code: Select all
sudo raspi-setup
Joshun
EDIT - oh, I saw you were just having problems starting xorg. Just run
Code: Select all
sudo raspi-setup
Re: New Raspbian Image
Since I do not know how to edit a previous post, this is an addendum to my previous post. Please see my post a few back for initial information.
If your partitions and swap do not use the entire flash device, when you dd at the device level you will be compressing data for the entire device. The flash controller uses any area not containing a partition for garbage collection and wear leveling. So make sure you dd /dev/zero any area not assigned to a partition (see dd man page, I believe it is the seek option). Better yet just tar/compress separately the vfat and ext4 partitions and let the user create the partitions and un-compress/tar the information into the proper partition (but this assume some Linux expertise). If you have an 8GB SDHC card and your partitions only use 4GB, when you dd at the device level the destination device will need to be at least 8Gb in size or you will get an error in dd. I believe most of the dd images available for the Pi are small and expand to 4G or more (magic) on the 1st boot to get around the potential problem described above.
Regards, Ron
If your partitions and swap do not use the entire flash device, when you dd at the device level you will be compressing data for the entire device. The flash controller uses any area not containing a partition for garbage collection and wear leveling. So make sure you dd /dev/zero any area not assigned to a partition (see dd man page, I believe it is the seek option). Better yet just tar/compress separately the vfat and ext4 partitions and let the user create the partitions and un-compress/tar the information into the proper partition (but this assume some Linux expertise). If you have an 8GB SDHC card and your partitions only use 4GB, when you dd at the device level the destination device will need to be at least 8Gb in size or you will get an error in dd. I believe most of the dd images available for the Pi are small and expand to 4G or more (magic) on the 1st boot to get around the potential problem described above.
Regards, Ron
Re: New Raspbian Image
Both methods would safe unused space, yours a bit more foolproof than mine. Although the advantage of skipping unused space is that the image would easily fit on a smaller SD Card (e.g. 2GB), with still leaving extra space.
-
- Posts: 96
- Joined: Fri Jun 29, 2012 2:27 pm
- Location: In the sticks of Haggisland
Re: New Raspbian Image
Maybe mpthompson (or another of the major distro packagers) could kindly explain how they produce their image(s).
Example: rpi_pisces_r3.zip is 447.7MB yet expands to 4Gb when dd'ed out to a card, without a reboot. I'd assume that an xz version would be a fair deal smaller - see Puppi for an example of what can be achieved.
Example: rpi_pisces_r3.zip is 447.7MB yet expands to 4Gb when dd'ed out to a card, without a reboot. I'd assume that an xz version would be a fair deal smaller - see Puppi for an example of what can be achieved.
- joe wraith
- Posts: 7
- Joined: Sat Jul 14, 2012 8:55 am
Re: New Raspbian Image
Sounds great. 

Re: New Raspbian Image
How does "df /" in your image compare to images that compress better? Most likely you are just including more stuff.ejsolutions wrote:Example: rpi_pisces_r3.zip is 447.7MB yet expands to 4Gb when dd'ed out to a card, without a reboot. I'd assume that an xz version would be a fair deal smaller - see Puppi for an example of what can be achieved.
Did you remember to "sudo apt-get clean" before the dd zero? Otherwise there may be unnecessary debs in /var/cache/apt/archives.
-
- Posts: 96
- Joined: Fri Jun 29, 2012 2:27 pm
- Location: In the sticks of Haggisland
Re: New Raspbian Image
Actually not that much more; wireless-utils and firmware-ralink, plus pre-configuring a few bits.jojopi wrote:How does "df /" in your image compare to images that compress better? Most likely you are just including more stuff.
Though obviously the xbmc derivative that I create is considerably bigger.

Aye, that's pretty basic stuff.jojopi wrote: Did you remember to "sudo apt-get clean" before the dd zero? Otherwise there may be unnecessary debs in /var/cache/apt/archives.

Kinda given up on creating images for now, due to it taking so long to do, even with Class 10 cards. Once I have a definitive approach, then I'll get back to it. Time to play with MATE some more, though I should really be doing 'work' (e-commerce/server stuff)!
EJ
- mpthompson
- Posts: 620
- Joined: Fri Feb 03, 2012 7:18 pm
- Location: San Carlos, CA
- Contact: Website
Re: New Raspbian Image
ejsolutions,
The things I do to create an image were mostly covered above. Some of the specific commands are below:
To clean any cached Debian packages:
To zero a disk partition are, do the following in a directory on the partition you want to zero:
Notes: the purpose of the small file is to reduce the time when free space is completely zero; the purpose of sync is to make sure the data is actually written.
To zero out the swap partition:
I also use fdisk to check the total number of blocks on the file system I want to image and then use that total count with the dd command:
Then I use zip on the final image. I know I can get better compression with other algorithms, but zip is a little easier for most people to deal with. Particularly those with Windows PCs.
The things I do to create an image were mostly covered above. Some of the specific commands are below:
To clean any cached Debian packages:
Code: Select all
apt-get clean
Code: Select all
dd if=/dev/zero of=zero.small.file bs=1024 count=102400
dd if=/dev/zero of=zero.file bs=1024
sync ; sleep 60 ; sync
rm zero.small.file
rm zero.file
To zero out the swap partition:
Code: Select all
swapoff -a
shred -n 0 -z /dev/mmcblk0p2
mkswap /dev/mmcblk0p2
Code: Select all
dd if=/dev/sdc of=rpi_pisces_r3.img bs=512 count=7618560
-
- Posts: 96
- Joined: Fri Jun 29, 2012 2:27 pm
- Location: In the sticks of Haggisland
Re: New Raspbian Image
Muchos gracias amigo. 
Yes, I spotted most of that from your bash history in the image, but missed some key points: small zeroed file and the shred.
I'm interested in why you only use the default 512 block size. When doing a a partition resize/move, gparted seems to optimise at 4M blocks, for the cards that I'm using.
Thanks again - appreciated.
EJ

Yes, I spotted most of that from your bash history in the image, but missed some key points: small zeroed file and the shred.
I'm interested in why you only use the default 512 block size. When doing a a partition resize/move, gparted seems to optimise at 4M blocks, for the cards that I'm using.
Thanks again - appreciated.
EJ