Page 1 of 2

STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Sun Jan 20, 2019 8:04 pm
by HawaiianPi
Wouldn't it be nice to have SSH enabled by default, or automatically connect to your WiFi without needing a screen and keyboard? How about some custom settings in config.txt or cmdline.txt, or some scripts to automate things?

And what if it could be part of the Raspbian image so you could just write your SD card and boot?

Sure you could boot up, manually configure everything, then make one of those huge SD card images with dd, and then try and figure out how to shrink that to fit on smaller cards. If only there was an easier way...

So this is how you can make your own custom burn-n-boot image, the easy way, and you can do it before you write the image to the card!

Cool! Can I do this in Windows?
Nope, you need a Linux computer, but luckily you have a Raspberry Pi, which is a Linux computer. You could also boot a live Linux on your x86 Windows or Mac PC from USB or optical media. This would be a good option: Debian Stretch with Raspberry Pi Desktop.

Anyone still here?
Good, let's get started.

What you need to know about the image:
First, extract the Raspbian image from the .zip archive, and then we need to find out what's in the image file.

Code: Select all

fdisk -lu 2018-11-13-raspbian-stretch-lite.img
This will return the structure of the Raspbian image.

Code: Select all

Disk 2018-11-13-raspbian-stretch-lite.img: 1.8 GiB, 1866465280 bytes, 3645440 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7ee80803

Device                                Boot Start     End Sectors  Size Id Type
2018-11-13-raspbian-stretch-lite.img1       8192   98045   89854 43.9M  c W95 FAT32 (LBA)
2018-11-13-raspbian-stretch-lite.img2      98304 3645439 3547136  1.7G 83 Linux
We can see sector size, partitions, the file systems, the offset to the beginning of the partitions and the partition sizes. With this information we can mount the image like a disk and make changes (add or edit files).

Mounting disk partitions from an image:
You mount an image partition as a loop device (a block device within a file) and mount needs to know where the partition starts with offset={Start X SectorSize}. For the first "boot" partition the start sector is 8192, and you can let the command do the math.

Code: Select all

sudo mkdir /mnt/disk
sudo mount -t vfat -o loop,offset=$((8192*512)) 2018-11-13-raspbian-stretch-lite.img /mnt/disk

Mounting disk images:
If you want to mount the entire Raspbian disk image, mount the 2nd partition first and put the first partition in the mount folder/boot. When mounting another partition into an already mounted one, you must also tell Linux how big the partition is (sizelimit=Sectors X SectorSize).

Code: Select all

sudo mkdir /mnt/disk
sudo mount -t ext4 -o loop,offset=$((98304*512)) 2018-11-13-raspbian-stretch-lite.img /mnt/disk
sudo mount -t vfat -o loop,offset=$((8192*512)),sizelimit=$((89854*512)) 2018-11-13-raspbian-stretch-lite.img /mnt/disk/boot

So you've made your changes, now what?
After making your changes you can un-mount the partitions and your custom image is ready (re-zip the image to save space).

Code: Select all

sudo umount /mnt/disk/boot
sudo umount /mnt/disk

So what can you do with this?
You can add a file named "ssh" (or shh.txt) to enable SSH logins on first boot. You can add a pre-configured wpa_supplicant.conf file to connect to your wireless network. You can edit config.txt or cmdline.txt, and that's only the first partition. There's even more you can do on the second partition (that's the one Windows and Mac OS can't see). Mounting the entire image (both partitions) allows you to edit other system files that control way more stuff than I can cover here. With this simple technique you can't install software, run updates or add new users, but you could add scripts for those and other complex operations.

The ssh and wpa_supplicant.conf files are what you typically need for a totally headless boot on a wireless network (you can then SSH into the system to enable VNC or make any other changes needed).

Your pre-configured wpa_supplicant.conf should look like this.

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
	ssid="Your network SSID"
	psk="Your WPA/WPA2 security key"
	key_mgmt=WPA-PSK
}
Edit country=, ssid= and psk= with your information and save the file.

Who will benefit from this?
If you only have a couple of Pi computers, it may not help you much. If you have several Pi computers, or manage a classroom full of them, having a custom burn-n-boot image should be a real time saver.

Are we there yet?
Yup, that's it. That's how you make changes to a Raspbian image before it's written to a card, for your very own custom burn-n-boot image.

One final note:
The Raspbian Desktop images contain proprietary software that RPF is licenced to distribute, but you are not. So if you want to make custom images for yourself, that's fine, but you probably shouldn't share them without proper licencing and permissions.

Re: Making your own custom burn-n-boot Raspbian image

Posted: Sun Jan 20, 2019 8:23 pm
by sakaki
Nice tutorial HawaiianPi!

If your Linux system supports kpartx, that makes things easier - as there's no need to work out the size of partitions before mounting them. Example workflow for modifying an image:

Code: Select all

# unzip raspbian_lite_latest 
Archive:  raspbian_lite_latest
  inflating: 2018-11-13-raspbian-stretch-lite.img 
  
# kpartx -sva 2018-11-13-raspbian-stretch-lite.img 
add map loop0p1 (253:12): 0 89854 linear 7:0 8192
add map loop0p2 (253:13): 0 3547136 linear 7:0 98304

(your device mapper location etc may differ)

# mkdir -p /mnt/rasp_p{1,2}

# mount -v /dev/mapper/loop0p1 /mnt/rasp_p1
mount: /dev/mapper/loop0p1 mounted on /mnt/rasp_p1.

# mount -v /dev/mapper/loop0p2 /mnt/rasp_p2
mount: /dev/mapper/loop0p2 mounted on /mnt/rasp_p2.

(do your work)

# umount -v /mnt/rasp_p{1,2}
umount: /mnt/rasp_p1 unmounted
umount: /mnt/rasp_p2 unmounted

# kpartx -vd 2018-11-13-raspbian-stretch-lite.img 
del devmap : loop0p2
del devmap : loop0p1
loop deleted : /dev/loop0

(all done, you can now recompress the image again if you wish)
best, sakaki

Re: Making your own custom burn-n-boot Raspbian image

Posted: Sun Jan 20, 2019 9:38 pm
by dshaw619
Extremely helpful and well written post. Thanks.

Doug

Re: Making your own custom burn-n-boot Raspbian image

Posted: Mon Jan 21, 2019 6:27 am
by bertlea
That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.

Re: Making your own custom burn-n-boot Raspbian image

Posted: Mon Jan 21, 2019 10:46 am
by jahboater
Helpful posts HawaiianPi and Sakaki, thanks.
Should be in a sticky!

Re: Making your own custom burn-n-boot Raspbian image

Posted: Mon Jan 21, 2019 11:43 am
by jamesh
jahboater wrote:
Mon Jan 21, 2019 10:46 am
Helpful posts HawaiianPi and Sakaki, thanks.
Should be in a sticky!
Done. Probably best to avoid posting in the thread unless really needed, just to keep it on topic.

Re: Making your own custom burn-n-boot Raspbian image

Posted: Mon Jan 21, 2019 12:10 pm
by ShiftPlusOne
losetup -P can be used instead of kpartx as well.

Re: Making your own custom burn-n-boot Raspbian image

Posted: Mon Jan 21, 2019 4:58 pm
by SteveSpencer
bertlea wrote:
Mon Jan 21, 2019 6:27 am
That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.
Yes, it is, within disk space limits. Remember that your root fs has not yet been expanded.
You can do it by mounting the rootfs part first, then the boot fs part to the <your-mount-point>/boot, and using chroot.
I have done this to perform an apt-get update / apt-get upgrade before first boot.

Re: Making your own custom burn-n-boot Raspbian image

Posted: Mon Jan 21, 2019 7:55 pm
by morticiaskeeper
I have done similar. Starting from a fresh Raspbian Lite on a 4gb card, I've added the packages I use a lot, LAMP server, Python Document Generator, Auto Hot Spot, WPA supplicant.

An image has been taken and stored on my server. When I need a new card, I write this image and expand the filesystem on first boot. It probably saves a couple of hours each time.

Re: Making your own custom burn-n-boot Raspbian image

Posted: Tue Jan 22, 2019 1:59 am
by Milliways
HawaiianPi wrote:
Sun Jan 20, 2019 8:04 pm
Wouldn't it be nice to have SSH enabled by default, or automatically connect to your WiFi without needing a screen and keyboard? How about some custom settings in config.txt or cmdline.txt, or some scripts to automate things?
You can do all these by simply copying a few files to the boot partition. You could even do it on Windows!

Re: Making your own custom burn-n-boot Raspbian image

Posted: Tue Jan 22, 2019 5:45 pm
by HawaiianPi
bertlea wrote:
Mon Jan 21, 2019 6:27 am
That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.
The problem with that is the root filesystem partition has limited space before it's expanded. So technically the answer is yes, but realistically you won't have room for much.

Milliways wrote:
Tue Jan 22, 2019 1:59 am
You can do all these by simply copying a few files to the boot partition. You could even do it on Windows!
Yup, you can, and I said in the OP that a custom image might not benefit everyone. It's mainly for people who manage a lot of Pi computers, or people who re-flash the OS a lot, or who need to go beyond the basics. It's not for everyone, but I thought it might be useful to some.


@ sakaki and ShiftPlusOne,
If Linux has a fault, it's that there are often too many ways to do something (not to mention, too many versions). Thanks for expanding on the topic and suggesting alternatives.

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Fri Jan 25, 2019 10:44 am
by LindaLoki_Moon
A very nice tutorial! Thanks! :mrgreen:

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Tue Jan 29, 2019 4:02 pm
by sakaki
bertlea wrote:
Mon Jan 21, 2019 6:27 am
That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.
As others have said, yes absolutely this can be done. For the sake of concreteness, I'll provide a compete workflow for doing so in this post.

This time, I'm going to:
  • use loseup (as suggested by ShiftPlusOne) rather than kpartx, since the former comes bundled by default on Raspbian, whereas the latter does not;
  • use systemd-nspawn to manage the chroot-ing, as it takes care of setting up and tearing down the necessary bind mounts, which are easy to do but also easy to forget;
  • use zerofree to maximize the root partition's compressibility post-modification; and
  • assume you wish to add an additional package to the current "Raspbian Stretch with desktop" image, and are going to be manipulating the image on an RPi, also running Raspbian.
Right, so begin by installing systemd-container (this will give you the systemd-nspawn utility) and zerofree, if you don't already have them. This will not take long:

Code: Select all

[email protected]:~ $ sudo apt-get update
[email protected]:~ $ sudo apt-get install -y systemd-container zerofree
Then, assuming you have already downloaded the zipped image (from here), unpack, and then loop mount it:

Code: Select all

[email protected]:~ $ unzip raspbian_latest 
Archive:  raspbian_latest
  inflating: 2018-11-13-raspbian-stretch.img 
[email protected]:~ $ sudo losetup --show -P -f 2018-11-13-raspbian-stretch.img
/dev/loop0
Your reported loop device may differ (loop1, loop2 etc.) in which case modify the following instructions accordingly.
This losetup command should have created partition mappings automatically:

Code: Select all

[email protected]:~ $ ls /dev/loop0*
/dev/loop0  /dev/loop0p1  /dev/loop0p2
Yep, two partitions, as expected. Now mount them:

Code: Select all

[email protected]:~ $ sudo mkdir -p /mnt/raspbian
[email protected]:~ $ sudo mount -v /dev/loop0p2 /mnt/raspbian
mount: /dev/loop0p2 mounted on /mnt/raspbian.
[email protected]:~ $ sudo mount -v /dev/loop0p1 /mnt/raspbian/boot
mount: /dev/loop0p1 mounted on /mnt/raspbian/boot.
Now we can chroot in. We'll use systemd-nspawn to do this; it will take care of all the necessary behind-the-scenes bind-mounts etc.:

Code: Select all

[email protected]:~ $ sudo systemd-nspawn --directory=/mnt/raspbian
Spawning container raspbian on /mnt/raspbian.
Press ^] three times within 1s to kill container.
[email protected]:~#
You are now operating 'inside' the image, as the root user. Since we're going to be installing stuff, let's first update our package metadata and also upgrade the installed packages on the image (will save time during startup for new users):

Code: Select all

[email protected]:~# apt-get update
[email protected]:~# apt-get -y upgrade
You can skip the upgrade command if you wish.
This may take a little while to complete.

OK, so we've arrived at the payload point: you can now install any packages you want on the image.
I'll add firefox-esr here as an example (the image has sufficient free space). Adapt as required:

Code: Select all

[email protected]:~# apt-get -y install firefox-esr
Once you're done installing, do some minimal cleanup:

Code: Select all

[email protected]:~# apt-get clean
[email protected]:~# apt-get autoremove
You can go further, and remove the apt source lists, remove root's bash history etc. Up to you.

Then exit the container, and unmount the partitions:

Code: Select all

[email protected]:~# logout
Container raspbian exited successfully.
[email protected]:~ $ sudo umount -v /mnt/raspbian/{boot,}
umount: /mnt/raspbian/boot unmounted
umount: /mnt/raspbian/ unmounted
Now ensure all unallocated blocks on the image's root file system are zeroed, to improve compressibility:

Code: Select all

[email protected]:~ $ sudo e2fsck -f /dev/loop0p2
[email protected]:~ $ sudo zerofree -v /dev/loop0p2
Finally, detach the loop mount from the image, and ensure everything is flushed:

Code: Select all

[email protected]:~ $ sudo losetup -d /dev/loop0
[email protected]:~ $ sync
All done, you can now recompress the image if you wish!

If you now write your modified image using a tool like Etcher to a microSD card and boot from it, you should find you have your new package/s available (firefox-esr, in this example) right from the off.

hth,

sakaki

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Wed Jan 30, 2019 1:51 pm
by RaspbianDK
Nice, thank you!

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Thu Jan 31, 2019 9:03 am
by bertlea
Thank you very much @sakaki for the step by step instructions!

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Fri Feb 22, 2019 10:00 am
by mitsunataliya
Thank you very much HawaiianPi, Its very helpful and good post.
http://kmspicodownloads.com/
I have a small issue this one not work properly
"
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
ssid="Your network SSID"
psk="Your WPA/WPA2 security key"
key_mgmt=WPA-PSK
}

"

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Fri Feb 22, 2019 11:04 am
by HawaiianPi
mitsunataliya wrote:
Fri Feb 22, 2019 10:00 am
Thank you very much HawaiianPi, Its very helpful and good post.

I have a small issue this one not work properly

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
	ssid="Your network SSID"
	psk="Your WPA/WPA2 security key"
	key_mgmt=WPA-PSK
}
What isn't working?

Did you edit the country code, SSID name and WPA passphrase with your info?

Which model of Raspberry Pi?

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Mon Feb 25, 2019 6:00 am
by mitsunataliya
HawaiianPi wrote:
Fri Feb 22, 2019 11:04 am
mitsunataliya wrote:
Fri Feb 22, 2019 10:00 am
Thank you very much HawaiianPi, Its very helpful and good post.

I have a small issue this one not work properly

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
	ssid="Your network SSID"
	psk="Your WPA/WPA2 security key"
	key_mgmt=WPA-PSK
}
What isn't working?

Did you edit the country code, SSID name and WPA passphrase with your info?

Which model of Raspberry Pi?
My bad I put the wrong SSID, Now its work fine. Thank you HawaiianPi

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Mon Feb 25, 2019 11:37 am
by HawaiianPi
mitsunataliya wrote:
Mon Feb 25, 2019 6:00 am
My bad I put the wrong SSID, Now its work fine. Thank you HawaiianPi
Blame it on auto-correct ... ;)

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Tue Feb 26, 2019 11:39 am
by Ramtec
Whats the best way to automate the 'touch ssh' command on Alpine? Parse the fdisk output?

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Sun Mar 17, 2019 6:28 am
by FrancisTurner
By the way if you want to make lots of custom images - and/or make the same one only with the latest s/w again and again - then there's this Github project - https://github.com/RPi-Distro/Pi-gen

It runs on *buntu and debian with docker. Its been updated a bit since I first started playing with it but it allows you to completely customize the image removing and adding the packages you want as well as customizing start up scripts etc.

If you have the space on your PC I strongly recommend running it in a VM that is a custom docker configured ubuntu server because it was a complete pig to get running without docker and docker did not play well for me on my desktop xubuntu system

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Sun Mar 17, 2019 8:56 am
by HawaiianPi
Ramtec wrote:
Tue Feb 26, 2019 11:39 am
Whats the best way to automate the 'touch ssh' command on Alpine? Parse the fdisk output?
Don't know. This thread is about Raspbian Linux.

From a quick peek at alpinelinux.org, Alpine Linux doesn't seem to use images, so it has nothing to do with this thread. You could ask if anyone here has any experience with Alpine in our Other OS sub-forum: viewforum.php?f=56, but you might find more help in the Alpine community: https://alpinelinux.org/community/

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Tue Mar 26, 2019 12:37 pm
by jessica10
I have run the same code but getting error.

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Wed Mar 27, 2019 6:07 am
by HawaiianPi
jessica10 wrote:
Tue Mar 26, 2019 12:37 pm
I have run the same code but getting error.
What code, and what was the error?

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Posted: Thu Apr 04, 2019 6:13 am
by Elizabeth7
Recently, me and my fellow team mate Tharindu Muhandiram ran into a bit of an issue with regard to building a custom image for our Platformer IoT Gateway Upsers.