User avatar
HawaiianPi
Posts: 4272
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

STICKY: Making your own custom burn-n-boot Raspbian image

Sun Jan 20, 2019 8:04 pm

Wouldn't it be nice to have SSH enabled by default, or automatically connect to your WiFi without needing a screen and keyboard? How about some custom settings in config.txt or cmdline.txt, or some scripts to automate things?

And what if it could be part of the Raspbian image so you could just write your SD card and boot?

Sure you could boot up, manually configure everything, then make one of those huge SD card images with dd, and then try and figure out how to shrink that to fit on smaller cards. If only there was an easier way...

So this is how you can make your own custom burn-n-boot image, the easy way, and you can do it before you write the image to the card!

Cool! Can I do this in Windows?
Nope, you need a Linux computer, but luckily you have a Raspberry Pi, which is a Linux computer. You could also boot a live Linux on your x86 Windows or Mac PC from USB or optical media. This would be a good option: Debian Stretch with Raspberry Pi Desktop.

Anyone still here?
Good, let's get started.

What you need to know about the image:
First, extract the Raspbian image from the .zip archive, and then we need to find out what's in the image file.

Code: Select all

fdisk -lu 2018-11-13-raspbian-stretch-lite.img
This will return the structure of the Raspbian image.

Code: Select all

Disk 2018-11-13-raspbian-stretch-lite.img: 1.8 GiB, 1866465280 bytes, 3645440 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7ee80803

Device                                Boot Start     End Sectors  Size Id Type
2018-11-13-raspbian-stretch-lite.img1       8192   98045   89854 43.9M  c W95 FAT32 (LBA)
2018-11-13-raspbian-stretch-lite.img2      98304 3645439 3547136  1.7G 83 Linux
We can see sector size, partitions, the file systems, the offset to the beginning of the partitions and the partition sizes. With this information we can mount the image like a disk and make changes (add or edit files).

Mounting disk partitions from an image:
You mount an image partition as a loop device (a block device within a file) and mount needs to know where the partition starts with offset={Start X SectorSize}. For the first "boot" partition the start sector is 8192, and you can let the command do the math.

Code: Select all

sudo mkdir /mnt/disk
sudo mount -t vfat -o loop,offset=$((8192*512)) 2018-11-13-raspbian-stretch-lite.img /mnt/disk

Mounting disk images:
If you want to mount the entire Raspbian disk image, mount the 2nd partition first and put the first partition in the mount folder/boot. When mounting another partition into an already mounted one, you must also tell Linux how big the partition is (sizelimit=Sectors X SectorSize).

Code: Select all

sudo mkdir /mnt/disk
sudo mount -t ext4 -o loop,offset=$((98304*512)) 2018-11-13-raspbian-stretch-lite.img /mnt/disk
sudo mount -t vfat -o loop,offset=$((8192*512)),sizelimit=$((89854*512)) 2018-11-13-raspbian-stretch-lite.img /mnt/disk/boot

So you've made your changes, now what?
After making your changes you can un-mount the partitions and your custom image is ready (re-zip the image to save space).

Code: Select all

sudo umount /mnt/disk/boot
sudo umount /mnt/disk

So what can you do with this?
You can add a file named "ssh" (or shh.txt) to enable SSH logins on first boot. You can add a pre-configured wpa_supplicant.conf file to connect to your wireless network. You can edit config.txt or cmdline.txt, and that's only the first partition. There's even more you can do on the second partition (that's the one Windows and Mac OS can't see). Mounting the entire image (both partitions) allows you to edit other system files that control way more stuff than I can cover here. With this simple technique you can't install software, run updates or add new users, but you could add scripts for those and other complex operations.

The ssh and wpa_supplicant.conf files are what you typically need for a totally headless boot on a wireless network (you can then SSH into the system to enable VNC or make any other changes needed).

Your pre-configured wpa_supplicant.conf should look like this.

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
	ssid="Your network SSID"
	psk="Your WPA/WPA2 security key"
	key_mgmt=WPA-PSK
}
Edit country=, ssid= and psk= with your information and save the file.

Who will benefit from this?
If you only have a couple of Pi computers, it may not help you much. If you have several Pi computers, or manage a classroom full of them, having a custom burn-n-boot image should be a real time saver.

Are we there yet?
Yup, that's it. That's how you make changes to a Raspbian image before it's written to a card, for your very own custom burn-n-boot image.

One final note:
The Raspbian Desktop images contain proprietary software that RPF is licenced to distribute, but you are not. So if you want to make custom images for yourself, that's fine, but you probably shouldn't share them without proper licencing and permissions.
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

User avatar
sakaki
Posts: 214
Joined: Sun Jul 16, 2017 1:11 pm

Re: Making your own custom burn-n-boot Raspbian image

Sun Jan 20, 2019 8:23 pm

Nice tutorial HawaiianPi!

If your Linux system supports kpartx, that makes things easier - as there's no need to work out the size of partitions before mounting them. Example workflow for modifying an image:

Code: Select all

# unzip raspbian_lite_latest 
Archive:  raspbian_lite_latest
  inflating: 2018-11-13-raspbian-stretch-lite.img 
  
# kpartx -sva 2018-11-13-raspbian-stretch-lite.img 
add map loop0p1 (253:12): 0 89854 linear 7:0 8192
add map loop0p2 (253:13): 0 3547136 linear 7:0 98304

(your device mapper location etc may differ)

# mkdir -p /mnt/rasp_p{1,2}

# mount -v /dev/mapper/loop0p1 /mnt/rasp_p1
mount: /dev/mapper/loop0p1 mounted on /mnt/rasp_p1.

# mount -v /dev/mapper/loop0p2 /mnt/rasp_p2
mount: /dev/mapper/loop0p2 mounted on /mnt/rasp_p2.

(do your work)

# umount -v /mnt/rasp_p{1,2}
umount: /mnt/rasp_p1 unmounted
umount: /mnt/rasp_p2 unmounted

# kpartx -vd 2018-11-13-raspbian-stretch-lite.img 
del devmap : loop0p2
del devmap : loop0p1
loop deleted : /dev/loop0

(all done, you can now recompress the image again if you wish)
best, sakaki

dshaw619
Posts: 10
Joined: Thu Jan 04, 2018 7:06 am
Location: San Diego, California, USA

Re: Making your own custom burn-n-boot Raspbian image

Sun Jan 20, 2019 9:38 pm

Extremely helpful and well written post. Thanks.

Doug

bertlea
Posts: 265
Joined: Wed Dec 07, 2016 6:33 am
Location: Hong Kong

Re: Making your own custom burn-n-boot Raspbian image

Mon Jan 21, 2019 6:27 am

That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.

jahboater
Posts: 4465
Joined: Wed Feb 04, 2015 6:38 pm

Re: Making your own custom burn-n-boot Raspbian image

Mon Jan 21, 2019 10:46 am

Helpful posts HawaiianPi and Sakaki, thanks.
Should be in a sticky!

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22742
Joined: Sat Jul 30, 2011 7:41 pm

Re: Making your own custom burn-n-boot Raspbian image

Mon Jan 21, 2019 11:43 am

jahboater wrote:
Mon Jan 21, 2019 10:46 am
Helpful posts HawaiianPi and Sakaki, thanks.
Should be in a sticky!
Done. Probably best to avoid posting in the thread unless really needed, just to keep it on topic.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5717
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Making your own custom burn-n-boot Raspbian image

Mon Jan 21, 2019 12:10 pm

losetup -P can be used instead of kpartx as well.

SteveSpencer
Posts: 351
Joined: Thu Mar 28, 2013 9:19 am
Location: Nottingham, UK

Re: Making your own custom burn-n-boot Raspbian image

Mon Jan 21, 2019 4:58 pm

bertlea wrote:
Mon Jan 21, 2019 6:27 am
That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.
Yes, it is, within disk space limits. Remember that your root fs has not yet been expanded.
You can do it by mounting the rootfs part first, then the boot fs part to the <your-mount-point>/boot, and using chroot.
I have done this to perform an apt-get update / apt-get upgrade before first boot.
Steve S
No, I can't think of anything funny that won't offend someone if they want it to...

morticiaskeeper
Posts: 101
Joined: Mon Aug 27, 2012 6:05 pm

Re: Making your own custom burn-n-boot Raspbian image

Mon Jan 21, 2019 7:55 pm

I have done similar. Starting from a fresh Raspbian Lite on a 4gb card, I've added the packages I use a lot, LAMP server, Python Document Generator, Auto Hot Spot, WPA supplicant.

An image has been taken and stored on my server. When I need a new card, I write this image and expand the filesystem on first boot. It probably saves a couple of hours each time.

Milliways
Posts: 409
Joined: Fri Apr 25, 2014 12:18 am
Location: Sydney, Australia

Re: Making your own custom burn-n-boot Raspbian image

Tue Jan 22, 2019 1:59 am

HawaiianPi wrote:
Sun Jan 20, 2019 8:04 pm
Wouldn't it be nice to have SSH enabled by default, or automatically connect to your WiFi without needing a screen and keyboard? How about some custom settings in config.txt or cmdline.txt, or some scripts to automate things?
You can do all these by simply copying a few files to the boot partition. You could even do it on Windows!

User avatar
HawaiianPi
Posts: 4272
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: Making your own custom burn-n-boot Raspbian image

Tue Jan 22, 2019 5:45 pm

bertlea wrote:
Mon Jan 21, 2019 6:27 am
That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.
The problem with that is the root filesystem partition has limited space before it's expanded. So technically the answer is yes, but realistically you won't have room for much.

Milliways wrote:
Tue Jan 22, 2019 1:59 am
You can do all these by simply copying a few files to the boot partition. You could even do it on Windows!
Yup, you can, and I said in the OP that a custom image might not benefit everyone. It's mainly for people who manage a lot of Pi computers, or people who re-flash the OS a lot, or who need to go beyond the basics. It's not for everyone, but I thought it might be useful to some.


@ sakaki and ShiftPlusOne,
If Linux has a fault, it's that there are often too many ways to do something (not to mention, too many versions). Thanks for expanding on the topic and suggesting alternatives.
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

LindaLoki_Moon
Posts: 2
Joined: Fri Jan 18, 2019 3:34 pm

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Fri Jan 25, 2019 10:44 am

A very nice tutorial! Thanks! :mrgreen:

User avatar
sakaki
Posts: 214
Joined: Sun Jul 16, 2017 1:11 pm

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Tue Jan 29, 2019 4:02 pm

bertlea wrote:
Mon Jan 21, 2019 6:27 am
That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.
As others have said, yes absolutely this can be done. For the sake of concreteness, I'll provide a compete workflow for doing so in this post.

This time, I'm going to:
  • use loseup (as suggested by ShiftPlusOne) rather than kpartx, since the former comes bundled by default on Raspbian, whereas the latter does not;
  • use systemd-nspawn to manage the chroot-ing, as it takes care of setting up and tearing down the necessary bind mounts, which are easy to do but also easy to forget;
  • use zerofree to maximize the root partition's compressibility post-modification; and
  • assume you wish to add an additional package to the current "Raspbian Stretch with desktop" image, and are going to be manipulating the image on an RPi, also running Raspbian.
Right, so begin by installing systemd-container (this will give you the systemd-nspawn utility) and zerofree, if you don't already have them. This will not take long:

Code: Select all

[email protected]:~ $ sudo apt-get update
[email protected]:~ $ sudo apt-get install -y systemd-container zerofree
Then, assuming you have already downloaded the zipped image (from here), unpack, and then loop mount it:

Code: Select all

[email protected]:~ $ unzip raspbian_latest 
Archive:  raspbian_latest
  inflating: 2018-11-13-raspbian-stretch.img 
[email protected]:~ $ sudo losetup --show -P -f 2018-11-13-raspbian-stretch.img
/dev/loop0
Your reported loop device may differ (loop1, loop2 etc.) in which case modify the following instructions accordingly.
This losetup command should have created partition mappings automatically:

Code: Select all

[email protected]:~ $ ls /dev/loop0*
/dev/loop0  /dev/loop0p1  /dev/loop0p2
Yep, two partitions, as expected. Now mount them:

Code: Select all

[email protected]:~ $ sudo mkdir -p /mnt/raspbian
[email protected]:~ $ sudo mount -v /dev/loop0p2 /mnt/raspbian
mount: /dev/loop0p2 mounted on /mnt/raspbian.
[email protected]:~ $ sudo mount -v /dev/loop0p1 /mnt/raspbian/boot
mount: /dev/loop0p1 mounted on /mnt/raspbian/boot.
Now we can chroot in. We'll use systemd-nspawn to do this; it will take care of all the necessary behind-the-scenes bind-mounts etc.:

Code: Select all

[email protected]:~ $ sudo systemd-nspawn --directory=/mnt/raspbian
Spawning container raspbian on /mnt/raspbian.
Press ^] three times within 1s to kill container.
[email protected]:~#
You are now operating 'inside' the image, as the root user. Since we're going to be installing stuff, let's first update our package metadata and also upgrade the installed packages on the image (will save time during startup for new users):

Code: Select all

[email protected]:~# apt-get update
[email protected]:~# apt-get -y upgrade
You can skip the upgrade command if you wish.
This may take a little while to complete.

OK, so we've arrived at the payload point: you can now install any packages you want on the image.
I'll add firefox-esr here as an example (the image has sufficient free space). Adapt as required:

Code: Select all

[email protected]:~# apt-get -y install firefox-esr
Once you're done installing, do some minimal cleanup:

Code: Select all

[email protected]:~# apt-get clean
[email protected]:~# apt-get autoremove
You can go further, and remove the apt source lists, remove root's bash history etc. Up to you.

Then exit the container, and unmount the partitions:

Code: Select all

[email protected]:~# logout
Container raspbian exited successfully.
[email protected]:~ $ sudo umount -v /mnt/raspbian/{boot,}
umount: /mnt/raspbian/boot unmounted
umount: /mnt/raspbian/ unmounted
Now ensure all unallocated blocks on the image's root file system are zeroed, to improve compressibility:

Code: Select all

[email protected]:~ $ sudo e2fsck -f /dev/loop0p2
[email protected]:~ $ sudo zerofree -v /dev/loop0p2
Finally, detach the loop mount from the image, and ensure everything is flushed:

Code: Select all

[email protected]:~ $ sudo losetup -d /dev/loop0
[email protected]:~ $ sync
All done, you can now recompress the image if you wish!

If you now write your modified image using a tool like Etcher to a microSD card and boot from it, you should find you have your new package/s available (firefox-esr, in this example) right from the off.

hth,

sakaki

User avatar
RaspbianDK
Posts: 16
Joined: Fri Oct 26, 2018 12:21 pm
Location: Denmark

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Wed Jan 30, 2019 1:51 pm

Nice, thank you!

bertlea
Posts: 265
Joined: Wed Dec 07, 2016 6:33 am
Location: Hong Kong

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Thu Jan 31, 2019 9:03 am

Thank you very much @sakaki for the step by step instructions!

mitsunataliya
Posts: 3
Joined: Thu Feb 07, 2019 6:25 am

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Fri Feb 22, 2019 10:00 am

Thank you very much HawaiianPi, Its very helpful and good post.
http://kmspicodownloads.com/
I have a small issue this one not work properly
"
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
ssid="Your network SSID"
psk="Your WPA/WPA2 security key"
key_mgmt=WPA-PSK
}

"
Last edited by mitsunataliya on Mon Feb 25, 2019 5:57 am, edited 1 time in total.

User avatar
HawaiianPi
Posts: 4272
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Fri Feb 22, 2019 11:04 am

mitsunataliya wrote:
Fri Feb 22, 2019 10:00 am
Thank you very much HawaiianPi, Its very helpful and good post.

I have a small issue this one not work properly

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
	ssid="Your network SSID"
	psk="Your WPA/WPA2 security key"
	key_mgmt=WPA-PSK
}
What isn't working?

Did you edit the country code, SSID name and WPA passphrase with your info?

Which model of Raspberry Pi?
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

mitsunataliya
Posts: 3
Joined: Thu Feb 07, 2019 6:25 am

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Mon Feb 25, 2019 6:00 am

HawaiianPi wrote:
Fri Feb 22, 2019 11:04 am
mitsunataliya wrote:
Fri Feb 22, 2019 10:00 am
Thank you very much HawaiianPi, Its very helpful and good post.

I have a small issue this one not work properly

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
	ssid="Your network SSID"
	psk="Your WPA/WPA2 security key"
	key_mgmt=WPA-PSK
}
What isn't working?

Did you edit the country code, SSID name and WPA passphrase with your info?

Which model of Raspberry Pi?
My bad I put the wrong SSID, Now its work fine. Thank you HawaiianPi

User avatar
HawaiianPi
Posts: 4272
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Mon Feb 25, 2019 11:37 am

mitsunataliya wrote:
Mon Feb 25, 2019 6:00 am
My bad I put the wrong SSID, Now its work fine. Thank you HawaiianPi
Blame it on auto-correct ... ;)
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

Ramtec
Posts: 1
Joined: Tue Feb 26, 2019 11:36 am

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Tue Feb 26, 2019 11:39 am

Whats the best way to automate the 'touch ssh' command on Alpine? Parse the fdisk output?

FrancisTurner
Posts: 4
Joined: Mon Jan 21, 2013 11:21 am

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Sun Mar 17, 2019 6:28 am

By the way if you want to make lots of custom images - and/or make the same one only with the latest s/w again and again - then there's this Github project - https://github.com/RPi-Distro/Pi-gen

It runs on *buntu and debian with docker. Its been updated a bit since I first started playing with it but it allows you to completely customize the image removing and adding the packages you want as well as customizing start up scripts etc.

If you have the space on your PC I strongly recommend running it in a VM that is a custom docker configured ubuntu server because it was a complete pig to get running without docker and docker did not play well for me on my desktop xubuntu system

User avatar
HawaiianPi
Posts: 4272
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Sun Mar 17, 2019 8:56 am

Ramtec wrote:
Tue Feb 26, 2019 11:39 am
Whats the best way to automate the 'touch ssh' command on Alpine? Parse the fdisk output?
Don't know. This thread is about Raspbian Linux.

From a quick peek at alpinelinux.org, Alpine Linux doesn't seem to use images, so it has nothing to do with this thread. You could ask if anyone here has any experience with Alpine in our Other OS sub-forum: viewforum.php?f=56, but you might find more help in the Alpine community: https://alpinelinux.org/community/
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

jessica10
Posts: 1
Joined: Tue Mar 26, 2019 12:34 pm

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Tue Mar 26, 2019 12:37 pm

I have run the same code but getting error.

User avatar
HawaiianPi
Posts: 4272
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Wed Mar 27, 2019 6:07 am

jessica10 wrote:
Tue Mar 26, 2019 12:37 pm
I have run the same code but getting error.
What code, and what was the error?
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

Elizabeth7
Posts: 1
Joined: Thu Apr 04, 2019 4:50 am

Re: STICKY: Making your own custom burn-n-boot Raspbian image

Thu Apr 04, 2019 6:13 am

Recently, me and my fellow team mate Tharindu Muhandiram ran into a bit of an issue with regard to building a custom image for our Platformer IoT Gateway Upsers.

Return to “General discussion”