STICKY: Making your own custom burn-n-boot Raspbian image

[pe78420]

Hi HawaiianPi,

Very helpfull

Everything is ok with the raspbian_2018-11-15 release even when I tried to add or modify any file (ssh, config.txt, cmdline.txt, wpa_supplicant.conf, etc.)... To do that, I copied them from the windows explorer or after mounting the image on Debian via kpartx or loseup.

But I got a problem
On the latest raspbian image (2019-04-09), booting stucks when I tried to add of modify the same files!
Without modification (the raw image), booting is always successfull...

What's wrong!
Any ideas?

Many thanks in advance and best regards,

Philippe
A french guy...

[LittleMike]

That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.

Right, so begin by installing systemd-container (this will give you the systemd-nspawn utility) and zerofree, if you don't already have them. This will not take long:

Code: Select all

[email protected]:~ $ sudo apt-get update
[email protected]:~ $ sudo apt-get install -y systemd-container zerofree

Then, assuming you have already downloaded the zipped image (from here), unpack, and then loop mount it:

Code: Select all

[email protected]:~ $ unzip raspbian_latest 
Archive:  raspbian_latest
  inflating: 2018-11-13-raspbian-stretch.img 
[email protected]:~ $ sudo losetup --show -P -f 2018-11-13-raspbian-stretch.img
/dev/loop0

Your reported loop device may differ (loop1, loop2 etc.) in which case modify the following instructions accordingly.
This losetup command should have created partition mappings automatically:

Code: Select all

[email protected]:~ $ ls /dev/loop0*
/dev/loop0  /dev/loop0p1  /dev/loop0p2

Yep, two partitions, as expected. Now mount them:

Code: Select all

[email protected]:~ $ sudo mkdir -p /mnt/raspbian
[email protected]:~ $ sudo mount -v /dev/loop0p2 /mnt/raspbian
mount: /dev/loop0p2 mounted on /mnt/raspbian.
[email protected]:~ $ sudo mount -v /dev/loop0p1 /mnt/raspbian/boot
mount: /dev/loop0p1 mounted on /mnt/raspbian/boot.

Now we can chroot in. We'll use systemd-nspawn to do this; it will take care of all the necessary behind-the-scenes bind-mounts etc.:

Code: Select all

[email protected]:~ $ sudo systemd-nspawn --directory=/mnt/raspbian
Spawning container raspbian on /mnt/raspbian.
Press ^] three times within 1s to kill container.
[email protected]:~#

Can this be done on the x86 Raspi image? The download is an iso, but I used dd to convert it to an img and tried the above steps. Most of it seemed to work, but whenever I get to the systemd-nspawn portion, it yells at me that there is no OS-tree. So I tried putting a /usr/bin and an /etc folder but then it tells me that it's mounted read-only.

My goal is to create a Live USB stick running Raspberry Pi Desktop that has some base packages installed and some minor configuration completed so that I have a "baseline" because I have a habit of breaking things and have needed to start all over from scratch each time as I learn and that's just gotten really annoying.

*EDIT*

So I tried a different method of just extracting all the contents of the ISO to a folder and added the /etc and /usr/bin folders manually. Now when I try to run systemd-nspawn it tells me execv() failed, no such file or directory.

[sakaki]

LittleMike,

in principle this technique should be usable with an x86 (or x86_64) guest linux image, running under an x86 (or x86_64) host linux system, such as the Debian with Raspberry Pi Desktop OS, for example.

Could you provide some more details about your host system (e.g., is it running the RPi Desktop for PC, or some other Linux variant) and what you see at /mnt/raspbian (if you run e.g. "ls -l /mnt/raspbian"; assuming you are using the same paths as given in the instructions).

For disclosure, I have used the systemd-nspawn approach extensively on PCs running Gentoo and Arch (with Gentoo, Arch and vanilla Debian guests) but haven't actually used the "Debian with Raspberry Pi Desktop" image in either capacity to date.

Best, sakaki

[rin67630]

You know, it is so easy to have a headless reference Raspberry Pi (can be a cheap PiZero) with all your loved software that you keep up-to date and from which you copy the SD card each time you need another one...

[LittleMike]

LittleMike,

in principle this technique should be usable with an x86 (or x86_64) guest linux image, running under an x86 (or x86_64) host linux system, such as the Debian with Raspberry Pi Desktop OS, for example.

Could you provide some more details about your host system (e.g., is it running the RPi Desktop for PC, or some other Linux variant) and what you see at /mnt/raspbian (if you run e.g. "ls -l /mnt/raspbian"; assuming you are using the same paths as given in the instructions).

For disclosure, I have used the systemd-nspawn approach extensively on PCs running Gentoo and Arch (with Gentoo, Arch and vanilla Debian guests) but haven't actually used the "Debian with Raspberry Pi Desktop" image in either capacity to date.

Best, sakaki

Sorry, I didn't get an email notification you responded.

I'm running Raspberry Pi Desktop using the Live USB distro, running it with persistence.

I removed/redid everything a few times, so I'll have to go back to the start and try again (I gave up on it, honestly) but there was a boot directory and a bunch of others. No /usr/bin, but I was able to create one, and no /etc either, but I was able to create one. That's how I got as far as I did.

I'm wondering if it's less the Raspberry Pi Desktop being the issue and more that it's the Live USB that's the issue (along with the gnome bootloader). No syslinux file, I don't think.

[MyDearDeveloper]

sakaki wrote: ↑

Right, so begin by installing systemd-container (this will give you the systemd-nspawn utility) and zerofree, if you don't already have them. This will not take long:

Code: Select all

[email protected]:~ $ sudo apt-get update
[email protected]:~ $ sudo apt-get install -y systemd-container zerofree

Then, assuming you have already downloaded the zipped image (from here), unpack, and then loop mount it:

Code: Select all

[email protected]:~ $ unzip raspbian_latest 
Archive:  raspbian_latest
  inflating: 2018-11-13-raspbian-stretch.img 
[email protected]:~ $ sudo losetup --show -P -f 2018-11-13-raspbian-stretch.img
/dev/loop0

Thanks for this informative post i was struggling but this helped me

[bmanske]

I'm going to resurrect this topic. I have decided to work on a project requiring the use of several raspberry pis and the thought of going through the setup for each one is hard to swallow. So I started looking for this topic.

First:
SET UP A HEADLESS RASPBERRY PI, ALL FROM ANOTHER COMPUTER’S COMMAND LINE
https://hackaday.com/2018/11/24/set-up- ... mand-line/

November 24 2018 this article proceeded this topic. I have to wonder why it was never brought up in this discussion.

Second: Peter Lorenzen deserves credit for this very efficient shell script
Trying to get to http://peter.lorenzen.us/linux/headless ... figuration it never responded. So I went to the way back machine to get the shell script.
https://web.archive.org/web/20190131013 ... figuration
Using the final snapshot of this page is where I started.

I don't know if I got the latest version of the code. I decided to build a VMware Debian 10 machine with the Cinnamon desktop. I don't know what he used. This script would no longer run. The website obviously changed as there is one more redirect now than was accounted for in his code. I had some problems with curl giving me decent results worked almost always from the command line when requesting the https URL and almost always failed when run from the shell script. The format of the redirect message had also changed to instead of doing an awk on the second http in the file I implemented a quick search for it. I also didn't like his defaults so some of those were changed. You get the picture, I really extended the length of the script by adding more error checking and comments. I've used features of bash prior to v4.0 to try to maintain compatibility.

Three: Going Forward
I want to replace the rsync command with code to look at where the file is being copied to and set the ownership and permissions without the need for getting all of the files set up in the right condition before hand. Adding a ability to specify an include file for configuration as the command line is getting long among other ideas.

HERE's THE QUESTION.
I don't want to just set up SSH I also want RealVNC server enabled. I use TightVNC on all of my local machines and have changed a RealVNC config file to allow the 2 to communicate so I know SED and AWK will get that for me in the script. Can someone point me to the source code for the rasberry pi config or just tell me how to do it? The source code would allow me to learn how to enable / disable other devices also.

I've been thinking about putting the project on GitHub. Any interest in this?

[RonR]

Can someone point me to the source code for the rasberry pi config or just tell me how to do it?

If you're referring to 'raspi-config', it's a script located at: /usr/bin/raspi-config

[abraham5]

Nice Write Up Man, keep up the good work.

[dmikester1]

This sounds like something I was just reading about the other day: pibakery. Does that do a similar thing?
https://www.raspberrypi.org/blog/pibakery/

[bmanske]

RonR - thank you for pointing to the raspi-config shell script. I thought it was a compiled executable so I never opened it. It is probably one of the best written shell scripts I have ever seen. I found the project online and it is under the MIT license so I'm free to steal... I mean copy - paste sections of the script.

dmikester1 - I did try out Pi Bakery. It was just too limited for what I wanted to do. For example, checking for a new version before downloading, exiting on error, editing arbitrary files in the file system. This could be fixed by writing custom cards but I really didn't want to learn.

Sorry for the delay with replies but I was out of town for about a week. I'll make another post when I put it on .Git Hub.

[bmanske]

I've put what I have so far at https://github.com/bwmanske/rpido/tree/master for those who are interested.

[bmanske]

It's been a couple of weeks since I last posted and nobody has come after me for copyrights. So I must be good there.

I have added a HostCount option so you can configure the card and then it will create the number of images indicated host HostCount. Each will have a suffix on the file name and the suffix will be on the HostName as well. I'm hoping this helps code clubs, people experimenting with clusters and classrooms. If you fall into one of these categories let me know if this helps you and let me know your suggestions.

I have set up the potential for config directories 1 thru 9. -c 0 attempting to get to config 0 will loop through the possible config directories and print the messages describing the config setup (of course, you have to add the descriptions yourself). If a config number is asked for and it doesn't exist then a template file and directories will be created. I'm trying to detect if there are valid options and all of the variables are set. If a problem is detected, a rpido-configX.sh file will be created so you can compare your file to this latest template to see what is missing.

There is also a rpido-target.sh file that will read the config and actually set the options. Right now nothing makes actual changes to the system but that 's coming in the next few days. I just needed to get all of this structure right first.

The -k for keep file system mounted and the arbitrary command line command have been removed. If you use the -s option then a shell will open and you will be in a chroot shell at the root of the file system. qemu will be running the bash command shell from the image. This should be a quick way to mount and change the image from your Linux system. If using -s with a config the config executes first and then you are dropped into the shell. Type "exit" to get out of the shell when you have finished and all will be unmounted. I felt this was better than just leaving the file system forever mounted.

My next step is to be able to configure SSH, WiFi, VNC after that is country, timezone, keyboard, etc...

While it may sound like I know what I'm doing... I really don't. I don't know what options are needed. So it you have some ideas, requests or suggestions please let me know. I'm really trying to put together something that is useful for more than just myself.

[jaromrax]

Hi,
I was looking for a SD mod solution and I have found very promising
- PiBakery , last activity Oct 2018 - https://github.com/davidferguson/pibakery . Javascript, so it is more complex than I like.
- this RPiDo - https://github.com/bwmanske/rpido - from bmanske, that is just active. And it is command-line, the style.

I didn't go too deep, but it seems to me, that both projects basically mount the existing image and copy stuff into /boot. It is great that somebody works on RPiDo, however it is very brief on usage and examples and thus it makes some barrier to use it.

1/ I would like to see the examples in README.md to see where it really is now.
2/ What combinations do work ? - jessie, stretch, buster, Pi2, 3B, ZeroW, Pi4 ?

3/ One would also need to add his own blocks - e.g. download and start docker. It seems that there is very nice organization model of blocks https://github.com/davidferguson/pibakery-blocks . Maybe using the efforts out of (orphaned?) PiBakery blocks and documenting Pido would make a new quality. And attracted people that would share their new blocks?

Waiting for a comment of bmanske or bwmanske
cheers, jarom

[bmanske]

Jarom,

Thank you for the feedback. I haven't had very much so I really appreciate it. I hope that some more people that it may be useful for would offer some. I will add some examples to the readme and maybe that will help others come up with questions that I could then write about.


Everyone,

Again, I don't have much experience with shell scripts or github projects so I appreciate all kinds of constructive criticism.

To answer another of Jarom's queries... I have tried this on the 3B and 4B. I tried the latest release of Raspbian and the one previous. I'll add this to the readme also. Hopefully, others can try it and let me know if it works for them.

Recently I have added enough code to set the country / keyboard / wifi. I ran into some issues with enabling RealVNC as the card wipes out the settings as the it boots.

I have been concentrating on trying to get this much working so that I can test without using my TV as a monitor. I'll post another update soon.

[jaromrax]

Dear bmanske,
I tried briefly your code.
1. The documentation is necessary. yes, `rpido -h` shows a help. Many things are not clear.
2. crash of curl in the beginning leaves one in doubts, if the retrieved image is ok
3. sudo dd and sudo rm are very dangerous commands, I added confirmations before I have had run the code
4. user must find the necessity to install qemu-user-static
5. chroot ${PWD}/${RPI_ROOT} - adding $PWD worked to me
6. apt upgrade crashed with openssh-server

On the other hand, your code is very clean, nice and you have dedicated lot of effort to it. Congratulations.

I will try to play with it more. It seems that it is possible to install some packages before writing the image. Modular design would make a big change and opened space for other contributors.

Good luck

[bmanske]

Jaromrax,

You are the first person I have heard from that has tried this code! Thank you for your suggestions and feedback.

I'll try writing some documentation over the next day or two in the wiki on github.

I have had problems with the curl command before. Would you please tell me how it crashed for you?

Can you tell me more about #5 & 6?

Have you looked at the sample config file generated? Use the "-c 0" command line option and rpido will make one for you. I tried to leave a bunch of notes in there on use of config files.

I'll put some thought into making more modules out of this...

As always I'm always open to suggestions or even lessons on writing shell scripts.

Thank you

[Lewis-H]

Thanks! These codes are helpful! It worked on my raspberry pi.
Regards,
Lewis