STICKY: PINN - An enhanced version of NOOBS.

[jharris1993]

Update:

I decided to do some additional digging.

I have run into extremely frustrating problems in the past that turned out to be something really stupid, like spaces, or the wrong line-endings, or other stuff that's easy to miss.

So, I examined all the configuration files for line-endings - and they're all correct.
I re-downloaded a fresh copy of the metadata as advertised in the article:
https://github.com/procount/pinn/wiki/H ... using-PINN

Code: Select all

$ cd ~/os/Raspbian  
$ wget -N "http://downloads.raspberrypi.org/raspbian/os.json"  
$ wget -N "http://downloads.raspberrypi.org/raspbian/Raspbian.png"  
$ wget -N "http://downloads.raspberrypi.org/raspbian/partitions.json"  
$ wget -N "http://downloads.raspberrypi.org/raspbian/marketing.tar"  
$ wget -N "http://downloads.raspberrypi.org/raspbian/partition_setup.sh" 

(Actually, I cheated. I created a script with all those commands in it so they'd run all at once)

Code: Select all

#!/bin/bash

wget -N "http://downloads.raspberrypi.org/raspbian/os.json"  
wget -N "http://downloads.raspberrypi.org/raspbian/Raspbian.png"  
wget -N "http://downloads.raspberrypi.org/raspbian/partitions.json"  
wget -N "http://downloads.raspberrypi.org/raspbian/marketing.tar"  
wget -N "http://downloads.raspberrypi.org/raspbian/partition_setup.sh"

. . . and I checked to see if the indents were tab characters or spaces. They're spaces in both my version and in the canonical versions I downloaded.

It is also instructive to note that the canonical version of the metadata files does not include a "supports_backup" statement in their os.json file either. If that statement in the os.json file is mandatory, than how are "normal" images allowed to backup?

I also noticed that the SETTINGS partition contains a "cache" folder that contains a "data7" folder that contains a whole list of folders named "0" all the way to "F", which contains one file that seems to have essential metadata in it. I don't know where these cache files come from or what they do. In actuality, only the folders from "0" to "A" contain data, the others are empty.

Because there are so many things that just don't add up, I am going to try an additional experiment:
I am going to take another 500 gig SSD device that I have, prime it with PINN, run it, and download several Raspbian-style operating systems and install them.

I will then see which can and cannot be backed up - and then examine the metadata for each of these operating systems to discover what the problem might be.

Starting with a "canonical" installation of "blessed" operating systems, I should be able to get SOMETHING that will allow a backup. Once I get that, it will crack wide open - I can check for "what's different" and/or "what's unusual".

More when I know more.

Attached:
A tarball of my settings partition with everything included in case there's something else you'd like to examine

Settings_Partition.tar.gz

(110.63 KiB) Downloaded 22 times

(Note to forum maintainer: It would be nice to know what the maximum file sizes for uploads are. . .)

[procount]

Re: my previous email just unzip & copy this attached installed_os.json file to /settings.

It is also instructive to note that the canonical version of the metadata files does not include a "supports_backup" statement in their os.json file either. If that statement in the os.json file is mandatory, than how are "normal" images allowed to backup?

It's also done by other methods (for OSes I have no control over).

I also noticed that the SETTINGS partition contains a "cache" folder

It's a cache for internet file downloads, like the web browser. Don't worry about it, it's just for performance reasons.

I am going to try an additional experiment:

Please just try this last update to installed_os.json first, as I think it will fix your backup issues. We've come so far...!

[procount]

Most commonly I'm rebooting "back" to the usual OS (libreelec) from raspberry pi OS, and I want to walk away while that's happening (have to keep the screen on though to make sure the video works properly...).
Anyways, I'm just wondering if I can affect which OS it will default to on the next boot by a shell script? That would I could put a shortcut on the raspberry pi OS desktop, and then I wouldn't have to wait for PINN to boot up (or miss it because I'm not paying attention) to change the target OS. Actually the blue sky wish would be, to be able to do something in libreelec as well to select the next OS, but that's probably asking too much
Is any of this possible? thx

Simple answer - yes.
How? It depends on exactly what you want to do, your usual working practice, and what model of PI you have.

1. The first thing to try is just `sudo reboot n`, but replace 'n' with the boot partition number of the OS you want to boot (see /settings/installed_os.json for the partition numbers to use). This may not work on a Pi4.
2. If you nearly always use LibreELEC and only occasionally use Raspberry Pi OS, you could set LIbreELEC as a Sticky Default. That way it will always boot into LibreELEC immediately after boot. To use Pi OS instead, you need to press Shift on startup, press ESC to go to the boot menu, then select Pi OS. But on the next boot out of Pi OS, it will boot into LibreELEC again.
3. If you want to switch between more OSes, or more randomly, you can write a script to modify the sticky default. It is stored in the /settings/noobs.conf file.
4. Alternatively, you can create an autoboot.txt file on the PINN partition, which should contain "boot_partition=n" where n is the boot partition to choose. This is actioned by the firmware and will bypass PINN altogether.

[bluenote]

1. The first thing to try is just `sudo reboot n`, but replace 'n' with the boot partition number of the OS you want to boot (see /settings/installed_os.json for the partition numbers to use). This may not work on a Pi4.
2. If you nearly always use LibreELEC and only occasionally use Raspberry Pi OS, you could set LIbreELEC as a Sticky Default. That way it will always boot into LibreELEC immediately after boot. To use Pi OS instead, you need to press Shift on startup, press ESC to go to the boot menu, then select Pi OS. But on the next boot out of Pi OS, it will boot into LibreELEC again.
3. If you want to switch between more OSes, or more randomly, you can write a script to modify the sticky default. It is stored in the /settings/noobs.conf file.
4. Alternatively, you can create an autoboot.txt file on the PINN partition, which should contain "boot_partition=n" where n is the boot partition to choose. This is actioned by the firmware and will bypass PINN altogether.

Thank you! I'm on a pi4 sounds like these should do me. Thx!

[jharris1993]

Please just try this last update to installed_os.json first, as I think it will fix your backup issues. We've come so far...!

I'll do that.

Right now my system is chewing on backing up the other 500 gig SSD in case I need it. It's about 20 minutes from being done.

P. S.
The information in the "how to install two operating systems" document is absolutely excellent as far as giving me the information needed to make compressed tar-balls of the installation partitions of a particular installation. I can put them away - do something with the media - then put them back after re-creating the partition structures and resetting the device ID if necessary.

Assuming that this fixes things, (and I go through and change all the os.json files to "true" without the quotes), what do I do to make this persist - as in, if I re-install from my images, or create new "as configured" images to reinstall from, how do we make the ability to back up persist so we don't have to go through this whole dog-and-pony show the next time I try to build something like this?

This also asks the question of what has to change in the documentation to make this happen? I'd really like to believe that this pain hasn't been entirely in vain - hopefully others will learn from my mistakes and the documentation can be updated to make this a smoother transition for everyone.

[procount]

what do I do to make this persist.

Like I said above, this latest pain only occurred because you didn't add this additional field to os.json before installing your OSes and it took me a while to remember all the places to put it back retrospectively. Now you have it in place (I hope), next time it should go smoothly.

This also asks the question of what has to change in the documentation to make this happen?

I shall be reviewing this long thread next time I update the documentation. It probably needs a new section on creating the meta-files from scratch. (And I might have to have a word with XECDesign about updating the Raspbian meta-files for PINN, or choose another example).

[jharris1993]

Update:

We're making progress!

We've now gotten past the point of it not allowing the backup. The backup process is still not working, but we've progressed a step.

What's happening now is that when I go to make the backup it allows the backup, creates a name, time-stamp, and description and then when I press "OK", it immediately errors out with "An error occurred backing up. Perhaps there is no disk space?"

I notice that at the bottom of the window, it gives the destination drive, (a flash drive), the amount of disk space needed for the backup and the amount available on the target device.

In every case, the download space "Needed" is always zero. Ergo, it fails.

Is there a way for me to take a screen-shot in PINN? This way I could show you want I see.

Another question:
I remember seeing somewhere that it was possible to download and save the installer images that PINN uses, without necessarily actually installing them - for example to install on a system that doesn't have access to the internet.

I'm sure it's in the documentation SOMEWHERE. . . . but so far, I haven't been able to locate it again.

Thanks!

P. S.
While this is cooking, I am trying the experiment where I download of several "stock" images to see what happens.

[procount]

I really thought that would be it, but at least another hurdle has been passed.

You can connect via VNC and take a screen shot from your PC if you like, but I'm not sure it will help.
The backup size is normally zero, because I don't know what it will be until it is done.
Take a look at https://github.com/procount/pinn/wiki/Troubleshooting which explains how you can capture debug logs in general.

But from your zip file, I can see the problem. You didn't create, or copy, any installation slides for your OSes, so there is no slides_vga folder for any of your OSes.
It might be sufficient just to create the following folders:

/settings/os/GoPiGo_OS/slides_vga/
/settings/os/Raspbian_64_bit/slides_vga/
/settings/os/Raspbian_for_robots/slides_vga/

But just in case, once you've created the folders, copy /mnt/defaults/slides/A.png into each of those slides_vga folders.
I guess I might have to make these optional in the next version.

[jharris1993]

But from your zip file, I can see the problem. You didn't create, or copy, any installation slides for your OSes, so there is no slides_vga folder for any of your OSes.
It might be sufficient just to create the following folders:

/settings/os/GoPiGo_OS/slides_vga/
/settings/os/Raspbian_64_bit/slides_vga/
/settings/os/Raspbian_for_robots/slides_vga/

But just in case, once you've created the folders, copy /mnt/defaults/slides/A.png into each of those slides_vga folders.
I guess I might have to make these optional in the next version.

Why is the lack of "installation slides" a problem for backing up? As I read in the documentation for creating a custom spin, installation slides were an "if you have them and/or they're desirable" type thing - not a requirement.

I don't know what you have on /mnt, but I was able to find the "marketing" tarball in the stock O/S stuff I downloaded and created the appropriate directories and added the one slide.

Oh, and BTW, I DID find the download instructions - eventually, located at:
https://github.com/procount/pinn/blob/m ... d#download

This way I can download "stock" images and see what they're supposed to look like.

[jharris1993]

Update:

SUCCESS!

Adding the "slides_vga" filder and adding a slide seems to have done the trick - backups are now working on my installed operating systems. Though, to be perfectly honest, I don't know if actually adding the one slide was required or not. I did it anyway.

I am assuming that I need to update my installation files with this directory and slide too.

Can I also assume that I can substitute the backup files for the original files on my original installation media - and have an installation media that has the current state of these operating systems?

Suspected answer:
Yes, but if I want to have the operating system have a name other than the funky backup-time-stamped name, I will need to change the folder name and the operating system name in the os.json file to whatever I want, so long as they match.


One thing seems to beg mentioning:
IMHO, this should not be/have been this complex.

Though we both learned a lot, can this information be gathered together somehow so that the next poor sod who wants to do something like this - packaging up a few custom OS's in a bootable and repeatable format - doesn't have to go through all this pain? IMHO, this was much more difficult than it should have been.

I do appreciate all the effort you put into making this happen. I just want to figure out a way for the next guy to avoid all the grief we went through.

Thanks!!!

[procount]

SUCCESS!
Thanks!!!

Phew! Finally!
I have to say that this has to be one of, if not the longest support requests I've ever had!
You're right, it should not have been this complex. I'm sure most users don't have to go through this pain, but you have managed to test virtually all the advanced OS creation features I've put in there and found every edge/corner case, missing piece of documentation and tested the limits of my memory!
(I should let you loose on the "flavours/noobsconfig" features next, because they need some serious testing, too! )

I don't know if actually adding the one slide was required or not. I did it anyway.

Best to be safe. It was a straight folder copy command that failed. I've no idea why I made that mandatory, so I think I will have to make it optional, or update the documentation.

I am assuming that I need to update my installation files with this directory and slide too.

Yes. You'll find the /mnt/defaults/slides/A.png I mentioned in the PINN partition in /defaults/slides/A.png. Or better still, create some of your own.

Can I also assume that I can substitute the backup files for the original files on my original installation media - and have an installation media that has the current state of these operating systems?

It should be standalone. It will appear in the backups tab.

Suspected answer:
Yes, but if I want to have the operating system have a name other than the funky backup-time-stamped name, I will need to change the folder name and the operating system name in the os.json file to whatever I want, so long as they match.

Not quite. It depends. There is a subtle difference between this backup OS and an original OS installation.
When an original OS is installed by PINN for the first time, the partition_setup.sh may have to do a lot of operations to make it compatible with PINN. However, many of these operations only need to be done once, so a backup will already contain these modifications. They do not need to be done a second time when it is re-installed, restored, replaced or installed at a later time.
The funky backup name has a particular format with special field delimiters that indicate it is a backup and maintain a link to the original installation files (e.g. to indicate if a new version is available etc). This causes PINN to set an environment variable to modify the behaviour of partition_setup.sh, as executing some of the operations a second time may break the OS.
You might be ok if all your OSes are based on Raspbian, but I didn't expect users to start changing backup names, so this would require investigation. Feel free to experiment on your OSes, but I don't think it will work for all OSes in general.

Your original goal of controlling the boot OS through GPIO should be a breeze after this....

[jharris1993]

SUCCESS!
Thanks!!!

Phew! Finally!
I have to say that this has to be one of, if not the longest support requests I've ever had!
You're right, it should not have been this complex. I'm sure most users don't have to go through this pain, but you have managed to test virtually all the advanced OS creation features I've put in there and found every edge/corner case, missing piece of documentation and tested the limits of my memory!
(I should let you loose on the "flavours/noobsconfig" features next, because they need some serious testing, too! )

Before I retired, I was in demand as a software QA analyst. Not only was I good at creating test cases and scenarios, but my specialty was what I call "poke-and-prod" testing - just get down-and-dirty with the application and see what falls off!

Even now, the lead development engineer over at Modular Robotics who handles all the GoPiGo software went so far as to give me her "inside line" and loves to toss stuff my way to "take a look at" because of my seeming talent at finding obscure artifacts that have been bugging them for the longest time, but they forgot what caused it.

I will be happy to look at your "flavours/noobsconfig" features, but they have to stand on line behind about five or six other things people want me to destroy - (er, "test"!).

P.S. First things first - we gotta get your spelling of words like "flavors" fixed! (laughing!)

Suspected answer:
Yes, but if I want to have the operating system have a name other than the funky backup-time-stamped name, I will need to change the folder name and the operating system name in the os.json file to whatever I want, so long as they match.

Not quite. It depends. There is a subtle difference between this backup OS and an original OS installation.
When an original OS is installed by PINN for the first time, the partition_setup.sh may have to do a lot of operations to make it compatible with PINN. However, many of these operations only need to be done once, so a backup will already contain these modifications. They do not need to be done a second time when it is re-installed, restored, replaced or installed at a later time.
The funky backup name has a particular format with special field delimiters that indicate it is a backup and maintain a link to the original installation files (e.g. to indicate if a new version is available etc). This causes PINN to set an environment variable to modify the behaviour of partition_setup.sh, as executing some of the operations a second time may break the OS.

The key takeaway from the above is:

This causes PINN to set an environment variable to modify the behaviour of partition_setup.sh, as executing some of the operations a second time may break the OS.

. . . which I think I ran into when I tried a "use this to make an original installation file instead of jumping through all the hoops".

When I tried to use a backup as a raw installation source, the subsequent installation behaves "oddly" in subtle ways. For example GoPiGo O/S's ability to change WiFi modes from being a stand-alone access-point to being able to participate in a local WiFi network has been subtly damaged and the WiFi isn't behaving quite right. There may be other subtle oddities I have not found yet, so your caution is well taken:

I didn't expect users to start changing backup names, so this would require investigation. Feel free to experiment on your OSes, but I don't think it will work for all OSes in general.

Or, in other words:
Originals are originals and backups are backups and never the twain shall meet.

I am sure that with enough investigation I could discover the issues and fix them, but it would probably be easier to just repackage a new installation candidate using the knowledge gained from all this effort.

Update:
Reinstalling the same image I tried to install before, installing the backup as a backup, (instead of installing a munged installer), and this time WiFi appears to be working properly, right out of the box.

========================================

At risk of adding something else to your already long, LOOOOONG list of feature requests. . . .

Perhaps this could exist as a stand-alone utility for those who are interested in it, rather than cluttering up PINN with more "stuff".

We know that creating an installation package isn't a trivial undertaking, even if all the documentation and subtleties line up correctly the first time. My idea is that, having done all that work, created a working installation package, installed it, and then spent time configuring it to meet whatever requirements there may be - if the person doing this wants to create another raw installation package, it seems kind of wasteful for him to have to do all that work again - when PINN already knows how it's done.

It would be convenient to the point of confetti and champagne if there were a way to use PINN's innate knowledge of what it needs to re-create a raw installer package from an existing installation.

Then again, this begs the question "is this even doable?" - that is, creating a raw installer package from a previously "PINN-installed" image since, as you say, the image has been "subtly modified" so if I re-package the installed package, won't that still break when re-installed?

[jharris1993]

A follow-up:

I noticed an interesting artifact of the installation process:

Given, an installation environment that contains four or five installation packages.
Assume that all the installation packages have ONE graphic image slide except one, that has five or six image slides.

During the installation of ALL the packages, ALL of the media slides are shown, it doesn't matter if they are relevant or not.

[procount]

When I tried to use a backup as a raw installation source, the subsequent installation behaves "oddly" in subtle ways. For example GoPiGo O/S's ability to change WiFi modes from being a stand-alone access-point to being able to participate in a local WiFi network has been subtly damaged and the WiFi isn't behaving quite right.

Just check your partition_setup.sh. The default action for a new Raspbian installation is to copy the wifi settings from PINN to the new installation so that it can connect straight away. Same for some dhcpd and ssh settings. It shouldn't need to do this on an installation of a backup. You might want to just check it isn't doing what you don't want it to do.

It would be convenient to the point of confetti and champagne if there were a way to use PINN's innate knowledge of what it needs to re-create a raw installer package from an existing installation.

Having learnt how to convert a particular distro, I add all that knowledge to a bash script, so I can just crank out a new conversion on the next release.
Sakaki, who maintained the Gentoo distro for the Raspberry Pi and kindly converted it to PINN, created her own PINNIFY tool (search for it on here) to do much the same. It is quite general purpose, but only caters for OSes that are similar to Gentoo and would fail on something like Android for example.
I used some ideas from PINNIFY to create V2 of my script, which is still a work in progress. I may release it at some point, but it's not ready by any means at the moment.

.... that is, creating a raw installer package from a previously "PINN-installed" image since, as you say, the image has been "subtly modified" so if I re-package the installed package, won't that still break when re-installed?

None of the OSes should break when you re-install their backups to a new installation. That's the whole point of the $restore variable, the funky os backup names and correctly written partition_setup.sh scripts, so that I can prevent OSes from being backed up if all the conditions are not met so that they don't break when you re-install them. If you had picked one of my distributed OSes, it should have sailed through this process, but when you start crafting your own, or renaming backup files, all bets are off and you have to take care of these things yourself (and yes, I need to document these nuances better).

During the installation of ALL the packages, ALL of the media slides are shown, it doesn't matter if they are relevant or not.

Only the slides of the OSes you are installing are shown. They are all relevant because they are all being installed, they are just not synchronised to the order or duration of each individual OS being installed at any particular point in time.

[bluenote]

Hey procount. I'm trying your suggestions on rebooting my pi4 into the other OS's via the methods you listed.
I wanted to try the reboot n method, since that would be simplest for me (no need to get libreelec to automount, for one). I looked at installed_os.json, and it's not clear to me the numbering system. PINN settings partition is mmcblk0p5 , raspbian mmcblk0p8 and 9.
So I tried reboot 8 and reboot 9, but no luck. I just want to know if I am doing that right, and it's failing because this is a pi 4, or if I am misinterpreting the parameter reboot expects.
thanks, sorry if this is a dumb question.

[procount]

If 'sudo reboot 8' just boots into PINN, then it won't work on your pi4.
The partition numbers you want will be the first ones in each list, so typically 6,8,10 etc.
One of the other methods might be better for you.

[bluenote]

If 'sudo reboot 8' just boots into PINN, then it won't work on your pi4.
The partition numbers you want will be the first ones in each list, so typically 6,8,10 etc.
One of the other methods might be better for you.

Thanks. I'll set the sticky as it will be the most convenient most of the time. I appreciate your work on PINN it's very useful

[jharris1993]

If 'sudo reboot 8' just boots into PINN, then it won't work on your pi4.

That's odd.

Procount, as we both know, the two of us have been working on my system - also a Pi-4 with 4 gigs - for what seems like forever, and the reboot commands work fine from the recovery console. I am not sure if "reboot", (from within a running installation), is even designed to pass parameters back to the kernel on reboot - or at least, I've never heard of it.

[procount]

`sudo reboot` will work on all RPI. `sudo reboot <n>` should soft reboot into partition n.
It always used to work prior to the Pi4, but hardware changes in the Pi4 concerning the PMIC and speeding up the SD card access meant it stopped working. You can make it work by setting sd quirks at the expense of slower SD card access (like is done in NOOBS/PINN), or it might have been fixed in later subtle revisions of the Pi4, I can't remember. Google will turn up some info on it.

[jharris1993]

I have one burning question about the way PINN sets up the various partitions of my - admittedly complex - configuration, though I bet this is relevant to virtually any PINN installation.

Looking at the partitions using GParted, I notice that the FAT32 "boot" partitions for each image are all bigger than 150 megs, yet the data within them tops out at, maybe, 60 megs for a complex installation image.

Now I totally understand that there needs to be extra space for updates, re-configuration, this, that, and whatever else APT decides to throw in there at the next update. Not to mention my own updates to the config files!

But - over twice the size of the basic data set? I'm puzzled.

Ok, you could argue that out of a 500 gig SSD drive, (or even a 64/128 gig SD card), an extra 90 or so megs here and there don't amount to much.

Possibly true, but to paraphrase Warren Buffet: "A few megs here, a few megs there, and pretty soon you're talking real storage space."

Maybe it's me.

I come from a day and age where memory, (and disk space!), was an extremely valuable and very limited resource that needed to be carefully managed and conserved. I remember when a program update that required ONE extra 512 byte sector on disk had to be carefully planned and justified to get approved.

So, maybe I'm more sensitive to this than others.

However - even a huge SSD has limits, and SD cards have even less space to casually throw away. So maybe it would be worth revisiting some of the default allocations?

I remember that when I was building my installation images, the documents said to specify the boot partition size as the unpacked tarball size plus 100 megs - which accounts for the extra space. Maybe 50 megs - or even less - is more appropriate?

Update: I just checked and my largest boot partition is just a hair short of 60 megs (GoPiGo O/S) and the one "native" install I have, (Raspberry Pi O/S - 64 bit, downloaded from the PINN/NOOBS database), weighs in at less than 30 megs.

If I assume that every single boot image file, (.dtb, .elf, .img, etc.), and every single overlay file gets replaced if there's a kernel update, then the largest possible update I could ever encounter, (again, using GoPiGo O/S - the largest image - as an example), would be less than 50 megs in size.

So, do you think that a 100 meg buffer for the boot partition is adequate? Can it be made smaller by default? Or are there additional subtleties that I don't understand or know about yet?

[jharris1993]

When I tried to use a backup as a raw installation source, the subsequent installation behaves "oddly" in subtle ways. For example GoPiGo O/S's ability to change WiFi modes from being a stand-alone access-point to being able to participate in a local WiFi network has been subtly damaged and the WiFi isn't behaving quite right.

Just check your partition_setup.sh. The default action for a new Raspbian installation is to copy the wifi settings from PINN to the new installation so that it can connect straight away. Same for some dhcpd and ssh settings. It shouldn't need to do this on an installation of a backup. You might want to just check it isn't doing what you don't want it to do.

I will probably do that, just not right now.

One thing that needs to be more carefully documented is "what happens when PINN installs an image" and compare it to the case where PINN restores a backup. This would be useful in the case where someone wants to take an existing PINN image and after configuration, release it as a cleanly installable PINN installer package.

Here's an idea that might be worth some thought:
Perhaps the "backup" PINN image should be exactly the same as a raw installer package. This way either restoring a backup or installing fresh are actually the same process and won't require special exceptions and variables that may - or may not - work.

One of the advantages of this is in the case where multiple people are working on a project and have decided that PINN images are the best way to distribute working images to the rest of the team for test and/or review. If the result of a backup is the same as a raw installer package, then it doesn't matter. Any installation becomes the same as every installation, regardless of system or configuration.

A parallel case is the one where a package maintainer wants to release an operating system as both an image file AND a PINN formatted installation package file for those who have PINN installed. After the basic configuration is done, beta testing is complete, and management approval for release is obtained, all that needs to be done is "backup" the installation, rename it, compress it if desired, and place it on the server for distribution.

What say ye?

[procount]

So, do you think that a 100 meg buffer for the boot partition is adequate? Can it be made smaller by default? Or are there additional subtleties that I don't understand or know about yet?

In the case of a plain Raspbian/Raspberry Pi OS image, the original boot partition (say Jessie or Stretch maybe) was smaller, but at some point the boot partition was increased to about 250MB. Mostly I think this is to do with apt upgrades of the firmware etc. It needs enough free space to store the new image, expand it, and keep a backup of the original so you could rollback in the event of an error. It also needs some free space to allow for future expansion. I've never seen a newer firmware or software version be smaller than the previous one! The 100MB is a guideline for any OS. It may be a bit under the recommended size for Raspbian/PiOS. If in doubt, you can keep it the same as the original boot partition size of OS you are backing up, since this partition is a fixed size, therefore no estimation is typically required.

The root partition is another matter, however, as you can't use the existing partition size since it depends on the size of your SD card. Here 400MB is the recommended guideline, but even this could well be a bit on the small side. Imagine if you install as many OSes as you can onto an SD card. Each will take up their minimum size since there is no free space to add to them. Does 400MB leave you enough free space for a full apt-update on Raspbian? Previous issues indicate it is not. Do I know enough about each OS to know the minimum size required? Even if I did, I don't know how big a future upgrade will be. Imagine updating Mathematica or LibreOFFICE, for example, they would require much more free space. Lack of free disk space often prevents the desktop environment from loading too. Don't forget you may need a swap file on there too.

Perhaps the "backup" PINN image should be exactly the same as a raw installer package.

How can that be?
Suppose you install a normal image of a distro to an SD card, then immediately take another image of it. That would be an exact image of the original.
But suppose you then boot that image and take another image. Would that be the same? No, because a lot of things have now been configured for it to work on your system. This is the same for any backup system.

If the result of a backup is the same as a raw installer package, then it doesn't matter. Any installation becomes the same as every installation, regardless of system or configuration.

From PINN's point of view, it is exactly the same process because it uses the same meta-files and tar files. Any differences are due to the nature of the partition_setup.sh script itself that is used to customise the image for use within the PINN multi-boot environment, and this is totally OS dependent. PINN only facilitates the script to have different behaviours to help it avoid unnecessarily performing the same customisations twice. In the case of Raspbian, their script can copy wifi and other network configuration files from NOOBS/PINN on first installation for convenience. But it doesn't need to do this a second time when restoring a backup, and probably shouldn't, hence why I provide the option for it to know the difference. PINN only provides the capability. What you do with it on your own custom image is up to you. And if you use an existing script, just make sure it still does what you want and doesn't conflict with anything you have customised.

There shouldn't be any reason why your backup wifi doesn't work the same after being restored. I have no idea what your image does or how it works. My suggestion was the only thing I could think of where PINN could potentially have an influence, but I doubt it is that. Mostly you can just use a backup as an standard installation.

One of the advantages of this is in the case where multiple people are working on a project and have decided that PINN images are the best way to distribute working images to the rest of the team for test and/or review. If the result of a backup is the same as a raw installer package, then it doesn't matter. Any installation becomes the same as every installation, regardless of system or configuration.

This is largely how it is intended to work already. The result should be an accurate image of what you originally backed up, which would mostly not be the same as raw installer package, because you customised it before backing it up.

A parallel case is the one where a package maintainer wants to release an operating system as both an image file AND a PINN formatted installation package file for those who have PINN installed. After the basic configuration is done, beta testing is complete, and management approval for release is obtained, all that needs to be done is "backup" the installation, rename it, compress it if desired, and place it on the server for distribution.

That's not how distro maintainers create raw installation images. They do not copy files to an SD card, customise it, then take an image from it.
The images are always created virtually and directly.
Take Gentoo64 for example, previously maintained by Sakaki. As the Gentoo maintainter, she created normal installation images and PINN installation images. The PINN installation images were made directly from the raw installation images. They were not backups of an SD card after the raw installation had been installed and executed.
It's the same with any PINN OS image that I create, or the gaming distros produced by Matt.

The main difference is that a raw installation (whether an .img file or a set of .tar.xz files) is a plain vanilla image with no personal data in it, no crypto keys, or hashes etc. However, a backup image (whether an .img file or a set of .tar.xz files) may contain plenty of these as they are normally created on first boot. Backups will also contain your own passwords, hostname, ssh keys etc.

So whilst backups can be installed in the same way as a raw installation, they should not be treated in the same way.
Yes, you can distribute them as you suggest, but just be aware of what information you are distributing inside them.
You may need to sanitize them first by deleting any of your own personal data you don't want to share, and maybe create a run-on-first-boot script for the user to personalise it to themselves, just like a real raw distro might.


Have you started with your GPIO boot selection code yet? How is it coming along?

[jharris1993]

Greetings!

I am not sure if you are understanding me correctly. Let me try again.

1. I already know that if I install an image, customize it, play with it, perform updates and installations on it, (etc.), it won't be exactly the same as the original installation. Obviously it has been changed and whatever image I create will contain whatever has been changed.

2. Not everybody creates images "virtually". The cases you describe are cases where an original image has already been created and all that is needed is to read the data from the installation image and convert it to the required format.

In the cases I meant to suggest were cases where actual, original, development was being done to create the original instance of the image. In this case it is obvious that the data has to be extracted from the SD card containing the final version of the image, appropriately sanitized as you mentioned, and then packaged as a raw binary image file that utilities like "dd" or Etcher can flash to the target system's SD card.

3. My suggestion was intended to answer the point you made about the installation script that is supposed to perform different actions, based on if the image is a "backup" or an original install.

My thought was if - at the time the backup image is created - it is created in the same way as an original installation file - not exactly like the initial file used for the initial installation, but in the same form and substance as if it were an original installation file - as if I had followed the steps contained in the article about making a multi-boot PINN image from customized installations.

That being done, and the resultant "backup" file being the same as if it were a new installation file, there would be no need for the installation script to behave differently. Every file would be - as far as the installer was concerned - the same as any other file. There would be no need for specialized variables or exceptions in your scripts.

Note that I do not mean that I would be creating an exact copy of the original installation file, I would be creating a brand new file, containing any and all customizations, and creating it as if it had never been installed.

As far as sanitizing the image is concerned, that could be done by the same process that strips out any socket files - or it can be left to the user creating the backup to remove them. <== Perhaps the best choice, this way the "installation package" created would serve as a backup with everything as it was on the system. The user could then sanitize the file in whatever way he wished prior to backup if he wished to make it more generic.

Perhaps the backup process could have an option to either create a "standard" backup as is done now, or create an installation package instead of a true backup.

Hopefully I have explained this properly.

Thanks!

[jharris1993]

Have you started with your GPIO boot selection code yet? How is it coming along?

Not yet, there are a few other things that have to be done before I can return to that particular aspect of the project.

In essence, I'm taking a short breather to clear my mind so that I can focus efficiently on the actual boot process once I get back to it.

Additionally, I am still recovering from the neurological damage done to me by my recent Coronavirus infection. Though I am now completely recovered, (from the Coronavirus part), there is still rather severe neurological damage affecting things like eye-hand coordination which makes working on this stuff a challenge. Even my ability to type is seriously affected, making a simple typing task that would normally take a few minutes before, now take over an hour or two to complete.

As a consequence, I may not have results for you in the very nearest future.


There is one concern I have with PINN in general, and in any possible automatic boot selection process in particular:

The normal PINN behavior is to display the wallpaper delaying for "x" number of seconds waiting for the shift key, then progress to the boot selection menu, waiting for an additional number of seconds before eventually timing out and booting either the "sticky default" or whatever was booted previously. This takes an unacceptably long time for a headless system to boot. Ideally it should boot as if there is an "autoboot.txt" file present - immediately booting to the requested O/S.

I know we discussed this before, perhaps in the "booting multiple O/S's" thread I created. Once my hands become less numb and I can do things with greater precision and less pain, I will re-examine this.

[THX1182]

Hi procount...

Installed Raspbian an Lineage 17 to SSD... Lineage will reboot after the boot animation.

The SSD is sda so it should work.

What can I do to resolve this???

Cheers!!!