bls
Posts: 713
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA
Contact: Twitter

SD Card Image Manager

Mon May 04, 2020 9:55 pm

Like many of you, I have a small fleet of Raspberry Pi devices now, and when I rebuild them, I want them all set up "just so", with the configuration, apps, and personal files all in their places. Identically, on each of the systems.

And, like some (many?) of you, I have accumulated a set of scripts over time to set up a new system. I just finished a tooling overhaul, which resulted in sdm, an SD Card Image Manager tool.

sdm lets you:
  • Easily build and maintain a Raspbian "golden image" (.IMG file) with all of your customizations, keymaps, locales, timezones, and favorite apps installed in it.
  • Write that shiny golden image to an SD Card with the host name of your choice in it such that it's ready to go.
  • Do these things quickly and easily with minimal think or wait time.
  • Easily include your own configuration scripts
Ready-to-go means that your system has all your personal customizations installed and all Raspbian packages and updates installed at first system boot. If you're building a new fleet of Pi systems, you can apt update the golden image, and each of the new Pis will boot up fully updated.

With sdm you'll spend a lot less time rebuilding SD cards, configuring your system, and installing packages, and more time on the things you really want to do with your Pi.

As a special bonus, sdm includes a configuration script for apt-cacher-ng (client and server). apt-cacher-ng enables you to update all your Pis quickly by caching local packages. This reduces install and update time, and can greatly reduce network consumption. And, it's cooler than a ginsu knife. :lol:

sdm only runs on Raspbian, and requires a USB SD Card reader for writing a new SD Card. No, you cannot use sdm to rewrite the running system's SD Card.

I know that there are several other tools like this around. I opted to write my own in the process of upgrading my tooling. It's super-simple to install and use, and has no external dependencies, other than systemd-nspawn, which you must install via

Code: Select all

 sudo apt install systemd-container
Read about it and grab it from https://github.com/gitbls/sdm. Or, if you want to just grab the files, download the EZ downloader at https://raw.githubusercontent.com/gitbl ... mInstaller

Hope you find it useful. Feedback welcome, of course.
Pi tools:
Easily and quickly build customized-just-for-you SD Cards: https://github.com/gitbls/sdm
Free your network from your router's DHCP/DNS:https://github.com/gitbls/ndm
Easy strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

User avatar
Botspot
Posts: 1342
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas
Contact: Website

Re: SD Card Image Manager

Tue May 05, 2020 1:31 am

bls wrote:
Mon May 04, 2020 9:55 pm
I just finished a tooling overhaul, which resulted in sdm, an SD Card Image Manager tool.
That looks pretty useful!

BTW I've made my own image manager: Pi-Power-Tools. So I hope it's OK to ask lots of questions and give suggestions.

For example, suggestions like this one:
On line 357 of your main sdm script, why not add the -b flag to your systemd-nspawn command? That way systemctl/systemd will be running, and means you can do a lot more stuff, for example running graphical apps.
Ever wished you could "open" a RaspiOS .img file? Well now you can! I've developed Pi-Power-Tools, the only RaspiOS img editor on the planet.
https://github.com/Botspot/Pi-Power-Tools
Want to try it out?

Code: Select all

wget -O - https://raw.githubusercontent.com/Botspot/Pi-Power-Tools/master/update | bash

bls
Posts: 713
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA
Contact: Twitter

Re: SD Card Image Manager

Tue May 05, 2020 3:25 am

Botspot wrote:
Tue May 05, 2020 1:31 am
bls wrote:
Mon May 04, 2020 9:55 pm
I just finished a tooling overhaul, which resulted in sdm, an SD Card Image Manager tool.
That looks pretty useful!

BTW I've made my own image manager: Pi-Power-Tools. So I hope it's OK to ask lots of questions and give suggestions.

For example, suggestions like this one:
On line 357 of your main sdm script, why not add the -b flag to your systemd-nspawn command? That way systemctl/systemd will be running, and means you can do a lot more stuff, for example running graphical apps.
Thanks, @Botspot for taking a look at it. I'd looked at your Power Tools previously...thought they were pretty cool, but I'm not usually a GUI guy
:roll:

Definitely appreciate the feedback. I was looking at the nspawn documentation for a way to run a command and then prompt for input. Looks like the --pipe switch might do the trick, but it appears that Buster's systemd is a bit behind on that.

I'll take a look at the -b flag tomorrow and see how it works. At a minimum, I'll make it an option...easy enough to do!
Pi tools:
Easily and quickly build customized-just-for-you SD Cards: https://github.com/gitbls/sdm
Free your network from your router's DHCP/DNS:https://github.com/gitbls/ndm
Easy strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

User avatar
Botspot
Posts: 1342
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas
Contact: Website

Re: SD Card Image Manager

Tue May 05, 2020 12:52 pm

bls wrote:
Tue May 05, 2020 3:25 am
Thanks, @Botspot for taking a look at it. I'd looked at your Power Tools previously...thought they were pretty cool, but I'm not usually a GUI guy
Completely understand.
Have you seen vdesktop? It's the CLI script that PPT uses to boot things. Link here: https://github.com/Botspot/vdesktop
bls wrote:
Tue May 05, 2020 3:25 am
I was looking at the nspawn documentation for a way to run a command and then prompt for input.
I'm not sure what you mean. Do you mean "I want to run a command in the container automatically"?
Or do you mean "I can't type anything in the container. Like if apt asks for [Y/n], It doesn't 'see' my keyboard input"?
Ever wished you could "open" a RaspiOS .img file? Well now you can! I've developed Pi-Power-Tools, the only RaspiOS img editor on the planet.
https://github.com/Botspot/Pi-Power-Tools
Want to try it out?

Code: Select all

wget -O - https://raw.githubusercontent.com/Botspot/Pi-Power-Tools/master/update | bash

bls
Posts: 713
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA
Contact: Twitter

Re: SD Card Image Manager

Tue May 05, 2020 1:57 pm

Botspot wrote:
Tue May 05, 2020 12:52 pm

I'm not sure what you mean. Do you mean "I want to run a command in the container automatically"?
Or do you mean "I can't type anything in the container. Like if apt asks for [Y/n], It doesn't 'see' my keyboard input"?
The former: I want to run a single command in the container immediately after it starts, and then prompt the user for further input. With this the full process can be "touchless" for the user, if they desire (no user input required).

I put this issue aside while I finished everything else, and then in my excitement to be "done" (ha! We know that never really happens!) neglected to revisit it before I posted. I revisited this last night and now have it working. So sweet!

vdesktop...looked at it a while ago and put it on my "investigate and try" list, which I apparently lost :o . I see there are a couple of things you've sorted that I need to investigate further.

Thanks for the feedback and encouragement! Will be updating sdm later today with a couple of key improvements.
Pi tools:
Easily and quickly build customized-just-for-you SD Cards: https://github.com/gitbls/sdm
Free your network from your router's DHCP/DNS:https://github.com/gitbls/ndm
Easy strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

bls
Posts: 713
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA
Contact: Twitter

Re: SD Card Image Manager

Tue May 05, 2020 8:48 pm

Just posted updates that are primarily cleanup and logging-related, but added a few new features:
  • sdm now runs the Phase 1 script automatically. In fact, the whole process is automatic except for typing in the Pi password
  • Added a few new switches
    • --bootconfig to uncomment and set values for /boot/config.txt parameters
    • --bootadd to add new keys/values at the end of /boot/config.txt
    • --batch which performs the customizations and exits
    • --ddsw allows you to replace the default 'dd' switches for trying different blocksizes, etc.
    • --nspawnsw allows you to specify additional switches for the systemd-nspawn command
I think it's in really good shape with this update. Now I can go play with nspawn -b and a few other things :lol:

This will be my last update here. Further updates will be appropriately posted on github at https://github.com/gitbls/sdm
Pi tools:
Easily and quickly build customized-just-for-you SD Cards: https://github.com/gitbls/sdm
Free your network from your router's DHCP/DNS:https://github.com/gitbls/ndm
Easy strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

bls
Posts: 713
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA
Contact: Twitter

Re: SD Card Image Manager

Thu May 21, 2020 8:52 pm

One last post. Here's a log from building an SD Card image from the base Raspbian OS image (-Lite in this case, but -Full works equally well). Build steps include:
  • Install a bunch of apps (list in the log)
  • Install the X11 windowing system with the icewm window manager and xdm display manager (wouldn't need to do this on -Full, but you might want to install some additional graphical-based applications)
  • Set hdmigroup and hdmimode in config.txt
  • Copy a bunch of personal files (all done in my personal customize script bls-sdm-customize)
  • Set the eeprom directory to 'stable'
All completed in 5 minutes on a Pi4, with a lot of help from apt-cacher-ng.

The lines with bls-sdm-customize in them are logging my personal customizations. Everything else is "standard", and under your control. Of course, the list of apps and X11 software to install (listed in a text file) is fully customizable as well.

After the IMG file was updated, I burned it to an SD card. This process, which is also logged, sets the target hostname. The log below includes this, as well as the steps taken on the first system boot.

Those of you who are time-aware might notice that the date/time is wrong on the last few lines of the log. These are added by the firstboot script, which apparently runs before Raspbian has updated the date/time. Sigh!

Code: Select all

2020-05-20 16:25:18 Configuration Started
2020-05-20 16:25:18 Command Line: /usr/local/sdm/sdm 2020-02-13-raspbian-buster-lite.img --poptions 
2020-05-20 16:25:18  firstboot,apps,xwindows --apps @bls-apps --xapps @bls-xapps --cscript bls-sdm-customize --csrc 
2020-05-20 16:25:18  /rpi --aptcache 192.168.92.4 --hdmigroup 2 --hdmimode 82 --eeprom stable --rootpwd --user bls 
2020-05-20 16:25:18  --uid 2400
2020-05-20 16:25:18 Phase 0 Starting
2020-05-20 16:25:18 Phase 0 Custom Script '/usr/local/sdm/bls-sdm-customize'
2020-05-20 16:25:18 bls-sdm-customize Phase 0 Starting
2020-05-20 16:25:18 bls-sdm-customize Phase 0 Copy wpa_supplicant.conf
2020-05-20 16:25:18 bls-sdm-customize Phase 0 Create NFS environment
2020-05-20 16:25:18 bls-sdm-customize Phase 0 Copy /usr/local/bin config scripts
2020-05-20 16:25:19 bls-sdm-customize Phase 0 Copy bls login files to /mnt/sdm/home/bls
2020-05-20 16:25:19 bls-sdm-customize Phase 0 Completed
2020-05-20 16:25:19 Phase 0 Completed
2020-05-20 16:25:19 Phase 1 Starting
2020-05-20 16:25:35 Phase 1 Custom Script '/usr/local/sdm/bls-sdm-customize'
2020-05-20 16:25:35 bls-sdm-customize Phase 1 Starting
2020-05-20 16:25:35 bls-sdm-customize Phase 1 Disable unused wpa_supplicants
2020-05-20 16:25:35 bls-sdm-customize Phase 1 Disabling wlan0
2020-05-20 16:25:35 bls-sdm-customize Phase 1 Disable rng-tools and install haveged
2020-05-20 16:25:39 bls-sdm-customize Phase 1 Switch cron to systemd
2020-05-20 16:25:39 bls-sdm-customize Phase 1 Enable systemd VNC sockets
2020-05-20 16:25:39 bls-sdm-customize Phase 1 Completed
2020-05-20 16:25:48 FirstBoot Set Locale to 'en_US.UTF-8'
2020-05-20 16:25:51 FirstBoot Set Keymap to 'us'
2020-05-20 16:25:51 FirstBoot Set Timezone to 'America/Los_Angeles'
2020-05-20 16:25:51 Application Installs Starting
2020-05-20 16:25:52 Application Installs Installing keychain vim emacs at-spi2-core dnsutils putty aptitude 
2020-05-20 16:25:52  qpdfview avahi-utils python3-pip uuid-runtime lftp ncftp tmux mc iftop nfs-kernel-server whois 
2020-05-20 16:25:52  telnet lsof iperf3 zip nmap feh links2 systemd-container zerofree
2020-05-20 16:27:07 Application Installs Completed
2020-05-20 16:27:07 XWindows Installs Starting
2020-05-20 16:27:07 XWindows Installs Installing xserver-xorg xserver-xorg-core xserver-common xterm x11-apps 
2020-05-20 16:27:07  xfonts-base xfonts-100dpi xfonts-75dpi xfonts-scalable xdm icewm icewm-themes 
2020-05-20 16:27:07  tigervnc-standalone-server chromium-browser firefox-esr xclip xcolors xcolorsel xcolmix retext
2020-05-20 16:28:15 XWindows Installs Completed
2020-05-20 16:28:59 Phase 1 Completed
2020-05-20 16:28:59 bls-sdm-customize Phase post-install Starting
2020-05-20 16:28:59 bls-sdm-customize Phase post-install Editing xdm config files
2020-05-20 16:28:59 bls-sdm-customize Phase post-install Disabling xdm on console
2020-05-20 16:28:59 bls-sdm-customize Phase post-install Completed
2020-05-20 16:28:59 Phase 1 Command Prompt
2020-05-20 16:36:18 Setting host name 'pix' onto SD Card image for sdm-firstboot
2020-05-20 16:36:19 Burn command line: /usr/local/sdm/sdm --burn /dev/sdb --host pix 
2020-05-20 16:36:19  2020-02-13-raspbian-buster-lite.img
2020-05-20 16:36:19 Created SD Card from 2020-02-13-raspbian-buster-lite.img
2020-02-13 08:10:22 FirstBoot Set Hostname to 'pix'
2020-02-13 08:10:23 FirstBoot Set WiFi Country to 'US'
2020-02-13 08:10:23 Executing custom FirstBoot script '/usr/local/sdm/thispi/1piboot/010-disable-triggerhappy.sh'
2020-02-13 08:10:26 Executing custom FirstBoot script '/usr/local/sdm/thispi/1piboot/020-ssh-switch.sh'
2020-02-13 08:10:28 Executing custom FirstBoot script '/usr/local/sdm/thispi/1piboot/030-disable-rsyslog.sh'
sdm "installation" (e.g., copying the scripts) and basic configuration won't take you very long. Adding your own personal customizations...that's a lifetime of edits, as you discover new stuff to add to your "standard" system image!

https://github.com/gitbls/sdm
Pi tools:
Easily and quickly build customized-just-for-you SD Cards: https://github.com/gitbls/sdm
Free your network from your router's DHCP/DNS:https://github.com/gitbls/ndm
Easy strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

Return to “Advanced users”