Image File Utilities

[dsyleixa123]

thank you very much for your advices! Appearently it's at least a possible way as it seems, but tbh, quite cumbersome though.
I actually was thinking about a tool like the Stretch GUI Menu SD Copy Tool, but not for GUI-start but for a cmd line start, and quite as easy to use like that...
I forgot to mention that I am using a Pi 2 v1.1, for non-GUI Stretch and Jessie and retropie (non-GUI Stretch), and I also patially have multiple additional partitions from Noobs which I want to include into that image.

[RonR]

@dsyleixa123

I doubt a tool exists that will accomplish what you are seeking.

I've attached noobs2img, which is a slightly modified version of image-backup, that will create a stand-alone image file of any running Raspbian-based system installed by NOOBS. The resulting image file(s) will be suitable for writing to an SD card or USB device with Etcher. Hopefully this will allow you to extract individual systems from NOOBS or assist you in rebuilding your existing NOOBS system on a new and smaller SD card.

I think your only other option is to undertake a procedure of using resize2fs and parted to shrink one or more of your existing partitions, followed by using dd to copy the result to another SD card or image file.

[iamme]

Hello! I hope this is not the wrong place to post a problem/comment, but since these scripts are for the RPi (vs. only Raspbian)
I hope someone (the author?) could help. If nothing else it may make the script more portable.

I like the image-backup scripts better than others I have looked at, and the image-shrink utility has saved me at least one. I use it on RPi4 Raspbian Buster and am trying to get it to run w/o errors on another Rpi 3 running latest OSMC/KODI. Creating a full image appears to work.

However, due to a difference in mounts/mount names, the update of an existing IMG causes an error. Before I do a data dump, 1) thanks for the code and sharing it, and 2) It would be great if the image-backup utility could 1) be made non-interactive, for use in cron, and 2) include the option to shutdown/restart user-defined services/task before running (i.e., sudo systemctl stop mediacenter, ("KODI"), etc. So here goes:

In buster, the result of "df | grep /dev/root" returns: /dev/root 10536812 9468860 512996 95% /
In OSMC, the result of "df | grep /dev/root" returns: NOTHING

Thus the line (~178): ROOTUSED=$(((${DFKSIZE} - ${DFKFREE}) * 1024))
./x.sh: line 40: ( - ) * 1024: syntax error: operand expected (error token is ") * 1024")
DFKSIZE and DFKFREE had no value ( nul - nul * 1024)
because on OSMC df | grep /dev/root returns nothing...

On OSMC a df might return this:

Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 376668 0 376668 0% /dev
tmpfs 381788 14760 367028 4% /run
/dev/mmcblk0p2 60531780 2920592 54513232 6% /
tmpfs 381788 0 381788 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 381788 0 381788 0% /sys/fs/cgroup
/dev/mmcblk0p1 244988 36542 208446 15% /boot
/dev/sda1 976761856 723000832 253761024 75% /media/USB3VIDS

In this case, I assume (and one knows what can happen if you do) OSMC's/dev/mmcblk0p[1|2] = Buster's /dev/root?

So, how can/should the hard-coded "df | grep /dev/root" call be enhanced to allow compatibility with OSMC?

Thanks.

[RonR]

since these scripts are for the RPi (vs. only Raspbian)

Actually, these scripts are for 'only Raspbian', or at least Raspbian-like installations, meaning a fat boot partition and an ext4 root partition.

It would be great if the image-backup utility could 1) be made non-interactive, for use in cron

image-backup is intended to be run interactively to create the initial full backup, and then non-interactively via cron for incremental backups.

and 2) include the option to shutdown/restart user-defined services/task before running (i.e., sudo systemctl stop mediacenter, ("KODI"), etc.

I don't believe any changes are required to accomplish this. Simply create a cron job that runs a backup script:

Code: Select all

0 4 * * * /root/image-bkup

and do your shutdown(s), incremental backup, and restart(s) in that backup script:

Code: Select all

#!/bin/bash

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# shutdown user-defined services/tasks here

image-backup /mnt/backups/backup.img

# restart user-defined services/tasks here

In this case, I assume (and one knows what can happen if you do) OSMC's/dev/mmcblk0p[1|2] = Buster's /dev/root?

So, how can/should the hard-coded "df | grep /dev/root" call be enhanced to allow compatibility with OSMC?

I run everything headless here and use SSH and/or VNC for access. Is there a way I can install and run OSMC/KODI without an attached keyboard and monitor so I can peer around a little? I put OSMC_TGT_rbp2_20191118.img on a 32 GB SD card and tried it in a Raspberry Pi 3B+, but it doesn't appear to even get started. Raspbian Buster Lite on the same SD card boots right up.

[dsyleixa123]

@dsyleixa123

I doubt a tool exists that will accomplish what you are seeking.

I've attached noobs2img, which is a slightly modified version of image-backup, that will create a stand-alone image file of any running Raspbian-based system installed by NOOBS. The resulting image file(s) will be suitable for writing to an SD card or USB device with Etcher. Hopefully this will allow you to extract individual systems from NOOBS or assist you in rebuilding your existing NOOBS system on a new and smaller SD card.

I think your only other option is to undertake a procedure of using resize2fs and parted to shrink one or more of your existing partitions, followed by using dd to copy the result to another SD card or image file.

thank you very much for your efforts!
So IIUC, I'll have to use the img-tools for simple Raspbian partitions and the new noobs2 tool just for noobs SDs, correct?

[RonR]

So IIUC, I'll have to use the img-tools for simple Raspbian partitions and the new noobs2 tool just for noobs SDs, correct?

noobs2img will simply backup a single running Raspbian system installed under NOOBS to an image file, not the whole NOOBS SD. The resulting image file will be a standalone backup of that single system, which if written to an SD card will run by itself.

[dsyleixa123]

ok, thank you, now I see!
what will happen to additional present FAT32 partitions with either backup program?

[RonR]

what will happen to additional present FAT32 partitions with either backup program?

image-backup and noobs2img both create 'normal' Raspbian images of the system that is currently executing, consisting of a single FAT32 boot partition followed by a single EXT4 root partition. Additional system partitions (in the case of NOOBS) or other partitions you have added are not dealt with.

noobs2img is simply image-backup with three additional lines that 'fix' fstab and cmdline.txt entries so that the extracted boot/root partitions will run standalone instead of as part of NOOBS.

[iamme]

RE:
I run everything headless here and use SSH and/or VNC for access. Is there a way I can install and run OSMC/KODI without an attached keyboard and monitor so I can peer around a little? I put OSMC_TGT_rbp2_20191118.img on a 32 GB SD card and tried it in a Raspberry Pi 3B+, but it doesn't appear to even get started. Raspbian Buster Lite on the same SD card boots right up.

Off-hand, I would start with downloading the correct version for your Pi... In a perfect world, you might try a "headless" install using
pi bakery or the like to pre-configure things, but it would probably still need a keyboard and mouse to install Kodi initially.

I personally think LIBELEC does a better job setting things up from scratch in a new install, with OSMC second. If you have a running system (and I have not tried this yet) can't one do a "apt-get install kodi" or similar?

In any case, if I do a df -Th | grep "^/dev" on my OSMC, it returns:

/dev/mmcblk0p2 ext4 58G 2.8G 52G 6% /
/dev/mmcblk0p1 vfat 240M 36M 204M 15% /boot
/dev/sda1 fuseblk 932G 690G 243G 75% /media/USB3VIDS

and cat /etc/os-release returns [edited]:

ID=osmc
ID_LIKE=debian

[RonR]

Off-hand, I would start with downloading the correct version for your Pi... In a perfect world, you might try a "headless" install using
pi bakery or the like to pre-configure things, but it would probably still need a keyboard and mouse to install Kodi initially.

All of the images under the heading of "Raspberry Pi 2 / 3 / 3+" have 'rbp2' in the filename.

I think I'll leave the task of supporting OSMC images to someone else.

So, how can/should the hard-coded "df | grep /dev/root" call be enhanced to allow compatibility with OSMC?

You might try using the following instead:

Code: Select all

df | grep "% /$"

[iamme]

re:
You might try using the following instead: df | grep "% /$"

Thanks! That got me past the error on OSMC. I was then able to create an IMG file and "image-check" appeared to report the IMG file was OK. Been busy and (it's a working media center) I have not tried loading in on a memory and trying to boot with it, so I can not give it a real final test. In any case, this is the code I added/modified to work on OSMC and Raspbian:

Code: Select all

if [ `uname -a | grep -i "osmc"` ]
  then
    INFO="$(df | grep "% /$")"    #OSMC
  else
    INFO="$(df | grep /dev/root)" #RASPBIAN
fi

[RonR]

In any case, this is the code I added/modified to work on OSMC and Raspbian:

Code: Select all

if [ `uname -a | grep -i "osmc"` ]
  then
    INFO="$(df | grep "% /$")"    #OSMC
  else
    INFO="$(df | grep /dev/root)" #RASPBIAN
fi

That's not necessary. Just:

Code: Select all

INFO="$(df | grep "% /$")"

works in boh cases (Raspbian and OSMC).

[mihol]

Hi Ron,

I appreciate the work you put into creating this utility for LINUX noobs like me

Unfortunately when I tried to copy my 6853 MB installation to a 7.5GB USB-stick, it gave below error.

Maybe I did something wrong, any hint appreciated

Code: Select all

root@rp1:~# . /home/pi/bin/noobs2img.sh
basename: missing operand
Try 'basename --help' for more information.

Image file to create? /media/pi/RP/rp-eve.3.img

Image ROOT filesystem size (MB) [20184]? 0

Requested image ROOT filesystem size (0) is too small (Minimum = 6853)

Image ROOT filesystem size (MB) [20184]? 6853

Create /media/pi/RP/rp-eve.3.img [6853 MB] (y/n)? y
truncate: failed to truncate '/media/pi/RP/rp-eve.3.img' at 7462715392 bytes: File too large

Unable to create image file

You have new mail in /var/mail/pi
pi@rp1:~ $ sudo su -
root@rp1:~# df
Filesystem      1K-blocks    Used Available Use% Mounted on
/dev/root        20212992 5967420  13195768  32% /
devtmpfs           217212       0    217212   0% /dev
tmpfs              221540       0    221540   0% /dev/shm
tmpfs              221540    6088    215452   3% /run
tmpfs                5120       4      5116   1% /run/lock
tmpfs              221540       0    221540   0% /sys/fs/cgroup
/dev/mmcblk0p8      64366   23010     41356  36% /boot
tmpfs               44308       4     44304   1% /run/user/1000
/dev/mmcblk0p6      47371    7729     39643  17% /media/pi/RISC_OS_Boo
/dev/mmcblk0p5      30701    3162     25246  12% /media/pi/SETTINGS
/dev/mmcblk0p10    163660  107876     55784  66% /media/pi/E32D-086E
/dev/mmcblk0p1     126608   28820     97788  23% /media/pi/RP1
/dev/sda1         7823488      96   7823392   1% /media/pi/RP
root@rp1:~# lsblk -p
NAME              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
/dev/sda            8:0    1  7.5G  0 disk
└─/dev/sda1         8:1    1  7.5G  0 part /media/pi/RP
/dev/mmcblk0      179:0    0 59.5G  0 disk
├─/dev/mmcblk0p1  179:1    0  124M  0 part /media/pi/RP1
├─/dev/mmcblk0p2  179:2    0    1K  0 part
├─/dev/mmcblk0p5  179:5    0   32M  0 part /media/pi/SETTINGS
├─/dev/mmcblk0p6  179:6    0   47M  0 part /media/pi/RISC_OS_Boo
├─/dev/mmcblk0p7  179:7    0  1.9G  0 part
├─/dev/mmcblk0p8  179:8    0   63M  0 part /boot
├─/dev/mmcblk0p9  179:9    0 19.7G  0 part /
├─/dev/mmcblk0p10 179:10   0  160M  0 part /media/pi/E32D-086E
└─/dev/mmcblk0p11 179:11   0 17.2G  0 part
root@rp1:~#

[RonR]

@miho,

Were you booted into a normal Raspbian OS on NOOBS when you ran noobs2img? noobs2img only works for extracting a single Raspbian OS partition on NOOBS to an image file.

There's something strange happening when you run noobs2img:

root@rp1:~# . /home/pi/bin/noobs2img.sh
basename: missing operand

Just renaming noobs2img to noobs2img.sh wouldn't cause that error.

[fbe]

@miho,

Were you booted into a normal Raspbian OS on NOOBS when you ran noobs2img? noobs2img only works for extracting a single Raspbian OS partition on NOOBS to an image file.

There's something string happening when you run noobs2img:

root@rp1:~# . /home/pi/bin/noobs2img.sh
basename: missing operand

Just renaming noobs2img to noobs2img.sh wouldn't cause that error.

This message comes from "sourcing" /home/pi/bin/noobs2img.sh instead of executing it.

Code: Select all

. /home/pi/bin/noobs2img.sh

and

Code: Select all

/home/pi/bin/noobs2img.sh

do different things.

[RonR]

This message comes from "sourcing" /home/pi/bin/noobs2img.sh instead of executing it.

Code: Select all

. /home/pi/bin/noobs2img.sh

and

Code: Select all

/home/pi/bin/noobs2img.sh

do different things.

Good catch! I wasn't seeing the space between the '.' and the rest of the path.

[mihol]

thanks for catching my fault, I'll have to read about the difference using a dot.

Unfortunately even with the corrected syntax I'm getting error 'File too large'.
Please assume that I might miss an 'obvious' action/command

Code: Select all

root@pi4:/home/pi# /home/pi/bin/noobs2img.sh 

Image file to create? /media/pi/D8G/rp4.1.img

/media/pi/D8G/rp4.1.img already exists, Ok to delete (y/n)? y

Image ROOT filesystem size (MB) [7340]? 0

Requested image ROOT filesystem size (0) is too small (Minimum = 3964)

Image ROOT filesystem size (MB) [7340]? 3964

Create /media/pi/D8G/rp4.1.img [3964 MB] (y/n)? y
truncate: failed to truncate '/media/pi/D8G/rp4.1.img' at 4433379328 bytes: File too large

Unable to create image file

#########################################
pi@pi4:~ $ lsblk -p
NAME             MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
/dev/sda           8:0    1  7.5G  0 disk 
└─/dev/sda1        8:1    1  7.5G  0 part /media/pi/D8G
/dev/mmcblk0     179:0    0  7.4G  0 disk 
├─/dev/mmcblk0p1 179:1    0  256M  0 part /boot
└─/dev/mmcblk0p2 179:2    0  7.2G  0 part /

[RonR]

@mihol,

That's simply saying that you don't have enough free space on the device mounted at /media/pi/D8G to create rp4.1.img (even after deleting the existing rp4.1.img). How is the destination device formatted? Keep in mind that FAT32 has a maximum file size of 4 GB, so a ROOT size of 3,964 MB + 256 MB of BOOT is going to exceed 4 GB. Use EXT4 or NTFS (which may require installing ntfs-3g).

[tinker2much]

@RonR - Thank you! I'm happily backing up my many pi's. And incremental backup is very cool.

[tinker2much]

Question - could these scripts deal with "Debian Buster with Raspberry Pi Desktop"? I think the structure on disk may be different?

[RonR]

Question - could these scripts deal with "Debian Buster with Raspberry Pi Desktop"? I think the structure on disk may be different?

These scripts are intended for use with Raspbian format images (two partitions, BOOT and ROOT).

[mihol]

Keep in mind that FAT32 has a maximum file size of 4 GB, so a ROOT size of 3,964 MB + 256 MB of BOOT is going to exceed 4 GB. Use EXT4 or NTFS (which may require installing ntfs-3g).

@RonR
Thanks you nailed the root cause.I failed to consider the fact '+ 256 MB of BOOT'.
Maybe a check could be added to the script to prevent others from hitting the same issue?

[tinker2much]

So, given these clever utilities, what's the best backup workflow for me? I wonder if I'm currently doing it the worst possible way...

I have a dozen Raspberry Pis, of many kinds - 256MB 1B, 2, 3, 3a+, 4, two zeros, five zero Ws. These have a mix of storage - mostly 16GB cards, a couple 32GB, while the 3 USB boots from a 500GB hard drive and the 4 boots from a 32GB card with root on a 120GB SSD (courtesy of RonR's other scripts).(NOTE: I didn't try the 3 and 4 at this time - they have too much data.)

Doing full backups obviously took quite a while, but incremental backups (given rsync) were super fast. I told the backup script to create the data at /mnt/pixxx.img. Maybe the intent was an external drive - but I just tried creating them locally. I'm used to compressed backups, so I then gzipped the images locally, and then scp'd them to my mac, where they will get backed up, along with everything else there, via Time Machine. So at least they're safe.

But this was all very slow - initial creation, gziping, transfer. I tried NOT gziping one of them, to avoid the gziping time, but the transfer therefore took even longer, and I wasn't scientifically recording the times to tell which case was longer overall. Both were annoying. The gziping is CPU and IO intensive, but so is the transfer at least with scp. Maybe straight ftp would be less intensive? The gziping (via the "progress" utility) was often slower than a Megabyte a second, though it spiked up to 10MB/sec at the end. The transfers via scp (as shown by the utility's own progress report) were mostly about 2MB/sec, as low as 1, rarely as high as 4. I ran several at a time, for part of the time.

So what's the bottleneck? The SD cards (mostly SanDisk Ultra+), or the slow processors (one the Zeros and 1B, not so much on the 2 or 3a+), or my slow wireless? Or all of the above...

If I left the images uncompressed, that's the only way to allow fast incremental backups, but they would keep taking up a lot of my small local storage space, and I would have to transfer them off again in full after each incremental.

RonR, were you assuming an external medium - a fast USB stick or external hard drive or even space on the network mounted over NFS or samba? Then the images could be created there, left there uncompressed, facilitating later incremental backups, and we avoid both compression and multiple transfers. Maybe this is a "duh!" - go ahead, I can take it...

[RonR]

RonR, were you assuming an external medium - a fast USB stick or external hard drive or even space on the network mounted over NFS or samba? Then the images could be created there, left there uncompressed, facilitating later incremental backups, and we avoid both compression and multiple transfers.

Yes. I run Raspbian off of fast USB 3 flash drives or SSD's and backup to either fast USB 3 flash drives or a Windows share across the LAN. The initial backup can take some time (especially if the source is a large SSD/HD), but incremental's are pretty quick.

[tinker2much]

I tried again, this time using an external USB 3 hard drive. Not as fast as an SSD, but much better. Thanks, RonR.