P13r0
Posts: 5
Joined: Mon Nov 11, 2019 6:26 pm

Re: Image File Utilities

Wed Nov 13, 2019 7:54 pm

Thank you so much, RonR :)
I'll try asap and let you know the result.

About Jessie... you're right. But in the last 3 years I've installed (and configured...) a lot of applications; so (a) I'm not really sure that everything would work as expected, even in a different environment, and (b) I'm not sure to remember everything I did (...to be honest, I'm sure of the contrary). So, I prefer to wait... a little longer ;)

Thank you again. Il'l reply soon.


Regards

P13r0
Posts: 5
Joined: Mon Nov 11, 2019 6:26 pm

Re: Image File Utilities

Fri Nov 15, 2019 6:38 pm

It worked flawlessly. I'll try the obtained image as soon as I can... but I've no doubt that it will run.

Thank you again, RonR. Really a great job ;)

Regards

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Fri Nov 15, 2019 9:38 pm

P13r0 wrote:
Fri Nov 15, 2019 6:38 pm
It worked flawlessly.

Thanks for the feedback. I'm pleased to hear of your success.
P13r0 wrote:
Fri Nov 15, 2019 6:38 pm
I'll try the obtained image as soon as I can... but I've no doubt that it will run.

Be patient on the first startup. The restored system will be auto-expanded which can take some extra time.

P13r0
Posts: 5
Joined: Mon Nov 11, 2019 6:26 pm

Re: Image File Utilities

Sun Nov 17, 2019 3:09 pm

I'm trying to use the image file to restore a (new) boot card, and I've a question.

Looking inside the image file, it seems to be OK. Both 0.fat (/boot) and 1.img (/) are present and readable; the content seems OK too.
So I've taken a new SD card -same model and size (...?...) of the installed one- and tried to burn the image using Etcher; but it complains about the size of the image, vs the size of the SD card. The image file is too big.
I didn't touch any default during image creation process; in other words, image's size has been automatically determined by image-backup.
On my Raspberry, image-backup suggests a root filesystem size as follows:

Code: Select all

Image ROOT filesystem size (MB) [30483]?
Eventually, the question: I can choose a different root filesystem size, in order to obtain a smaller image; but will it expand correctly on a SD card which is (probably) smaller than the original one...?

Another way to ask the same question: can I use image-backup to change the size of SD card (bigger | smaller than the original one)? Will the "inflate" process set all the free SD room for / ?

Thank you in advance for your answer :)

Regards

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sun Nov 17, 2019 7:28 pm

P13r0 wrote:
Sun Nov 17, 2019 3:09 pm
Another way to ask the same question: can I use image-backup to change the size of SD card (bigger | smaller than the original one)? Will the "inflate" process set all the free SD room for / ?

SD cards of the 'same size' can vary considerably. image-backup allows you to choose the size of the image file you are creating, with the only restriction being that it has to be at least as large the the actual data contained on the source media. Selecting an image file size of '0' will display the minimum size that can be used. That size is acceptable if you don't plan to do incremental updates to that image file. When restored, the image will auto-expand to fill the media on the first boot.

You can also use image-shrink to shrink your existing image file so that it will fit onto smaller media.

P13r0
Posts: 5
Joined: Mon Nov 11, 2019 6:26 pm

Re: Image File Utilities

Mon Nov 18, 2019 8:58 am

Thank you, RonR. I was almost sure, but... asking is better :)

The SD card restored from the new (smaller) image works correctly.

I have a last question: I've noticed that /boot has been resized too

Code: Select all


(original SD)
 [email protected]:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       30722812 1816380  27632856   7% /
devtmpfs          468152       0    468152   0% /dev
tmpfs             472760       0    472760   0% /dev/shm
tmpfs             472760   47708    425052  11% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             472760       0    472760   0% /sys/fs/cgroup
/dev/mmcblk0p1     41322   21382     19940  52% /boot


(restored SD)
[email protected]:~# df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       30445928 1819156  27342264   7% /
devtmpfs          468152       0    468152   0% /dev
tmpfs             472760       0    472760   0% /dev/shm
tmpfs             472760    6312    466448   2% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs             472760       0    472760   0% /sys/fs/cgroup
/dev/mmcblk0p1    261872   21696    240176   9% /boot

...I suppose this is normal, isn't it?

Thank you again for these super-useful tools ;)

Regards

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Mon Nov 18, 2019 9:05 am

P13r0 wrote:
Mon Nov 18, 2019 8:58 am
I have a last question: I've noticed that /boot has been resized too
.
.
.
...I suppose this is normal, isn't it?

That's normal. Buster requires and uses a larger boot partition, so all the image file utilities create a larger boot partition to accommodate it.

dennyfmn
Posts: 33
Joined: Thu Aug 16, 2012 1:36 pm
Location: Minnesota/California USA

Re: Image File Utilities

Tue Nov 19, 2019 9:45 pm

RonR - thanks for the great Image file utilities!

I have a question about cloning a new system from an image created by image-backup. It is simple to use Etcher to write the image to a new device as long as the /boot and /root in the image were originally on the same device, i.e. both on an SD card or both on a USB device.

What if the /boot partition is on an SD card and the /root partition is on a USB device, especially a very large SSD. You can use Etcher to clone the image to a new large SSD (/boot and /root), but how do you create a new (small) SD card with only the small /boot from the (large) .img file?

Thanks!

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Tue Nov 19, 2019 10:25 pm

dennyfmn wrote:
Tue Nov 19, 2019 9:45 pm
What if the /boot partition is on an SD card and the /root partition is on a USB device, especially a very large SSD. You can use Etcher to clone the image to a new large SSD (/boot and /root), but how do you create a new (small) SD card with only the small /boot from the (large) .img file?

image-backup uses fdisk to create an image file consisting of a 256M boot partition and a root partition of the size you specify. When Etcher writes that image file to any device, that device will then have those partition sizes. Any previously existing partitions and data on the device will be overwritten by the contents of the image file. On the first boot, the root partition will be auto-expanded to use all available space. The boot partition will remain unchanged at 256M.

dennyfmn
Posts: 33
Joined: Thu Aug 16, 2012 1:36 pm
Location: Minnesota/California USA

Re: Image File Utilities

Tue Nov 19, 2019 10:39 pm

Right, but the /boot partition needs to be on a new (small) SD card, not on the new SSD. It doesn't hurt anything for the SSD to also have the /boot partition, but the system needs /boot on an SD card to boot. This is a cloning operation to new bare metal, not a restore operation. The question is how to get the /boot partition to a new SD card. Especially if the image file is very large.

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Tue Nov 19, 2019 11:33 pm

dennyfmn wrote:
Tue Nov 19, 2019 10:39 pm
This is a cloning operation to new bare metal, not a restore operation. The question is how to get the /boot partition to a new SD card. Especially if the image file is very large.

I'm assuming the image file was created by image-backup on an up-to-date Raspbian system with an SD card that booted to an attached SSD.

I would suggest the following steps:

1. Use Etcher to write the backup image file to the new SSD.

2. Use Etcher to write a virgin Raspbian image to the new SD card.

3. Plug everything in and boot the SD card.

4. Update the SD card.

5. Use sdc-boot to set the SSD as the boot device.

6. Reboot to the SSD.

If my assumptions about your source and target are incorrect, please describe in detail where you're coming from and trying to get to.

dennyfmn
Posts: 33
Joined: Thu Aug 16, 2012 1:36 pm
Location: Minnesota/California USA

Re: Image File Utilities

Thu Nov 21, 2019 1:25 pm

Hi RonR,

Instead of downloading and using an image from the Pi repository, I modified a copy of image-backup to only backup the /boot partition, calling it image-backup-boot. This way it can be written to a very small SD card and will be in sync with the kernel version of the /root backup created with image-backup.

I also found a couple of small bugs in image-backup while I was working on image-backup-boot. If the image file already exists, you ask if it is OK to remove it, but don't actually rm it. I added rm ${IMGFILE} above break 2. I also had a couple of losetup resource unavailable errors in mkloop1(). Adding sleep 2 just above LOOP1=.... seems to fix that. I'm running on a PI4 with 4GB of ram.

I've attached a ZIP file with image-backup-boot and patched image-backup, please check them out.

These utilities are great! I have been using a very similar image backup strategy originally gleaned from raspbx, customizing it to my local needs, but without the benefit of all the other utilities you have created. I find image-shrink to be particularly useful.

One additional comment: It would be nice if automatically expanding the root filesystem on restore would be optional.

Best regards, and thanks!

Edit 11/23/2019:

To comply with RONR's wishes, I have removed my uploaded zip file from this posting and created a new topic. See https://www.raspberrypi.org/forums/view ... 9&t=257549
Furthermore, after looking at the code in image-backup more carefully, I see how the file remove was intended to work. I have also removed the sleep 2 statement from mkloop1(). The errors I was seeing probably were due to having extra loop() devices left over from failed testing of my new script. Sorry for any confusion.
Last edited by dennyfmn on Sat Nov 23, 2019 1:38 pm, edited 1 time in total.

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Thu Nov 21, 2019 6:57 pm

dennyfmn wrote:
Thu Nov 21, 2019 1:25 pm
Instead of downloading and using an image from the Pi repository, I modified a copy of image-backup to only backup the /boot partition, calling it image-backup-boot. This way it can be written to a very small SD card and will be in sync with the kernel version of the /root backup created with image-backup.

I'm a firm believer that the SD card should remain an intact standalone bootable entity. Your image-backup-boot omits the root partition and renders the SD card unbootable. The image-backup provided in the first post of this topic is perfectly suitable for backing up the SD card which will be in sync with the kernel version of the /root backup. There is no need for a modified version.
dennyfmn wrote:
Thu Nov 21, 2019 1:25 pm
I also found a couple of small bugs in image-backup while I was working on image-backup-boot. If the image file already exists, you ask if it is OK to remove it, but don't actually rm it. I added rm ${IMGFILE} above break 2. I also had a couple of losetup resource unavailable errors in mkloop1(). Adding sleep 2 just above LOOP1=.... seems to fix that. I'm running on a PI4 with 4GB of ram.

Your addition of 'rm ${IMGFILE}' is an inappropriate modification. The image file is intentionally not deleted at that point. By doing so, the existing image file is destroyed even if the user decides to abort and not proceed with the backup. An existing image file is properly deleted just before the actual backup is started.

I have not encountered any need for the addition of 'sleep 2 just above LOOP1='.
dennyfmn wrote:
Thu Nov 21, 2019 1:25 pm
I've attached a ZIP file with image-backup-boot and patched image-backup, please check them out.

I strongly disagree with your changes. You are welcome to modify the scripts for your own use, but I cannot recommend the use of the modified files and would prefer they not be posted in this topic.

dennyfmn
Posts: 33
Joined: Thu Aug 16, 2012 1:36 pm
Location: Minnesota/California USA

Re: Image File Utilities

Sat Nov 23, 2019 1:52 pm

RonR wrote:
I'm a firm believer that the SD card should remain an intact standalone bootable entity. Your image-backup-boot omits the root partition and renders the SD card unbootable. The image-backup provided in the first post of this topic is perfectly suitable for backing up the SD card which will be in sync with the kernel version of the /root backup. There is no need for a modified version.
I happen to like the idea that the boot partition can be written to a very small SD card. Yes, it needs to be paired with its companion /root device, but I do not consider that to be a problem.

RonR wrote:
Your addition of 'rm ${IMGFILE}' is an inappropriate modification. The image file is intentionally not deleted at that point. By doing so, the existing image file is destroyed even if the user decides to abort and not proceed with the backup. An existing image file is properly deleted just before the actual backup is started.

I have not encountered any need for the addition of 'sleep 2 just above LOOP1='.

I strongly disagree with your changes. You are welcome to modify the scripts for your own use, but I cannot recommend the use of the modified files and would prefer they not be posted in this topic.
After looking at your code more carefully, I now see how it works. You're right, the addition of rm ${IMGFILE} was inappropriate. I have also removed the sleep 2 statement; I think I had leftover loop() devices from failed runs of my script as I was working on it. Sorry for the confusion.

I have removed my upload to your topic and edited my posting above. I've created a new topic at
https://www.raspberrypi.org/forums/view ... 9&t=257549 for image-backup/root

Best regards...

jgscott987
Posts: 4
Joined: Sat Nov 30, 2019 5:08 pm

Re: Image File Utilities

Sat Nov 30, 2019 6:09 pm

Thanks so much for this code, RonR!

I've got everything installed and running well, but I can't get my restored image to boot...

I backed up my Pi using image-backup, which completed normally with no errors. I backed up the file to a windows Samba share. I burned the resulting image file, pihole.img to an SD card using Etcher, which completed and verified successfully. Unfortunately the card won’t boot. I’ve tried multiple cards and multiple backup runs from image-backup. When I try to boot the card, the activity light on the Pi zero flashes for a couple of seconds and then just stays lit.

When I check the card from another Pi, it has two partitions that contain the following:

Code: Select all

/media/USBdrive
     bin   dev  home  lost+found  mnt  proc  run   srv  tmp  var
     boot  etc  lib   media       opt  root  sbin  sys  usr

/media/USBdrive1
      bcm2708-rpi-0-w.dtb        config.txt         LICENSE.oracle
      bcm2708-rpi-b.dtb          COPYING.linux      os_config.json
      bcm2708-rpi-b-plus.dtb     fixup_cd.dat       overlays
      bcm2708-rpi-cm.dtb         fixup.dat          start_cd.elf
      bcm2709-rpi-2-b.dtb        fixup_db.dat       start_db.elf
      bcm2710-rpi-3-b.dtb        fixup_x.dat        start.elf
      bcm2710-rpi-3-b-plus.dtb   issue.txt          start_x.elf
      bcm2710-rpi-cm3.dtb        kernel7.img       'System Volume Information'
      bootcode.bin               kernel.img
      cmdline.txt                LICENCE.broadcom
The boot.log on the card shows no errors. It’s about 75 lines and ends with the following:

Code: Select all

[^[[0;32m  OK  ^[[0m] Started LSB: Autogenerate and use a swap file.^M
[^[[0;32m  OK  ^[[0m] Started Permit User Sessions.^M
[^[[0;32m  OK  ^[[0m] Started /etc/rc.local Compatibility.^M
         Starting Hold until boot process finishes up...^M
         Starting Terminate Plymouth Boot Screen...^M

Any thoughts on what might be going on? Thanks so much for your help!

jgscott987
Posts: 4
Joined: Sat Nov 30, 2019 5:08 pm

Re: Image File Utilities

Sat Nov 30, 2019 9:52 pm

Here's some additional information that might be helpful regarding my SD card restored image that won't boot. The following is the output from fdisk -l. The first disk listed is the active system SD card. The second disk listed is the SD card with the image created by image-backup (that won't boot).

Active system disk

Code: Select all

Disk /dev/mmcblk0: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00066bbf

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1         8192  3779296  3771105  1.8G  e W95 FAT16 (LBA)
/dev/mmcblk0p2      3779297 31116287 27336991   13G  5 Extended
/dev/mmcblk0p5      3784704  3850237    65534   32M 83 Linux
/dev/mmcblk0p6      3850240  3991551   141312   69M  c W95 FAT32 (LBA)
/dev/mmcblk0p7      3997696 31116287 27118592   13G 83 Linux

SD Card with restored image from image-backup

Code: Select all

Disk /dev/sda: 29.7 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00066bbf

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sda1         8192   532480   524289  256M  c W95 FAT32 (LBA)
/dev/sda2       540672 27660287 27119616   13G 83 Linux
I notice that the imaged boot partition is much smaller than than the original. I've read that the Pi can hang on boot if the boot partition is completely full...

Thanks again for any thoughts.

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sat Nov 30, 2019 10:39 pm

@jgscott987

Image File Utilities are intended for use with Raspbian images. Image File Utilities are not compatible with NOOBS.

jgscott987
Posts: 4
Joined: Sat Nov 30, 2019 5:08 pm

Re: Image File Utilities

Sat Nov 30, 2019 10:46 pm

This is a straight Raspian Stretch install. What makes you think it is NOOBS?

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sat Nov 30, 2019 10:55 pm

jgscott987 wrote:
Sat Nov 30, 2019 10:46 pm
This is a straight Raspian Stretch install. What makes you think it is NOOBS?

This:
jgscott987 wrote:
Sat Nov 30, 2019 9:52 pm
Active system disk

Code: Select all

Disk /dev/mmcblk0: 14.9 GiB, 15931539456 bytes, 31116288 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00066bbf

Device         Boot   Start      End  Sectors  Size Id Type
/dev/mmcblk0p1         8192  3779296  3771105  1.8G  e W95 FAT16 (LBA)
/dev/mmcblk0p2      3779297 31116287 27336991   13G  5 Extended
/dev/mmcblk0p5      3784704  3850237    65534   32M 83 Linux
/dev/mmcblk0p6      3850240  3991551   141312   69M  c W95 FAT32 (LBA)
/dev/mmcblk0p7      3997696 31116287 27118592   13G 83 Linux

Raspbian only uses two partitions, with the first being boot and the second being root.

NOOBS places the Raspbian boot partition in partition 6 and the Raspbian root partition in partition 7, as you illustrated above.

Am I confused about the above being your SD card that you were trying to backup with image-backup?

jgscott987
Posts: 4
Joined: Sat Nov 30, 2019 5:08 pm

Re: Image File Utilities

Sat Nov 30, 2019 10:59 pm

Interesting. I haven't used NOOBS in years, but I guess I must have for this particular install. Thanks for your help.

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sat Nov 30, 2019 11:07 pm

jgscott987 wrote:
Sat Nov 30, 2019 10:59 pm
Interesting. I haven't used NOOBS in years, but I guess I must have for this particular install. Thanks for your help.

Time for a fresh install of pihole on vanilla Buster? :idea:

dsyleixa123
Posts: 580
Joined: Mon Jun 11, 2018 11:22 am

Re: Image File Utilities

Mon Dec 16, 2019 1:32 pm

will the image-utils also work for multiple partitions like e.g. Noobs ?

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Mon Dec 16, 2019 1:48 pm

dsyleixa123 wrote:
Mon Dec 16, 2019 1:32 pm
will the image-utils also work for multiple partitions like e.g. Noobs ?

Image File Utilities are intended for use with Raspbian images. Image File Utilities are not compatible with NOOBS.

dsyleixa123
Posts: 580
Joined: Mon Jun 11, 2018 11:22 am

Re: Image File Utilities

Mon Dec 16, 2019 2:35 pm

how to make images of entire multiple partion SDs not having the GUI SD copy tool and also fitting for slightly different sized SDs (e.g. 16 GB vs 15.9 GB)?
And which also will work with Stretch and possibly Jessie?

edit:
I forgot to mention that I am using a Pi 2 v1.1, for non-GUI Stretch and Jessie and retropie (non-GUI Stretch).
Last edited by dsyleixa123 on Wed Dec 18, 2019 9:23 am, edited 2 times in total.

RonR
Posts: 787
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Tue Dec 17, 2019 6:42 am

dsyleixa123 wrote:
Mon Dec 16, 2019 2:35 pm
how to make images of entire multiple partion SDs not having the GUI SD copy tool and also fitting for slightly different sized SDs (e.g. 16 GB vs 15.9 GB)?
And which also will work with Stretch and possibly Jessie?

I'm not aware of a 'quick-and-dirty' way to image a NOOBS SD card such that it can be restored to a smaller SD card.

If the only OS on NOOBS is Raspbian and/or you're willing to copy only the Raspbian portion of NOOBS to another (possibly smaller) SD card, usb-boot can be used:

1. Remove all USB devices

2. Boot the SD card you want to copy to another (possibly smaller) SD card

3. Run "sudo cat /boot/cmdline.txt" and make a note of the "root=" value

4. Insert the new (possibly smaller) SD card in a USB adapter and plug it into a Raspberry Pi USB port

5. Run "sudo ./usb-boot"

6. If prompted "Use SD card to boot the USB device (recommended)?", select "Yes"

7. Select "/dev/sda" and "Replicate BOOT/ROOT contents from /dev/mmcblk0 to /dev/sda"

8. When usb-boot completes, !!! DO NOT REBOOT !!!

9. Run "sudo nano /boot/cmdline.txt" and restore the original "root=" value

10. Run "sudo mount /dev/sda2 /mnt/"

11. Run "sudo nano /mnt/etc/fstab" and change the /boot entry to /dev/mmcblk0p1

12. Run "sudo halt" to shut down the Raspberry Pi

13. Remove power from the Raspberry Pi

14. Remove the USB adapter from the Raspberry Pi USB port

15. Remove the SD card from the Raspberry Pi SD card slot and set it aside

16. Insert the new SD card from the USB adapter in the Raspberry Pi SD card slot and boot it

17. Your original NOOBS Raspbian should now be running as a normal standalone Raspbian

I just tested this procedure and it worked as expected.

Return to “Advanced users”