tinker2much
Posts: 126
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Mon Jan 06, 2020 8:16 pm

@RonR, I have a PI 3b which is booting entirely from a hard drive. There's a lot of data on the hard drive, including approximately 150GB from one specific application which I don't want it to be included in the backups I hope to take with image-backup. It would make the backup just too large, and I have it backed up elsewhere.

i can see the rsync command in the code, and how you exclude certain directories from ALL backups that image-backup takes. I fiddled with excluding MY directory there too, and that was a trivial change, but getting past the check around lines 220 to 229 where the requested size is compared to the minimum size is harder. The computed minimum size still includes the stuff I don't intend to back up.

I suppose that I could separately hack your IRFSMIN calculation, so as to subtract the size of the data I don't intend to back up. Or, I could establish a custom variable for the directory I intend to exclude and, if present, both subtract its size from IRFSMIN and slap it onto the rsync command line as an exclude.

RonR
Posts: 786
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Tue Jan 07, 2020 3:21 am

@tinker2much

Since acquiring larger and larger storage devices, I've become very unhappy with the way image-backup computes a minimum image size. It's reasonable with 16 or 32 GB flash drives, but isn't on a larger SSD. Try as I may, I haven't been able to come up with a formula that comes out satisfactory on both small and large sized devices due to the way ext4 filesystems are constructed (if you happen to be an expert on ext4 innards, I'm open to suggestions :) ).

I finally decided to make a significant change to the dialog that image-backup presents for an initial full-backup (there's no change to incremental backups):

The first prompt is for "Initial image file ROOT filesystem size (MB)" which defaults to what was previously the minimum image size (which is known to be enough on smaller drives, and is likely much much more than is needed on larger drives). Any value can be entered and is not enforced in any way. It's up to you to not pick a value that's too small (df is your friend). If you pick too small a value, you'll ultimately get a 'No space left on device' and have to start over.

The second prompt is for "Added space for incremental updates after shrinking (MB)" which defaults to 0.

After a confirmation prompt to continue, image-backup creates an image file of the size specified by the first prompt and proceeds with a full backup using rsync. image-backup then shrinks the image to it's smallest possible size using resize2fs. If an added space for incremental backups is specified by the second prompt, the image file is then enlarged by that amount using resize2fs.

This probably sounds a bit much, but it only increases the time it takes for the initial full backup by a minute or so. The advantage is you always end up with the smallest possible image file regardless of what you chose for the initial image size.

I've attached a beta test copy of the revamped image-backup. Please give it a try and let me know what you think.
Last edited by RonR on Tue Feb 11, 2020 9:42 am, edited 1 time in total.

tinker2much
Posts: 126
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Wed Jan 08, 2020 12:53 am

@RonR- I gave the beta a try. I did hack in an exclude for the huge directory I wanted to skip. I guesstimated 130GB for that, so I could have reduced my initial size from 161954 to 31954, but I rounded up to 40,000. I guess I could have let it create the whole giant size, and then we'd really see the shrinking, but we'll see.

Here's how it went:

Code: Select all

[email protected]:/mnt/backup $ sudo rm pi3.img 
Press any key to continue...

[email protected]:~/bin $ sudo ./image-backup-beta 

Image file to create? /mnt/backup/pi3.img

Initial image file ROOT filesystem size (MB) [161954]? 40000

Added space for incremental updates after shrinking (MB) [0]? 

Create /mnt/backup/pi3.img (y/n)? y

Starting full backup (for incremental backups, run: ./image-backup-beta /mnt/backup/pi3.img)
At this point while it was backing up, the image file was 40264M.

Something around 15 minutes in, it was here:

Code: Select all

e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0: 165235/2564096 files (0.2% non-contiguous), 3426375/10240000 blocks

resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/loop0 to 3686867 (4k) blocks.
The filesystem on /dev/loop0 is now 3686867 (4k) blocks long.

resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/loop0 to 3673771 (4k) blocks.
The filesystem on /dev/loop0 is now 3673771 (4k) blocks long.

resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/loop0 to 3673747 (4k) blocks.
The filesystem on /dev/loop0 is now 3673747 (4k) blocks long.

e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0: 165235/925696 files (0.3% non-contiguous), 3321521/3673747 blocks
Partition #2 contains a ext4 signature.

It finished, and the resulting image file is now 15324M.

I'll now try one where I DON'T set a lower initial amount, and we'll see how that goes.

tinker2much
Posts: 126
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Wed Jan 08, 2020 12:57 am

@RonR -

why the three calls:

Code: Select all

    
    resize2fs -M "${LOOP2}"
    resize2fs -M "${LOOP2}"
    resize2fs -M "${LOOP2}"

RonR
Posts: 786
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Jan 08, 2020 1:09 am

tinker2much wrote:
Wed Jan 08, 2020 12:57 am
@RonR -

why the three calls:

Code: Select all

    
    resize2fs -M "${LOOP2}"
    resize2fs -M "${LOOP2}"
    resize2fs -M "${LOOP2}"

Just to squeeze everything possible out. :) The third one gets almost nothing, but doesn't take any time, so...

tinker2much
Posts: 126
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Wed Jan 08, 2020 2:34 am

Here's another run, all defaults:

Code: Select all

[email protected]:~/bin $ ./image-backup-beta 

./image-backup-beta must be run as root user

Press any key to continue...

[email protected]:~/bin $ sudo ./image-backup-beta 

Image file to create? /mnt/backup/pi3.img

Initial image file ROOT filesystem size (MB) [161954]? 

Added space for incremental updates after shrinking (MB) [0]? 

Create /mnt/backup/pi3.img (y/n)? y

Starting full backup (for incremental backups, run: ./image-backup-beta /mnt/backup/pi3.img)

e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0: 165235/10371072 files (0.2% non-contiguous), 4116144/41460224 blocks

resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/loop0 to 3945646 (4k) blocks.
The filesystem on /dev/loop0 is now 3945646 (4k) blocks long.

resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/loop0 to 3870511 (4k) blocks.
The filesystem on /dev/loop0 is now 3870511 (4k) blocks long.

resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/loop0 to 3870361 (4k) blocks.
The filesystem on /dev/loop0 is now 3870361 (4k) blocks long.

e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0: 165235/974848 files (0.3% non-contiguous), 3521219/3870361 blocks
Partition #2 contains a ext4 signature.

The image file was initially 162218M, the final size was 16130M, slightly larger than the first run.

tinker2much
Posts: 126
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Wed Jan 08, 2020 2:45 am

correction and caution - i had two windows open, one showing 1024ish units, one 1000ish units.

The final sizes of the two attempts were (in units of 1000) 14615m and 15383m.

tinker2much
Posts: 126
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Wed Jan 08, 2020 2:47 am

correcting the correction:

14615m and 15383m.by the 1024s

15324m and 16130m by the 1000s

RonR
Posts: 786
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Jan 08, 2020 2:55 am

As you've probably figured out, image-backup has no control over the final outcome. resize2fs -M (shrink to minimum) does whatever Linux does. I've found after 3 passes, nothing changes with more passes. The filesystem structures in EXT4 are rather complex and depending on the size of the volume, the number and size of these internal structures can change.

Do you find the new approach to do the initial full backup acceptable?

tinker2much
Posts: 126
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Wed Jan 08, 2020 3:11 am

Third run of the beta code:

Code: Select all

[email protected]:~/bin $ sudo ./image-backup-beta 

Image file to create? /mnt/backup/pi3.img

/mnt/backup/pi3.img already exists, Ok to delete (y/n)? y

Initial image file ROOT filesystem size (MB) [161954]? 

Added space for incremental updates after shrinking (MB) [0]? 2000

Create /mnt/backup/pi3.img (y/n)? y

Starting full backup (for incremental backups, run: ./image-backup-beta /mnt/backup/pi3.img)

e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0: 165236/10371072 files (0.3% non-contiguous), 4116151/41460224 blocks

resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/loop0 to 3945653 (4k) blocks.
The filesystem on /dev/loop0 is now 3945653 (4k) blocks long.

resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/loop0 to 3870513 (4k) blocks.
The filesystem on /dev/loop0 is now 3870513 (4k) blocks long.

resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/loop0 to 3870363 (4k) blocks.
The filesystem on /dev/loop0 is now 3870363 (4k) blocks long.

e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0: 165236/974848 files (0.3% non-contiguous), 3521221/3870363 blocks
Partition #2 contains a ext4 signature.

e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0: 165236/974848 files (0.3% non-contiguous), 3521221/3870363 blocks

resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/loop0 to 4382363 (4k) blocks.
The filesystem on /dev/loop0 is now 4382363 (4k) blocks long.

e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/loop0: 165236/1097728 files (0.3% non-contiguous), 3529958/4382363 blocks

final sizes of the three runs :

by the 1,024s - 14615m, 15383m, 17383m

by the 1,000s (if anyone wants to know) - 15324m, 16130m, 18227m

Thanks, RonR!

tinker2much
Posts: 126
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Wed Jan 08, 2020 2:08 pm

RonR wrote:
Wed Jan 08, 2020 2:55 am
Do you find the new approach to do the initial full backup acceptable?
Your changes work for me, but I wouldn't want to be the only one with a vote.

RonR
Posts: 786
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Jan 08, 2020 5:47 pm

tinker2much wrote:
Wed Jan 08, 2020 2:08 pm
Your changes work for me, but I wouldn't want to be the only one with a vote.

Thanks for all your testing and feedback.

tinker2much
Posts: 126
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Fri Jan 10, 2020 6:24 pm

I thought of providing you with some more testing, by trying to restore any of those images I took of my 3B.

But I don't think I could use etcher directly, with boot on the SD and root on the hard drive. Perhaps I would need to use etcher to write both to the SD, then use your USB booting scripts to copy the SD card's restored root over the hard drive?

RonR
Posts: 786
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Fri Jan 10, 2020 6:40 pm

tinker2much wrote:
Fri Jan 10, 2020 6:24 pm
But I don't think I could use etcher directly, with boot on the SD and root on the hard drive. Perhaps I would need to use etcher to write both to the SD, then use your USB booting scripts to copy the SD card's restored root over the hard drive?

It's true the SSD backup image file will contain the boot partition of the SD card and Etcher will restore that boot partition to the SSD (which never gets accessed). But unless your SD card has gotten clobbered somehow, there's no problem. I make a point of using sdc-boot to boot into the SD card periodically (especially after kernel updates) and doing an apt-get update/upgrade followed by an image-backup (usually incremental) of the SD card just in case something untoward ever happens to it.

tinker2much
Posts: 126
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Fri Jan 10, 2020 7:07 pm

I was typing and submitting faster than I was thinking, and I was confused. Let me try again.

In the case of a currently purely SD card based system, the images taken with your utilities could easily be restored via etcher to that or another SD card, or even to a USB stick or drive for USB booting.

In the case of a purely SSD or Hard Drive based system (like my 3b), I think the images taken with your utilities could easily be restored to those drives (or to an SD card assuming there was space).

In the case of what I'll call a hybrid system - like my Pi4 - set up by your usb booting scripts - where boot is on the SD and root is on a USB drive - I don't think i could restore such a system using Etcher, because it would want to restore both boot and root to one place.

So I would have to use etcher to burn it to one place - say the SD card - then use your other scripts to copy root to the USB device?

There, I think I'm saying what I think I'm thinking.

RonR
Posts: 786
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Fri Jan 10, 2020 7:23 pm

tinker2much wrote:
Fri Jan 10, 2020 7:07 pm
In the case of what I'll call a hybrid system - like my Pi4 - set up by your usb booting scripts - where boot is on the SD and root is on a USB drive - I don't think i could restore such a system using Etcher, because it would want to restore both boot and root to one place.

So I would have to use etcher to burn it to one place - say the SD card - then use your other scripts to copy root to the USB device?

My reply was addressing this case. Simply use Etcher to restore the SSD backup image to the SSD and you should be done.

When you restore an SSD backup image to the SDD, yes, you will be restoring a copy of the SD card's boot partition to the SSD's boot partition. In a hybrid system, the SSD's boot partition is not used (when the RPi 4 gets a native boot capability, the SSD should be ready to take off without an SD card in place).

Unless your SD card has gotten wrecked somehow, the existing SD card shouldn't need anything done to it. As I noted, I keep a separate image backup of my SD card just in case.

User avatar
Botspot
Posts: 477
Joined: Thu Jan 17, 2019 9:47 pm
Location: Texas

Re: Image File Utilities

Tue Jan 14, 2020 1:25 am

RonR wrote:
Sun Jan 05, 2020 12:36 am
These scripts are intended for use with Raspbian format images (two partitions, BOOT and ROOT).
Suggestion for image-mount:
Wouldn't it be better to just have losetup create a block device for every partition in the img file like this:

Code: Select all

LOOP="$(losetup -fP --show $1)"
mount -o rw "${LOOP}p2" "$MNT"
mount -o rw "${LOOP}p1" "$MNT"
Instead of having to calculate the start and end of each partition manually?

Then Raspbian with a user-added third partition can still mount, instead of exiting with a syntax error. BTW I now use this method in Vdesktop.
Ever wanted to run an .img file before flashing it to an SD card?
Or wished you could run Stretch on a Pi 4?
Or wanted to run two versions of Raspbian on a single Pi simultaneously?
You can do all of that, and more, with my Vdesktop script - http://bit.ly/VDESKTOP

RonR
Posts: 786
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Tue Feb 11, 2020 9:38 am

image-utils.zip (in the first post of this topic) has been updated.

image-backup now creates image files that have been shrunk to the smallest size possible:

-----

image-backup creates a backup of a running Raspbian system to a standard 'raw' image file that can be written to an SD card or a USB device with Etcher, imageUSB, etc. It will also perform incremental updates to an existing backup image file.

Running image-backup with no parameters will create a full backup. image-backup will prompt for an "Initial image file ROOT filesystem size (MB)", which can be any size as long as (1) it's large enough to hold the data contained on the device being backed up and (2) that amount of space is available on the destination device. image-backup will also prompt for an "Added space for incremental updates after shrinking (MB)" which will be added to the image file size after the full backup completes and the image file has been shrunk to the smallest size possible. The resulting image file will auto-expand the first time it's run.

Return to “Advanced users”