RonR
Posts: 2260
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Tue Jun 08, 2021 8:19 pm

npaisnel wrote:
Tue Jun 08, 2021 7:13 pm
Still does not offer a reduced size option..but does this time offer to add extra if a small image size is manually created.

Code: Select all

pi@homevision:~ $ blkid
/dev/mmcblk0p1: SEC_TYPE="msdos" UUID="D2D9-C6B4" TYPE="vfat" PARTUUID="d704444e-01"
/dev/mmcblk0p2: UUID="9b77c019-7802-416f-81e7-267560473405" TYPE="ext4" PARTUUID="d704444e-02"
pi@homevision:~ $ cd /usr/bin
pi@homevision:/usr/bin $ sudo ./image-backup

Image file to create? /media/pi/2f41b0ce-44b2-4f63-8080-862574e2d6fc/210608-RPi2_HV-GUI-HAB.img

Initial image file ROOT filesystem size (MB) [4143]? 1800

Added space for incremental updates after shrinking (MB) [0]? 

Create /media/pi/2f41b0ce-44b2-4f63-8080-862574e2d6fc/210608-RPi2_HV-GUI-HAB.img (y/n)? y

I think you're confused. image-backup does not utilize compression and cannot make the resulting image file smaller than the actual filesystem space used.

It appears you're loading Raspberry Pi OS Desktop onto an 8GB SD card. The used space was approximately 4.134 GB. You cannot stuff 4.134 GB into a 1.8 GB image file. If you try, you will get (this was on an 8GB SD card loaded with Raspberry Pi OS Desktop):

Code: Select all

root@raspberrypi:~# ./image-backup

Image file to create? /mnt/xxx.img

Initial image file ROOT filesystem size (MB) [3562]? 1800

Added space for incremental updates after shrinking (MB) [0]?

Create /mnt/xxx.img (y/n)? y

Starting full backup (for incremental backups, run: ./image-backup /mnt/xxx.img)
rsync: write failed on "/tmp/img-backup-mnt/usr/lib/python2.7/dist-packages/jinja2/visitor.pyc": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]

Unable to create backup

npaisnel
Posts: 229
Joined: Sun Aug 05, 2018 4:23 pm

Re: Image File Utilities

Tue Jun 08, 2021 8:35 pm

RonR wrote:
Tue Jun 08, 2021 8:19 pm


I think you're confused. image-backup does not utilize compression and cannot make the resulting image file smaller than the actual filesystem space used.

It appears you're loading Raspberry Pi OS Desktop onto an 8GB SD card. The used space was approximately 4.134 GB. You cannot stuff 4.134 GB into a 1.8 GB image file. If you try, you will get (this was on an 8GB SD card loaded with Raspberry Pi OS Desktop):

No, I get that

What I was trying to achieve by typing in a too small size was to offer it a smaller size than it needed..and then it would hopefully come back and say t was TOO small and offer a bigger size..

This is what happened the first time this occurred...i'll explain

First time it did this I was imaginng an 8gb card
it had about 2.2gb on it if you used df -h
when it started up ..it offered to create an image size of 7350..or there about.
I told it to make 1800 size.
it came back with an "offer" to make the image file of 2350.

I purposely put a smaller "offer" of a size to create..but this time it did not offer to increase the size it needed. I t just tried and then said the device had run out of space.

All those SD cards now deleted..the cards have now been gparted to no file system and in the morning I will start fresh..installing the lite image on one.of the devices the RPi2, ..sudo apt update-upgrade list upgrade etc etc ..... ..image-backup -samba - then creating and image as a new starting point ..then adding my aps that i want HVXL HA-Bridge noon e.. RTL-SDR RTL-SDR software on on e..etc etc..go back to where i was 2 week sago.

Or the alternative is to take a bloody big hammer tot he lot of them..rip out all the home automation devices in my house...if that bloody Alexa responds to another voice on the TV she is take-in a flying lesson out the bloody window.

As you can telll..not in the best of moods this evening :twisted: :twisted:


thanks for your help


i am off to befd th a large brandy before I do serious physical damage to one of these ruddy machines :D :D :D

npaisnel
Posts: 229
Joined: Sun Aug 05, 2018 4:23 pm

Re: Image File Utilities

Tue Jun 08, 2021 8:38 pm

Or maybe before I go to bed I'll just double check and download AGAIN the image-backup softener ..and backup my OLD original running for 3 years SD cards....

RonR
Posts: 2260
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Tue Jun 08, 2021 9:00 pm

npaisnel wrote:
Tue Jun 08, 2021 8:35 pm
This is what happened the first time this occurred...i'll explain

First time it did this I was imaginng an 8gb card
it had about 2.2gb on it if you used df -h
when it started up ..it offered to create an image size of 7350..or there about.
I told it to make 1800 size.
it came back with an "offer" to make the image file of 2350.

I purposely put a smaller "offer" of a size to create..but this time it did not offer to increase the size it needed. I t just tried and then said the device had run out of space.

That was the behavior of a very old version of image-backup (which also didin't offer an "Added space for incremental updates after shrinking (MB)" prompt). Later versions don't enforce a minmum initial image file size. From the README:

"If no -i/--initial option is included, image-backup will interactively prompt for an "Initial image file ROOT filesystem size (MB)", which can be any size as long as (1) it's large enough to hold the data contained on the device being backed up and (2) that amount of space is available on the destination device."

I would like to be able to present and enforce the exact minimum size required, but rsync'ing from one media to another makes it virtually impossible to do so. Consequently, I left the decision up to the user. Newer versions present a 'safe' minimum size for the default, but if you make it too small, it will simply fail and you will have to try again. It's really not useful to try for a smaller value than the 'safe' default as it will be shrunk to the smallest size possible using resize2fs.

npaisnel
Posts: 229
Joined: Sun Aug 05, 2018 4:23 pm

Re: Image File Utilities

Tue Jun 08, 2021 9:09 pm

well like a dog with a bone..I am still at it..

I had not come across this utility until this post of mine
some one pointed me this way.

so ti is not likely that i had the older version kicking around .
my First pi was 2018 after a motorcycle crash in Prague left me bed bound for 3 months..so I took the opportunity to play with them as SDR receivers. I am only here now hitting this hard because I have been off flying duties with work for a week due a (another !) leg injury

emma1997
Posts: 1551
Joined: Sun Nov 08, 2015 7:00 pm
Location: New England (not that old one)

Re: Image File Utilities

Fri Jun 11, 2021 6:09 pm

Hi Ron,

A brief scan through the threads did not seem to answer my question so let me try here. I'm attempting to get an idea how much 'space' is left when using image-backup. Unless with big add-on installs I typically specify 100mb extra but wonder if there's an easy way to tell how close to the limit.

Image-info seems to do this but I'm not sure. IE it always says 22% boot but root seems to start at about 85% and grows into the 90's. Is this the indicator of how much of that 100mb I'm using and time to start a new image?

RonR
Posts: 2260
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Fri Jun 11, 2021 7:36 pm

emma1997 wrote:
Fri Jun 11, 2021 6:09 pm
Hi Ron,

A brief scan through the threads did not seem to answer my question so let me try here. I'm attempting to get an idea how much 'space' is left when using image-backup. Unless with big add-on installs I typically specify 100mb extra but wonder if there's an easy way to tell how close to the limit.

Image-info seems to do this but I'm not sure. IE it always says 22% boot but root seems to start at about 85% and grows into the 90's. Is this the indicator of how much of that 100mb I'm using and time to start a new image?

image-info should be giving accurate numbers. The image file is mounted as a LOOP device and the space info presented (Used/Available/Use%) is simply the output of a 'df -T' command:

Code: Select all

INFO="$(df -T)"
echo "$(sed -n 's|^\(.*\)\s*Mounted on.*|\1|p' <<< ${INFO})"
echo "$(sed -n "s|^${LOOP}p1\(.*\)${MNTPATH}.*|boot        \1|p" <<< ${INFO})"
echo "$(sed -n "s|^${LOOP}p2\(.*\)${MNTPATH}.*|root        \1|p" <<< ${INFO})"

The 'Use%' value approaching 100% should be interpreted on an image file as you would on a physical storage device (IOW, it's time for concern).

emma1997
Posts: 1551
Joined: Sun Nov 08, 2015 7:00 pm
Location: New England (not that old one)

Re: Image File Utilities

Mon Jun 14, 2021 7:00 pm

Thanks, I was hoping that was the case. Much better than WAG causing many boot failures back in the old days. Skimming through Image-Info code didn't make it obvious to me at my skill level. On the other hand, the Image-Backup script was very educational but did take some time.

BTW does your full name happen by any chance to be Ronald Rsync? lol

User avatar
procount
Posts: 2528
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: Image File Utilities

Fri Jun 18, 2021 1:10 pm

RonR wrote:
Fri Jun 11, 2021 7:36 pm
The image file is mounted as a LOOP device and the space info presented (Used/Available/Use%) is simply the output of a 'df -T' command:
Hey RonR,
as you seem pretty knowledgable on imaging, I wondered if you knew the answer to a related query? I hope it is not too Off topic.
Suppose you have an image file of an SD card, containing several partitions. It is possible to dd the image to an SD card and extract an image of each individual partition by referencing each partition, such as `dd if=/dev/sdd2 ...` etc. The image of each partition will be the same size as the related partition.

Now suppose you LOOP mount each partition from the original image file instead, using the appropriate offset to the start of each partition. Each partition can now be accessed using /dev/loop1 /dev/loop2 or similar. If you then try to extract an image of each separate partition with `dd if=/dev/loop1 ...` etc, the partition images are far too big, up to the size of the original image file. I haven't studied further but I suspect the size is roughly the original image size minus the partition offset (?).

My guess is that DD does not know the size of a partition when it is loop-mounted, and goes off the size of the original image file. Do you know any way to correct this and extract an image of the correct partition size? Google failed in my search.
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

npaisnel
Posts: 229
Joined: Sun Aug 05, 2018 4:23 pm

Re: Image File Utilities

Fri Jun 18, 2021 4:49 pm

Backing up across the network to a smb share?

is that possible ? I keep getting the does not start with /mnt or /media error

I copy the path from the local machine file browser
Copy path.png
Copy path.png (113.75 KiB) Viewed 1047 times


and then ssh to the remote machine and use the path previously copied in the sudo /bin/./image-backup command

Will be nice to be able to back up my remote machines in the attic down to key local machine without having to get the ladder out , crawl in the attic to plug in the USB HDD

result.png
result.png (191.78 KiB) Viewed 1047 times
Screenshot 2021-06-18 at 17.16.14.jpg
Screenshot 2021-06-18 at 17.16.14.jpg (34.91 KiB) Viewed 1047 times

RonR
Posts: 2260
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Fri Jun 18, 2021 5:58 pm

procount wrote:
Fri Jun 18, 2021 1:10 pm
Suppose you have an image file of an SD card, containing several partitions. It is possible to dd the image to an SD card and extract an image of each individual partition by referencing each partition, such as `dd if=/dev/sdd2 ...` etc. The image of each partition will be the same size as the related partition.

Yes
procount wrote:
Fri Jun 18, 2021 1:10 pm
Now suppose you LOOP mount each partition from the original image file instead, using the appropriate offset to the start of each partition. Each partition can now be accessed using /dev/loop1 /dev/loop2 or similar. If you then try to extract an image of each separate partition with `dd if=/dev/loop1 ...` etc, the partition images are far too big, up to the size of the original image file. I haven't studied further but I suspect the size is roughly the original image size minus the partition offset (?).

My guess is that DD does not know the size of a partition when it is loop-mounted, and goes off the size of the original image file. Do you know any way to correct this and extract an image of the correct partition size? Google failed in my search.

If you use the '-P' option of losetup to create a LOOP device of an image file, you don't need to get involved with the offsets and lengths of partitions. The kernel will take care of that and you can simply reference the partitions as /dev/loop0p1, /dev/loop0p2, etc., just as you would a normal device.

RonR
Posts: 2260
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Fri Jun 18, 2021 6:37 pm

npaisnel wrote:
Fri Jun 18, 2021 4:49 pm
Backing up across the network to a smb share?

is that possible ? I keep getting the does not start with /mnt or /media error

image-backup requires that the target device for the image file be mounted somewhere under /media or /mnt/. That's because image-backup excludes /media and /mnt from the backup. This prevents rsync from trying to backup the image file that's being created back into the image file (which would keep going until the target device fills up).

User avatar
procount
Posts: 2528
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: Image File Utilities

Fri Jun 18, 2021 6:41 pm

RonR wrote:
Fri Jun 18, 2021 5:58 pm
If you use the '-P' option of losetup to create a LOOP device of an image file, you don't need to get involved with the offsets and lengths of partitions.
That sounds a bit like Kpartx. I used to use that but it didn't always work on some images, so I defaulted to using losetup the hard way instead. I'll give it a go, thanks.

[ASIDE: BTW, I found out how I might have misquoted you in the past. If you highlight some text in a post, and then accidentally scroll up too far to another person's post and click the quote button, it quotes the text correctly, but attributes it to the wrong poster!]
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

npaisnel
Posts: 229
Joined: Sun Aug 05, 2018 4:23 pm

Re: Image File Utilities

Fri Jun 18, 2021 7:10 pm

RonR wrote:
Fri Jun 18, 2021 6:37 pm
npaisnel wrote:
Fri Jun 18, 2021 4:49 pm
Backing up across the network to a smb share?

is that possible ? I keep getting the does not start with /mnt or /media error

image-backup requires that the target device for the image file be mounted somewhere under /media or /mnt/.



So that brings us to the question if it is possible to mount a network share under /mnt or /media

Just off to bed now
Will look at that tomorrow

RonR
Posts: 2260
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Fri Jun 18, 2021 7:42 pm

npaisnel wrote:
Fri Jun 18, 2021 7:10 pm
So that brings us to the question if it is possible to mount a network share under /mnt or /media

/media or /mnt are the common places to mount external media or network shares. That's why they're excluded from backups.

npaisnel
Posts: 229
Joined: Sun Aug 05, 2018 4:23 pm

Re: Image File Utilities

Sat Jun 19, 2021 6:23 am

RonR wrote:
Fri Jun 18, 2021 7:42 pm

/media or /mnt are the common places to mount external media or network shares. That's why they're excluded from backups.
external media yes, I do that using sudo mount regularly .to mount my Ext USB etc....but not as yet looked at the syntax for doing that to a smb:// share.

other things on the cards today ...working out the syntax to get a variable in an http get string to allow dimming/ brightening of 1990's 2000 era X-10 stuff wtih a similar era control board... few resources easily to hand..might have to come back here for some generic help if I can't get it working..it is running on a Raspberry Pi.....

dshaw619
Posts: 38
Joined: Thu Jan 04, 2018 7:06 am
Location: San Diego, California, USA

Re: Image File Utilities

Sat Jun 19, 2021 6:29 am

RonR wrote:
Fri Jun 18, 2021 7:42 pm
npaisnel wrote:
Fri Jun 18, 2021 7:10 pm
So that brings us to the question if it is possible to mount a network share under /mnt or /media

/media or /mnt are the common places to mount external media or network shares. That's why they're excluded from backups.
@npaisnel - I use image-backup in just such a manner. 2 RPi3b+'s and an RPi0W backup to a Samba share on a hard disk attached to an RPi4B. I mount the share to a subdirectory under /mnt via an entry in /etc/fstab when the RPi's are booted. See @thagrol's excellent guide to fstab at viewtopic.php?f=91&t=302752&p=1815316#p1815316. FYI ... my fstab entry looks like:

Code: Select all

//dutchapple/HD2share  /mnt/HD2  cifs  defaults,credentials=/etc/cifs-user,nofail,_netdev,file_mode=0777,dir_mode=0777  0  0

@RonR - I used Open Media Vault to set up file sharing. When I got to setting up image-backup on the RPi4B file server, I encountered the same "... does not begin with /mnt/ or /media/" error that @npaisnel encountered. I dug into it and found that OMV had set it up so that the HD was mounted under /srv which image-backup was rejecting. I modified the image-backup script to exclude /srv from the rsync command and allow it as a location for the backup file. It seems to be working fine, but are there any pitfalls or reasons I shouldn't do this?

dshaw619
Posts: 38
Joined: Thu Jan 04, 2018 7:06 am
Location: San Diego, California, USA

Re: Image File Utilities

Sat Jun 19, 2021 6:36 am

npaisnel wrote:
Sat Jun 19, 2021 6:23 am
...working out the syntax to get a variable in an http get string to allow dimming/ brightening of 1990's 2000 era X-10 stuff wtih a similar era control board... few resources easily to hand..might have to come back here for some generic help if I can't get it working..it is running on a Raspberry Pi.....
I am controlling my X-10 system from a RPi. Start a new topic if you'd like to discuss.

RonR
Posts: 2260
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sat Jun 19, 2021 6:51 am

dshaw619 wrote:
Sat Jun 19, 2021 6:29 am
@RonR - I used Open Media Vault to set up file sharing. When I got to setting up image-backup on the RPi4B file server, I encountered the same "... does not begin with /mnt/ or /media/" error that @npaisnel encountered. I dug into it and found that OMV had set it up so that the HD was mounted under /srv which image-backup was rejecting. I modified the image-backup script to exclude /srv from the rsync command and allow it as a location for the backup file. It seems to be working fine, but are there any pitfalls or reasons I shouldn't do this?

It sounds like you solved your problem well. As long as the resulting backup image contents look like what you're backing up, I would think you're in good shape.

motomouse
Posts: 14
Joined: Fri Feb 06, 2015 11:36 am

Re: Image File Utilities

Mon Jun 21, 2021 3:36 pm

dshaw619 wrote:
Sat Jun 19, 2021 6:29 am
I modified the image-backup script to exclude /srv from the rsync command and allow it as a location for the backup file. It seems to be working fine, but are there any pitfalls or reasons I shouldn't do this?
What you can also do ( without modifying script, and then running into issues when new version comes ) is :

1. use image-backup with

Code: Select all

--options --exclude-from=/home/pi/exclude.txt option.
2. Keep in exclude.txt list of locations you want to exclude from being backed up. This is what I have in mine : it covers all devices mounted via OMV

Code: Select all

/srv/dev*
3. create a symlink ( if you are backing up a machine with OMV install ) to /media or /mnt from OMV shared folder or mount to /media /mnt folder network location of OMV folder to save backup image there.

Jineu
Posts: 2
Joined: Mon Jul 05, 2021 1:52 pm

Re: Image File Utilities

Fri Jul 09, 2021 6:47 am

Many thanks for making these available. This does exactly what we need on a Pi48GB running 64-bit RPI OS (beta). We copied the unzipped files into /usr/local/bin and then ran chmod +x on them. Mounted a 1TB USB My Passport drive with an 250gb EXT32FAT formatted partition for full clone/image backups of the 240 Kingston SATA drive in the Pi4 M.2 Argon One case.. USB drive automounts /media/pi/<partition name> and gives the target location. Just remember to use the -i flag for the first run. Works quickly and easily.

Jineu
Posts: 2
Joined: Mon Jul 05, 2021 1:52 pm

Re: Image File Utilities

Sun Jul 11, 2021 11:21 am

System - RPI OS 64-bit beta version on RPi48Gb running multiple Docker containers including apache2/php

Problem - The image-backup command would not complete, though there was ample space on the target USB drive partition.

Background - The command had completed previously without error. The new .img file is after installing multiple docker hosts on the RPi48Gb.

Command -

Code: Select all

sudo /usr/local/bin/image-backup -i /media/targetfolderlocation/itsatest.img
Error Message -
sync: write failed on "/tmp/img-backup-mnt/var/lib/docker/overlay2/<container_ID>/merged/usr/src/php.tar.xz": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]
Note: The error messsage pointed to the apache2/php container that was running.

Fix:
1. Stop all running docker containers with the command:

Code: Select all

docker container stop $(docker container list -q)
2. Run image-backup command again (e.g.

Code: Select all

 sudo /usr/local/bin/image-backup -i /media/targetfolderlocation/itsatest.img
)
Result:
image-backup completed as expected and drive reports actual/accurate usage data

RonR
Posts: 2260
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sun Jul 11, 2021 6:48 pm

Jineu wrote:
Sun Jul 11, 2021 11:21 am
Result:
image-backup completed as expected and drive reports actual/accurate usage data

Thanks for the feedback. I'm sure others will benefit from your instructions when Docker (which I have no experience with) is involved.

Dan1jel
Posts: 38
Joined: Wed Aug 08, 2018 6:21 pm
Location: Sweden
Contact: Twitter

Re: Image File Utilities

Tue Jul 13, 2021 8:47 pm

I made a script with dd to do a backup every month but i found out that DD is not good to run when the system is on. So is there any way I could change the start location from /mnt or /media to /server instead? I have my Plex library and everything (3tb of 6tb) in /server drive so this is not working out that great for me atm :/

And i would like to do a cronjob, would it get stuck when image-backup is prompting some questions?

RonR
Posts: 2260
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Jul 14, 2021 2:38 am

Dan1jel wrote:
Tue Jul 13, 2021 8:47 pm
So is there any way I could change the start location from /mnt or /media to /server instead? I have my Plex library and everything (3tb of 6tb) in /server drive so this is not working out that great for me atm :/

Simply make a symbolic link in /mnt or /media that points to /server. For example:

Code: Select all

sudo ln -s /server /mnt/server

Then references to /mnt/server will be equivalent to /server.

You will also need to add an exclusion option for the /server directory or image-backup will try to include it in the backups:

Code: Select all

sudo image-backup --options --exclude-from=/home/pi/exclude.txt

In /home/pi/exclude.txt put:

Code: Select all

/server
Dan1jel wrote:
Tue Jul 13, 2021 8:47 pm

And i would like to do a cronjob, would it get stuck when image-backup is prompting some questions?

image-backup has a number of options to aid scripting (no interactive prompting needed).

See the description of image-backup in the first post of this topic.

Return to “Advanced users”