Dan1jel
Posts: 38
Joined: Wed Aug 08, 2018 6:21 pm
Location: Sweden
Contact: Twitter

Re: Image File Utilities

Wed Jul 14, 2021 9:08 am

RonR wrote:
Wed Jul 14, 2021 2:38 am
Dan1jel wrote:
Tue Jul 13, 2021 8:47 pm
So is there any way I could change the start location from /mnt or /media to /server instead? I have my Plex library and everything (3tb of 6tb) in /server drive so this is not working out that great for me atm :/

Simply make a symbolic link in /mnt or /media that points to /server. For example:

Code: Select all

sudo ln -s /server /mnt/server

Then references to /mnt/server will be equivalent to /server.

You will also need to add an exclusion option for the /server directory or image-backup will try to include it in the backups:

Code: Select all

sudo image-backup --options --exclude-from=/home/pi/exclude.txt

In /home/pi/exclude.txt put:

Code: Select all

/server
Dan1jel wrote:
Tue Jul 13, 2021 8:47 pm

And i would like to do a cronjob, would it get stuck when image-backup is prompting some questions?

image-backup has a number of options to aid scripting (no interactive prompting needed).

See the description of image-backup in the first post of this topic.
Thank you for the clarification and helping me get a hold of this, i tried a simple backup script and it work, now my main backup script is ready, if curious, see my first post about my "issue" here - viewtopic.php?t=315698

Edit: is there a page or GitHub where we could get t any updates of image-utils or is it just by zip file when available? Was thinking if apt maybe could have it, we get the updated version when "apt update" is done?

RonR
Posts: 2388
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Jul 14, 2021 6:57 pm

Dan1jel wrote:
Wed Jul 14, 2021 9:08 am
is there a page or GitHub where we could get t any updates of image-utils or is it just by zip file when available?

I update the ZIP file attached to the first post of this topic whenever any problems are found or enhancements are made. Fortunately, the scripts are pretty mature at this point and haven't required any changes in a while.

Dan1jel
Posts: 38
Joined: Wed Aug 08, 2018 6:21 pm
Location: Sweden
Contact: Twitter

Re: Image File Utilities

Wed Jul 14, 2021 9:36 pm

RonR wrote:
Wed Jul 14, 2021 6:57 pm
Dan1jel wrote:
Wed Jul 14, 2021 9:08 am
is there a page or GitHub where we could get t any updates of image-utils or is it just by zip file when available?

I update the ZIP file attached to the first post of this topic whenever any problems are found or enhancements are made. Fortunately, the scripts are pretty mature at this point and haven't required any changes in a while.
Ah i see, then i just need to keep eye on updates from this post then :) thank you for this great app btw and keep up the good work!

pidd
Posts: 2518
Joined: Fri May 29, 2020 8:29 pm
Location: Wirral, UK
Contact: Website

Re: Image File Utilities

Wed Jul 14, 2021 10:01 pm

One minor change I would suggest to image-backup is that it excludes /var/swap by default, the swap file has no purpose in the image file and it increases both the initial and incremental backup times.

RonR
Posts: 2388
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Jul 14, 2021 10:43 pm

pidd wrote:
Wed Jul 14, 2021 10:01 pm
One minor change I would suggest to image-backup is that it excludes /var/swap by default, the swap file has no purpose in the image file and it increases both the initial and incremental backup times.

That's a good idea. I've added that to my TODO list. Thanks!

In the meantime, you can exclude it manually with an rsync option on the command line:

Code: Select all

image-backup --options --exclude=/var/swap ...

Kitulous
Posts: 2
Joined: Sat Jul 17, 2021 12:47 pm

Re: Image File Utilities

Sun Jul 18, 2021 5:11 pm

Hi, sorry if it's written somewhere in this chat already, but I'm new to Raspberry Pi backing up (or backupping, idk), and I have a couple of questions

1. I have a 32 GB MicroSD card that the Pi is running off of. Is the command

Code: Select all

sudo ./image-backup --initial {{FILENAME}},32000,0
sufficient? What exactly do these two parameters after the name mean? I suppose the first one is the target ISO size (and if it's smaller than the used space on an SD card it will fail)

I want to point out that I will not be using the incremental updates of image-backup.

2. Can I gzip-compress the backup right when it is being done, not afterwards?

npaisnel
Posts: 230
Joined: Sun Aug 05, 2018 4:23 pm

Re: Image File Utilities

Sun Jul 18, 2021 5:55 pm

Just sudo ./ image-backup will work too, and it goes interactive and asks you file name to save as/ size/extra space etc

I never use the options , I just do as I posted above

RonR
Posts: 2388
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sun Jul 18, 2021 7:57 pm

Kitulous wrote:
Sun Jul 18, 2021 5:11 pm
Is the command

Code: Select all

sudo ./image-backup --initial {{FILENAME}},32000,0
sufficient?

Actually,

Code: Select all

sudo ./image-backup --initial IMAGE_FILENAME

should be sufficient for non-interactive mode.
Kitulous wrote:
Sun Jul 18, 2021 5:11 pm
What exactly do these two parameters after the name mean?

image-backup usage is described in the first post of this topic. If you need further clarification about some item, let me know.
Kitulous wrote:
Sun Jul 18, 2021 5:11 pm
Can I gzip-compress the backup right when it is being done, not afterwards?

No, gzip-compress will need to be a separate pass.

typxxi
Posts: 58
Joined: Sat Mar 05, 2016 6:11 pm
Location: GERMANY

Re: Image File Utilities

Wed Jul 21, 2021 8:47 pm

I had tried to shrink an image I had created before using dd but I got an error while shrinking

Code: Select all

sudo ./image-shrink /media/pi/Backup-20210505-200000.img 100

Shrink /media/pi/Backup-20210505-200000.img [Additional MB = 100] (y/n)? y

e2fsck 1.44.5 (15-Dec-2018)
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 3772 has zero dtime.  Fix? no

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  +342212 +570128 -570129 +570271 -570272 -1191013
Fix? no

Free blocks count wrong for group #17 (1343, counted=1345).
Fix? no

Free blocks count wrong for group #36 (5660, counted=5657).
Fix? no

Free blocks count wrong (173062, counted=315868).
Fix? no

Inode bitmap differences:  -3772
Fix? no

Free inodes count wrong for group #16 (49, counted=48).
Fix? no

Free inodes count wrong (223495, counted=223411).
Fix? no


rootfs: ********** WARNING: Filesystem still has errors **********

rootfs: 220025/443520 files (0.4% non-contiguous), 1651706/1824768 blocks

Filesystem appears corrupted before resize2fs
I do not get it why it should be corrupted if it is working quite well if I use an imager to create a new sd card from it.

I need a solution to reduce my current 32 GB SD card partition to a 15 GB partition to get that onto a 16 GB SD card as target.
My old 8 GB one had died and my spare ones where all a few bytes smaller so I had to use the smallest other I had which was bigger than 8 GB and ended up using a a 32 GB with just 6 GB being used. Now I want to go for a 16 GB card using just 15 GB partition to avoid any hazzles of having a different just few bytes to small spare SD card.

And looking for such a solution I came across this solution which unfortunatelly does not work.
WOLF

RonR
Posts: 2388
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Thu Jul 22, 2021 3:41 am

typxxi wrote:
Wed Jul 21, 2021 8:47 pm
I had tried to shrink an image I had created before using dd but I got an error while shrinking
...
I do not get it why it should be corrupted if it is working quite well if I use an imager to create a new sd card from it.

Was the image file created by using dd to copy a running system? If so, that is likely the cause of the corrupted image file.

I just created an image file of a 32GB SD card that was offline (not mounted) using dd and image-shrink worked perfectly on it.

image-shrink runs fsck on an image file before it does anything else to ensure it's not corrupted. Proceeding with shrinking a corrupted image file would most likely only make the corruption worse.

The bottom line is: image-shrink isn't the problem.
Last edited by RonR on Thu Jul 22, 2021 5:01 am, edited 2 times in total.

RonR
Posts: 2388
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Thu Jul 22, 2021 4:57 am

.

RonR
Posts: 2388
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sun Jul 25, 2021 11:34 pm

image-utils.zip (in the first post of this topic) has been updated.

This update adds image-chroot.

image-chroot performs a linux 'chroot' to an image file. The current user will be 'root' and the current directory will be '/' in the image file. The host's root filesystem will be available at /host-root-fs. Use exit or ^D to terminate chroot. Usage is:

image-chroot [imagefile]

This utility allows you to do things things such as updating an image file's Raspberry Pi OS (use image-shrink before and after to add and remove additional space to the image file for the update to take place in):

Code: Select all

pi@raspberrypi:~# sudo image-shrink 2021-05-07-raspios-buster-armhf.img 1000

Shrink 2021-05-07-raspios-buster-armhf.img [Additional MB = 1000] (y/n)? y
...

pi@raspberrypi:~# sudo image-chroot 2021-05-07-raspios-buster-armhf.img

chroot to 2021-05-07-raspios-buster-armhf.img (y/n)? y

Attempting to chroot to 2021-05-07-raspios-buster-armhf.img
Host root filesystem is available at /host-root-fs
Use exit or ^D to terminate chroot

root@raspberrypi:/# apt -y update && apt -y full-upgrade
Hit:1 http://raspbian.raspberrypi.org/raspbian buster InRelease
Hit:2 http://archive.raspberrypi.org/debian buster InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@raspberrypi:/# exit
exit

Exited chroot to 2021-05-07-raspios-buster-armhf.img

pi@raspberrypi:~# sudo image-shrink 2021-05-07-raspios-buster-armhf.img 0

Shrink 2021-05-07-raspios-buster-armhf.img [Additional MB = 0] (y/n)? y
...

pi@raspberrypi:~#

jccat
Posts: 16
Joined: Tue Aug 03, 2021 6:13 am

Re: Image File Utilities

Wed Aug 04, 2021 2:42 am

Hi I just found this RE:Image File Utilities I'm using Linux mint and was following a how to but ran into trouble with fdisk not showing data that is required ?

the post sun jul 25 image-utils.zip
I tried using the bottom sudo image-shrink (name of File my case 5aug2021.img2) and it spat the dummy
I see that it is being done on the pi so that is most likely why

Had hoped it would work on Mint

do you know any thing about fdisk

http://www.aoakley.com/articles/2015-10 ... images.php

Lets look at these .img
sudo fdisk -l imagename.img
This should show something like:
Device Boot Start End Blocks Id System
imagename.img1 8192 122879 57344 c W95 FAT32 (LBA)
imagename.img2 122880 15415295 7646208 83 Linux

Thanks jccat

AleXSR700
Posts: 56
Joined: Mon Jun 22, 2020 6:13 am

Re: Image File Utilities

Mon Aug 16, 2021 5:47 am

Good morning,
I wanted to use image-backup after freshly setting up my crashed RasPi but I always get an error.

Code: Select all

Starting full backup (for incremental backups, run: ./image-backup /mnt/ssh_192_168_0_102/mnt/sdc1/Backup/RaspberryPi/02_Backups/backup_15-08-2021.img)
file has vanished: "/var/cache/cups/Samsung_C3_Series_SEC897_.data"
rsync: write failed on "/tmp/img-backup-mnt/var/lib/docker/overlay2/bf5b93e993050812e32639cc052644ed81c94e921fdd3ff1e57dcaa35dcffa1d/diff/usr/local/lib/python3.9/site-packages/pycountry/locales/de/LC_MESSAGES/iso3166.mo": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]
I am not sure what to make of this. The target HDD on my NAS has moooore than enough disc space. Is this a problem of the internal tmp folder filling up or something?

I am running Raspberry OS from the SD card with no USB drive attached. Maybe necessary to use the target NAS also as tmp folder target?

Code: Select all

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       30356612 7074176  21994112  25% /
devtmpfs         3834532       0   3834532   0% /dev
tmpfs            4000260       0   4000260   0% /dev/shm
tmpfs            4000260   25780   3974480   1% /run
tmpfs               5120       4      5116   1% /run/lock
tmpfs            4000260       0   4000260   0% /sys/fs/cgroup
/dev/mmcblk0p1    261868   30888    230980  12% /boot
tmpfs             800052       8    800044   1% /run/user/1000
overlay         30356612 7074176  21994112  25% /var/lib/docker/overlay2/344479e441beae73c9507c5dc8b056c18488e870e026d43e72dfc4b4160f6b9d/merged
overlay         30356612 7074176  21994112  25% /var/lib/docker/overlay2/0122b3ccf5fbabe02b2c880e9ece224fa7638c93087a8cff66e6a289d8e197d5/merged
overlay         30356612 7074176  21994112  25% /var/lib/docker/overlay2/c366804508f3e925d045ddef14f1619de7688c33ef42ddb1d647ae8355663498/merged
overlay         30356612 7074176  21994112  25% /var/lib/docker/overlay2/f6f984e23d3480048a7cbba2353956eecc3f78a7269e6324a773eac3c0c972b5/merged
overlay         30356612 7074176  21994112  25% /var/lib/docker/overlay2/b1a64fd19db0cd2202b85e60d41b3b1d9277fc43789830175044d93cb651fe6f/merged
overlay         30356612 7074176  21994112  25% /var/lib/docker/overlay2/3fbece06cb1303c7b799b36223d4617cc88f70292fb25fe74876037f711ce012/merged
overlay         30356612 7074176  21994112  25% /var/lib/docker/overlay2/7c19b8befddec74df98da91da54a84dc8dc6bc7e5b357335d9e83e7f99cb0286/merged
overlay         30356612 7074176  21994112  25% /var/lib/docker/overlay2/0388667a6429d60bfa6e25129f4671ec02cc60db0e8e498d5bf1fcbb3dd21e08/merged
overlay         30356612 7074176  21994112  25% /var/lib/docker/overlay2/e8940e498e027698007217daa427fa83e510342038c913f887c72fcd9b5da334/merged

motomouse
Posts: 15
Joined: Fri Feb 06, 2015 11:36 am

Re: Image File Utilities

Tue Aug 17, 2021 10:05 am

AleXSR700 wrote:
Mon Aug 16, 2021 5:47 am
rsync: write failed on "/tmp/img-backup-mnt/var/lib/docker/overlay2/bf5b93e993050812e32639cc052644ed81c94e921fdd3ff1e57dcaa35dcffa1d/diff/usr/local/lib/python3.9/site-packages/pycountry/locales/de/LC_MESSAGES/iso3166.mo": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]
[/code]
Stop all docker containers before running image-backup.

w4d3
Posts: 1
Joined: Mon Aug 23, 2021 6:44 am

Re: Image File Utilities

Mon Aug 23, 2021 6:47 am

RonR wrote:
Sun Jul 25, 2021 11:34 pm
image-utils.zip (in the first post of this topic) has been updated.
Thank you for the update, your work is appreciated.

MaddoxWO
Posts: 1
Joined: Mon Aug 23, 2021 12:24 pm

Re: Image File Utilities

Mon Aug 23, 2021 12:34 pm

Has anyone tried these utilities on Ubuntu?

RonR
Posts: 2388
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Tue Aug 24, 2021 5:33 pm

MaddoxWO wrote:
Mon Aug 23, 2021 12:34 pm
Has anyone tried these utilities on Ubuntu?

These scripts are intended for use on Raspberry Pi OS. Ubuntu does several things differently that make them incompatible.

cope
Posts: 7
Joined: Sun Nov 29, 2020 4:48 pm

Re: Image File Utilities

Mon Aug 30, 2021 2:19 pm

RonR wrote:
Wed Jul 14, 2021 10:43 pm
pidd wrote:
Wed Jul 14, 2021 10:01 pm
One minor change I would suggest to image-backup is that it excludes /var/swap by default, the swap file has no purpose in the image file and it increases both the initial and incremental backup times.

That's a good idea. I've added that to my TODO list. Thanks!

In the meantime, you can exclude it manually with an rsync option on the command line:

Code: Select all

image-backup --options --exclude=/var/swap ...
I want to go on with image-backup and exclude the swap file.
First I make a full backup with the exclude option. In the crontab I do a incremental backup every day of this image file. I wonder, do I have to repeat the exclude option in the crontab command?
-- Cor --

RonR
Posts: 2388
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Mon Aug 30, 2021 5:38 pm

cope wrote:
Mon Aug 30, 2021 2:19 pm
I want to go on with image-backup and exclude the swap file.
First I make a full backup with the exclude option. In the crontab I do a incremental backup every day of this image file. I wonder, do I have to repeat the exclude option in the crontab command?

The currently posted version of image-backup has been updated to exclude /var/swap by default.

Any options you wish to use must be included each time image-backup is run (they are not remembered in the image file).

cope
Posts: 7
Joined: Sun Nov 29, 2020 4:48 pm

Re: Image File Utilities

Tue Aug 31, 2021 12:10 pm

Great, thanks for the information!
-- Cor --

guruabyss
Posts: 72
Joined: Wed Sep 25, 2019 7:54 pm

Re: Image File Utilities

Thu Sep 09, 2021 3:08 am

Why would I get this error when trying to do a backup?
Failed to add #2 partition: Invalid argument
mkfs.vfat: unable to open /dev/Ioop0p1: No such file or directory
Unable ta create image BOOT filesystem

RonR
Posts: 2388
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Thu Sep 09, 2021 4:28 am

guruabyss wrote:
Thu Sep 09, 2021 3:08 am
Why would I get this error when trying to do a backup?
Failed to add #2 partition: Invalid argument
mkfs.vfat: unable to open /dev/Ioop0p1: No such file or directory
Unable ta create image BOOT filesystem

I can't begin to even speculate without detailed information about the system you're trying to backup as well as the destination of the backup.

guruabyss
Posts: 72
Joined: Wed Sep 25, 2019 7:54 pm

Re: Image File Utilities

Thu Sep 09, 2021 1:48 pm

I can't begin to even speculate without detailed information about the system you're trying to backup as well as the destination of the backup.
- Raspberry pi OS
- Backing it up to /media/pi/usb/image.img
- I put image-backup in home/pi


During the process it asks the following:

- Image file to create? which i put /media/pi/usb/image.img
- Initial image file ROOT filesystem size (MB) [4583]? I said "y"
- Added space for incremental updates after shrinking (MB) [0]? I said "n"
- Create /media/pi/usb/image.img (y/n)? I said "y"

RonR
Posts: 2388
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Thu Sep 09, 2021 5:14 pm

guruabyss wrote:
Thu Sep 09, 2021 1:48 pm
During the process it asks the following:

- Initial image file ROOT filesystem size (MB) [4583]? I said "y"
- Added space for incremental updates after shrinking (MB) [0]? I said "n"

The answers to these questions must be a numeric value or simply left blank to accept the default value.

Return to “Advanced users”