Backup SD card to .IMG file

[strawberry]

With the piclone program that comes with the standard installation of the Raspberry Pi OS you can create a backup of your SD card to a second SD card while it's active. Now I've modified this program a little bit so it will create backups to an image (.img) file on an external USB drive or shared network drive.

I use this to create backups of SD cards of Raspberry's that are installed somewhere in a box that is not easy accessible. I mount a shared network drive and run the program to transfer entire SD card over Wifi to .img file (takes a few hours though).

If anyone finds this useful too you can download it here:
https://github.com/tom-2015/imgclone

[TommyG007]

I've just installed this and ran a backup job and it works brilliantly!

As you can see from my original thread, I was looking for a way to streamline my backup procedure: viewtopic.php?f=63&t=281858&p=1707213#p1707213

This does everything I was hoping for and more, now I just need to figure out how to automate it so it creates a backup once a month lol.

Thanks very much!

[CaptainMidnight]

If anyone finds this useful too you can download it here:
https://github.com/tom-2015/imgclone

Thanks for sharing, I've currently been using a modified/forked version of rpi-clone among others I've found over time - preferring cli versions, as they can be scripted/scheduled.

I'll take a look at this and add it to my collection, I'll see how it compares ......

[CaptainMidnight]

@strawberry your code works well - nice and quick.

I'm just surprised you didn't add any compression options to the image file output, initial tests when creating an image backup of a 32GB SDcard resulted in a file size of 7.73GB - after running through bzip2, the image file size reduced down to 2.05GB.

I suppose in these days of cheap and plentiful storage it's not a major issue anymore, although it would be a nice option to have built-in.

[Bosse_B]

With the piclone program that comes with the standard installation of the Raspberry Pi OS you can create a backup of your SD card to a second SD card while it's active. Now I've modified this program a little bit so it will create backups to an image (.img) file on an external USB drive or shared network drive.

I use this to create backups of SD cards of Raspberry's that are installed somewhere in a box that is not easy accessible. I mount a shared network drive and run the program to transfer entire SD card over Wifi to .img file (takes a few hours though).

If anyone finds this useful too you can download it here:
https://github.com/tom-2015/imgclone

Seems like this is what I really need!
But a few questions regarding the procedure outlined in the Readme file on github:

Code: Select all

On the raspberry you open a terminal window and type following commands:
- sudo apt-get update
- sudo apt-get install gcc make git
- git clone https://github.com/tom-2015/imgclone.git
- cd imageclone
- make
- sudo make install

Why the two first lines of commands?
On my Pi-OS fresh install everything is up-to-date and gcc, make and git all are available...
I skipped these and just performed the remaining items and it worked just fine.

Concerning usage:
In the Readme.md file is stated that:

Code: Select all

Attach external storage device (USB, Hard disk or mount a Samba share). cd to the drive:
    cd /media/pi/<external drive> Start backup:
    imgclone -d mybackup.img

Is it necessary to have the working dir positioned in the image target dir?
It does not seem so if viewing the command to image to a network drive:

Code: Select all

Make sure you have a NAS or other Samba shared drive in your network, then just mount it:
    mkdir /tmp/backup
    sudo mount -t cifs //<share_drive_ip>/<share_folder_name> /tmp/backup
    imgclone -d /tmp/backup/mybackup.img

Question:
Do I have to run imgclone as sudo or does it switch internally?

I just want to get this right when I build the image to send to the tech on location..

Of course I also need to compress the image file to reduce its file for transmission.
The gzip command could be used for that, I guess....
Finally:
It seems like this will really help me, thank you for posting!

[hippy]

Code: Select all

On the raspberry you open a terminal window and type following commands:
- sudo apt-get update
- sudo apt-get install gcc make git
- git clone https://github.com/tom-2015/imgclone.git
- cd imageclone
- make
- sudo make install

Why the two first lines of commands?
On my Pi-OS fresh install everything is up-to-date and gcc, make and git all are available...
I skipped these and just performed the remaining items and it worked just fine.

It's just for completeness and courtesy to those who haven't updated recently, haven't for whatever reason got the prerequisite items needed.

[nliviu]

If anyone finds this useful too you can download it here:
https://github.com/tom-2015/imgclone

I've created a PR with some fixes.

[Bosse_B]

Concerning the target for the img file:
I have failed to use an NTFS hard drive directly connected by USB as the target for the output img file.
But I have succeeded in using an SDcard in an USB carrier directly inserted into the RPi after reformatting it as ext4.

Now I am facing a problem where the RPi I want to image is on a remote network which I can access via OpenVPN.
On this network I also have an Ubuntu server (the OpenVPN server) available.
So I would like to use the option to create the img file on the Ubuntu file system.

What kind of file share should I create in Ubuntu such that the RPi image file can be deposited there?
My alternatives seem to be NFS or Samba.

[strawberry]

Do I have to run imgclone as sudo or does it switch internally?

yes you need to run as sudo it will not switch internally and you don't need to run inside the directory if you provide full path to the backup destination file.

[strawberry]

Concerning the target for the img file:
I have failed to use an NTFS hard drive directly connected by USB as the target for the output img file.
But I have succeeded in using an SDcard in an USB carrier directly inserted into the RPi after reformatting it as ext4.

Now I am facing a problem where the RPi I want to image is on a remote network which I can access via OpenVPN.
On this network I also have an Ubuntu server (the OpenVPN server) available.
So I would like to use the option to create the img file on the Ubuntu file system.

What kind of file share should I create in Ubuntu such that the RPi image file can be deposited there?
My alternatives seem to be NFS or Samba.

Make sure you can write to the NTFS hard drive! By default NTFS is read-only on the Raspberry.
You just need to have a network drive you can mount and read/write files to from the command line. Samba should work fine.

[Bosse_B]

I used an Ubuntu server I have as the file store.
Just had to install NFS server and then it worked like magic using an nfs share!
Much easier to install and configure than samba....

[DAI-X]

I tried to use this to create an .IMG on a drive attached to another networked Pi but when I enter

sudo mount -t cifs //192.168.0.22/media/[450GB]/ /tmp/backup

it then asks for the ROOT password to the remote machine

"Password for root@192.168.0.22/media/[450GB]/: "

What am I doing wrong here?

[pidd]

Excellent, I've spent a lot of time over the last few weeks trying to get a backup system going, given up all hopes of rsync/incremental backups for use with the whole system. I've also been cursing piclone for being gui.

Performed imgclone over an sshfs mount and it worked fine.

Thank you!

[CaptainMidnight]

@strawberry - both gzip and bzip2 compression options working fine, thank you

[TommyG007]

I tried to use this to create an .IMG on a drive attached to another networked Pi but when I enter

sudo mount -t cifs //192.168.0.22/media/[450GB]/ /tmp/backup

it then asks for the ROOT password to the remote machine

"Password for root@192.168.0.22/media/[450GB]/: "

What am I doing wrong here?

This is the command I use to mount my NAS share to my /home/pi/OMVNAS folder before creating the backup. If your network share requires a password, type that in after password= and delete the double quotes.

Code: Select all

sudo mount.cifs //10.82.107.2/Shared /home/pi/OMVNAS -o user=root,password="" && exit

[jiml8]

I backup with this script:
tar -zcf - / | ssh jiml@192.168.2.33 "cat >/mnt/sda1/raspberrypi.bak.tgz"

This backs up my pi to a location on another computer on the network. Set up SSH keys to avoid password prompts, and this is easily automated, resulting in a compressed copy on the other machine.

[Bosse_B]

I backup with this script:
tar -zcf - / | ssh jiml@192.168.2.33 "cat >/mnt/sda1/raspberrypi.bak.tgz"

This backs up my pi to a location on another computer on the network. Set up SSH keys to avoid password prompts, and this is easily automated, resulting in a compressed copy on the other machine.

Does it preserve the structure of the RPi SDcard?
I.e. does it properly create the boot partition as vfat separate from all the rest, which is ext4?
This is what one of my RPi system looks like:

Code: Select all

$ sudo lsblk -o UUID,NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL,MODEL
UUID                                 NAME        FSTYPE  SIZE MOUNTPOINT LABEL  MODEL
                                     mmcblk0            29.8G
9F2D-0578                            ├─mmcblk0p1 vfat    256M /boot      boot
efc2ea8b-042f-47f5-953e-577d8860de55 └─mmcblk0p2 ext4   29.6G /          rootfs

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        30G   17G   12G  59% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           2.0G   17M  1.9G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   52M  202M  21% /boot
tmpfs           391M     0  391M   0% /run/user/1000

To me it looks like tar is simply copying all of the files recursively from / downwards and this will include the boot partition as just another directory, right?

My experience
On Sunday I have used imgclone successfully on a prototype RPi with the target set on an nfs share in my Ubuntu server.
Then gzipped that img file on Ubuntu and moved it into the webserver documents area.
Finally on the remote location I could use wget to download the img.gz file to the local server there.
This was all done remotely.
Then a tech on location grabbed the file and used Balena Etcher to burn it to an SDcard which started up just fine on an RPi3 over there (in Texas as opposed to Sweden where I live).

I think very highly of imgclone!
Thanks for creating it and posting this forum thread!!

[wildestpixel]

Hi there,

Not adding this as a GH issue, but putting here first instance, post compression seems to work with some commands but not others, backing up from ssd as /dev/sda :

WORKS

Code: Select all

sudo imgclone -s /dev/sda -d /home/pi/mnt/Main/Raspbian/backupimages/argon.img -gzip

Outcome - Completes making .img file and then proceeds to make .gz file succesfully

DOESN'T WORK

Code: Select all

sudo imgclone -s /dev/sda -d /home/pi/mnt/Main/Raspbian/backupimages/$(hostname).backup.$(date +%Y%m%d%H%M).img -gzip

Outcome - Completes making .img file but doesnt attempt to make gz file.

Obviously would be nice to datestamp these in the file name as well as the attributes

Also - if a gzipped image already exists with the filename chosen, would be good to be able to switch so that it force overwrites rather than having to select yes manually at the terminal.

[jiml8]

Does it preserve the structure of the RPi SDcard?
I.e. does it properly create the boot partition as vfat separate from all the rest, which is ext4?

No, it does not. For myself, I don't consider it to be necessary; I only ever use a backup when I want to restore the system or the data. If I'm restoring the structure as well, that usually means that I am installing on a new media, I prefer to format that media appropriately then copy the right stuff into the new partitions. I find it takes very little extra time on initial setup and saves a lot of headaches in adapting to the undoubtedly different size of the new media.

To me it looks like tar is simply copying all of the files recursively from / downwards and this will include the boot partition as just another directory, right?

Correct.

It does have the advantage of being very simple to implement.
My experience

[Bosse_B]

It does have the advantage of being very simple to implement.
My experience

I see the advantage as a backup system since I guess it can be set to do incrementals etc.
That way it is a lot faster too.
But I was looking for a way to duplicate a configured RPi with all the settings and installations etc,
I then transfer the zipped img file from here to Texas where it is loaded onto a new SDcard and they can start it up.

Now running there and I can access it from here via VPN and VNC and do my coding and tests on it like it was sitting in my home office.

[jiml8]

Well, duplicating an installation is certainly different than backing up. What I described here would not be anything close to the best solution if you are seeking to clone systems.

[Bosse_B]

I have a question regarding imgclone:
Is there a way to define the source SDcard for the operation?
I would like to use this utility to create an image file of an RPi SDcard, which is put into an USB SDcard reader plugged into the host RPi.
So I want the clone source to be this USB mounted SDcard rather than the running system SD.

This is how the system looks like with the SDcard in a USB reader:

Code: Select all

$ sudo lsblk -o UUID,NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL,MODEL
UUID                                 NAME        FSTYPE  SIZE MOUNTPOINT LABEL  MODEL
                                     sda                29.8G                   SDDR-339
E5B7-FEA1                            ├─sda1      vfat   41.8M            boot
b4ea8e46-fe87-4ddd-9e94-506c37005ac5 └─sda2      ext4   29.8G
                                     mmcblk0            29.8G
70A2-8001                            ├─mmcblk0p1 vfat    256M /boot      boot
a1fafd2b-1ef0-4fe8-8ac1-ad33bbb48642 └─mmcblk0p2 ext4   29.6G /          rootfs

I want to clone sda onto an image file that is shrunk to the size of actual data.

Can this be done with imgclone?
If so how?

[Eirikur]

This tool worked for me a while ago. I stopped working on my own image shrinker when I discover this.

https://github.com/Drewsif/PiShrink



Straight to the script:
1https://raw.githubusercontent.com/Drewsif/PiShrink/master/pishrink.sh
Just save the page as pishrink.sh

[HvdW]

What about this well written document?

A tip:
When making backups sometimes execution of cron stands in the way or in my case Domiticz is very busy.
Write a bash script like this for example:

Code: Select all

#!/bin/bash

# https://www.raspberrypi.org/documentation/linux/filesystem/backup.md
# command to find attached drives
# sudo lsblk -o UUID,NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL,MODEL
# df -h
# command sh -x rpi-homedir-backup.sh so you can watch what's going on

sudo service cron stop
sudo service domoticz stop
sudo systemctl stop SBFspotUpload

# create snapshot archive of the home folder
cd /home/
sudo tar czf /mnt/backup_1/rpi_backup/$(date +\%Y\%m\%d)_RPI3_homedir_backup.tar.gz hein
cd /home/hein/

# to install as a cronjob: crontab -e
# # m h  dom mon dow   command
#3 1 * * * sudo rpi-homedir-backup.sh

sudo service cron start
sudo service domoticz start
sudo systemctl start SBFspotUpload

BTW
I'm fond of rpi-clone for which I'm using this script:

Code: Select all

#!/bin/bash

# https://github.com/billw2/rpi-clone
# command to find attached drives
# sudo lsblk -o UUID,NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL,MODEL
# df -h
# command sh -x rpi-clone.sh

sudo service cron stop
sudo service domoticz stop
sudo service lighttpd stop
#sudo service pihole-FTL stop
sudo service unifi stop
sudo service unattended-upgrades stop
sudo service unbound stop
sudo service fail2ban stop
sudo systemctl stop SBFspotUpload

# first setup
# -f is force initialise -U is unattended
sudo rpi-clone -f -v sdx

# all following
#sudo rpi-clone -v -u sdx

# to install as a cronjob: crontab -e
# # m h  dom mon dow   command
#6 6 * * 6 sudo rpi-clone -u sdx

sudo service cron start
sudo service domoticz start
sudo service lighttpd start
sudo service unifi start
#sudo service pihole-FTL start
sudo service unattended-upgrades start
sudo service unbound start
sudo service fail2ban start
sudo systemctl start SBFspotUpload

To prevent myself from making mistakes the device is named /dev/sdx instead of sda or sdb
Every time I'm using the script I'm watching carefully which drive is which.

O, almost forgot; rpi-clone takes 8 minutes SD to SD and 4 minutes SD to SSD

[Bosse_B]

It turns out that imgclone has a command line argument setting the source disk too...

Code: Select all

sudo imgclone -s /dev/sda -d /mnt/ext/<path to>/mybackup.img

So the tool I have used successfully before seems to be able to do also this.