fmarzocca
Posts: 26
Joined: Fri May 25, 2018 12:42 pm

Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 9:31 am

I have seen many posts and "howto" concerning backing up Raspi's SD from remote, with the use of dd on the main mmcbl0 device (which has 2 partitions inside).

As an old time LInux user, I know how unreliable is it to backup mounted and running partitions: services, /sys, /proc, etc... I mean, you CAN backup mounted and running partitions, but what will be unreliable it is restoring that backup.


I am using my Raspberries as components of larger projects, all enclosed in large cases with other components, and opening the box to remove the SD is a real PITA.

Thoughts on how to reliable backup a Raspi mounted in a larger enclosure from remote?

Thanks

User avatar
DarkPlatinum
Posts: 842
Joined: Thu Nov 02, 2017 2:30 pm
Location: Unknown
Contact: Website YouTube

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 10:23 am

You probably would either have to code, or use software, or manually VNC into the Pi and backup to a SMB share.
1 * Raspberry Pi Zero W, 1 * Raspberry Pi 2, 1 * Raspberry Pi 3 1 * Raspberry Pi 3B + :mrgreen:

Check Out My Raspberry Site (Run on a Raspberry Pi 3B :) ): https://html.dynu.net

fmarzocca
Posts: 26
Joined: Fri May 25, 2018 12:42 pm

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 10:39 am

DarkPlatinum wrote:
Thu Mar 14, 2019 10:23 am
You probably would either have to code, or use software, or manually VNC into the Pi and backup to a SMB share.
I have ssh accesso to the Raspi, that's not the point.

User avatar
B.Goode
Posts: 8227
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 11:00 am

Thoughts on how to reliable backup a Raspi mounted in a larger enclosure from remote?


Recent releases of the Raspberry Pi Foundation Raspbian Stretch Operating System come with an SDCardCopier utility accessible from the Raspberry Pi Desktop main menu system.

I understand your concern based on conventional wisdom about backing up active(mounted) filesystems, and I can't comment on how SDCardCopier deals with this for reliability.

So it might be worth investigating, but like all backup solutions it needs assessment to ensure it meets your needs.

User avatar
procount
Posts: 1689
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 11:18 am

If you had installed your OS via PINN, then you could have made use of its backup facilities.
PINN provides a small linux recovery shell which can run independently of your OS.
It includes SSH and VNC support to allow remote administration of your OS.

It includes a port of Raspbian's SD Card Copier utility, so you can use it to make a complete clone of your SD card. The difference is that it is cloning your OS whilst it is offline, if you are worried about that aspect.

Alternatively, it can backup your OS partitions to a USB disk as compressed tar files. These can be transferred off to a remote location using scp or something afterwards. I haven't considered remote file systems up to now, but it may be possible to mount an external drive using sshfs or I could possibly add nfs if there is interest in that and it doesn't require too many resources.

As you probably haven't installed your OS via NOOBS or PINN, it would still be possible to copy your OS to another SD card with PINN installed on it, in order to make use of these administration features.

Although PINN is primarily an os-installer/boot-manager, you can also hide PINN so you don't even know it is there in normal use. When you want to perform a remote backup, you could enable it, do the backup and then hide it again. It only takes up about 60MB of your SD card.
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

User avatar
rpdom
Posts: 14984
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 11:18 am

fmarzocca wrote:
Thu Mar 14, 2019 9:31 am
I have seen many posts and "howto" concerning backing up Raspi's SD from remote, with the use of dd on the main mmcbl0 device (which has 2 partitions inside).

As an old time LInux user, I know how unreliable is it to backup mounted and running partitions: services, /sys, /proc, etc... I mean, you CAN backup mounted and running partitions, but what will be unreliable it is restoring that backup.
There's no point in backing up /sys, /proc, /dev etc as they are virtual or RAM based and created dynamically by the system.

I'd prefer to shutdown a system to at least single user mode, or better still, boot to ramdisk, in order to do a full backup. However that is not always possible.

The supplied desktop SD Card Copier does a good job. I believe it uses rsync to copy the files (could be wrong, it might just do a cp -r type thing). It also has a command line equivalent called piclone. Both of those require you to put a spare SD card in a USB adaptor in the Pi that is being backed up.

I'm a bit old school Unix. I use a couple of methods on my Pis. I first tar up the contents of /boot and save to a file in the main file system. Then I use the old "dump" command to dump that file system to a remote file. I do this weekly. On the first Sunday of the month I do a full (level 0) backup, which takes quite a while for some of my systems, then on the other Sundays I do an incremental (level 1) backup, which backs up just the changes since the level 0 backup. I've used this method for years across various Unix and Linux systems and have had to (successfully) rely on it a few times for restores of full systems (hard disk/SD Card fails) or individual files (finger troubles).

fmarzocca
Posts: 26
Joined: Fri May 25, 2018 12:42 pm

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 11:19 am

@B.Goode
Thanks for your reply. I don’t know how SDCardCopier works as I do not have a GUI, but I have investigated other cli tools and all of then make a brutal dd copy of the whole SD, so I believe the GUI tool acts the same way.

fmarzocca
Posts: 26
Joined: Fri May 25, 2018 12:42 pm

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 11:22 am

procount wrote:
Thu Mar 14, 2019 11:18 am


Although PINN is primarily an os-installer/boot-manager, you can also hide PINN so you don't even know it is there in normal use. When you want to perform a remote backup, you could enable it, do the backup and then hide it again. It only takes up about 60MB of your SD card.
I will investigate this PINN. It sounds good. Thanks.

User avatar
B.Goode
Posts: 8227
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 11:43 am

I have investigated other cli tools and all of then make a brutal dd copy of the whole SD, so I believe the GUI tool acts the same way.


I think your assumption in respect of the SDCardCopier might be unjustified.

If you are not in a position to use a GUI utility (remote access alone need not be an impediment) then I think there is a near-equivalent cli version that acts in a similar way.

fmarzocca
Posts: 26
Joined: Fri May 25, 2018 12:42 pm

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 11:50 am

I have checked the cli version and it is using dd without unmounting devices.
I don’t want to install a GUI as it is not needed in the project and I don’t like GUIs for embedded scopes.

User avatar
procount
Posts: 1689
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 11:53 am

fmarzocca wrote:
Thu Mar 14, 2019 11:19 am
I don’t know how SDCardCopier works as I do not have a GUI, but I have investigated other cli tools and all of then make a brutal dd copy of the whole SD, so I believe the GUI tool acts the same way.
As rpdom stated, SD card copier (and the similar clone tool in PINN) work by replicating the partition layout of your disk, and then copying (via rsync or cp - I can't remember either, but it is immaterial as they are similar) all of the files from your source drive partitions to the target drive partitions. The last partition is adjusted to fill the remainder of the target disk rather than being an exact duplicate size in order to account for the small variations in sizes of SD card of the same nominal capacity. This also allows cloning to bigger or smaller drives.
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

fruitoftheloom
Posts: 20432
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 11:53 am

fmarzocca wrote:
Thu Mar 14, 2019 11:50 am
I have checked the cli version and it is using dd without unmounting devices.
I don’t want to install a GUI as it is not needed in the project and I don’t like GUIs for embedded scopes.

Do you mean rpi-clone ??

https://github.com/billw2/rpi-clone
adieu

Asus CS10 Chromebit / HP Envy 4500 Wireless Printer / Raspberry Pi Model 2B v1.1 / RealVNC Software...

Ernst
Posts: 1214
Joined: Sat Feb 04, 2017 9:39 am
Location: Germany

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 12:15 pm

fmarzocca wrote:
Thu Mar 14, 2019 9:31 am
I have seen many posts and "howto" concerning backing up Raspi's SD from remote, with the use of dd on the main mmcbl0 device (which has 2 partitions inside).

As an old time LInux user, I know how unreliable is it to backup mounted and running partitions: services, /sys, /proc, etc... I mean, you CAN backup mounted and running partitions, but what will be unreliable it is restoring that backup.


I am using my Raspberries as components of larger projects, all enclosed in large cases with other components, and opening the box to remove the SD is a real PITA.

Thoughts on how to reliable backup a Raspi mounted in a larger enclosure from remote?

Thanks
It took me some time but I found something that may help you.

Many months ago, almost one year, a member with the name RonR wrote a post with a link to a very interesting script named "image-backup".

viewtopic.php?t=210202#p1298239
The road to insanity is paved with static ip addresses

fmarzocca
Posts: 26
Joined: Fri May 25, 2018 12:42 pm

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 12:17 pm

Thank you, I’ll have a look


fmarzocca
Posts: 26
Joined: Fri May 25, 2018 12:42 pm

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 1:30 pm

I want to thank you all, giving me suggestions for backup code, but maybe my first post in the thread was not clear, sorry.

I perfectly know how to backup a partition, a whole device, etc. What I am asking (maybe to myself) is: is it reliable to hot backup a live and working block device?

Thank you for sharing your comments.

User avatar
rpdom
Posts: 14984
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 2:35 pm

fmarzocca wrote:
Thu Mar 14, 2019 1:30 pm
is it reliable to hot backup a live and working block device?
I agree with you that it is not the best way to do things.

For preference I wouldn't use something like dd to do a physical copy of an active partition (I don't consider the boot partition to be active as it usually only gets written to during software updates) as that will lead to file system corruption on restore as inodes change during the copy, but a file by file copy is fairly safe.

Ideally I would stop any "disk heavy" applications if possible while the backup is running.

rsync is a good bet for safety as you can run it twice. The first time it will copy all files and take quite a while, the second time it will only copy files that changed while the first pass was running and should be a lot quicker. I've never found a backup I've made that has resulted in significant loss of data (yet).

ghans
Posts: 7871
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Is it reliable to backup Raspi from remote?

Thu Mar 14, 2019 5:37 pm

The official SD Card Copier does not use dd because of this exact issue.

It uses "cp" instead. In line with this, i would suggest using rsync to backup your Pi.
rsync can transfer data over network by itself and also can be used over SSH for convenience.

Obviously, i can't guarantee it works if both ends are not running Linux.

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: Is it reliable to backup Raspi from remote?

Fri Mar 15, 2019 7:18 pm

fmarzocca wrote:
Thu Mar 14, 2019 1:30 pm
is it reliable to hot backup a live and working block device
Of course it's not! And the official SD Card Copier which does file-by-file copy is not safe either! It may or it may not work. You will never know for sure until actually running your backup.
Author of the robust Nard distro http://www.nard.se

fmarzocca
Posts: 26
Joined: Fri May 25, 2018 12:42 pm

Re: Is it reliable to backup Raspi from remote?

Fri Mar 15, 2019 8:49 pm

@rln: So, in other words, there's no way to backup a Raspberry embedded in a big project without stopping everything, unscrew, unplug connectors and remove the raspberry in order to get access to the SD card?
If it's true, Raspi should not be so popular.

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: Is it reliable to backup Raspi from remote?

Fri Mar 15, 2019 9:54 pm

fmarzocca wrote:
Fri Mar 15, 2019 8:49 pm
So, in other words, there's no way to backup a Raspberry embedded in a big project without stopping everything
In embedded systems one usually don't need full system backups. What you need to copy is only the unique data the embedded system collects/generates. In the event of a SD card corruption one replaces the card with a new one, initialized from a common template/image. (All devices in you project are 99.99% alike. Only a few settings in /etc should differ.)
Author of the robust Nard distro http://www.nard.se

BeauSlim
Posts: 46
Joined: Mon Jul 31, 2017 10:02 am

Re: Is it reliable to backup Raspi from remote?

Fri Mar 15, 2019 11:43 pm

The usual solution to the problem of having your storage in an indeterminate state is to keep temporary files in RAM and to synchronize them to storage on a schedule. This also helps keep flash storage bootable after unexpected power losses, etc.

There are many guides out there for doing this with Linux in general. Here are two specifically for the Pi:
https://hallard.me/raspberry-pi-read-only/
https://learn.adafruit.com/read-only-ra ... i/overview

Since storage is read-only except during synchronization, you should be able to back it up safely.

fmarzocca
Posts: 26
Joined: Fri May 25, 2018 12:42 pm

Re: Is it reliable to backup Raspi from remote?

Sat Mar 16, 2019 9:13 am

rln wrote:
Fri Mar 15, 2019 9:54 pm
In embedded systems one usually don't need full system backups.
Correct, "usually", but that's not my case.
BeauSlim wrote: The usual solution to the problem of having your storage in an indeterminate state is to keep temporary files in RAM and to synchronize them to storage on a schedule. This also helps keep flash storage bootable after unexpected power losses, etc.
This is interesting, but I am afraid it does not apply to my case. I should have a lot more RAM than 1GB.

Anyway, I have studied and tested rpi-clone: https://github.com/billw2/rpi-clone
It does use dd only to copy the /boot partition (which is rarely active), while the / partition is rsync-ed. The result is another SD which is a perfect clone of the one installed inside. I have also made test swapping the 2 SDs, and it worked every time.

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: Is it reliable to backup Raspi from remote?

Sat Mar 16, 2019 10:24 am

fmarzocca wrote:
Sat Mar 16, 2019 9:13 am
This is interesting, but I am afraid it does not apply to my case. I should have a lot more RAM than 1GB.
You need to provide additional information about your use case for us to understand your overall problem. Why all the RAM?
Author of the robust Nard distro http://www.nard.se

Return to “Advanced users”