Page 1 of 1

Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 9:31 am
by fmarzocca
I have seen many posts and "howto" concerning backing up Raspi's SD from remote, with the use of dd on the main mmcbl0 device (which has 2 partitions inside).

As an old time LInux user, I know how unreliable is it to backup mounted and running partitions: services, /sys, /proc, etc... I mean, you CAN backup mounted and running partitions, but what will be unreliable it is restoring that backup.


I am using my Raspberries as components of larger projects, all enclosed in large cases with other components, and opening the box to remove the SD is a real PITA.

Thoughts on how to reliable backup a Raspi mounted in a larger enclosure from remote?

Thanks

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 10:23 am
by DarkPlatinum
You probably would either have to code, or use software, or manually VNC into the Pi and backup to a SMB share.

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 10:39 am
by fmarzocca
DarkPlatinum wrote:
Thu Mar 14, 2019 10:23 am
You probably would either have to code, or use software, or manually VNC into the Pi and backup to a SMB share.
I have ssh accesso to the Raspi, that's not the point.

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 11:00 am
by B.Goode
Thoughts on how to reliable backup a Raspi mounted in a larger enclosure from remote?


Recent releases of the Raspberry Pi Foundation Raspbian Stretch Operating System come with an SDCardCopier utility accessible from the Raspberry Pi Desktop main menu system.

I understand your concern based on conventional wisdom about backing up active(mounted) filesystems, and I can't comment on how SDCardCopier deals with this for reliability.

So it might be worth investigating, but like all backup solutions it needs assessment to ensure it meets your needs.

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 11:18 am
by procount
If you had installed your OS via PINN, then you could have made use of its backup facilities.
PINN provides a small linux recovery shell which can run independently of your OS.
It includes SSH and VNC support to allow remote administration of your OS.

It includes a port of Raspbian's SD Card Copier utility, so you can use it to make a complete clone of your SD card. The difference is that it is cloning your OS whilst it is offline, if you are worried about that aspect.

Alternatively, it can backup your OS partitions to a USB disk as compressed tar files. These can be transferred off to a remote location using scp or something afterwards. I haven't considered remote file systems up to now, but it may be possible to mount an external drive using sshfs or I could possibly add nfs if there is interest in that and it doesn't require too many resources.

As you probably haven't installed your OS via NOOBS or PINN, it would still be possible to copy your OS to another SD card with PINN installed on it, in order to make use of these administration features.

Although PINN is primarily an os-installer/boot-manager, you can also hide PINN so you don't even know it is there in normal use. When you want to perform a remote backup, you could enable it, do the backup and then hide it again. It only takes up about 60MB of your SD card.

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 11:18 am
by rpdom
fmarzocca wrote:
Thu Mar 14, 2019 9:31 am
I have seen many posts and "howto" concerning backing up Raspi's SD from remote, with the use of dd on the main mmcbl0 device (which has 2 partitions inside).

As an old time LInux user, I know how unreliable is it to backup mounted and running partitions: services, /sys, /proc, etc... I mean, you CAN backup mounted and running partitions, but what will be unreliable it is restoring that backup.
There's no point in backing up /sys, /proc, /dev etc as they are virtual or RAM based and created dynamically by the system.

I'd prefer to shutdown a system to at least single user mode, or better still, boot to ramdisk, in order to do a full backup. However that is not always possible.

The supplied desktop SD Card Copier does a good job. I believe it uses rsync to copy the files (could be wrong, it might just do a cp -r type thing). It also has a command line equivalent called piclone. Both of those require you to put a spare SD card in a USB adaptor in the Pi that is being backed up.

I'm a bit old school Unix. I use a couple of methods on my Pis. I first tar up the contents of /boot and save to a file in the main file system. Then I use the old "dump" command to dump that file system to a remote file. I do this weekly. On the first Sunday of the month I do a full (level 0) backup, which takes quite a while for some of my systems, then on the other Sundays I do an incremental (level 1) backup, which backs up just the changes since the level 0 backup. I've used this method for years across various Unix and Linux systems and have had to (successfully) rely on it a few times for restores of full systems (hard disk/SD Card fails) or individual files (finger troubles).

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 11:19 am
by fmarzocca
@B.Goode
Thanks for your reply. I don’t know how SDCardCopier works as I do not have a GUI, but I have investigated other cli tools and all of then make a brutal dd copy of the whole SD, so I believe the GUI tool acts the same way.

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 11:22 am
by fmarzocca
procount wrote:
Thu Mar 14, 2019 11:18 am


Although PINN is primarily an os-installer/boot-manager, you can also hide PINN so you don't even know it is there in normal use. When you want to perform a remote backup, you could enable it, do the backup and then hide it again. It only takes up about 60MB of your SD card.
I will investigate this PINN. It sounds good. Thanks.

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 11:43 am
by B.Goode
I have investigated other cli tools and all of then make a brutal dd copy of the whole SD, so I believe the GUI tool acts the same way.


I think your assumption in respect of the SDCardCopier might be unjustified.

If you are not in a position to use a GUI utility (remote access alone need not be an impediment) then I think there is a near-equivalent cli version that acts in a similar way.

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 11:50 am
by fmarzocca
I have checked the cli version and it is using dd without unmounting devices.
I don’t want to install a GUI as it is not needed in the project and I don’t like GUIs for embedded scopes.

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 11:53 am
by procount
fmarzocca wrote:
Thu Mar 14, 2019 11:19 am
I don’t know how SDCardCopier works as I do not have a GUI, but I have investigated other cli tools and all of then make a brutal dd copy of the whole SD, so I believe the GUI tool acts the same way.
As rpdom stated, SD card copier (and the similar clone tool in PINN) work by replicating the partition layout of your disk, and then copying (via rsync or cp - I can't remember either, but it is immaterial as they are similar) all of the files from your source drive partitions to the target drive partitions. The last partition is adjusted to fill the remainder of the target disk rather than being an exact duplicate size in order to account for the small variations in sizes of SD card of the same nominal capacity. This also allows cloning to bigger or smaller drives.

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 11:53 am
by fruitoftheloom
fmarzocca wrote:
Thu Mar 14, 2019 11:50 am
I have checked the cli version and it is using dd without unmounting devices.
I don’t want to install a GUI as it is not needed in the project and I don’t like GUIs for embedded scopes.

Do you mean rpi-clone ??

https://github.com/billw2/rpi-clone

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 12:15 pm
by Ernst
fmarzocca wrote:
Thu Mar 14, 2019 9:31 am
I have seen many posts and "howto" concerning backing up Raspi's SD from remote, with the use of dd on the main mmcbl0 device (which has 2 partitions inside).

As an old time LInux user, I know how unreliable is it to backup mounted and running partitions: services, /sys, /proc, etc... I mean, you CAN backup mounted and running partitions, but what will be unreliable it is restoring that backup.


I am using my Raspberries as components of larger projects, all enclosed in large cases with other components, and opening the box to remove the SD is a real PITA.

Thoughts on how to reliable backup a Raspi mounted in a larger enclosure from remote?

Thanks
It took me some time but I found something that may help you.

Many months ago, almost one year, a member with the name RonR wrote a post with a link to a very interesting script named "image-backup".

viewtopic.php?t=210202#p1298239

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 12:17 pm
by fmarzocca
Thank you, I’ll have a look

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 12:19 pm
by fmarzocca

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 1:30 pm
by fmarzocca
I want to thank you all, giving me suggestions for backup code, but maybe my first post in the thread was not clear, sorry.

I perfectly know how to backup a partition, a whole device, etc. What I am asking (maybe to myself) is: is it reliable to hot backup a live and working block device?

Thank you for sharing your comments.

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 2:35 pm
by rpdom
fmarzocca wrote:
Thu Mar 14, 2019 1:30 pm
is it reliable to hot backup a live and working block device?
I agree with you that it is not the best way to do things.

For preference I wouldn't use something like dd to do a physical copy of an active partition (I don't consider the boot partition to be active as it usually only gets written to during software updates) as that will lead to file system corruption on restore as inodes change during the copy, but a file by file copy is fairly safe.

Ideally I would stop any "disk heavy" applications if possible while the backup is running.

rsync is a good bet for safety as you can run it twice. The first time it will copy all files and take quite a while, the second time it will only copy files that changed while the first pass was running and should be a lot quicker. I've never found a backup I've made that has resulted in significant loss of data (yet).

Re: Is it reliable to backup Raspi from remote?

Posted: Thu Mar 14, 2019 5:37 pm
by ghans
The official SD Card Copier does not use dd because of this exact issue.

It uses "cp" instead. In line with this, i would suggest using rsync to backup your Pi.
rsync can transfer data over network by itself and also can be used over SSH for convenience.

Obviously, i can't guarantee it works if both ends are not running Linux.

ghans

Re: Is it reliable to backup Raspi from remote?

Posted: Fri Mar 15, 2019 7:18 pm
by rln
fmarzocca wrote:
Thu Mar 14, 2019 1:30 pm
is it reliable to hot backup a live and working block device
Of course it's not! And the official SD Card Copier which does file-by-file copy is not safe either! It may or it may not work. You will never know for sure until actually running your backup.

Re: Is it reliable to backup Raspi from remote?

Posted: Fri Mar 15, 2019 8:49 pm
by fmarzocca
@rln: So, in other words, there's no way to backup a Raspberry embedded in a big project without stopping everything, unscrew, unplug connectors and remove the raspberry in order to get access to the SD card?
If it's true, Raspi should not be so popular.

Re: Is it reliable to backup Raspi from remote?

Posted: Fri Mar 15, 2019 9:54 pm
by rln
fmarzocca wrote:
Fri Mar 15, 2019 8:49 pm
So, in other words, there's no way to backup a Raspberry embedded in a big project without stopping everything
In embedded systems one usually don't need full system backups. What you need to copy is only the unique data the embedded system collects/generates. In the event of a SD card corruption one replaces the card with a new one, initialized from a common template/image. (All devices in you project are 99.99% alike. Only a few settings in /etc should differ.)

Re: Is it reliable to backup Raspi from remote?

Posted: Fri Mar 15, 2019 11:43 pm
by BeauSlim
The usual solution to the problem of having your storage in an indeterminate state is to keep temporary files in RAM and to synchronize them to storage on a schedule. This also helps keep flash storage bootable after unexpected power losses, etc.

There are many guides out there for doing this with Linux in general. Here are two specifically for the Pi:
https://hallard.me/raspberry-pi-read-only/
https://learn.adafruit.com/read-only-ra ... i/overview

Since storage is read-only except during synchronization, you should be able to back it up safely.

Re: Is it reliable to backup Raspi from remote?

Posted: Sat Mar 16, 2019 9:13 am
by fmarzocca
rln wrote:
Fri Mar 15, 2019 9:54 pm
In embedded systems one usually don't need full system backups.
Correct, "usually", but that's not my case.
BeauSlim wrote: The usual solution to the problem of having your storage in an indeterminate state is to keep temporary files in RAM and to synchronize them to storage on a schedule. This also helps keep flash storage bootable after unexpected power losses, etc.
This is interesting, but I am afraid it does not apply to my case. I should have a lot more RAM than 1GB.

Anyway, I have studied and tested rpi-clone: https://github.com/billw2/rpi-clone
It does use dd only to copy the /boot partition (which is rarely active), while the / partition is rsync-ed. The result is another SD which is a perfect clone of the one installed inside. I have also made test swapping the 2 SDs, and it worked every time.

Re: Is it reliable to backup Raspi from remote?

Posted: Sat Mar 16, 2019 10:24 am
by rln
fmarzocca wrote:
Sat Mar 16, 2019 9:13 am
This is interesting, but I am afraid it does not apply to my case. I should have a lot more RAM than 1GB.
You need to provide additional information about your use case for us to understand your overall problem. Why all the RAM?