davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

SD backup on the Pi during runtime: rpi-clone? alternatives?

Tue May 10, 2016 2:55 pm

hi,

I now just wanted to start using rpi-clone
https://github.com/billw2/rpi-clone

but I now read:
rpi-clone does not work with NOOBS.
I have a Jessie Build, created from NOOBS.
So it woudn't work then? :cry:
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

Dutch_Master
Posts: 362
Joined: Sat Jul 27, 2013 11:36 am

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Tue May 10, 2016 3:32 pm

You cannot back-up a running system :roll: If you want a backup of a working system, copy the contents of the SD card to a file on a separate PC. If you want to store important data from a running system, use rsync to copy the contents to another medium, like NAS, USB stick or said extra PC :)

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Tue May 10, 2016 3:56 pm

Dutch_Master wrote:You cannot back-up a running system :roll: If you want a backup of a working system, copy the contents of the SD card to a file on a separate PC. If you want to store important data from a running system, use rsync to copy the contents to another medium, like NAS, USB stick or said extra PC :)
rpi-clone says it can backup a running system!
rpi-clone is a shell script that will back up (clone using dd and rsync) a running Raspberry Pi file system to a destination SD card 'sdN' plugged into a Pi USB port (via a USB card reader).
please read before replying! ;)

so again:
davenull wrote:hi,

I now just wanted to start using rpi-clone
https://github.com/billw2/rpi-clone

but I now read:
rpi-clone does not work with NOOBS.
I have a Jessie Build, created from NOOBS.
So it woudn't work then? :cry:
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 7:33 am

now is rpi-clone working with NOOBS-created Jessie SD cards or not?

if not:

how to create a full SD image copy by the Raspi itself during runtime instead, without removing the SD card ?
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

User avatar
Pilosopong Tasyo
Posts: 116
Joined: Wed Apr 06, 2016 11:43 am
Location: /root

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 8:47 am

davenull wrote:I have a Jessie Build, created from NOOBS.
So it woudn't work then?
Short answer: No.

The rpi-clone script -- less $(which rpi-clone) -- is designed to work with 2 partitions only: the boot (/boot) and root (/) partitions. The project page says it all:
Raspberry Pi distributions which have a VFAT boot partition 1 and a Linux root partition 2. Tested on Raspbian but should work on other distributions which have this same two partition structure. rpi-clone does not work with NOOBS.
Raspbian installs via NOOBS create a minimum of 4 or 5 partitions (IIANM). If you want to make a "clone" (and I use that term loosely, since rpi-clone is partially a misnomer as far as cloning is concerned) of a NOOBS-based installation, based on rpi-clone, fork rpi-clone to make it suitable for making NOOBS-based backups.

Or wait for someone to make a fork that enhances rpi-clone.

Or submit a feature request to the author of the script to make it possible to create NOOBS-based backups.

Sorry I can't recommend any alternatives to rpi-clone ATM. I don't use NOOBS (only tried it once and noticed it created several partitions). Perhaps someone else can chime in.
Give a man a ghoti and he'll eat for a day; teach him how to ghoti and he'll eat for life. Ghoti?! - http://i.imgur.com/mAfBUdi.png
How To Ask Questions The Smart Way - http://www.catb.org/esr/faqs/smart-questions.html

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 9:20 am

that's a pity, but thank you very much for your reply!
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

User avatar
RaTTuS
Posts: 10539
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 9:29 am

learn how to use
fdisk , mkfs , dd
and your prayers will be answered
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 9:38 am

please be more specific!
davenull wrote:how to create a full SD image copy by the Raspi itself during runtime instead, without removing the SD card ?
RaTTuS wrote:learn how to use fdisk , mkfs , dd and your prayers will be answered
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

User avatar
B.Goode
Posts: 9586
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 10:04 am

Just to say, in an attempt at clarity, that there is a conundrum at the heart of what you are attempting...

A linux system, at least in the simple form that you are likely to be running, is constantly writing data to its filestore in the form of logs, journals, cron job updates, etc.

Any 'backup' operation will take a finite time to complete.

It follows that the state of the filestore at the end of the backup will be different to what it was when the backup started running. So what, precisely, is in your backup? Probably a mixture of files as they existed at various times while the backup operation took place. So you don't have a single coherent and consistent snapshot of your filestore at a single point in time.

If you understand what is happening, that may be acceptable. But it is the reason why some people say, with technical accuracy, that you cannot backup a running system. (Which is to say: you can't do it and get a coherent, consistent reliable outcome.) This is not a particular vice of the RPi, or Raspbian: it has plagued unix system administrators since the beginning of time.

The author of rpi-clone has produced a tool which meets his needs, and has shared it in case it meets the needs of others. It does not disprove the general explanation given above.

The code of rpi-clone is publically available for inspection on github. https://github.com/billw2/rpi-clone/blo ... /rpi-clone It appears to be constructed largely using the tools mentioned by @RaTTuS. In principle, you could adapt it for your own needs.

The Issues tracker for the package https://github.com/billw2/rpi-clone/issues shows that there are others seeking the same outcome.

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 10:34 am

I have time to wait for the SD image copy to complete (e.g., over night), and I cannot remove the card without disassembling 80% of my mobile robot.
It's just for backup reasons and to have a updated SD clone in case of urgent emergency issues (hope they'll never happen, but who knows...)

As I am a [MOD EDIT] beginner to Raspi and Linux, I don't understand what the rpi-clone code does, so I have not the skills to change that, I just need to have something that works.

so the rpi-clone thing would be fine (if it works) - or a reliable substitute.
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

User avatar
procount
Posts: 2049
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 11:45 am

@DaveNull - so to paraphrase your issue - You have an SD card in a robot, that you want to backup without removing it, because that is difficult to do on a regular basis.
Your SD card is based on a NOOBS installed Raspbian Jessie image.

So the issue is not really about how to backup your SD card with Raspbian running, but how to back it up without removing it from the robot? (although it is best to backup when the system is not running for the reasons already stated)
I suppose whatever backup system you choose to use, it would be acceptable to remove the SD card once in order to install the solution (if necessary)?

Where do you intend to store the backup? Can you plug a USB stick into the Pi in your Robot? Or do you intend to store the backup over the network?
Can you plug a keyboard and monitor into your robot (easily, without dismantling it) in order to perform the backup, or do you access it only via ssh/vnc over a wifi network? If so, does your wifi dongle work with NOOBS?
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 1:16 pm

I want to create a backup of my SD card (a 1:1 copy), e.g. to a 2nd SD card which is plugged to a USB SD card reader on my Pi.
This SD card reader has a long wire, so I can handle the 2nd SD card fine.
Just the 1st SD card is hard to handle because the Pi is built deep in the innermost of my mobile robot.

In emergency cases of course I can reach the 1st SD card by removing the motor shield and the extra IO multiplexer boards, the TFT, and so on.
Then I want to exchange the broken SD card by the last backup SD card I had created.

So my intention is to have always a 2nd SD card which can replace the 1st one in case I need to.

I don't use ssh at all.
(but yes, a wireless keyboard and a mouse are plugged to my Pi).

rpi-clone is supposed to do that perfectly, but not of my NOOBS SD.
So I need something as a substitute to get a most pefect SD card copy.

(the 2nd SD card has an additional benefit: I can use it as an updated version on a 2nd Pi, too, for experimental purposes.)
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

User avatar
procount
Posts: 2049
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 1:53 pm

Since you already have NOOBS on the card (which is a linux OS in its own right) I would suggest using that as part of your strategy.

When you boot into NOOBS, you can start a rescue shell, and use that to backup your Raspbian partitions to your USB card reader.
This avoids the problem of backing up a live system and having to remove the SD card.
When the backup is complete, you can then let NOOBS reboot into Raspbian.

There are a few issues to solve:
1. NOOBS does not support USB devices.
2. NOOBS does not include rsync

But If you use my compatible fork of NOOBS called PINN, it already includes USB support.
Since you want a 1:1 backup, you could use 'cp' (as you mention time is not an issue), or I could include 'rsync' in PINN which will only copy the changed files.
PINN already has a background script facility that you could hijack to perform the backup, so you could even automate the backup. But to do this, I think PINN would need a little mod to allow the boot process to be temporarily halted to allow the backup to proceed, and then allowed continue to boot into Raspbian. Actually, my next version will have a small mod that would at least be suitable for a manual backup.
How does that sound?

EDIT: I was asking about ssh/vnc keyboard display etc. to determine how you can access NOOBS in your robot to perform any manual interactions with NOOBS to start the backup on demand.
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 2:06 pm

no, no rescue shell wanted.

I have to spend 1 complete week (or perhaps even more) for re-installing additional software and re-adjusting my OS and the hardware pins and everything, so I need to make a complete card copy to a 2nd card to be useable out of the box, in case the 1st system breaks, or for a 2nd system as a test platform.

I never use ssh or Putty, that's too weird to me.

All SD image backups must be able to be created from the running 1st system, backing up my live system, like rpi-clone does.
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

User avatar
procount
Posts: 2049
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 2:31 pm

Don't be afraid of the term "rescue-shell". It is just one way of initiating the backup process. (e.g. login and type 'backup') But there are alternatives.
How do you expect to start the backup process?

If you want a complete, coherent backup of your Raspbian system, you need to do it whilst it is not running.

Why is it so important to back it up from the "running live system"?
Do you never turn it off?
What is it doing at night when you have "time to wait" for the backup to complete?
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 2:41 pm

maybe I misunderstood how you want to create the 2nd SD card (which must be a 1:1 copy), can you please tell me again?
I do need this 2nd complete copy.

(short + clear sentences,
single step by step,
my English is quite shaky)
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

mikerr
Posts: 2814
Joined: Thu Jan 12, 2012 12:46 pm
Location: UK
Contact: Website

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 2:58 pm

procount wrote:
If you want a complete, coherent backup of your Raspbian system, you need to do it whilst it is not running.
Well you could mount the root filesystem read-only. Then you CAN do a complete, coherent backup while running...
Android app - Raspi Card Imager - download and image SD cards - No PC required !

User avatar
procount
Posts: 2049
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 3:21 pm

OK, nice & slow :)
This is how I think it would work (but you haven't answered my previous questions, so some aspects may need to be modified to suit you)

1. Turn off robot
2. Insert 2nd SD card in USB SD Card reader
3. Turn on Robot.
4. Press SHIFT key on wireless keyboard (?) to get into PINN.
5. Run backup program
6. (the backup program copies robot SD card to 2nd SD card.)
7. Remove 2nd backup SD card.
7. Return to PINN program.
8. Boot into your robot Raspbian (by selection, after timeout, by rebooting etc)

The backup program is just a script, like rpi-clone that can backup your SD card however you like:
a) use dd for a bit image (but may not work on different sized SD cards) [1:1 copy]
b) Create identical partitions on 2nd card then use cp or rsync to copy the files [1:1 Copy]
c) Create (compressed) tar files of your partitions (but then you need to untar them to create an identical SD card) [Needs an extra step to create 1:1 copy]
d) Something else of your choosing.
...your choice!

The important point is that you can use what you already have (NOOBS) to backup your main system when it is not running, so that you have better confidence that your backup SD will work. You will not have such confidence if you back it up whilst it is running.

@mikerr - Yes you could, but I don't know what his robot is doing during this time or what issues write failures may have on a read-only file system. I think something simple, easy and dependable is what he requires.
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 4:14 pm

ok, thank you now it's a bit more clear.

pressing shift shortly after the Pi started booting I get into NOOBS OS restore mode?
I am a little scared about damaging my SD card then by accidentally reinstalling Raspbian anew and overwriting everything...?

and which exactly is the backup program which creates a complete bootable new SD card, an exact 1:1 copy of my 1st one?
How do I call it after having pressed shift?

(I would need the exact command(s), ready to use, e.g. a Linux executable or several system commands written into a bash script, as I have no idea about dd or cp or rsync or untar or partitions or mount root file system or what ever)
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

User avatar
procount
Posts: 2049
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 8:29 pm

If you like my proposal and can be patient, I will modify PINN for you, as I think it will be a useful feature for others too.
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Wed May 11, 2016 9:59 pm

thank you, sure I will wait and I'm gladly looking forward!
8-)
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}


User avatar
procount
Posts: 2049
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Fri May 13, 2016 9:24 am

Cool! 8-) That saves me some work.
Although I will probably go ahead with PINN mods because it will allow it to work with all OSes, not just Raspbian.
I must dig into the SD card copier to see how Simon cloned the running system....
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

davenull
Posts: 1159
Joined: Thu Oct 22, 2015 7:22 am
Location: a small planet close to Betelgeuze

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Fri May 13, 2016 11:33 am

thank you, I will try ASAP!
Can I get the new Raspbian by

Code: Select all

sudo apt-get update
sudo apt-get upgrade
sudo reboot
?

actually I did that already but there is no SD copier in "Zubehoer" (Accessories)

BTW, I have a Pi 2, not 3.
#define S sqrt(t+2*i*i)<2
#define F(a,b) for(a=0;a<b;++a)
float x,y,r,i,s,j,t,n;int main(){F(y,64){F(x,99){r=i=t=0;s=x/33-2;j=y/32-1;F(n,50&S){t=r*r-i*i;i=2*r*i+j;r=t+s;}if(S){PointOut(x,y);}}}for(;;);}

fruitoftheloom
Posts: 22037
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: SD backup on the Pi during runtime: rpi-clone? alternati

Fri May 13, 2016 11:38 am

davenull wrote:thank you, I will try ASAP!
Can I get the new Raspbian by

Code: Select all

sudo apt-get update
sudo apt-get upgrade
sudo reboot
?

actually I did that already but there is no SD copier in "Zubehoer" (Accessoires)
So you did not read the BLOG that B.Goode pointed towards, if you had you would of not posted above :shock: :shock: :shock: :shock: :shock: :shock:
Retired disgracefully.....
......to an uncomplicated life !

Rather than negativity think outside the box !

Return to “Beginners”