steveb4pi
Posts: 62
Joined: Sun Aug 11, 2013 6:12 pm

Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sat Oct 21, 2017 8:08 am

'Everyone' says, "To Backup your Pi, use Win32 Disk Imager to read the SD card and make a new .img" ..

WRONG WRONG WRONG == what they DON'T say is, 9 times out of 10, when you come to 'restore' the backup .img) to a new 'same sized' SD card, Win32 Disk Imager refuses, saying 'Destination insufficient space' !!!

The newbie pops in a 'double sized' card and forgets about it .. then makes another Backup and finds exactly the same problem .. at some stage, when going from 8Gb to 16Gb to 32Gb to 64Gb most people realise that restoring the backup is soon going to be impossible (not to mention it's getting very expensive :-) )

The PROBLEM is three-fold -
1) first all SD cards are not the same = they all contain slightly different 'block' counts due to manufacturers having to 'map out' the failed blocks ('bad sectors' in disk speak) and cheap cards shipping with more and more 'bad sectors'. This you can't fix.
2) second, for some insane reason, on the very first boot, the Pi expands it's file system to use ALL the available space on the SD card i.e. every last block. You need to stop the Pi doing this BEFOR you put the SD card in the Pi.
3) When Win32 Disk Imager reads the SD card, it copies each and every block .. so later, when you try to write to a 'same sized' SAD card, you might get lucky (with one that has fewer bad blocks) or, more likley, you get the 'insufficient space' error (what you NEED is a tool that will ONLY read up to the end of the last partitiom, NOT the end of the card)

SO the solution (I've only actually done this on a Pi Zero with stretch LITE == someone more clever than me will have to check that it works for NOOBS .. )

1) STOP the Pi using all the space at first boot
After burning the new Stretch LITE to SD card, in Windows, open the SD card (Windows will only see it as about 40Mb i.e. just the FAT32 partition), find the file cmdline.txt and DELETE the entry " init=/usr/lib/raspi-config/init_resize.sh"
2) BOOT the Pi, go find the file /usr/lib/raspi-config/init_resize.sh. EDIT the file so the line :-
"TARGET_END=$((ROOT_DEV_SIZE - 1))" now reads :-
"TARGET_END=$((ROOT_DEV_SIZE * 0.95))"
3) Run the modified init_resize.sh script (or just add " init=/usr/lib/raspi-config/init_resize.sh" back into cmdline.txt and reboot the Pi)
4) To make a backup .img YOU MUST NOT USE Win32 Disk Imager (if you do, it reads ALL the blocks and you are back to 'insufficient space') == INSTEAD use 'DiskInternals Linux Reader' (https://www.diskinternals.com/linux-reader/) == this ONLY reads to the end of the last partition
NOT the end of the SD card.

The backup .img made by Linux Reader can now be 'restored' (i.e. written using Win32 Disk Imager) to any of your 'same sized' SD cards (assuming they are at least 95% (0.95) of the size of the first one you used)

User avatar
rpdom
Posts: 17723
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sat Oct 21, 2017 9:01 am

steveb4pi wrote:
Sat Oct 21, 2017 8:08 am
'Everyone' says, "To Backup your Pi, use Win32 Disk Imager to read the SD card and make a new .img" ..
No they don't.

They say "use the piclone or SD card copier program to copy your running system to another SD card using a USB adaptor in your Pi". That way you can copy across different card sizes and end up with a fully bootable backup card.

Best practice is to copy your running system to a new card, shut down, swap the cards and start running of the new one. That way you know you have the old card in a good known state to fall back on, and you have proved that the new card is bootable. Ideally you'll cycle this around a set three or more cards.

Win32 Disk Imager isn't recommended for anything now.

User avatar
jahboater
Posts: 6290
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sat Oct 21, 2017 9:03 am

steveb4pi wrote:
Sat Oct 21, 2017 8:08 am
2) second, for some insane reason, on the very first boot, the Pi expands it's file system to use ALL the available space on the SD card i.e. every last block. You need to stop the Pi doing this BEFORE you put the SD card in the Pi.
Its not insane, its very useful and saves a lot of problems.

You should use the provided "SD Card Copier" which was written to solve this problem.

I personally don't use the full image backup method, its mostly copying empty space (zeros) and the OS with all the apps. The OS can quickly and easily be replaced, so there is no point in saving that. I just back up my data, and/or any project I am working on.

Yes, you might say that it took me ages to set the Pi up as I want it. I'd say script it.

Edit: sorry @rpdom for crossed post - +1 - good advice for best practice.

User avatar
rpdom
Posts: 17723
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sat Oct 21, 2017 9:18 am

jahboater wrote:
Sat Oct 21, 2017 9:03 am
steveb4pi wrote:
Sat Oct 21, 2017 8:08 am
2) second, for some insane reason, on the very first boot, the Pi expands it's file system to use ALL the available space on the SD card i.e. every last block. You need to stop the Pi doing this BEFORE you put the SD card in the Pi.
Its not insane, its very useful and saves a lot of problems.
This is totally true. When the Raspbian images were first released they didn't do this. The result was many people installing the software, then finding they had run out of space on the card and upgrades were failing or they weren't able to install additional software. The "expand to fill card" option was there in the raspi-config menu, but most users (quite rightly) expected that they would just be able to use their Pi without any major additional steps. Much better now with the automatic expansion.

Just out of interest, how many other OS installs are there that don't automatically use the maximum available space unless overridden? Windows, Ubuntu, Debian, MacOS...?

User avatar
jahboater
Posts: 6290
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sat Oct 21, 2017 9:28 am

One difference with the Pi is that it is not an "OS install" in the usual sense of the word.
The actual installation is done by the RPF engineers first, which they then make an image of and publish. What the user does is simply copy the pre-installed image over the raw disk. It would be an extremely bad idea for the engineers to provide an image of their entire disk which would be mostly zero's, and then 16 million users download and write all that empty space . So they shrink it first as much as possible. The user then expands it to fit their particular disk.

User avatar
KLL
Posts: 1453
Joined: Wed Jan 09, 2013 3:05 pm
Location: thailand
Contact: Website

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sat Oct 21, 2017 10:19 am

yes, over the time strategies changed,
and i also use the SD card copier tool now ( using full RASPBIAN )
to make copies ?backup? to other SD card and USB stick.

still it can make sense to backup a RPI system to a .img file to a PC HD ( and zip it down )
but zip and unzip gets unreasonable time consuming on bigger cards.

so the Win 32 Disk Imager ( now in revision 1.0 )
and the new [ ] "read only allocated partitions" option
is not obsolete.

remember: you will be able to backup and restore to SAME SD card,
to a bigger one ( and do a manual resize from raspi-config later )
but not sure to a other card with only NOMINAL same size.

UNLESS:
-a- you could use a tool like "gparted"... to shrink your last big partition by some ? MB
-b- and as that problem has been around it could long be solved by putting like a "-1M" in the resize command, instead using full space ( by leaving size info empty ).

steveb4pi
Posts: 62
Joined: Sun Aug 11, 2013 6:12 pm

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sat Oct 21, 2017 10:43 pm

I stand by my original post :-
Google 'How to back-up my Pi' (just like any mewbie would) and see how many pages will suggest ''Win32 Disk Imager' .. on my 'top page' of results it was all of them ..

Now try google 'pi SD card copier' (not that a newbie knows to 'ask' this). On my first page of results, of the top 7, :5 say 'use 'Win32 DiskImager' (for those who doubt, see below ... note the 'top' hit == The Pi Hut ! Yes yes I know, Google 'personalises' the results (plainly I'm spending too much time at the Pi Hut :-) )

https://thepihut.com/blogs/raspberry-pi ... is-sd-card
https://computers.tutsplus.com/articles ... -mac-59294
https://lifehacker.com/how-to-clone-you ... 1261113524
https://beebom.com/how-clone-raspberry- ... nux-macos/
http://www.makeuseof.com/tag/easily-clo ... computing/

Of the other 2, one is an 'advert' from a propriatory software vendor promoting their own (pay for) back-up software, which leaves only ONE that actually suggests using 'SD card copier' (it's https://diyprojects.io/backup-clone-sd-raspberry-pi/ in case anyone is interested)


Expanding to use each and every block is, to say the least, 'not clever' .. the change I make reduces the resize to 95% (which should be 'safe') == I'm NOT saying expand is insane, I'm say USING EVERY BLOCK is insane ..

OK, perhaps 'insane' is a bit harsh, however I'm betting whoever programmed the 'expand' code never realised what effect that would have on back-ups ... plus I'm betting they never actually tried to restore from back-up ..

For me, 'SD card copier' is NOT 'a fix' to MY problem == for me, and, I would suggest, most other newbies, the problem with Pi backup is that I can't backup to my PC (and thus keep the .img safe on my RAID box).

Hopefully at least some of you can see my logic here .. having a series of back-up .img files on your PC (and a copy on your RAID / Cloud back-up repository and/or wherever else you might decide to hide it in the name of 'keeping it safe') means it's so much easier to find when you need it ...

You can hold dozens and dozens of Pi back-up .img named by 'date and system version' and hold them in fully descriptive folders (like 'my photoframe', 'my KODI system' etc) . Indeed, if all else fails I bet you could even find the backup you want by doing a 'search' on the .img file creation date ...

So, yes, whilst you can use your Pi to make a back-up on another SDHC card, but how many micro-SSD cards did you ever manage to write a date on ?? (let alone a Project name) ... as I write this I'm looking at a dozon or two USB sticks, wondering which one has my fav. music tracks on .. and wondering if it will faster to just re-write the whole lot onto a new USB stick rather than plug each one in looking for the 'right' one .. so maybe you will understand why I would have a problem wiith a dozen or two micro-SD cards ????).

Finally, for the vast majority of my projects, I use the Pi Zero as a headless 'controller'. That means PuTTY control from my PC .. those who use the Pi B3 as some sort of 'substitute PC' will have attached a HD display, keyboard. mouse, SD card reader/writers, CD/DVD and hard drives etc. etc. and may well find it easier to use 'SD card copier' ... however the B3 is not for me and I refuse to give up my PC :-)

User avatar
jahboater
Posts: 6290
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 8:08 am

steveb4pi wrote:
Sat Oct 21, 2017 10:43 pm
You can hold dozens and dozens of Pi back-up .img
All full of empty space and easily replaced stuff. At least compress them, which should remove the empty space, as it will all be zero's and be easily compressible.

How large is you project? "du -sh ~" should tell you how big your home folder is.

tar Jcf myproject<date>.tar.xz ~

creates a highly compressed backup of your home area in a single file (change <date> to the current date of course). Then copy it to your PC with scp or something.

I trust the RPF to continue providing reliable OS images and app repositories, so I don't bother backing up anything except stuff in $HOME, and certainly not gigabytes of zeros.

User avatar
rpdom
Posts: 17723
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 8:14 am

jahboater wrote:
Sun Oct 22, 2017 8:08 am
steveb4pi wrote:
Sat Oct 21, 2017 10:43 pm
You can hold dozens and dozens of Pi back-up .img
All full of empty space and easily replaced stuff. At least compress them, which should remove the empty space, as it will all be zero's
Will it? It depends on if it has been used for anything previously. Deleted files don't get zeroed out unless you do it manually.

User avatar
jahboater
Posts: 6290
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 8:19 am

rpdom wrote:
Sun Oct 22, 2017 8:14 am
jahboater wrote:
Sun Oct 22, 2017 8:08 am
steveb4pi wrote:
Sat Oct 21, 2017 10:43 pm
You can hold dozens and dozens of Pi back-up .img
All full of empty space and easily replaced stuff. At least compress them, which should remove the empty space, as it will all be zero's
Will it? It depends on if it has been used for anything previously. Deleted files don't get zeroed out unless you do it manually.
True. Either way it seems daft to back it up.
In the past I have done "dd | gzip" which has reduced the size dramatically, but yes, probably not on heavily used old disks.

I was looking on Amazon recently for Samsung EVO cards and nowhere could I find one for sale less than 32GB. Lots of empty space for most Pi projects.
Last edited by jahboater on Sun Oct 22, 2017 8:44 am, edited 1 time in total.

User avatar
rpdom
Posts: 17723
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 8:41 am

I'm a bit old fashioned. I use the ancient Unix "dump" and "restore" commands to back up and restore file systems. "dump" allows for incremental backups, so I can do a full (level 0) backup once a month, and do incremental backups each week which just backup the changes made since the last full one. I also take a gzipped copy of the contents of the /boot partition, as dump doesn't handle FAT filesystems, and a copy of the partition table too. It needs a Linux system to restore the contents, but I have no shortage of those. I've had to rely on my backups on a number of occasions (mostly hard disk failures) and they have worked every time.

User avatar
jahboater
Posts: 6290
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 8:59 am

It might be old fashioned but its the proper way of doing it.

It does seem an overkill for the average Pi user though!

k-pi
Posts: 929
Joined: Sun Feb 12, 2017 1:46 pm
Location: Upper Hale, Surrey, UK.

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 11:26 am

You should only be backing up your data, (& maybe some config files) - no point in backing up the system as that can be restored any time, but if you lost your data, that's it gone forever! ;)

steveb4pi
Posts: 62
Joined: Sun Aug 11, 2013 6:12 pm

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 4:13 pm

Without wishing to get into a flame war :-)

1) '... lots of empty space'

OK, if you don't know that on a PC, Windows will 'transparently' compress stuff for you, then storing big files of 'empty space' can be a concern
(FYI, when you setup your Pi Backups folder, right click on the folder 'Properties', in Attributes click 'Advanced', then set ''Compress contents to save disk space' - when asked respond 'This folder and all sub-folders')

On the other hand, when I run out of space, I will just stick another 3Tb (or whatever size the current sweet spot of 'pence per Gb' works out at) NAS RAID box on my LAN ...or maybe I'll just switch to the 'Cloud' (Amazon Prime is advertising 'unlimited space' again..) == a 4Gb .img for one of my headless Pi Zero's is way smalller than a DVD Movie ...

2) "I trust the RPF to continue providing reliable OS images... "

Yes, but they KEEP CHANGING HOW STUFF WORKS ...

Previously (Jessie) I could burn the SD card, plug my (headless) Pi Zero into my LAN and PuTTY into it .. with 'Stretch' I can't ... and that's REALLY BASIC == plus what have they done to eth0 setup ????? Not that I ever used NOOBS, but I gather they actually changed the PARTITIONING on NOOBS ....

Now I'm sure you can still download Jessie Lite from RPF, howver (so far), despite a lot of poking around on their website and wasting some quality time on google, it seems I'm just too dumb to find the download page containing a list of 'old' system images ...

This means you MUST keep .img backups if you want to actually restore your Project (since, chances are, your old Project isn't going to run on any other version of the Operating System)

SO == no .img back-up means each new Project is yet another steep learning curve whilst I start again with the very basics (like trying to find out why PuTTy isn't working anymore and how to set up a static IP address again ...)

drgeoff
Posts: 11239
Joined: Wed Jan 25, 2012 6:39 pm

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 6:14 pm

steveb4pi wrote:
Sun Oct 22, 2017 4:13 pm
Yes, but they KEEP CHANGING HOW STUFF WORKS ...
Ah yes, I forgot that Microsoft NEVER does that. :) :)
Quis custodiet ipsos custodes?

User avatar
jahboater
Posts: 6290
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 6:33 pm

steveb4pi wrote:
Sun Oct 22, 2017 4:13 pm

1) '... lots of empty space'

OK, if you don't know that on a PC, Windows will 'transparently' compress stuff for you, then storing big files of 'empty space' can be a concern
Of course it will. However all that "empty space" has to be read off the slow SD card which takes time, it then needs to be compressed which takes a little time, and when the backup is restored it all has to be written back to the SD card which takes time, wears out the SD card, and probably screws up any wear leveling the card might do. All a 100% waste of time.

Sure you can easily buy more disk space, or use the cloud - but why? Just to store gigabytes of nothing. I really am sorry, but I just don't see the point.
steveb4pi wrote:
Sun Oct 22, 2017 4:13 pm
Previously (Jessie) I could burn the SD card, plug my (headless) Pi Zero into my LAN and PuTTY into it .. with 'Stretch' I can't ... and that's REALLY BASIC
Rubbish. Sorry but all my Pi's run headless over ssh including two Zero's. Never had the slightest problem with the move to stretch, neither did I change the way I set it up.
steveb4pi wrote:
Sun Oct 22, 2017 4:13 pm
and how to set up a static IP address again ...)
Its not changed from Jessie.

If you rely on preserving old OS images, your system will just get more and more out of date.

RonR
Posts: 1705
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 7:59 pm

@steveb4pi ,

You may find image-backup useful:

download/file.php?id=19918

image-backup will create a backup of the running SD card to a standard 'raw' image file that can be written to a new SD card with etcher, imageUSB, etc. It will also do incremental updates to the backup image.

To create the initial full backup, run image-backup with no parameters:

sudo image-backup

image-backup will prompt:

Image file to create? /media/usb/whatever.img

Image ROOT filesystem size (MB) [xxxxx]?

(Hitting return will create an image file capable of holding a totally full ROOT filesystem. If you want to create an absolute minimum size image file, enter 0 to see the minimum size permitted. If you want to allow for incremental updates, enter a size larger than the minimum size to allow for expansion.)

Create /media/usb/whatever.img [xxxxx MB] (y/n)? y

To incrementally update the image file, run:

sudo image-backup /media/usb/whatever.img

image-backup will silently add, delete, and update files in the image file to match your running system.

I use cron to run image-backup daily at 4 AM so that I always have an up-to-date backup.

You may also find image-shrink useful:

download/file.php?id=19915

Usage:

image-shrink imagefile [Additional MB]

The image file will be shrunk to its absolute minimum size plus an optional number of additional megabytes.

steveb4pi
Posts: 62
Joined: Sun Aug 11, 2013 6:12 pm

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 8:24 pm

Point I'm trying to make is that even half-an-hour (or however long it takes) making a back-up is time well spent if you ever have to get things working again after a SD corruption ..
.. and I would bet my .img against any 'incremental' or 'user files' alternative back-up anyday .. after a year or two, I would bet my .img back-up even against a 'SD copy' (cloned microSD card), on the grounds that I can't even find the USB sticks I used 6 months ago, so there's no chance I'll ever find a microSD card after a year or two .. indeed, my NAS still has back-ups of my Windows 98 system (not bad after almost 20 years ! )

But wait - a new install of Stretch, the naming convention for network interfaces was changed to an amalgam that includes the MAC address of the Ethernet port. This didn't happen to those who 'upgrade' only those who did a new install

So, if your system SD is corrupted and you have to do a fresh install of Stretch you are in for a shock when eth0 isn't eth0 anymore = and your static address doesn't 'take' .. but hey, you would start by installing Jessie and then redo the 'upgrade' .... wouldn't you ?

Of course, if you took a complete .img back-up ...

I hope I've made my point with this specific example of how an old system+updates can't be replaced by a new install with the same User files if you need 'everything' to start working 'as usual' ..
[ although last I heard, Stretch will revert to the old naming convention if you add "net.ifnames=0" to "/boot/cmdline.txt" ]


Now lets consider clever ways of 'saving' just the 'used' portion of each Partition ..etc.

Quite apart from the multiple commands that need to be entered V's "unplug microSD, insert into reader, save all partitions complete to .img"

Yes, they may 'save space' .. but once again, lets think of the long term .. in a year or two you will have forgotten the 'original' size of the partitions - and the fact that you will need to 'expand' them again after doing the 'restore' ... although when your restored Pi falls over that could jog your memory ...

YES, 'incremental backups' (and CRON jobs) are what a 'general purpose' computer might well benefit from .. however (for example) when I got my Pi Zero + Pi camera running, I built them into a small box for use as a CCTV camera (with just one single cable connection = either Ethernet with PoE (for iSpy) or analogue TV out (+power over co-ax) for connection to a 'classical' CCTV 'recorder' box), I made a final back-up .img of each ....


PS Bit off-topic, but how do you manage to get Stretch running headless without adding the 'ssh' file to the /boot/ partition ?
(according to https://www.raspberrypi.org/documentati ... /README.md it's been disabled since Nov 2016)

RonR
Posts: 1705
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 8:52 pm

steveb4pi wrote:
Sun Oct 22, 2017 8:24 pm
PS Bit off-topic, but how do you manage to get Stretch running headless without adding the 'ssh' file to the /boot/ partition ?
You don't. An ssh file in the boot partition is required for a first time headless startup.

In order to avoid having to repeatedly add an ssh file when I write fresh SD cards/USB drives, I add the ssh file to the Raspbian image file (by mounting the image file).

User avatar
B.Goode
Posts: 10725
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sun Oct 22, 2017 9:07 pm

PS Bit off-topic, but how do you manage to get Stretch running headless without adding the 'ssh' file to the /boot/ partition ?

Just the same as with versions of Jessie that preceded Stretch...

One possibility is to boot the microSD card in a 'headfull' (ie. with display and keyboard) system and use the option provided in the raspi-config utility. (The system used to do this does not have to be the target headless one.)

User avatar
KLL
Posts: 1453
Joined: Wed Jan 09, 2013 3:05 pm
Location: thailand
Contact: Website

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Mon Oct 23, 2017 2:44 am

if you work headless you need for setup
++ ssh file
++ wpa_supplicant.conf file
and that should be prepared on your PC.
if you want save time and you use etcher for writing a image
you go etcher setup and can DISABLE
-- eject on success ( so not need to take out / in sd card for the file copy)
-- validate write on success

the copy of the 2 files to a
burned image || a copied NOOBS
SD card is a 10sec thing,
so your question must have some other background ? missunderstanding

User avatar
jahboater
Posts: 6290
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Mon Oct 23, 2017 7:44 am

KLL wrote:
Mon Oct 23, 2017 2:44 am
the copy of the 2 files to a
burned image || a copied NOOBS
SD card is a 10sec thing,
so your question must have some other background ? misunderstanding
5 seconds, the ssh file can be empty and doesn't need to be copied.
I just do this to create the file:

>ssh

User avatar
KLL
Posts: 1453
Joined: Wed Jan 09, 2013 3:05 pm
Location: thailand
Contact: Website

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Mon Oct 23, 2017 9:06 am

jahboater wrote: 5 seconds, the ssh file can be empty and doesn't need to be copied.
I just do this to create the file:

>ssh
as i am a slow "typer" i prefer drag/copy,
but yes, many way again to do it....

____________________________________________________________________
one may be stupid way add to the MAIN point,
if you have a image file on PC ( lets say from a 8GB SD card ) and restore ( to a other 8GB SD card ) fails,
you can burn it to a 16GB SD card, boot it and with SD card copier tool write to that smaller ( 8GB SD ) card
( and clean the big card.. again )
you loose ?20min?

ejolson
Posts: 6016
Joined: Tue Mar 18, 2014 11:47 am

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Mon Oct 23, 2017 8:04 pm

rpdom wrote:
Sun Oct 22, 2017 8:41 am
I'm a bit old fashioned. I use the ancient Unix "dump" and "restore" commands to back up and restore file systems. "dump" allows for incremental backups, so I can do a full (level 0) backup once a month, and do incremental backups each week which just backup the changes made since the last full one. I also take a gzipped copy of the contents of the /boot partition, as dump doesn't handle FAT filesystems, and a copy of the partition table too. It needs a Linux system to restore the contents, but I have no shortage of those. I've had to rely on my backups on a number of occasions (mostly hard disk failures) and they have worked every time.
As demonstrated by how long it takes companies to recover from ransomware, basic knowledge how to efficiently, securely and reliably back up data is a computing skill that is very much in need. The Raspberry Pi is a real computer designed for children which is suitable for anyone learning almost anything in computer science.

An interesting study of backup techniques was performed by Elizabeth Zwicky in 1991. By that time, simply copying the entire disk image to tape was impractical and more efficient methods were needed. The results of her study indicated that "dump comes out ahead." Today, copying an entire sdcard image to a Window's PC is impractical and, in my opinion, cards bigger than 4GB should be backed up using more efficient methods. The dump program has been kept up to date and works well for incremental backups of the Raspberry Pi even when the destination is not tape. The development of snapshots for copy-on-write file systems has led to another approach more suitable when the backup media is another disk.

Given an external USB disk drive that works with your Raspberry Pi, format that disk with BTRFS. Then use a command like "btrfs sub create pibackup" to create a BTRFS subvolume on the USB drive. Now use rsync with options such as "-glopqrtxDH --numeric-ids --delete" to copy the files from the sdcard to the pibackup subvolume. After rsync is done use the command "btrfs sub snap -r pibackup pibackup-oct23" to create a read-only snapshot of the backup. The next time you issue the rsync command, the changed files will be updated in the "pibackup" subvolume but the copy-on-write semantics means all the original files in the "pibackup-oct23" snapshot will remain unchanged. Create a new snapshot using "btrfs sub snap -r pibackup pibackup-anotherdate" and keep going. This results in incremental backups with easily available historical snapshots.

The advantage over dump is that there is no need to worry about the dump level against which the incremental backup is performed. Old subvolumes can be deleted at any time in any order, whereas deleting a dump level n backup invalidates all subsequent level n+1 backups. With BTRFS, when a snapshot is deleted only drive space filled with data no longer referred to by any of the other subvolumes is freed and made available for future snapshots. Similar results can be achieved with other copy-on-write filesystems such as ZFS and HAMMER as well as thinly provisioned block devices and storage appliances that support snapshots.

JackElliott
Posts: 35
Joined: Wed May 03, 2017 11:57 pm

Re: Newbie Pitfall - BACKUPS don't restore (insufficient space)

Sat Aug 24, 2019 3:02 pm

RonR wrote:
Sun Oct 22, 2017 7:59 pm
@steveb4pi ,

You may find image-backup useful:

https://www.raspberrypi.org/forums/down ... p?id=19918
That link no longer works. Do we not like image-backup? Is image-backup available elsewhere?

Return to “Beginners”