skyraven
Posts: 3
Joined: Mon Apr 19, 2021 8:52 am

Snapshot idea/solution to revert easily to clean / working Raspbian

Mon Apr 19, 2021 8:58 am

Hi everyone,

I'm having a bit of an impossible dream maybe, despite this though I decided to give it a shot:). Maybe someone hit the same dilemma and has a better solution than what I found.

I have a cluster of 8 x PI 4 with Raspbian OS and K8S on them.
This is the state of the current clean install.
I want to play with different CNIs, CSIs, service meshes BUT in case something goes wrong to easily revert to the clean state of the devices (just as they were after the installation). This more in a VM snapshot style of working :).

I know I can use backup tools: rsync, rdiff-backup (uses rsync), clonezilla (full image? might take some time), timeshift (not an option, I am not a fan of UIs or installing X11 just for this; I know it uses rsync behind the scenes) but most of these will just save my files => I still need time to reinstall everything.
Filesystem snapshots / LVM style I do not have experience with at the moment, not sure if that would be a solution. If yes, then I'd give it a go with installing Ubuntu instead of Raspbian (Raspbian does not use LVM).
Getting the SD-cards out of all 8 x PIs, doing a dd (they are 64gb in size) for each, then putting them back in is also not the best way to go about it.

Currently I am a bit lost in the sea of tools / possibilities and would need a few pointers from people that already hit this need (to get fast back to a working state) and managed to find a clean/straightforward way to solve it.

Any ideas ?:)
Thank you !!

User avatar
thagrol
Posts: 4714
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Mon Apr 19, 2021 11:13 am

A sanpshot capable files system such as BTRFS might be one way to go. You'll need an initrd to use it as your root FS as it's not built into the kernel.

Snapshot on a regular basis and rsync the snapshot to external storage.

If the OS is bootable but broken, restore from the snapshot.
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

skyraven
Posts: 3
Joined: Mon Apr 19, 2021 8:52 am

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Mon Apr 19, 2021 11:55 am

wonderfully summed up and described for both snapshots and backup!
thank you very much !!!

never played with BTRFS but reading a bit on the snapshot capability of the FS reminded me of ZFS in the day of Solaris :)

I'll make my own image with BTRFS in this case and use the built-in snapshot functionality.
Sounds reasonable enough.

User avatar
thagrol
Posts: 4714
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Mon Apr 19, 2021 12:25 pm

ZFS is still around and available on linix it's not official though and has issues when run on 32 bit RPiOS.

If you want to try it, you'll need to build the kernel module from source which may involve a custom kernel too. And you'll still need an initrd.

Last time I tried I couldn't get it to dectect the zpools on reboot but that was on the 64 bit beta OS.
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

skyraven
Posts: 3
Joined: Mon Apr 19, 2021 8:52 am

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Mon Apr 19, 2021 1:46 pm

Limited time for debugging too much with the actual workload :( otherwise it would be a good challenge.
I still have ZFS on my FreeNAS setup at home :)
I was having Nexenta Stor in the past and there also ZFS:).
I do like it.

Found this nice tutorial (long but good) about setting up the PI with BTRFS and will follow up on it:
https://hackmd.io/FP-7sHiPTJGaJvzSa3nw8A

Thank you! I owe you one!

andrum99
Posts: 1415
Joined: Fri Jul 20, 2012 2:41 pm

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Tue Apr 20, 2021 7:24 pm

thagrol wrote:
Mon Apr 19, 2021 12:25 pm
ZFS is still around and available on linix it's not official though and has issues when run on 32 bit RPiOS.
ZFS has issues on all 32-bit OSes because Sun stopped supporting 32-bit systems from Solaris 11, which was the first release with ZFS. Even before they did that it had issues because certain things that are atomic on 64-bit CPUs are not atomic on 32-bit CPUs. When ZFS was designed it was clear the server and workstation worlds was going 64-bit fast, so it was not a problem for Sun to drop 32-bit support. While ZFS will work on 32-bit Linux if you give it enough kernel virtual memory, it is not recommended: that's not specific to Raspberry Pi OS.

andrum99
Posts: 1415
Joined: Fri Jul 20, 2012 2:41 pm

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Tue Apr 20, 2021 7:26 pm

skyraven wrote:
Mon Apr 19, 2021 1:46 pm
Found this nice tutorial (long but good) about setting up the PI with BTRFS and will follow up on it:
https://hackmd.io/FP-7sHiPTJGaJvzSa3nw8A
I found the UI on BTRFS to be complete rubbish compared to ZFS, but some people do use it successfully.

User avatar
thagrol
Posts: 4714
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Tue Apr 20, 2021 7:41 pm

andrum99 wrote:
Tue Apr 20, 2021 7:26 pm
skyraven wrote:
Mon Apr 19, 2021 1:46 pm
Found this nice tutorial (long but good) about setting up the PI with BTRFS and will follow up on it:
https://hackmd.io/FP-7sHiPTJGaJvzSa3nw8A
I found the UI on BTRFS to be complete rubbish compared to ZFS, but some people do use it successfully.
Can't say I have a preference for one over the other when it comes to UI.

I had ZFS working fine on a 64 bit kernel/32 bit userland RPiOS but couldn't get it to work reliably on a full 64 bit RPiOS. Either way it needed a custom kernel and custom kernel module as well as an initrd if "/" was on it. BTRFS just needs the initrd and then only if "/" is on it.
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

swampdog
Posts: 621
Joined: Fri Dec 04, 2015 11:22 am

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Tue Apr 20, 2021 11:14 pm

LVM works fine on rpi4 but you need an initrd for it to boot an LVM partition.

I use timeshift on an rpi+ssd (and use the gui to configure it) but I don't think it requires it..

Code: Select all

foo@pi18:~ $ sudo timeshift --list
Device : /dev/dm-3 (sda3)
UUID   : 51262452-bfe0-4b3f-94fd-aef5de1eb946
Path   : /timeshift
Mode   : RSYNC
Device is OK
7 snapshots, 96.0 GB free

Num     Name                 Tags  Description  
------------------------------------------------------------------------------
0    >  2020-11-19_19-21-36  O                  
1    >  2020-11-19_23-32-59  O                  
2    >  2021-04-16_22-00-01  D                  
3    >  2021-04-17_22-00-02  D                  
4    >  2021-04-18_22-00-01  D                  
5    >  2021-04-19_22-00-02  D                  
6    >  2021-04-20_22-00-02  D
Initial backup takes a few minutes, thereafter it's incremental and quick (for me). It barfs trying to restore grub but otherwise works.

Code: Select all

foo@pi18:~ $ timeshift -h

Timeshift v19.01 by Tony George (teejeetech@gmail.com)

Syntax:

  timeshift --check
  timeshift --create [OPTIONS]
  timeshift --restore [OPTIONS]
  timeshift --delete-[all] [OPTIONS]
  timeshift --list-{snapshots|devices} [OPTIONS]

Options:

List:
  --list[-snapshots]         List snapshots
  --list-devices             List devices

Backup:
  --check                    Create snapshot if scheduled
  --create                   Create snapshot (even if not scheduled)
  --comments <string>        Set snapshot description
  --tags {O,B,H,D,W,M}       Add tags to snapshot (default: O)

Restore:
  --restore                  Restore snapshot
  --clone                    Clone current system
  --snapshot <name>          Specify snapshot to restore
  --target[-device] <device> Specify target device
  --grub[-device] <device>   Specify device for installing GRUB2 bootloader
  --skip-grub                Skip GRUB2 reinstall

Delete:
  --delete                   Delete snapshot
  --delete-all               Delete all snapshots

Global:
  --snapshot-device <device> Specify backup device (default: config)
  --yes                      Answer YES to all confirmation prompts
  --btrfs                    Switch to BTRFS mode (default: config)
  --rsync                    Switch to RSYNC mode (default: config)
  --debug                    Show additional debug messages
  --verbose                  Show rsync output (default)
  --quiet                    Hide rsync output
  --scripted                 Run in non-interactive mode
  --help                     Show all options

Examples:

timeshift --list
timeshift --list --snapshot-device /dev/sda1
timeshift --create --comments "after update" --tags D
timeshift --restore 
timeshift --restore --snapshot '2014-10-12_16-29-08' --target /dev/sda1
timeshift --delete  --snapshot '2014-10-12_16-29-08'
timeshift --delete-all 

Notes:

  1) --create will always create a new snapshot
  2) --check will create a snapshot only if a scheduled snapshot is due
  3) Use --restore without other options to select options interactively
  4) UUID can be specified instead of device name
  5) Default values will be loaded from app config if options are not specified


Back to LVM (or perhaps BTRFS but I dunno that)..
..if you separate your OS from the data that's the way I'd proceed currently. Have the OS boot normally and copy it "offsite" somehow. If the cluster images are nigh on identical you may be able to script it so a single image can be restored/patched to any of the cluster.

You can play with LVM without doing anything special ("lvm2" package) and start using it on a live system without a reboot. The module will just load. All you need in preparation is a blank partition (say /dev/sda2) and (loosely) read up on these commands in order..

pvs,vgs,lvs
pvcreate,vgcreate,lvcreate

..once you've created a logical volume (LV) use mkfs.blah upon it. You'll find two types of /dev/ entries (/dev/VG/LV and /dev/mapper-VG-LV) but I'll leave it there for now.

Snapshots work in the same fashion as you'd backup a database. eg: with oracle it's *not* like using rman which can snapshot a running DB. You have to pause it. Whilst paused you create the snaphot. Unpause and backup the snapshot at your leisure (then delete snapshot). Thus, the downtime of your K8S (never used K8S either) will depend how quickly you can bounce/pause it. Taking a snaphot is quick. It's more correct to think of it as initiating a snapshot. LVM keeps track of what's changing between the snapshot and its associated LV. The snapshot looks like a full LV for the purposes of backing up and will grow in size on the disk as more of the associated LV alters. I suspect the same for BTRFS though others will need to comment

eg:
OS on /dev/sda1 <-backup seperately.
LVM (K8S/data) on /dev/sda2
..this way you don't need an initrd. Boot the OS with /etc/fstab not automounting LVM. Script an LVM command to invoke an LVM command late in boot (service or /etc/rc.local) then mount the LVM partition(s).

Note: the problem with an initrd currently is when the kernel changes it's likely the 'update-initramfs' won't see the new kernel so when you reboot you end up with no LVM module.

Note: don't recall if timeshift does /boot/ correctly or not.

bls
Posts: 1361
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Wed Apr 21, 2021 3:09 am

The other approach is to make it super-easy to burn an SD Card that's configured "just so"...that is, exactly the way you want it.

With this approach, when you want to try something new that might be risky, you can just burn a new card that's already set up exactly perfect, and if it fails to meet your goals, you can go back to your "known good card" or simply restart from a "just so" card again.

This seems much easier to me than wrestling with an absolutely huge "how to use BTRFS" guide filled with caveats. YMMV
Pi tools:
Quickly and easily build customized-just-for-you SD Cards: https://github.com/gitbls/sdm
Easily run your network's DHCP/DNS on a Pi: https://github.com/gitbls/ndm
Easy strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

andrum99
Posts: 1415
Joined: Fri Jul 20, 2012 2:41 pm

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Wed Apr 21, 2021 11:55 am

thagrol wrote:
Tue Apr 20, 2021 7:41 pm
andrum99 wrote:
Tue Apr 20, 2021 7:26 pm
skyraven wrote:
Mon Apr 19, 2021 1:46 pm
Found this nice tutorial (long but good) about setting up the PI with BTRFS and will follow up on it:
https://hackmd.io/FP-7sHiPTJGaJvzSa3nw8A
I found the UI on BTRFS to be complete rubbish compared to ZFS, but some people do use it successfully.
Can't say I have a preference for one over the other when it comes to UI.

I had ZFS working fine on a 64 bit kernel/32 bit userland RPiOS but couldn't get it to work reliably on a full 64 bit RPiOS. Either way it needed a custom kernel and custom kernel module as well as an initrd if "/" was on it. BTRFS just needs the initrd and then only if "/" is on it.
The problem I had with BTRFS was that when I tried to mount a mirrored filesystem with just one half of the mirror present it silently did nothing. There was no feedback whatsoever about what went wrong. I decided it wasn't worth messing with, when I knew ZFS had a decent UI.

I'm running ZFS on Ubuntu 20.04 LTS on my NAS Pi and it works well with my spinning rust. It has the ZFS modules built in, and they've fixed the daft bugs that cause errors when you install zfsutils-linux on a systemd-based system. Ubuntu 21 also has OpenZFS 2.0, which I will be trying out tomorrow once Ubuntu 21 is released.

I didn't need a custom kernel when I ran ZFS on Raspberry Pi OS - just an out of tree kernel module built against the exact same kernel.

User avatar
thagrol
Posts: 4714
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Wed Apr 21, 2021 12:09 pm

andrum99 wrote:
Wed Apr 21, 2021 11:55 am
I didn't need a custom kernel when I ran ZFS on Raspberry Pi OS - just an out of tree kernel module built against the exact same kernel.
In hind sight I probably had a custom kernel because of problems I had (due to my limited knowledge at that time) build just the out oftree module.

I'm using one ATM because I need the SATA modules (CM4 with PCIe SATA card).
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

andrum99
Posts: 1415
Joined: Fri Jul 20, 2012 2:41 pm

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Wed Apr 21, 2021 2:51 pm

thagrol wrote:
Wed Apr 21, 2021 12:09 pm
andrum99 wrote:
Wed Apr 21, 2021 11:55 am
I didn't need a custom kernel when I ran ZFS on Raspberry Pi OS - just an out of tree kernel module built against the exact same kernel.
In hind sight I probably had a custom kernel because of problems I had (due to my limited knowledge at that time) build just the out oftree module.

I'm using one ATM because I need the SATA modules (CM4 with PCIe SATA card).
A CM4 with a bunch of SATA SSDs would make an awesome NAS, particularly with ZFS. There's some discussion about enabling more SATA in the kernel on Raspberry Pi OS at the moment if you want to chip in at https://github.com/raspberrypi/linux/pull/4256. The holy grail would be an array of NVMe SSDs, but this requires a PCIe switch which I'm not sure is currently fully supported.

User avatar
thagrol
Posts: 4714
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Wed Apr 21, 2021 3:49 pm

andrum99 wrote:
Wed Apr 21, 2021 2:51 pm
A CM4 with a bunch of SATA SSDs would make an awesome NAS, particularly with ZFS.
I'm not bothered about a pure NVMe solution. I'm quite happy with my SATA and spinning rust NAS (2 x 2.5" and 2 x 3.5" HDD).

Frankly, I'm not conviced that for my use case (centralised media and document storage) NVMe would improve performance enough to justify the extra cost and complexity. Sure, random access will likely be faster but in the end you're still limited by the x1 link back to the SoC. And that's more than enough to saturate the gigabit ethernet interface.
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

ejolson
Posts: 7093
Joined: Tue Mar 18, 2014 11:47 am

Re: Snapshot idea/solution to revert easily to clean / working Raspbian

Wed Apr 21, 2021 4:16 pm

thagrol wrote:
Wed Apr 21, 2021 3:49 pm
andrum99 wrote:
Wed Apr 21, 2021 2:51 pm
A CM4 with a bunch of SATA SSDs would make an awesome NAS, particularly with ZFS.
I'm not bothered about a pure NVMe solution. I'm quite happy with my SATA and spinning rust NAS (2 x 2.5" and 2 x 3.5" HDD).

Frankly, I'm not conviced that for my use case (centralised media and document storage) NVMe would improve performance enough to justify the extra cost and complexity. Sure, random access will likely be faster but in the end you're still limited by the x1 link back to the SoC. And that's more than enough to saturate the gigabit ethernet interface.
I have used BTRFS in two different ways to create clusters of Pi computers with system images that can have snapshots.
  • Read write BTRFS snapshots of root filesystems exported over NFS.
    • BTRFS as backing store for iSCSI root images copied using reflinks.
    The iSCSI method has the advantage that the nodes in the cluster can employ any native Linux filesystem on the resulting network block device as if it were a locally attached disk. The advantage of NFS is that files can be added from the NAS side while the export is mounted.

    Performance is generally better with iSCSI because the nodes use their local Linux buffer cache. Since iSCSI is strictly one target per initiator there is also no need to synchronize state across the network for concurrent access. This provides the effect of having local disks mounted on each node with the advantage that those disks can be accessed and restored from the NAS in case of an emergency.

    The NAS that I use for the iSCSI network block devices is based on an older AMD APU with spinning disks. It supports 20 systems. I suspect a Raspberry Pi 4B with suitable USB disk enclosure could also function as the server.

    andrum99
    Posts: 1415
    Joined: Fri Jul 20, 2012 2:41 pm

    Re: Snapshot idea/solution to revert easily to clean / working Raspbian

    Thu Apr 22, 2021 11:04 am

    thagrol wrote:
    Wed Apr 21, 2021 3:49 pm
    andrum99 wrote:
    Wed Apr 21, 2021 2:51 pm
    A CM4 with a bunch of SATA SSDs would make an awesome NAS, particularly with ZFS.
    I'm not bothered about a pure NVMe solution. I'm quite happy with my SATA and spinning rust NAS (2 x 2.5" and 2 x 3.5" HDD).

    Frankly, I'm not conviced that for my use case (centralised media and document storage) NVMe would improve performance enough to justify the extra cost and complexity. Sure, random access will likely be faster but in the end you're still limited by the x1 link back to the SoC. And that's more than enough to saturate the gigabit ethernet interface.
    Yes, the only real speed up with NVMe straight onto a PCIe bus versus SATA is to random IO. Even with the limitations of the single lane, random IO is faster, at least with a single SSD. I've not seen what happens if you use a PCIe switch to get more than one NVMe drive connected to the Pi's PCIe bus - not sure if anyone has tried this yet.

    Return to “Advanced users”