richdyer_2000
Posts: 4
Joined: Mon Sep 26, 2016 9:09 pm

Running Raspberry Pis Cold Redundant

Tue Sep 27, 2016 10:34 am

So, I've set up a Raspberry Pi to control all our shutters at home. It's been working flawlessly (with the exception of my own coding bugs) for 6 months, but this week I got hit by a corrupt SD Card. I was savvy enough to have saved an image, so getting it going again wasn't a problem. My concern is really about reliability for when we're on holidays etc.

My idea was to have 2 Raspberry Pis in cold redundancy with an Arduino acting as a watchdog with some very simple logic. If it gets no response from Pi A after X seconds, it sends a signal for the Pi to do an orderly shutdown (seems possible after a very quick search through posts) and then toggles the power via a physical relay to reboot it. Still no response, it cuts Power to Pi A and switches Pi B on.

My questions are:
- Does anyone have experience with this sort of setup or is there anything obviously wrong with it?
- Does anyone know where I can get a split 40 pin ribbon, or if they even exist?
- Is there anyway to improve my SD card reliability? The only thing I can think of is that I'm using a pair of DS18B20 Temperature sensors which are continuously writing to a file, other than that there's very little activity.

Cheers

Rich

User avatar
DougieLawson
Posts: 40214
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Running Raspberry Pis Cold Redundant

Tue Sep 27, 2016 11:38 am

How do you plan to redundently share your data from Pi A to Pi B? What failover support do you plan to use for your data server?
Criticising any questions is banned on this forum.

Any DMs sent on Twitter will be answered next month.
All fake doctors are on my foes list.

Note: Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

peterlite
Posts: 720
Joined: Sun Apr 17, 2016 4:00 am

Re: Running Raspberry Pis Cold Redundant

Tue Sep 27, 2016 11:56 am

An SD card can look bad after a momentary power fluctuation. It is usually a corrupt write. Reloading the system from your backup image usually fixes it. A UPS is one answer.

If you have two machines running from one mains power, they can both die at the same time. A UPS could help. There are many posts on using small batteries for a 20 minute standby. I am looking at a 12 volt motor cycle battery, charger combo for outages lasting hours.

richdyer_2000
Posts: 4
Joined: Mon Sep 26, 2016 9:09 pm

Re: Running Raspberry Pis Cold Redundant

Tue Sep 27, 2016 2:28 pm

DougieLawson wrote:How do you plan to redundently share your data from Pi A to Pi B? What failover support do you plan to use for your data server?
Hi Doug,

There's only one file I would potentially need to share - it describes the dynamic config (default open close times etc which are changed from a web GUI) and allows the system to resume from a power down.

I could share this file on my home NAS (Synology with RAID 5) to allow a seamless swap, however my backup would just be to use a default location on the SD card if the NAS wasn't available, accepting the limitation that I might not have the latest configuration. The project has various bells and whistles which need internet access etc (e.g for public holidays), but my philosophy is that basic functionality should be preserved when it's isolated.

Cheers

Rich

richdyer_2000
Posts: 4
Joined: Mon Sep 26, 2016 9:09 pm

Re: Running Raspberry Pis Cold Redundant

Tue Sep 27, 2016 2:46 pm

peterlite wrote:An SD card can look bad after a momentary power fluctuation. It is usually a corrupt write. Reloading the system from your backup image usually fixes it. A UPS is one answer.
Thanks - i'll take a look at that!
peterlite wrote:If you have two machines running from one mains power, they can both die at the same time
Actually, I was planning to have them cold redundant - i.e. only one on at a time. Their GPIOs would be joined which I guess could cause all manner of issues if they're on at the same time. My idea was to have the Arduino (which I'm assuming is more reliable than the RPi/SD Card combo) controlling 3 relays as attached. One to select between PiA and PiB, and then one each to independently control power.
Attachments
IMG_2969.JPG
Power Relays
IMG_2969.JPG (32.71 KiB) Viewed 3014 times

User avatar
davidcoton
Posts: 5503
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK
Contact: Website

Re: Running Raspberry Pis Cold Redundant

Tue Sep 27, 2016 3:29 pm

The point about a UPS is that it will reduce the likelihood of SD corruption. There are other solutions (make the SD read only), but they introduce other problems.

With a UPS you are unlikely to need two Pis and a switchover monitor. However, if you do still want redundancy, two power relays (one for each Pi) is enough -- the changeover relay is redundant. The Arduino can make sure that both are not simultaneously active. Yopu will also need some way to safely combine the Pi outputs, so that a failed or unpowered Pi will not prevent correct operation of the system.
Location: 345th cell on the right of the 210th row of L2 cache

richdyer_2000
Posts: 4
Joined: Mon Sep 26, 2016 9:09 pm

Re: Running Raspberry Pis Cold Redundant

Thu Sep 29, 2016 3:53 pm

davidcoton wrote: However, if you do still want redundancy, two power relays (one for each Pi) is enough -- the changeover relay is redundant. The Arduino can make sure that both are not simultaneously active.
You're right and that was my original idea...but the changeover relay means that this is ensured at a H/W level which is safer, especially considering who will be writing the S/W! It'll also help during testing, or any software updates etc. Putting the power relay before the changeover relay would mean I just have 2 relays and can still do the same job though.
davidcoton wrote: Yopu will also need some way to safely combine the Pi outputs, so that a failed or unpowered Pi will not prevent correct operation of the system.
This is the real crux I think - I was wondering how important it would be to isolate the un-powered Pi from the system, but now I realise the "un-powered" Pi would actually be powered via the 5V and GND pins of the other Pi.

I'm guessing that means a changeover relay for every GPIO pin I'm using....

MarkTF
Posts: 318
Joined: Tue Mar 03, 2015 4:59 pm

Re: Running Raspberry Pis Cold Redundant

Thu Sep 29, 2016 4:23 pm

RPi has a reset pin so the backup could be powered, but held in the reset state by the Arduino until needed. Since RPi GPIOs should be in the input state while the board is held in reset, you could simply wire the outputs of the two RPi boards together if it can be guaranteed that no more than one will be allowed to run at a time.

Some potential hazards that come to mind with this approach:
1) Is the sdcard susceptible to corruption on loss of power if the RPi is held in reset? (I think not)
2) Does the Arduino boot into user code quickly enough to keep the RPis in reset before they do something that writes to the sdcard?
3) What happens if the Arduino fails?

User avatar
davidcoton
Posts: 5503
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK
Contact: Website

Re: Running Raspberry Pis Cold Redundant

Thu Sep 29, 2016 4:36 pm

richdyer_2000 wrote:m guessing that means a changeover relay for every GPIO pin I'm using....
First things first -- unless you connect the 5V (or 3V3) rails, one Pi will not power the other through GPIO connections.
But an unpowered Pi connected to a powered one is not a good idea. The GPIOs of the unpowered Pi are not guaranteed to be in a high impedance state.

Relays are not the best choice for switching inputs and outputs in this context. Mechanical devices are inherently less reliable than electronics, so you will soon remove any benefit of a standby Pi. Ideally you need some suitable logic gates, either from a suitable chip family or good old diode-resistor logic. You just need to be careful that the inactive Pi is not affecting the output. MarkTF's suggestion is also worth following up.

Although standby designs a re instructive and fun to build, I would suggest trying a UPS with a single Pi first and seeing if that will meet your reliability goal. Using the Arduino watchdog to monitor and reboot (power cycle) the Pi would be the next step. A dual Pi solution would only come into the equation if that was still insufficiently reliable -- but at that level redundancy is not as easy as it sounds -- it is extremely difficult to anticipate and allow for every possible fault, and the one you miss will be the one that happens.
Location: 345th cell on the right of the 210th row of L2 cache

mutley
Posts: 61
Joined: Sat Jan 02, 2016 8:06 pm

Re: Running Raspberry Pis Cold Redundant

Thu Sep 29, 2016 5:12 pm

richdyer_2000 wrote: - Is there anyway to improve my SD card reliability? The only thing I can think of is that I'm using a pair of DS18B20 Temperature sensors which are continuously writing to a file, other than that there's very little activity.
My personal preference with fighting numerous SD card corruptions is to use read-only root (either CF or USB) / ramdisks for tmp, log & lock files and have a data partition on a USB stick in RW mode. Very important to have 2 physically different pieces of hardware for root mounded ro and data mounted rw, do not use one disk with two different partitions (unless you are using a spinning hard disk).
I never worry about power issues now and have not had one bad root fs in 3 years on 4 different pi's running 24/7 after switching to this method.

ejolson
Posts: 6019
Joined: Tue Mar 18, 2014 11:47 am

Re: Running Raspberry Pis Cold Redundant

Thu Sep 29, 2016 5:29 pm

richdyer_2000 wrote:I got hit by a corrupt SD Card.
It seems the sdcard is a common point of failure for Pi. Many embedded processors and control systems that have been commercially designed rely on flash memory. However, during normal operation the flash memory is read only. The flash memory is written only when the firmware is updated or the settings changed. If you've ever updated the firmware in a DVD player or router you will notice ask sorts of warning not to turn of the power during the update.

Flash cards are generally quite reliable, because when written to, the wear leveling algorithms in the card copy often modified data to less used parts of the card and seldom modified data to areas of the card that are wearing out. Infuriatingly, the seldom modified parts of a Raspbian image are often core components of the operating system. Therefore, any write to an sdcard introduces a possibly that those core files are corrupted. Unlike traditional harddrives, this problem persists even if the files are in separate partitions.

Reliability could be increased by mounting the sdcard read only. Most Linux distributions don't tolerate this; however, PiNet, live DVDs for PCs and the method mentioned in the previous post employ a read/write overlay in RAM to make a read-only root filesystem appear writable. Simple modifications allow Raspbian to run for days in this way and scheduled reboots can alleviate the tendency for RAM to run out. Putting swap and persistent user-writable parts of the filesystem on a separate thumb drive as suggested in the previous post is also a good idea.
Last edited by ejolson on Thu Sep 29, 2016 5:43 pm, edited 2 times in total.

Heater
Posts: 16846
Joined: Tue Jul 17, 2012 3:02 pm

Re: Running Raspberry Pis Cold Redundant

Thu Sep 29, 2016 5:39 pm

Make your Raspbian file system on SD read only. It's pretty straight forward: https://wiki.debian.org/ReadonlyRoot

Keep any data you have to write on a different media. A USB stick say.

Preferably send such that data out of the machine, over the net, and to a server.
Memory in C++ is a leaky abstraction .

ejolson
Posts: 6019
Joined: Tue Mar 18, 2014 11:47 am

Re: Running Raspberry Pis Cold Redundant

Thu Sep 29, 2016 5:49 pm

Heater wrote:It's pretty straight forward: https://wiki.debian.org/ReadonlyRoot
The method in this link dates back to before writable overlay support in the Linux kernel was reliable. Using a writable overlay is much easier with no special cases to be considered on an application by application level. There appears to be a link to using Aufs on Ubuntu at the end of the howto. I recommend overlayfs with a recent kernel.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9917
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Running Raspberry Pis Cold Redundant

Thu Sep 29, 2016 6:07 pm

davidcoton wrote:First things first -- unless you connect the 5V (or 3V3) rails, one Pi will not power the other through GPIO connections.
But an unpowered Pi connected to a powered one is not a good idea. The GPIOs of the unpowered Pi are not guaranteed to be in a high impedance state.
Worse is that the ESD diodes on GPIO lines may well end up conducting, because what is normally at 3.3V or 5V is now at 0V. You need to really know the internals of each pin before you hold them high with the device off.
I have had a device where that parasitic leakage back onto the power rails has been enough to upset device operation when it is powered up.
Last edited by 6by9 on Thu Sep 29, 2016 7:30 pm, edited 1 time in total.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Heater
Posts: 16846
Joined: Tue Jul 17, 2012 3:02 pm

Re: Running Raspberry Pis Cold Redundant

Thu Sep 29, 2016 6:17 pm

ejolson,

It's true that newer file systems can do what we want without the hassle of the instructions I linked to. Which are not much hassle actually.

I was thinking that a change of root file system type is a bit beyond most Pi users new to Linux. Especially if it needs a kernel update.
Memory in C++ is a leaky abstraction .

ejolson
Posts: 6019
Joined: Tue Mar 18, 2014 11:47 am

Re: Running Raspberry Pis Cold Redundant

Thu Sep 29, 2016 7:42 pm

Heater wrote:ejolson,

It's true that newer file systems can do what we want without the hassle of the instructions I linked to. Which are not much hassle actually.

I was thinking that a change of root file system type is a bit beyond most Pi users new to Linux. Especially if it needs a kernel update.
While read-only root may be beyond most Pi users, it should be normal for anyone developing embedded applications. It appears overlayfs is already compiled into the kernel. I've just created a thread describing how to do use overlayfs with a read-only sdcard.

Return to “General discussion”