The problem that we are facing is SD card corruption, specifically with the Raspbian operating system. We are using the Raspberry Pis (3B+) in environments where they may lose power at any moment. We need to use them headless and have no formal process to switch them off, we just pull the power.
We knew this was a problem and likely to be causing the corruption so introduced a read-only root file system within Raspbian. This is something I found online and we tested it as thoroughly as we could:
[removed bad link]
However, after several months of no corruption related issues, recently we have had several units fail. Despite protecting the root file system, this hasn’t been enough, so I believe we need to look at protecting the rest of the partitions (boot etc).
It also appears to be a time related issue. If a Pi is restarted in a short period of time, it will normally recover and restart. If one has been running for a long period of time and it then power cycled, there's a very slim chance (almost none) that it will reboot. For me, this points at some sort of internal logging building up over time. An issue which I thought would have been removed with a read-only root file system.
I am not a programmer by trade, I’m a product designer and have done what I can but have unfortunately reached the end of my knowledge. We’re looking for an expert in Linux and Raspbian to help us build a robust, incorruptible file system. The world seems to be full of examples of people running Raspberry Pis for years but we have struggled to get ours to run consistently.
I appreciate this is slightly unorthodox but I would really appreciate any help you can provide.