schadha
Posts: 5
Joined: Sun Dec 14, 2014 8:23 am

24x7x365 Products using RPI2-How to get rid of Kernel Panic

Thu Jan 28, 2016 2:48 am

About 2 years ago, we committed building a 24x7 product based on RPI and currently our product uses RPI2. We have not able to make our product reliable - We are not able to 100% get rid of - "End kernel panic - no syncing: VFS: Unable to mount fs on unknown-block ".

Our RPI based System:
1. RPI 2 -with USB WiFi, BLE, USB Flash, 4G SD Card
2. HAT board with Battery Backup + voice codex
3. 5v 2amp Power adapter - powered through the HAT board

We have done following to make our system robust
1. HAT board has a battery back up - power outages will not be noticed by the RPI
2. Our SD card is read only and at boot, uses the OS on the USB to boot.
3. USB Flash has 2 partitions
a. /dev/sda1 OS Partition - which is Read Only
b. /dev/sda2 Data Partition which is ReadWrite and contains all our data.
c. All applications have been parted on RW and RO basis. RW of data is done in the RW partition.

We can confirm that the SD card is intact. It is the USB Flash which gets corrupted. We are using the latest version of Debian
We have spent months to accomplish both 1 and 2 above. However we still about every 10-15 days get " End kernel panic - no syncing: VFS: Unable to mount fs on unknown-block" on one of our 10 RPI 2's used for testing.

We would appreciate if one of more these questions can be addressed:
1. Is there a known 24x7 system developed based on RPI 2 which is successfully running for months?
2. Is it possible to built commercially viable 24x7x365 products with RPI2?
3. What are steps we can perform to make our product 24x7x365 product?
4. What is the potential problem:[/list]
a. Raspian - is it robust
b. USB Flash it is made in China
c. Power supply - made in China

User avatar
rpdom
Posts: 15049
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: 24x7x365 Products using RPI2-How to get rid of Kernel Pa

Thu Jan 28, 2016 7:39 am

That message occurs during boot if the file system is damaged. If your systems are supposed to be 24x7x365, why are they rebooting? Power issues? That's the most common cause of failure in a Pi.

I only have six Pis (A, 2xB, B+, 2x2B). Four of those run 24x7 without problems using Raspbian (three on normal SD card, one on SDcard + USB hard disk). The only time they've rebooted in the last two years is when I've done a manual reboot, or during power failure.

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: 24x7x365 Products using RPI2-How to get rid of Kernel Pa

Thu Jan 28, 2016 10:31 am

Hi
How often and how much data is written to the USB flash? Is it possible that you have weared it up? Have you tried with USB sticks of different brands?
Author of the robust Nard distro http://www.nard.se

User avatar
RaTTuS
Posts: 10418
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: 24x7x365 Products using RPI2-How to get rid of Kernel Pa

Thu Jan 28, 2016 10:54 am

Code: Select all

uprecords
     #               Uptime | System                                     Boot up
----------------------------+---------------------------------------------------
->   1   302 days, 18:45:52 | Linux 3.18.7+             Tue Mar 31 17:05:46 2015
     2   130 days, 15:49:43 | Linux 3.6.11+             Wed Dec 18 19:52:36 2013
     3   112 days, 08:14:35 | Linux 3.6.11+             Wed Aug 28 12:33:35 2013
     4    91 days, 05:20:26 | Linux 3.12.24+            Mon Jul 21 10:54:30 2014
     5    77 days, 00:36:37 | Linux 3.12.24+            Mon Oct 20 16:15:16 2014
     6    71 days, 22:59:51 | Linux 3.6.11+             Tue Mar 12 08:37:44 2013
     7    69 days, 20:43:58 | Linux 3.12.35+            Mon Jan  5 15:52:24 2015
     8    43 days, 23:35:53 | Linux 3.10.25+            Mon Apr 28 12:44:21 2014
     9    39 days, 21:42:05 | Linux 3.12.20+            Wed Jun 11 12:20:40 2014
    10    33 days, 23:26:17 | Linux 3.6.11+             Wed Jul 24 12:35:39 2013
----------------------------+---------------------------------------------------
NewRec   172 days, 02:56:08 | since                     Sun Aug  9 08:55:29 2015
    up   1268 days, 12:29:3 | since                     Tue Jul 17 12:46:07 2012
  down    21 days, 10:35:52 | since                     Tue Jul 17 12:46:07 2012
   %up               98.338 | since                     Tue Jul 17 12:46:07 2012
don't reboot if you want to stay up.
if you have battery backed system then when you switch to battery shutdown the system cleanly
when the power comes back reboot.

usb flash will corrupt as much as sdcard [they are the same type of system] if you remove power then you have a chance of corruption if the system is running.
you can force a filesystem check each boot if you require
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

JF002
Posts: 94
Joined: Sat Feb 04, 2012 8:49 am
Contact: Website

Re: 24x7x365 Products using RPI2-How to get rid of Kernel Pa

Thu Jan 28, 2016 9:23 pm

USB Flash drives use the same memory technology than SDCard, and they have the same "weaknesses": they support a limited number of write/erase cycles on each memory cells.

The error message you get could be caused by a filesystem corruption because of a faulty flash drive.

I once tried to set up 2 rpi as servers running H24 : one using a very low-cost SDcard, and the second one using a Sandisk extreme pro (something like that) SDcard.
The first one died within 2 weeks, the second one is still running for more than a year.

Conclusion : do not try to save money on SDcard/Flash drives!
My web site : https://codingfield.com

FM81
Posts: 518
Joined: Wed Apr 17, 2013 4:33 pm

Re: 24x7x365 Products using RPI2-How to get rid of Kernel Pa

Sat Jan 30, 2016 12:10 am

@ TO - three ideas from my side:
- mount with option sync - especially the RW-parts
- write "small pieces of data" to a RAM disk instead and copy them to USB in bigger blocks only
- make very sure, that during reboot/shutdown everything is REALLY synced and clean unmounted

Greetings, FM_81
A: What does the command 'cat /dev/urandom', can you tell me please?
B: Yeah, that's very simple: It feeds your cat with radioactive material!

MaxK1
Posts: 1043
Joined: Sun Aug 26, 2012 11:34 pm

Re: 24x7x365 Products using RPI2-How to get rid of Kernel Pa

Sat Jan 30, 2016 1:04 pm

You want space/industrial/medical grade relliability? From parts and pieces you can buy at Walmart?
You are in a maze of twisty little passages, all alike.
When General Failure and Major Disaster get together, Private Parts usually suffers.

User avatar
GTR2Fan
Posts: 1601
Joined: Sun Feb 23, 2014 9:20 pm
Location: South East UK

Re: 24x7x365 Products using RPI2-How to get rid of Kernel Pa

Sat Jan 30, 2016 1:39 pm

schadha wrote:Our RPI based System:
1. RPI 2 -with USB WiFi, BLE, USB Flash, 4G SD Card
2. HAT board with Battery Backup + voice codex
3. 5v 2amp Power adapter - powered through the HAT board
This doesn't actually tell us a great deal.

1/ Which exact make and model of microSD card? There are plenty of cheap 4GB cards from China that I wouldn't trust in a camera, let alone in a Pi.

2/ Which HAT board with battery backup? Are you sure that the supply to the Pi isn't glitching if/when the mains power fails and the battery takes over?

3/ Which 5V 2A power adapter? Have you checked that it actually delivers 5V (+/- 0.25V) at all times, preferably at the power pins on the GPIO header?

Being able to answer these questions would be a good start.
Pi2B Mini-PC/Media Centre: ARM=1GHz (+3), Core=500MHz, v3d=500MHz, h264=333MHz, RAM=DDR2-1200 (+6/+4/+4+schmoo). Sandisk Ultra HC-I 32GB microSD card on '50=100' OCed slot (42MB/s read) running Raspbian/KODI16, Seagate 3.5" 1.5TB HDD mass storage.

Return to “Advanced users”