Raspberry Pi 3 B+ lockups

[ejolson]

What could you advice? Thanks.

Reflash the Raspbian image to your SD card, unplug network cable, boot Pi, change default password for pi user to something secure, then plug network cable back in.

[jbaumann]

Hi,

strangely, I observed the same problem on one of my RPi 3 (not B+). Is there a thread discussing this hardware or is this thread the only one?

NB: I'm using an original RPI Power Supply with 2.5A. I have tested this power supply with a number of other RPi3 with attached SSDs and other USB devices without any problems whatsoever. No HDMI connection, Ethernet, no WLAN or Bluetooth connected. Raspbian Lite (newest version from the server). Executed rpi-update as well and "sudo apt update; sudo apt upgrade" as well. The SD-card I use is a new SANDISK 32GB that works without problems in my other RPi3 systems, with or without load.

My problem with the RPi 3 (not B+):

In the default configuration the system crashes (with a kernel oops) after a short time if used with a USB device (e.g., an SSD). If I use an SD-card the system works until I put load on it. This can be something like "apt update, apt upgrade", a tensorflow object detection or memtester.

Code: Select all

arm_freq=1100
over_voltage on default value
over_voltage_sdram on default value

works for 24 hours under load (bonnie++ on usb ssd)

Code: Select all

arm_freq=1150 or default i.e., 1200
over_voltage on default value
over_voltage_sdram on default value

crashes after some time

Code: Select all

arm_freq on default value (1200)
over_voltage=1
over_voltage_sdram=1

works for 24 hours under load (memtester - 70 loops)

Code: Select all

arm_freq on default value (1200)
over_voltage=1
over_voltage_sdram on default value

currently under test

So it seems that either lowering the frequency from 1200-1100MHz is working or a moderate over voltage (if it needs the sdram overvoltage as well remains to be seen, my working hypothesis is that it should not be necessary).

Is there anything else I can do or any experiments that could help me determine how to run this board in a stable manner?

Regards, Joachim

[5ft24dave]

I'll give that a shot and post results

I haven't had any crashes with overvoltage set at 2.

Still no crashes/reboots, and arm_freq is at 1100 with the over voltages set to 2

[mattmiller]

Executed rpi-update

Never do this if you expect support - it gets you a bleedin edge kernel
Only do it if suggested by an RPi employee

PS And you should start a new thread as this one is for the pi3B+

[ejolson]

I'll give that a shot and post results

I haven't had any crashes with overvoltage set at 2.

Still no crashes/reboots, and arm_freq is at 1100 with the over voltages set to 2

Probably this thread is the most relevant for Pi 3B crashing under load. If you have the time, reading the entire thread is interesting. In summary, the trouble with the Pi 3B appears to be that the built-in 3V power supply was too simple to handle current fluctuations that resulted when switching clock rates while running optimized computational codes on all cores. The redesigned 3B+ has much more advanced 3V power circuitry.

My Pi 3B worked with overvolt 2, but I decided to underclock it to 900MHz for power savings and stability.

[5ft24dave]

I haven't had any crashes with overvoltage set at 2.

Still no crashes/reboots, and arm_freq is at 1100 with the over voltages set to 2

Probably this thread is the most relevant for Pi 3B crashing under load. If you have the time, reading the entire thread is interesting. In summary, the trouble with the Pi 3B appears to be that the built-in 3V power supply was too simple to handle current fluctuations that resulted when switching clock rates while running optimized computational codes on all cores. The redesigned 3B+ has much more advanced 3V power circuitry.

My Pi 3B worked with overvolt 2, but I decided to underclock it to 900MHz for power savings and stability.

Mine is a 3B+

[Tozzi1]

Well, I'm really sorry, but here in my scenario the B+ are still not running reliably.
It helps to do the rpi-update, but all in all, there has to be at least one bug still hidden somewhere in the drivers.
My Model 3B (non-plus) in the basement works completely reliably for months without reboots, still on Jessie.
Exactly what I used to love about Raspy Pi.
With the 3B+'s, I have to pull the plug every other day, and they get boiling hot whenever that happens.

I'll put my 3B+'s in a drawer for now and replace them with something else meanwhile until this has been figured out.
Can't really be bothered anymore and have got no more time to waste on this.
I'd love to help but can't see how...
Over and out.

[jamesh]

Well, I'm really sorry, but here in my scenario the B+ are still not running reliably.
It helps to do the rpi-update, but all in all, there has to be at least one bug still hidden somewhere in the drivers.
My Model 3B (non-plus) in the basement works completely reliably for months without reboots, still on Jessie.
Exactly what I used to love about Raspy Pi.
With the 3B+'s, I have to pull the plug every other day, and they get boiling hot whenever that happens.

I'll put my 3B+'s in a drawer for now and replace them with something else meanwhile until this has been figured out.
Can't really be bothered anymore and have got no more time to waste on this.
I'd love to help but can't see how...
Over and out.

We are not seeing any major problems with the 3b+ now - have them running on my desk 24/7 with no problems. So, would be useful to determine your exact circumstances. Any details you can share? Without them we won't be able to fix your issue.

[Tozzi1]

OK, I need to take back what I had said before, and I wish to apologise.

After your recent reply, I gave it a last chance, updated and upgraded to the latest Raspbian, including rpi-update, and now, all of a sudden, all my 3B+ are running flawlessly.
I don't really know what you fixed and when, but it looks like you've killed the bug(s) for good (fingers crossed).
No more "hickups", no more freezes so far. In fact, the overall performance has been awesome for the past 8 days 24/7.
Hope it stays that way.

Anyway, seems like I was wrong, and I wanted to admit to that.
The journey was a tough one, but "all's well that ends well"...
Keep up the good work!

[Tozzi1]

Well, and it still locks up. Sometimes it would run flawlessly for a week or two, then lock up every other day.
By now I am quite convinced that it must have to do with ethernet somehow.
I'll switch to Wifi and see what happens then. I've eliminated everything else by now and software side is being kept up-to-date, including rpi-update.

[DougieLawson]

You shouldn't run rpi-update unless an expert tells you to run it. You don't know what you're getting with rpi-update, it pulls the latest experimental kernel and bootcode. Every so often that will kill your system or disable something (last rpi-update failure stopped my AIY voicehat from working).

Use apt update ; apt -y dist-upgrade to keep current with the stable kernel & bootcode.

[e-raser]

Hi there,

I´m almost out of ideas and need expert´s help on this.

After many months of troubles since April 2018 (first bad hardware, than the power throttling with sdram frequency...) my Pi 3 B+ still is not running stable. And I´m very very very sure it´s because of my specific setup, having several hard disks attached to the pi via USB. I wanna remark that with the former Pi 2 B this NAS-Pi combo worked perfectly stable for more than 180 days constant uptime without any issues.

Back to now, first of all (so you expertes know this performance thing is sorted out): according to /boot/config.txt and "vcgencmd get_config int" I have "sdram_freq=450". This way the Pi runs stable if no USB HDDs are connected.

When all USB HDDs are connected and swapping is turned off, the following is happening every 12 hours and sometimes leads to a system freeze&reboot (after 3 to 4 days at the latest!):
- process kswapd0 is (most) active, consuming about up to 77 % of CPU and 0 % of RAM
- together with kswapd0 mmcqd/0 is/are also active
- system freezes totally (no screen freshup on SSH and even console)
- watching the Pi´s LEDs the microSD is under heavy/full load, leading to a almost permanent green LED
- sometimes the Pi survives this "kswapd" terror and comes back responsive after short time (about 5 minutes), and sometimes it reboots (I think because of the watchdog, which is set very friendly by the way)

uptime_month_2018-12-10.png (23 KiB) Viewed 1887 times

As it ran for 5 days now without having any USB hard disks attached to the Pi I have proof for this.

I have plenty of log data, webminstats graphics etc. and ran out of ideas. Currently I´m checking if maybe
- one specific SATA-to-USB-controller or
- any attached USB device or
- a combination of two or more USB hard disks
might be the cause, always taking 5 days (if it doesn´t freeze&reboot) each test.

I have no idea what kswapd is doing every 12 hours and why sd card is under heavy load, especially cause swapping is disabled.

Code: Select all

[email protected]: free -h
              total        used        free      shared  buff/cache   available
Mem:           939M        484M         84M         93M        370M        301M
Swap:            0B          0B          0B

Anyway, to summarize:

1) freezes&reboots occure when kswapd is punching in every 12 hours (at the latest after 3 to 4 days) if USB drives are attached and swapping is turned off.
2) so I point the finger to kswapd and USB (interrupt handling or sth. like that) on the Pi 3 B+ (firmware)

Your turn! Appreciate any help to get this Pi 3 B+ stable!!!

[Gavinmc42]

SDRAM clock is one useful tip, the other is lower the max CPU clock.
1.4GHz is the default but try 1.2GHz, I had to do both on one of my 3B+.

But how old is your 3B+? Early ones seemed to have more issues.
I think they do more testing these days.

[e-raser]

As I wrote several times: I sorted that clocking out already (intensively), Pi runs stable without USB HDDs attached. It‘s a RAM/swapping/USB issue, let‘s focus on that.

[e-raser]

No ideas? Guys I‘m totally lost and ran out ideas. What‘s the problem with handling USB devices for the Pi 3 B+?!?

[e-raser]

I think I "fixed" it by reverting the last change and enabling swapping again. Pi´s running for more than 6 days now (which is more than x2 the time it did before with those freeze/lock indicated reboots). Not 100 % sure system is running stable without any temporary issues or "only a minute lasting lockup", but anyway the few minutes freezes and reboots seem to be gone.

Off-topic: Don´t know if it´s the holiday season time or the fact this topic now gots 18 pages, but: no response after more than 14 days, so... thanks for nothing.

[jamesh]

I think I "fixed" it by reverting the last change and enabling swapping again. Pi´s running for more than 6 days now (which is more than x2 the time it did before with those freeze/lock indicated reboots). Not 100 % sure system is running stable without any temporary issues or "only a minute lasting lockup", but anyway the few minutes freezes and reboots seem to be gone.

Off-topic: Don´t know if it´s the holiday season time or the fact this topic now gots 18 pages, but: no response after more than 14 days, so... thanks for nothing.

Holiday season, yes, and perhaps no-one actually knows what the problem is?

[e-raser]

Holiday season, yes, and perhaps no-one actually knows what the problem is?

Happy holidays
Maybe "no-one" can or wants to read? See bold text (summary) at the end of viewtopic.php?p=1402659#p1402659, I spent a lot of time to collect data, diagnose behaviour and finally write all these information. No one read, no one responded. That´s life

But as I wrote: I don´t mind anymore and just updated this (by basically answering to myself) so anyone who might someday run into the same or similiar set of symptoms might have a chance.

[jamesh]

Holiday season, yes, and perhaps no-one actually knows what the problem is?

Happy holidays
Maybe "no-one" can or wants to read? See bold text (summary) at the end of viewtopic.php?p=1402659#p1402659, I spent a lot of time to collect data, diagnose behaviour and finally write all these information. No one read, no one responded. That´s life

But as I wrote: I don´t mind anymore and just updated this (by basically answering to myself) so anyone who might someday run into the same or similiar set of symptoms might have a chance.

Remember this is a general questions forum, we also have a GitHub bug tracking system that is probably more suitable for your report, and which Pi engineers are more likely to read.

[jlallen75]

I got a new 3 B+ a few days ago and am having a similar issue. I tried to install Hassio on it right out of the box and the install would never finish. It it just seemed to lock up at some point. I also wound up trying to install LibreElec and Raspbian and those two installed , but would eventually lock up after a minute of two. I noticed that they will only lock up using ethernet. I can boot into Raspbian and run a constant ping, but anymore traffic than that and it locks up solid. I even updating over ethernet locks it up.

To sum up what I've tried.
1. Tried the 2 SD Scards I had and then purchased 2 more to try.
2. Installed Hassio, LibreElec & Rasbian
3. Purchased an additional power supply and tried that.
4. Changed ethernet cable
5. Moved the connection from my Netgear switch to my Cisco switch.

I wound up getting a 2nd 3 B+ and that's had no issues using the same SD, Power supply and network connection so I'm guessing bad batch of hardware? Should I just send it back and try my luck on the return?

[jlallen75]

I returned it.

[jlallen75]

I got a new 3 B+ a few days ago and am having a similar issue. I tried to install Hassio on it right out of the box and the install would never finish. It it just seemed to lock up at some point. I also wound up trying to install LibreElec and Raspbian and those two installed , but would eventually lock up after a minute of two. I noticed that they will only lock up using ethernet. I can boot into Raspbian and run a constant ping, but anymore traffic than that and it locks up solid. I even updating over ethernet locks it up.

To sum up what I've tried.
1. Tried the 2 SD Scards I had and then purchased 2 more to try.
2. Installed Hassio, LibreElec & Rasbian
3. Purchased an additional power supply and tried that.
4. Changed ethernet cable
5. Moved the connection from my Netgear switch to my Cisco switch.

I wound up getting a 2nd 3 B+ and that's had no issues using the same SD, Power supply and network connection so I'm guessing bad batch of hardware? Should I just send it back and try my luck on the return?

I returned it.

[e-raser]

I´m trying with a different hardware (I guess my 3rd or 4th Pi 3 B+... what a pity) with same settings and same system (microSDXC) etc. to rule out any possible hardware cause. I think within 7 to 10 days I know if it´s a hardware of software thing.

[jamesh]

I got a new 3 B+ a few days ago and am having a similar issue. I tried to install Hassio on it right out of the box and the install would never finish. It it just seemed to lock up at some point. I also wound up trying to install LibreElec and Raspbian and those two installed , but would eventually lock up after a minute of two. I noticed that they will only lock up using ethernet. I can boot into Raspbian and run a constant ping, but anymore traffic than that and it locks up solid. I even updating over ethernet locks it up.

To sum up what I've tried.
1. Tried the 2 SD Scards I had and then purchased 2 more to try.
2. Installed Hassio, LibreElec & Rasbian
3. Purchased an additional power supply and tried that.
4. Changed ethernet cable
5. Moved the connection from my Netgear switch to my Cisco switch.

I wound up getting a 2nd 3 B+ and that's had no issues using the same SD, Power supply and network connection so I'm guessing bad batch of hardware? Should I just send it back and try my luck on the return?

I returned it.

Did any replacement work OK?

[jlallen75]

I got a new 3 B+ a few days ago and am having a similar issue. I tried to install Hassio on it right out of the box and the install would never finish. It it just seemed to lock up at some point. I also wound up trying to install LibreElec and Raspbian and those two installed , but would eventually lock up after a minute of two. I noticed that they will only lock up using ethernet. I can boot into Raspbian and run a constant ping, but anymore traffic than that and it locks up solid. I even updating over ethernet locks it up.

To sum up what I've tried.
1. Tried the 2 SD Scards I had and then purchased 2 more to try.
2. Installed Hassio, LibreElec & Rasbian
3. Purchased an additional power supply and tried that.
4. Changed ethernet cable
5. Moved the connection from my Netgear switch to my Cisco switch.

I wound up getting a 2nd 3 B+ and that's had no issues using the same SD, Power supply and network connection so I'm guessing bad batch of hardware? Should I just send it back and try my luck on the return?

I returned it.

Did any replacement work OK?

I didn't opt for a replacement, but the other one I got the same day has been going strong for days with no issues. Raspbian and Hassio had no issue using Ethernet.