ejolson
Posts: 3791
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi 3 B+ lockups

Fri Aug 03, 2018 6:02 am

kostik2022 wrote:
Fri Aug 03, 2018 2:11 am
What could you advice? Thanks.
Reflash the Raspbian image to your SD card, unplug network cable, boot Pi, change default password for pi user to something secure, then plug network cable back in.

jbaumann
Posts: 2
Joined: Mon Sep 17, 2018 9:14 am

Re: Raspberry Pi 3 B+ lockups

Mon Sep 17, 2018 2:55 pm

Hi,

strangely, I observed the same problem on one of my RPi 3 (not B+). Is there a thread discussing this hardware or is this thread the only one?

NB: I'm using an original RPI Power Supply with 2.5A. I have tested this power supply with a number of other RPi3 with attached SSDs and other USB devices without any problems whatsoever. No HDMI connection, Ethernet, no WLAN or Bluetooth connected. Raspbian Lite (newest version from the server). Executed rpi-update as well and "sudo apt update; sudo apt upgrade" as well. The SD-card I use is a new SANDISK 32GB that works without problems in my other RPi3 systems, with or without load.

My problem with the RPi 3 (not B+):

In the default configuration the system crashes (with a kernel oops) after a short time if used with a USB device (e.g., an SSD). If I use an SD-card the system works until I put load on it. This can be something like "apt update, apt upgrade", a tensorflow object detection or memtester.

Code: Select all

arm_freq=1100
over_voltage on default value
over_voltage_sdram on default value
works for 24 hours under load (bonnie++ on usb ssd)

Code: Select all

arm_freq=1150 or default i.e., 1200
over_voltage on default value
over_voltage_sdram on default value
crashes after some time

Code: Select all

arm_freq on default value (1200)
over_voltage=1
over_voltage_sdram=1
works for 24 hours under load (memtester - 70 loops)

Code: Select all

arm_freq on default value (1200)
over_voltage=1
over_voltage_sdram on default value
currently under test

So it seems that either lowering the frequency from 1200-1100MHz is working or a moderate over voltage (if it needs the sdram overvoltage as well remains to be seen, my working hypothesis is that it should not be necessary).

Is there anything else I can do or any experiments that could help me determine how to run this board in a stable manner?

Regards, Joachim

5ft24dave
Posts: 41
Joined: Tue Jun 12, 2018 3:38 am

Re: Raspberry Pi 3 B+ lockups

Mon Sep 17, 2018 7:24 pm

5ft24dave wrote:
Fri Jul 13, 2018 2:34 am
5ft24dave wrote:
Sun Jul 08, 2018 9:42 pm
I'll give that a shot and post results
I haven't had any crashes with overvoltage set at 2.
Still no crashes/reboots, and arm_freq is at 1100 with the over voltages set to 2

mattmiller
Posts: 2125
Joined: Thu Feb 05, 2015 11:25 pm

Re: Raspberry Pi 3 B+ lockups

Mon Sep 17, 2018 9:01 pm

Executed rpi-update
Never do this if you expect support - it gets you a bleedin edge kernel
Only do it if suggested by an RPi employee

PS And you should start a new thread as this one is for the pi3B+

ejolson
Posts: 3791
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi 3 B+ lockups

Tue Sep 18, 2018 5:11 am

5ft24dave wrote:
Mon Sep 17, 2018 7:24 pm
5ft24dave wrote:
Fri Jul 13, 2018 2:34 am
5ft24dave wrote:
Sun Jul 08, 2018 9:42 pm
I'll give that a shot and post results
I haven't had any crashes with overvoltage set at 2.
Still no crashes/reboots, and arm_freq is at 1100 with the over voltages set to 2
Probably this thread is the most relevant for Pi 3B crashing under load. If you have the time, reading the entire thread is interesting. In summary, the trouble with the Pi 3B appears to be that the built-in 3V power supply was too simple to handle current fluctuations that resulted when switching clock rates while running optimized computational codes on all cores. The redesigned 3B+ has much more advanced 3V power circuitry.

My Pi 3B worked with overvolt 2, but I decided to underclock it to 900MHz for power savings and stability.

5ft24dave
Posts: 41
Joined: Tue Jun 12, 2018 3:38 am

Re: Raspberry Pi 3 B+ lockups

Tue Sep 18, 2018 5:26 am

ejolson wrote:
Tue Sep 18, 2018 5:11 am
5ft24dave wrote:
Mon Sep 17, 2018 7:24 pm
5ft24dave wrote:
Fri Jul 13, 2018 2:34 am


I haven't had any crashes with overvoltage set at 2.
Still no crashes/reboots, and arm_freq is at 1100 with the over voltages set to 2
Probably this thread is the most relevant for Pi 3B crashing under load. If you have the time, reading the entire thread is interesting. In summary, the trouble with the Pi 3B appears to be that the built-in 3V power supply was too simple to handle current fluctuations that resulted when switching clock rates while running optimized computational codes on all cores. The redesigned 3B+ has much more advanced 3V power circuitry.

My Pi 3B worked with overvolt 2, but I decided to underclock it to 900MHz for power savings and stability.
Mine is a 3B+

Tozzi1
Posts: 21
Joined: Thu Apr 20, 2017 11:05 pm

Re: Raspberry Pi 3 B+ lockups

Fri Oct 12, 2018 12:06 am

Well, I'm really sorry, but here in my scenario the B+ are still not running reliably.
It helps to do the rpi-update, but all in all, there has to be at least one bug still hidden somewhere in the drivers.
My Model 3B (non-plus) in the basement works completely reliably for months without reboots, still on Jessie.
Exactly what I used to love about Raspy Pi.
With the 3B+'s, I have to pull the plug every other day, and they get boiling hot whenever that happens.

I'll put my 3B+'s in a drawer for now and replace them with something else meanwhile until this has been figured out.
Can't really be bothered anymore and have got no more time to waste on this.
I'd love to help but can't see how...
Over and out.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24116
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Fri Oct 12, 2018 9:01 am

Tozzi1 wrote:
Fri Oct 12, 2018 12:06 am
Well, I'm really sorry, but here in my scenario the B+ are still not running reliably.
It helps to do the rpi-update, but all in all, there has to be at least one bug still hidden somewhere in the drivers.
My Model 3B (non-plus) in the basement works completely reliably for months without reboots, still on Jessie.
Exactly what I used to love about Raspy Pi.
With the 3B+'s, I have to pull the plug every other day, and they get boiling hot whenever that happens.

I'll put my 3B+'s in a drawer for now and replace them with something else meanwhile until this has been figured out.
Can't really be bothered anymore and have got no more time to waste on this.
I'd love to help but can't see how...
Over and out.
We are not seeing any major problems with the 3b+ now - have them running on my desk 24/7 with no problems. So, would be useful to determine your exact circumstances. Any details you can share? Without them we won't be able to fix your issue.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

Tozzi1
Posts: 21
Joined: Thu Apr 20, 2017 11:05 pm

Re: Raspberry Pi 3 B+ lockups

Sat Oct 20, 2018 12:27 am

OK, I need to take back what I had said before, and I wish to apologise.

After your recent reply, I gave it a last chance, updated and upgraded to the latest Raspbian, including rpi-update, and now, all of a sudden, all my 3B+ are running flawlessly.
I don't really know what you fixed and when, but it looks like you've killed the bug(s) for good (fingers crossed).
No more "hickups", no more freezes so far. In fact, the overall performance has been awesome for the past 8 days 24/7.
Hope it stays that way. 8-)

Anyway, seems like I was wrong, and I wanted to admit to that.
The journey was a tough one, but "all's well that ends well"...
Keep up the good work!

Tozzi1
Posts: 21
Joined: Thu Apr 20, 2017 11:05 pm

Re: Raspberry Pi 3 B+ lockups

Mon Oct 29, 2018 2:48 am

Well, and it still locks up. Sometimes it would run flawlessly for a week or two, then lock up every other day.
By now I am quite convinced that it must have to do with ethernet somehow.
I'll switch to Wifi and see what happens then. I've eliminated everything else by now and software side is being kept up-to-date, including rpi-update.

User avatar
DougieLawson
Posts: 36512
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Raspberry Pi 3 B+ lockups

Mon Oct 29, 2018 9:00 am

You shouldn't run rpi-update unless an expert tells you to run it. You don't know what you're getting with rpi-update, it pulls the latest experimental kernel and bootcode. Every so often that will kill your system or disable something (last rpi-update failure stopped my AIY voicehat from working).

Use apt update ; apt -y dist-upgrade to keep current with the stable kernel & bootcode.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

e-raser
Posts: 71
Joined: Sat Apr 04, 2015 1:30 pm

Re: Raspberry Pi 3 B+ lockups

Mon Dec 10, 2018 5:31 pm

Hi there,

I´m almost out of ideas and need expert´s help on this.

After many months of troubles since April 2018 (first bad hardware, than the power throttling with sdram frequency...) my Pi 3 B+ still is not running stable. And I´m very very very sure it´s because of my specific setup, having several hard disks attached to the pi via USB. I wanna remark that with the former Pi 2 B this NAS-Pi combo worked perfectly stable for more than 180 days constant uptime without any issues.

Back to now, first of all (so you expertes know this performance thing is sorted out): according to /boot/config.txt and "vcgencmd get_config int" I have "sdram_freq=450". This way the Pi runs stable if no USB HDDs are connected.

When all USB HDDs are connected and swapping is turned off, the following is happening every 12 hours and sometimes leads to a system freeze&reboot (after 3 to 4 days at the latest!):
- process kswapd0 is (most) active, consuming about up to 77 % of CPU and 0 % of RAM
- together with kswapd0 mmcqd/0 is/are also active
- system freezes totally (no screen freshup on SSH and even console)
- watching the Pi´s LEDs the microSD is under heavy/full load, leading to a almost permanent green LED
- sometimes the Pi survives this "kswapd" terror and comes back responsive after short time (about 5 minutes), and sometimes it reboots (I think because of the watchdog, which is set very friendly by the way)
uptime_month_2018-12-10.png
uptime_month_2018-12-10.png (23 KiB) Viewed 1872 times
As it ran for 5 days now without having any USB hard disks attached to the Pi I have proof for this.

I have plenty of log data, webminstats graphics etc. and ran out of ideas. Currently I´m checking if maybe
- one specific SATA-to-USB-controller or
- any attached USB device or
- a combination of two or more USB hard disks
might be the cause, always taking 5 days (if it doesn´t freeze&reboot) each test.

I have no idea what kswapd is doing every 12 hours and why sd card is under heavy load, especially cause swapping is disabled.

Code: Select all

[email protected]: free -h
              total        used        free      shared  buff/cache   available
Mem:           939M        484M         84M         93M        370M        301M
Swap:            0B          0B          0B
Anyway, to summarize:

1) freezes&reboots occure when kswapd is punching in every 12 hours (at the latest after 3 to 4 days) if USB drives are attached and swapping is turned off.
2) so I point the finger to kswapd and USB (interrupt handling or sth. like that) on the Pi 3 B+ (firmware)


Your turn! Appreciate any help to get this Pi 3 B+ stable!!!
1x Nextcloud & Pi-hole & ... on Raspbian @ Pi (4 4 GB)
1x Kodi media center on LibreELEC @ Pi 3 B+

User avatar
Gavinmc42
Posts: 4016
Joined: Wed Aug 28, 2013 3:31 am

Re: Raspberry Pi 3 B+ lockups

Mon Dec 10, 2018 11:41 pm

SDRAM clock is one useful tip, the other is lower the max CPU clock.
1.4GHz is the default but try 1.2GHz, I had to do both on one of my 3B+.

But how old is your 3B+? Early ones seemed to have more issues.
I think they do more testing these days.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

e-raser
Posts: 71
Joined: Sat Apr 04, 2015 1:30 pm

Re: Raspberry Pi 3 B+ lockups

Tue Dec 11, 2018 8:33 am

As I wrote several times: I sorted that clocking out already (intensively), Pi runs stable without USB HDDs attached. It‘s a RAM/swapping/USB issue, let‘s focus on that.
1x Nextcloud & Pi-hole & ... on Raspbian @ Pi (4 4 GB)
1x Kodi media center on LibreELEC @ Pi 3 B+

e-raser
Posts: 71
Joined: Sat Apr 04, 2015 1:30 pm

Re: Raspberry Pi 3 B+ lockups

Fri Dec 21, 2018 12:24 pm

No ideas? Guys I‘m totally lost and ran out ideas. What‘s the problem with handling USB devices for the Pi 3 B+?!?
1x Nextcloud & Pi-hole & ... on Raspbian @ Pi (4 4 GB)
1x Kodi media center on LibreELEC @ Pi 3 B+

e-raser
Posts: 71
Joined: Sat Apr 04, 2015 1:30 pm

Re: Raspberry Pi 3 B+ lockups

Fri Dec 28, 2018 8:07 pm

I think I "fixed" it by reverting the last change and enabling swapping again. Pi´s running for more than 6 days now (which is more than x2 the time it did before with those freeze/lock indicated reboots). Not 100 % sure system is running stable without any temporary issues or "only a minute lasting lockup", but anyway the few minutes freezes and reboots seem to be gone.

Off-topic: Don´t know if it´s the holiday season time or the fact this topic now gots 18 pages, but: no response after more than 14 days, so... thanks for nothing.
1x Nextcloud & Pi-hole & ... on Raspbian @ Pi (4 4 GB)
1x Kodi media center on LibreELEC @ Pi 3 B+

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24116
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Fri Dec 28, 2018 11:32 pm

e-raser wrote:
Fri Dec 28, 2018 8:07 pm
I think I "fixed" it by reverting the last change and enabling swapping again. Pi´s running for more than 6 days now (which is more than x2 the time it did before with those freeze/lock indicated reboots). Not 100 % sure system is running stable without any temporary issues or "only a minute lasting lockup", but anyway the few minutes freezes and reboots seem to be gone.

Off-topic: Don´t know if it´s the holiday season time or the fact this topic now gots 18 pages, but: no response after more than 14 days, so... thanks for nothing.
Holiday season, yes, and perhaps no-one actually knows what the problem is?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

e-raser
Posts: 71
Joined: Sat Apr 04, 2015 1:30 pm

Re: Raspberry Pi 3 B+ lockups

Sat Dec 29, 2018 12:49 am

jamesh wrote:
Fri Dec 28, 2018 11:32 pm
Holiday season, yes, and perhaps no-one actually knows what the problem is?
Happy holidays :-)
Maybe "no-one" can or wants to read? See bold text (summary) at the end of viewtopic.php?p=1402659#p1402659, I spent a lot of time to collect data, diagnose behaviour and finally write all these information. No one read, no one responded. That´s life :D

But as I wrote: I don´t mind anymore and just updated this (by basically answering to myself) so anyone who might someday run into the same or similiar set of symptoms might have a chance.
1x Nextcloud & Pi-hole & ... on Raspbian @ Pi (4 4 GB)
1x Kodi media center on LibreELEC @ Pi 3 B+

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24116
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Sat Dec 29, 2018 10:24 am

e-raser wrote:
Sat Dec 29, 2018 12:49 am
jamesh wrote:
Fri Dec 28, 2018 11:32 pm
Holiday season, yes, and perhaps no-one actually knows what the problem is?
Happy holidays :-)
Maybe "no-one" can or wants to read? See bold text (summary) at the end of viewtopic.php?p=1402659#p1402659, I spent a lot of time to collect data, diagnose behaviour and finally write all these information. No one read, no one responded. That´s life :D

But as I wrote: I don´t mind anymore and just updated this (by basically answering to myself) so anyone who might someday run into the same or similiar set of symptoms might have a chance.
Remember this is a general questions forum, we also have a GitHub bug tracking system that is probably more suitable for your report, and which Pi engineers are more likely to read.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

jlallen75
Posts: 4
Joined: Sat Dec 29, 2018 6:08 pm

Re: Raspberry Pi 3 B+ lockups

Sat Dec 29, 2018 6:32 pm

I got a new 3 B+ a few days ago and am having a similar issue. I tried to install Hassio on it right out of the box and the install would never finish. It it just seemed to lock up at some point. I also wound up trying to install LibreElec and Raspbian and those two installed , but would eventually lock up after a minute of two. I noticed that they will only lock up using ethernet. I can boot into Raspbian and run a constant ping, but anymore traffic than that and it locks up solid. I even updating over ethernet locks it up.

To sum up what I've tried.
1. Tried the 2 SD Scards I had and then purchased 2 more to try.
2. Installed Hassio, LibreElec & Rasbian
3. Purchased an additional power supply and tried that.
4. Changed ethernet cable
5. Moved the connection from my Netgear switch to my Cisco switch.

I wound up getting a 2nd 3 B+ and that's had no issues using the same SD, Power supply and network connection so I'm guessing bad batch of hardware? Should I just send it back and try my luck on the return?

jlallen75
Posts: 4
Joined: Sat Dec 29, 2018 6:08 pm

Re: Raspberry Pi 3 B+ lockups

Tue Jan 08, 2019 5:30 pm

I returned it.

jlallen75
Posts: 4
Joined: Sat Dec 29, 2018 6:08 pm

Re: Raspberry Pi 3 B+ lockups

Tue Jan 08, 2019 5:31 pm

jlallen75 wrote:
Sat Dec 29, 2018 6:32 pm
I got a new 3 B+ a few days ago and am having a similar issue. I tried to install Hassio on it right out of the box and the install would never finish. It it just seemed to lock up at some point. I also wound up trying to install LibreElec and Raspbian and those two installed , but would eventually lock up after a minute of two. I noticed that they will only lock up using ethernet. I can boot into Raspbian and run a constant ping, but anymore traffic than that and it locks up solid. I even updating over ethernet locks it up.

To sum up what I've tried.
1. Tried the 2 SD Scards I had and then purchased 2 more to try.
2. Installed Hassio, LibreElec & Rasbian
3. Purchased an additional power supply and tried that.
4. Changed ethernet cable
5. Moved the connection from my Netgear switch to my Cisco switch.

I wound up getting a 2nd 3 B+ and that's had no issues using the same SD, Power supply and network connection so I'm guessing bad batch of hardware? Should I just send it back and try my luck on the return?
I returned it.

e-raser
Posts: 71
Joined: Sat Apr 04, 2015 1:30 pm

Re: Raspberry Pi 3 B+ lockups

Tue Jan 08, 2019 7:15 pm

I´m trying with a different hardware (I guess my 3rd or 4th Pi 3 B+... what a pity) with same settings and same system (microSDXC) etc. to rule out any possible hardware cause. I think within 7 to 10 days I know if it´s a hardware of software thing.
1x Nextcloud & Pi-hole & ... on Raspbian @ Pi (4 4 GB)
1x Kodi media center on LibreELEC @ Pi 3 B+

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24116
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Tue Jan 08, 2019 8:05 pm

jlallen75 wrote:
Tue Jan 08, 2019 5:31 pm
jlallen75 wrote:
Sat Dec 29, 2018 6:32 pm
I got a new 3 B+ a few days ago and am having a similar issue. I tried to install Hassio on it right out of the box and the install would never finish. It it just seemed to lock up at some point. I also wound up trying to install LibreElec and Raspbian and those two installed , but would eventually lock up after a minute of two. I noticed that they will only lock up using ethernet. I can boot into Raspbian and run a constant ping, but anymore traffic than that and it locks up solid. I even updating over ethernet locks it up.

To sum up what I've tried.
1. Tried the 2 SD Scards I had and then purchased 2 more to try.
2. Installed Hassio, LibreElec & Rasbian
3. Purchased an additional power supply and tried that.
4. Changed ethernet cable
5. Moved the connection from my Netgear switch to my Cisco switch.

I wound up getting a 2nd 3 B+ and that's had no issues using the same SD, Power supply and network connection so I'm guessing bad batch of hardware? Should I just send it back and try my luck on the return?
I returned it.
Did any replacement work OK?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

jlallen75
Posts: 4
Joined: Sat Dec 29, 2018 6:08 pm

Re: Raspberry Pi 3 B+ lockups

Tue Jan 08, 2019 10:04 pm

jamesh wrote:
Tue Jan 08, 2019 8:05 pm
jlallen75 wrote:
Tue Jan 08, 2019 5:31 pm
jlallen75 wrote:
Sat Dec 29, 2018 6:32 pm
I got a new 3 B+ a few days ago and am having a similar issue. I tried to install Hassio on it right out of the box and the install would never finish. It it just seemed to lock up at some point. I also wound up trying to install LibreElec and Raspbian and those two installed , but would eventually lock up after a minute of two. I noticed that they will only lock up using ethernet. I can boot into Raspbian and run a constant ping, but anymore traffic than that and it locks up solid. I even updating over ethernet locks it up.

To sum up what I've tried.
1. Tried the 2 SD Scards I had and then purchased 2 more to try.
2. Installed Hassio, LibreElec & Rasbian
3. Purchased an additional power supply and tried that.
4. Changed ethernet cable
5. Moved the connection from my Netgear switch to my Cisco switch.

I wound up getting a 2nd 3 B+ and that's had no issues using the same SD, Power supply and network connection so I'm guessing bad batch of hardware? Should I just send it back and try my luck on the return?
I returned it.
Did any replacement work OK?
I didn't opt for a replacement, but the other one I got the same day has been going strong for days with no issues. Raspbian and Hassio had no issue using Ethernet.

Return to “Troubleshooting”