PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2302
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Sun May 27, 2018 2:27 pm

Until this is a problem that we can recreate in the lab we will continue to be unable to understand and fix it. We want a perfect product and happy customers, so this is almost as frustrating for us as it is for those affected.

I'm not going to speculate about the possible cause, but as Gavinmc42 said, there are many new elements in 3B+ - far more than are shared with the original 3B. What appears as an evolutionary step is virtually a redesign.

wybielacz
Posts: 29
Joined: Wed Nov 06, 2013 9:37 pm

Re: Raspberry Pi 3 B+ lockups

Sun May 27, 2018 2:43 pm

PhilE wrote:
Sun May 27, 2018 2:27 pm
Until this is a problem that we can recreate in the lab we will continue to be unable to understand and fix it. We want a perfect product and happy customers, so this is almost as frustrating for us as it is for those affected.

I'm not going to speculate about the possible cause, but as Gavinmc42 said, there are many new elements in 3B+ - far more than are shared with the original 3B. What appears as an evolutionary step is virtually a redesign.

I can happily send you my already second RPi with this issue for investigation. This can't be a coincident that my replacement Pi has exactly the same issue...

For me it is just frustrating... My home automation and media server depends on this RPi so you must understand that not even 24 hours uptime just makes them completely unreliable and worthless. In the past 2 months i had just trouble with it so i was looking forward for my replacement Pi to finally stabilize everything again but instead i got the same issue all over again. It was a mistake to trade my rock solid RPi 3B over this one...

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2302
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Sun May 27, 2018 9:15 pm

I'm waiting for an answer for you on what might be the best replacement strategy. I would want your SD card (or a clone) as well, in case software proves to be the culprit.

ejolson
Posts: 3386
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi 3 B+ lockups

Mon May 28, 2018 2:46 am

jamesh wrote:
Sun May 06, 2018 7:57 pm
There is no design flaw on the Pi itself, but as has been explained before, it appears that some SoC's have come off the fab outside the specification of all the initial run of chips we received for testing. We tested all the initial batches and came up with the SD ram/CPU settings that worked. However, if you suddenly get a batch from the fab outside those parameters, some might have problems, which cannot be caught on the production line since the problem only appears after some time, and with 5000 a day being made, thee is no chance of long term testing.
There exist manufacturer reliability diagnostic suites which are not only used on the assembly line but can identify later malfunctions in the field. A good example of such software is provided by the company pc-doctor. My understanding is that pc-doctor software was originally used by Dell, Hewlett-Packard and Lenovo.

While the pc-doctor diagnostics do not, a priori, run on the Raspberry Pi, there exists ARM versions. From what I understand the software is modular so making a version for Raspberry Pi might be easy. Just to make things clear, I do not work for pc-doctor, but know people who do and would be happy to put the appropriate people in contact. Please send me a private message to follow up on this or contact pc-doctor directly.

At any rate, any similar diagnostic tool would allow Raspberry Pi engineers to distinguish what kinds and malfunctions are occurring. The idea is, after factory testing, to further provide a complete set of diagnostic tools to each customer. Complete manufacturer supplied diagnostics used to be quite common during the days of minicomputers. Today, such tools could either be run on demand when trouble occurs as in the past or set to automatically report anonymised metrics over the internet which would then be continuously monitored using big-data statistical analysis.

User avatar
bensimmo
Posts: 4152
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: Raspberry Pi 3 B+ lockups

Mon May 28, 2018 9:15 am

At which point there would be a screaming uproar the RaspberryPi were stealing and watching what you are doing, watching your every keystroke.
They do it with any company that is trying to see and improve how their software works or doesn't when things break.

ejolson
Posts: 3386
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi 3 B+ lockups

Mon May 28, 2018 11:08 am

bensimmo wrote:
Mon May 28, 2018 9:15 am
At which point there would be a screaming uproar the RaspberryPi were stealing and watching what you are doing, watching your every keystroke.
They do it with any company that is trying to see and improve how their software works or doesn't when things break.
Debian Linux has a built-in popularity contest that tracks which packages are used and how often, but nobody is screaming. The trick to avoid misunderstanding is for such data collection to be opt-in and to make it clear what is being collected. For those who don't opt in, having a complete set of manufacturer-supplied diagnostics is still useful.

Performance metrics that ensure correct functioning of hardware seem to be a big enough win for the end user (especially large corporations) that a significant number opt in. Over time this allows a manufacturer to obtain a clear picture of how slight variations in manufacturing and sourcing of components affect long-term reliability, which in turn can be used to increase reliability and brand value. If things go as planned, even hobbyists start to see the manufacturer as a strategic business partner and further cooperation is natural.
Last edited by ejolson on Mon May 28, 2018 2:26 pm, edited 4 times in total.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23311
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Mon May 28, 2018 1:10 pm

ejolson wrote:
Mon May 28, 2018 11:08 am
bensimmo wrote:
Mon May 28, 2018 9:15 am
At which point there would be a screaming uproar the RaspberryPi were stealing and watching what you are doing, watching your every keystroke.
They do it with any company that is trying to see and improve how their software works or doesn't when things break.
Debian Linux has a built-in popularity contest that tracks which packages are used and how often, but nobody is screaming. The trick to avoid misunderstanding is for such data collection to be opt-in and to make it clear what is being collected. For those who don't opt-in, having a complete set of vendor supplied diagnostics is still useful.

Performance metrics that ensure correct functioning of hardware seem to be a big enough win for the end user (especially large corporations) that a significant number opt in. Over time this allows a manufacturer to obtain a clear picture of how slight variations in manufacturing and sourcing of components affect long-term reliability, which in turn can be used to increase reliability and brand value. If things go as planned, even hobbyists start to see the manufacturer as a strategic business partner and further cooperation is natural.
I dunno. Ubuntu recently stated they were going to capture some system metrics, anonymise them, and send them back to base camp.

The outcry was considerable, even though there is no trackable information at all being captured.

http://www.theregister.co.uk/2018/05/24 ... urbulence/
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

ejolson
Posts: 3386
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi 3 B+ lockups

Mon May 28, 2018 2:31 pm

jamesh wrote:
Mon May 28, 2018 1:10 pm
The outcry was considerable, even though there is no trackable information at all being captured.
Right. I think that example shows how differently people view opt-in compared to opt-out when it comes to data collection.

User avatar
bensimmo
Posts: 4152
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: Raspberry Pi 3 B+ lockups

Mon May 28, 2018 5:31 pm

I know the reasons, but plenty don't seem to understand it. Especially with the Media behind them.
I use the Preview edition of Windows, where some scream murder becsuse they collect anonymous information or if you tick the even more info version maybe not so anonymous.

But you are testing the software, they analyse everything, the get reports, usage, hardware and can best target their efforts.
Of course you can never collect every setup in the PC world.
Still some tester seem to think they can test it and not have any data collected and whatnot.
This Android phone is collecting data as I type, all used to improve the OS in some way and hardware if there are major problems it'll dump and send back if I want.

One side effect of the GDPR is it should all be opt in unless it cannot function (or w.r.t. Win10 Preview is required for its purpose).
There are going to be some cases.in the courts come up to see how this actually works.

But for me if Raspbian had a testing version (apt install?) and the engineers could and would use the data fedback, I'd be ticking the boxes. Just make sure there are the boxes :-).

pudding
Posts: 95
Joined: Sat Jan 12, 2013 3:55 pm
Location: UK

Re: Raspberry Pi 3 B+ lockups

Mon May 28, 2018 6:10 pm

Had regular crashing with a 3B+ I bought when they were first released from CPC. Regular crashing with Libreelec (latest version). Never experienced any crashing on any previous RPis. I got a replacement from CPC and the new 3B+ works perfectly, no crashing, on exactly the same SD card running Libreelec.
Author of MAME4ALL for Pi, PiSNES, PiFBA & PiMenu

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23311
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Mon May 28, 2018 9:22 pm

pudding wrote:
Mon May 28, 2018 6:10 pm
Had regular crashing with a 3B+ I bought when they were first released from CPC. Regular crashing with Libreelec (latest version). Never experienced any crashing on any previous RPis. I got a replacement from CPC and the new 3B+ works perfectly, no crashing, on exactly the same SD card running Libreelec.
Glad to read some good news in this thread!
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Tozzi1
Posts: 21
Joined: Thu Apr 20, 2017 11:05 pm

Re: Raspberry Pi 3 B+ lockups

Mon May 28, 2018 10:26 pm

Mine are all working pretty well now, with the latest rpi-update(s) and keep improving in stability (@original clock speeds!).
For now, it seems to me, we'll need to stay on the "bleeding edge" with drivers (rpi-updates) and things will eventually be sorted.
Those who use latest Raspbian with latest rpi-update and still have machines crashing every night or so should probably really seek hardware replacements...
For critical stuff like home automation and such it's probably a good idea to downgrade and use an older 3B, until the new ones become as rock solid as these older machines were...
Got one in my basement that hasn't been rebooted for at least 180 days (still on Jessie)...

ejolson
Posts: 3386
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi 3 B+ lockups

Tue May 29, 2018 3:47 am

Tozzi1 wrote:
Mon May 28, 2018 10:26 pm
it's probably a good idea to downgrade and use an older 3B, until the new ones become as rock solid as these older machines were...
I'm happy to report that at default clock speeds the particular 3B+ I managed to procure is much more reliable than my 3B ever was.

My 3B+ is currently functioning as a masquerade firewall: upstream is through the built-in WiFi while a second WiFi adapter functions as an access point with an additional downstream connection over the wired network. I've run some pretty intensive NEON optimized code on it as well and it doesn't produce incorrect results or crash. While this may not seem like news, I post here to let people know that, even though some 3B+ computers are having problems, it is also possible to obtain one that just works.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2302
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Tue May 29, 2018 3:20 pm

Thanks for the feedback, ejolson. These kind of threads can, for obvious reasons, get a bit one-sided, so a bit of balance is welcomed.

We've been working behind the scenes on changes to a number of calibration settings that should make all(*) 3B+s that have been crashing under load stable and performant. There remains a separate issue of some units/use cases that have a tendency to lock up overnight when they ought to be idle. We've initiated a number of recalls for boards of this type and learnt nothing - one is now my "daily driver" and has shown no strange behaviour.

If anyone in this thread has a 3B+ that is suffering from "overnight" lockups - units than, on return to them after a period of absence, are completely unresponsive - please (re)post a brief description of your observations: frequency of failure, software installed and in use, etc. I'm particularly interested in whether you've noticed a high temperature on the device and, if so, where was hottest.

(*)There is the possibility that a very small number of out-of-spec units escaped production testing that should have been rejected - in those cases the usual return channels are available, but feel free to mail me ([email protected]) in case we want to test them ourselves.

Tozzi1
Posts: 21
Joined: Thu Apr 20, 2017 11:05 pm

Re: Raspberry Pi 3 B+ lockups

Mon Jun 04, 2018 8:07 pm

Well, for the past few days (after the most recent rpi-update) I'm experiencing nightly lockups once again.
This particular machine is running Kodi, NZBGet and Sonarr, on Raspbian Stretch with PiXel.
Connected to one network via Ethernet and another one via onboard WLAN.
Connected to USB are a 2 TB Notebook HD, a Raspberry microphone, a Logitech wireless mouse & keyboard dongle.
It's cooled by sitting inside a FLIRC aluminium case, and during normal operation, even when I force heavy load, the CPU never exceeds 55˚C.
During the lockups, the case feels pretty warm though.
I'll test this again on my other 2 B+ machines, too, which are still running "naked" Raspbian Stretch, w/o any additional software installed.
The power supply can be ruled out since I've swapped it multiple times and recently replaced it with a 3,5A model.

//EDIT: Another lockup just now. Case feels burning hot, like never before. rpi-update and raspbian stretch up-to-date.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2302
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Tue Jun 05, 2018 8:26 am

The closer to a cleanly installed image - ideally Raspbian, but Kodi would be OK - the more likely we are to be able to reproduce the problem here at Pi Towers.

1. Is there anything that might be active on your system overnight that you are aware of?

2. Does it connect to any external storage or network shares?

3. When it locks up Is there anything visible on screen, or is it blank (or headless)?

Tozzi1
Posts: 21
Joined: Thu Apr 20, 2017 11:05 pm

Re: Raspberry Pi 3 B+ lockups

Thu Jun 07, 2018 12:26 am

1. Yes, most of the network traffic is at night.
2. No shares, but an USB drive. Several folders on the Raspi are shared via SMB, though. This might be a common denominator, as now I realize this is the case with all 3 machines I am using. Never thought of that before.
3. Will have to check. Right now, with latest rpi-update, everything seems to be stable once again.
Fingers crossed.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2302
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: Raspberry Pi 3 B+ lockups

Thu Jun 07, 2018 8:31 am

I'm pretty sure there is some issue with network shares in one direction or the other, and I'll be delighted when we find a configuration that allows us to reproduce the problem and begin to solve it.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23311
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Thu Jun 07, 2018 11:30 am

PhilE wrote:
Thu Jun 07, 2018 8:31 am
I'm pretty sure there is some issue with network shares in one direction or the other, and I'll be delighted when we find a configuration that allows us to reproduce the problem and begin to solve it.
Indeed. Have recently tried Samba shares, one on the Pi, one on Windows, transferred over 30GB, no issues at all, so a guaranteed way of replicating this is the requirement at the moment.

Anything unusual about your network? Do you have a particular router brand?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Tozzi1
Posts: 21
Joined: Thu Apr 20, 2017 11:05 pm

Re: Raspberry Pi 3 B+ lockups

Thu Jun 07, 2018 3:38 pm

Router is AVM Fritz!Box 7490.
No Windows machines, all Mac and Debian.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23311
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Thu Jun 07, 2018 4:03 pm

Bit of a loss to explain why you see the issues I I do not. What is the share definition in /etc/samba/smb.conf?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Tozzi1
Posts: 21
Joined: Thu Apr 20, 2017 11:05 pm

Re: Raspberry Pi 3 B+ lockups

Thu Jun 07, 2018 4:09 pm

Nothing out of the ordinary there...
(Except perhaps the shared external USB drive)...

[boot]
path = /boot
guest ok = yes
read only = no
force user = pi
browseable = yes

[hd]
path = /mnt/usbhd
guest ok = yes
read only = no
force user = pi
browseable = yes

[nzb]
path = /home/pi/nzbget/nzb
guest ok = yes
read only = no
force user = pi
browseable = yes

[system-logs]
path = /var/log
guest ok = yes
read only = yes
force user = root
browseable = yes

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23311
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 3 B+ lockups

Thu Jun 07, 2018 4:30 pm

I have a similar share on my USB/SD card adaptor so that's not the issue. I haven't used force user, which is the only other real difference.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Tozzi1
Posts: 21
Joined: Thu Apr 20, 2017 11:05 pm

Re: Raspberry Pi 3 B+ lockups

Thu Jun 07, 2018 5:21 pm

On the other hand, I've updated again after the most recent lockup (June 4) and since then, no crashes.
I don't know yet if this is simply a rare occurence (several cofactors) or if it's been actually fixed meanwhile.

Proxx
Posts: 6
Joined: Thu Jun 07, 2018 6:22 pm

Re: Raspberry Pi 3 B+ lockups

Thu Jun 07, 2018 6:25 pm

Got the same lockups, on raspbian and HypriotOS

at random the pi locks.
Maximum of 3days uptime

Update:
No USB devices
Connected via ethernet cable

Tried 2 SDcards (Sandisk ultra/extreme)

With original rpi powersupply

Return to “Troubleshooting”