jaceson
Posts: 5
Joined: Fri Aug 05, 2016 4:47 pm

RPI3 Hangs. No new processes can be started.

Fri Aug 05, 2016 5:54 pm

Background: I have a headless raspberry pi 3 with the second version of the noir camera module running motion 24/7. I'm using the bcm2835-v4l2 driver for the camera. I'm running the latest, updated raspbian jessie with all the latest firmware updates. I access it solely over ssh using the built-in wireless. (yes, power save mode is disabled)

Problem: after a day, sometimes several, the raspberry pi will no longer accept new logins over ssh. It'll simply prompt for a password, show the motd, then hang indefinitely. I never get a command prompt.

Initially I thought this was a problem SSH/PAM but since then I've actually been logged into the pi when the problem started. What occurs is that no new processes can be started. For example, I was ssh'ed in on one terminal and I attempted to ssh from another. The hang occurred for the new session, so I was sitting there with an already active ssh session in one terminal while in the other I was experiencing the ssh hang. In the terminal that still had the active connection, I attempted to diagnose what was going on. Any command I tried to run simply hung and never completed. I was able to interrupt the hung commands with ctrl-c/SIGINT briefly before finally I attempted bash tab-completion for a command and it hung permanently while no longer responding.

There is never any clue in syslog or any other log as to what occurred. All processes that were already running prior to the hang, continue to run and function normally as long as they don't require network access. Network communication no longer functions properly. (I have motion set up to copy pictures remotely when motion is detected, I use task-spooler as a queue to queue up the scp commands so that only one at a time actually executes). The pictures continue to be stored locally on the sdcard but can no longer be copied over the network. I'm still able to ping the rpi and it responds.

I've seen very weird behavior due to power issues from my first-gen rpi but in this case I have one of the beefier power supplies running the pi3 (2.4 amp). Aside from the noir camera module, there is no other hardware attached, usb or otherwise.

So yeah, I was actually logged into the rpi when this problem happened and could not run a single command to begin to diagnose things. I put a daily reboot in root's crontab and that won't run when this problem occurs. Cron attempted to run the reboot will be logged by syslog but the reboot doesn't happen. Just hangs like any other new process.

I also switched sdcards just to rule that out as a problem source.

TL;DR: rpi 3 runs for a couple of days, eventually won't allow any new processes to run and can no longer login locally or over ssh. Responds to pings, network otherwise unusable. Already running processes continue to run, write to sdcard, but can't send data over the network.

Ideas?

User avatar
topguy
Posts: 5570
Joined: Tue Oct 09, 2012 11:46 am
Location: Trondheim, Norway

Re: RPI3 Hangs. No new processes can be started.

Sat Aug 06, 2016 12:38 am

My Tip: Too many "open files"

Read about "lsof" command and "ulimit".

jaceson
Posts: 5
Joined: Fri Aug 05, 2016 4:47 pm

Re: RPI3 Hangs. No new processes can be started.

Sun Aug 07, 2016 6:08 pm

Initially that was my suspicion as well but is not the case. (also the reason I'm using task-spooler to make sure it doesn't happen).

I reproduced the problem again while I had an active login with htop running. I had one process (gpg) peg a cpu core then hang with that core pegged. No new processes could be forked but the kill command would work (I guess a new process isn't forked for kill however the killall command would hang). I killed several processes but could never get the system back into a state where I could run any new processes. So strange.

ssteel
Posts: 3
Joined: Thu Feb 12, 2015 8:34 pm

Re: RPI3 Hangs. No new processes can be started.

Fri Aug 12, 2016 12:08 pm

I've been observing the _exact_ same thing on my Pi 2 with Raspbian and latest firmware, nginx+php-fpm and a Noir camera.
SSH login will hang at Last login line without ever giving a prompt randomly, usually after several hours (24ish). Startd happening with last raspbian and firmware updates (not saying it's surely related)

Also not sure if related: Additionally, starting a SSH session any command not found (e.g. "asdf") will always throws a segfault:

Code: Select all

[email protected]$ asdf
-bash: asdf: command not found
Segmentation fault
After this, every cat command for a valid file will show file contents and end with segfault. Some other commands also execute properly but last line is Segmentation fault. They seem to execute fine. Exiting the SSH session and login in again fixes it until next command not found.

No hints in syslog either. I'm not an expert but not a noob either.
Weird, will probably reinstall raspbian and packages and see if it comes back.

jaceson
Posts: 5
Joined: Fri Aug 05, 2016 4:47 pm

Re: RPI3 Hangs. No new processes can be started.

Thu Aug 18, 2016 5:18 pm

I still haven't figured out how to prevent this from occurring.

I've set my ulimit for open files and processes ridiculously low for my primary non-root user. No effect.

I've disabled the internal ir camera module and ran motion with an external usb camera for a while just to rule out any hardware issues associated with that.

I run htop continuously and see nothing suspicious when the hang occurs. Of course I can't fork any new processes to do much other investigation and there is still nothing being logged.

I can sometimes force a reboot if I have an open root shell with

Code: Select all

echo b > /proc/sysrq-trigger
but sometimes even that will hang.

JVARL
Posts: 1
Joined: Thu Aug 18, 2016 7:45 pm

Re: RPI3 Hangs. No new processes can be started.

Thu Aug 18, 2016 8:03 pm

Hi, I'm trying to pull a raspberry pI 3B (my first experience) with jessie image Raspbian wrote in a micro-SD 16Gb (Class6) using Win32diskimager. After starting the first boot, I observe that wrote…
[OK] Started LSB: Resize the root filesystem to fill partition.
And he did nothing more. I do not know what is going on. I doubt if I made an error in the copy of the image as it did no prior preparation of the micro-SD (I had data in NTF format) , I assumed it would W32DImager smart enough to FAT32 format and to boot. Thank you.

calaf
Posts: 1
Joined: Tue Nov 29, 2016 1:31 pm

Re: RPI3 Hangs. No new processes can be started.

Tue Nov 29, 2016 1:46 pm

Exactly the same symptoms: hangs, segmentation fault… I tried the same tests with the same (lack of) results.
This affects all my RPI3 and RPI2 models running latest Jessie and firmware.
Surprisingly, there are very few entries on the net about this topic, and this thread is the only one I found describing exactly my problem.
Some threads mention bluetooth causing freezes, or WiFi, or USB peripheral devices, or a weak power supply... But removing USB devices and disabling all waves don't solve the problem. I could not test without network, though, since I'm using the Pi as a server. The only way I found to avoid hangs is to switch off the RPI!

starfire
Posts: 4
Joined: Thu Nov 29, 2012 12:53 pm

Re: RPI3 Hangs. No new processes can be started.

Fri Jan 13, 2017 1:36 pm

I'm also suffering this issue with a noir camera attached, happens typically after 24-48 hours.

I can ssh but it hangs after motd:

Code: Select all

login as: pi
[email protected]'s password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Jan  5 23:43:17 2017 from 192.168.1.31
nothing more.

However it still produces network traffic fine and zoneminder connects and continuously captures the camera output fine.

Bit stumped, but at least I'm not the only one.

channelz
Posts: 5
Joined: Mon Feb 09, 2015 8:24 pm

Re: RPI3 Hangs. No new processes can be started.

Tue Mar 21, 2017 5:09 pm

I am also having this issue after upgrading to PIXEL. Before PIXEL it was running without fail and uptime was basically forever.

Driving me mad. After 24-48 hours the ssh login with hang indefinitely after MOTD on multiple login accounts.
ALSO commands issued on the local terminal will not execute. UNLESS I issue "sudo -i" (which is switching to interactive root) Commands run fine. When I exit from interactive mode. Commands again will not run.

Raspberry pi 2 is connected via ethernet.

Please Help!

(https://expresshosting.net/ssh-hanging-authentication/ does not fix my issue)

cr5
Posts: 1
Joined: Wed Apr 26, 2017 2:06 pm

Re: RPI3 Hangs. No new processes can be started.

Wed Apr 26, 2017 2:14 pm

Just updated my pi to Jessy and got exactly this problem.
SSH hangs after a while at last login, motion and a webserver still running fine on it.

Is there any new ideas regarding this topic?


Waszker
Posts: 6
Joined: Sun Sep 11, 2016 1:09 pm

Re: RPI3 Hangs. No new processes can be started.

Fri May 05, 2017 11:19 pm

Hi, sorry for the grave digging but I have exactly the same problem...
I've been trying different solutions for the last 1.5 year but just can't find anything working...
Have you managed to deal with that problem? There's nothing on the Internet except for this thread...

One thing that I've noticed was that the hanging usually occurs with 100% CPU load and many `/bin/sh ...` commands running. This is strange because there's really nothing on my system other than the Tornado server running. Also, the problem occurs after rather random amount of time.
I'll post more info about it in a week's time as I have this error on my Raspberry currently but I'm away and can't ssh login or anything at all...

Waszker
Posts: 6
Joined: Sun Sep 11, 2016 1:09 pm

Re: RPI3 Hangs. No new processes can be started.

Sat May 06, 2017 8:14 am

Okay quick update.
After ~10 hours of waiting the ssh command: ssh [email protected] 'sudo reboot' managed to execute! Yay! :D

jeanfrancoisp
Posts: 2
Joined: Thu Jul 27, 2017 3:08 pm

Re: RPI3 Hangs. No new processes can be started.

Mon Jul 31, 2017 1:40 pm

If this is still an issue to some of you - I have experienced the same situation using a Camera Noir 2 on a PI; after much digging around and debugging, I have isolated the issue to UV4L's extension library, /usr/lib/uv4l/uv4lext/armv6l/libuv4lext.so.

The library has a bug in its use of boost::interprocess which degrades over time, eventually hanging.

The library is used to interface the camera with Video4Linux software (not raspi tools such as raspistill) and is installed *into* bash with the uv4l streaming server package (and maybe others). It is the fact that this library is loaded into the bash environment that explains all the issues (crashes when returning from commands and eventually complete hang).

The issue disappears when the file /etc/environment is edited to comment out the line:
LD_PRELOAD=/usr/lib/uv4l/uv4lext/armv6l/libuv4lext.so like so:

#LD_PRELOAD=/usr/lib/uv4l/uv4lext/armv6l/libuv4lext.so

I was wondering if anyone on the PI boards knew who best to contact regarding this issue either within the PI community or the uv4l project?

Thanks,
Jean-François "Jeff" Poirier,
Senior Software Developer,
Moment Factory
Jean-François "Jeff" Poirier
Senior Developer
Moment Factory

starfire
Posts: 4
Joined: Thu Nov 29, 2012 12:53 pm

Re: RPI3 Hangs. No new processes can be started.

Tue Aug 15, 2017 8:45 am

I will try this out, thanks.

BTW there is a contact form on the uv4l website, see https://www.linux-projects.org/contact/

jaceson
Posts: 5
Joined: Fri Aug 05, 2016 4:47 pm

Re: RPI3 Hangs. No new processes can be started.

Sat Oct 14, 2017 4:37 pm

jeanfrancoisp wrote:
Mon Jul 31, 2017 1:40 pm
after much digging around and debugging, I have isolated the issue to UV4L's extension library, /usr/lib/uv4l/uv4lext/armv6l/libuv4lext.so.
Nice! I'd love to know how you went about tracking that down.

jeanfrancoisp
Posts: 2
Joined: Thu Jul 27, 2017 3:08 pm

Re: RPI3 Hangs. No new processes can be started.

Mon Oct 16, 2017 7:50 pm

Hi there!

I ended up noticing that the PIs would eventually hang, but only within the bash shell (ssh would still answer incoming connections but hang after the MOTD) and only after running for a while.

More specifically, bash would hang after accepting new input but before launching a new command, not while running executing ones. And, even more tellingly, *all* running instances of bash would freeze seemingly together - once one froze, all others would suffer the same fate on the next command launch.

One time, while investigating, I happened to launch sh from an about-to-freeze shell and it worked correctly, allowing me to soft-reboot the system. So that gave me a hint that the only shell affected seemed to be bash.

I figured it was something to do with launching commands, not with bash itself.

I rebooted the PI and launched a bash session then attached gdb to it; I used the shell until the hanging behaviour appeared again, then broke into the bash process and looked at the stack trace - it was a deadlock in boost, in uv4l. I then went to confirm that there was usage of boost in uv4l.

Following that, looked in the environment everywhere for where uv4l would be used.

I eventually found it in /etc/environment, commented it out, and it solved the issue.

That was the process, in a nutshell.
:)
Jean-François "Jeff" Poirier
Senior Developer
Moment Factory

jaceson
Posts: 5
Joined: Fri Aug 05, 2016 4:47 pm

Re: RPI3 Hangs. No new processes can be started.

Mon Oct 16, 2017 8:19 pm

jeanfrancoisp wrote:
Mon Oct 16, 2017 7:50 pm
eventually hang, but only within the bash shell
Well done. I wasn't able to track it down. Never suspected anything being thrown into LD_PRELOAD like that. Thanks!

Return to “Troubleshooting”