unwedging a pi

[quitequick]

I got my pi the other day and its up and running just fine. I'm running it headless and have installed Apache2, MySQL, PHP, phpmyadmin and its running as not an unreasonable web server for dev work. I even got a vnc server to work.

I just installed Chromium for the hell of it - surfed to a html5 video demo page (you can see what coming, can't you)... and my pi is wedged (swaptastically). Poor little SD card being asked to shovel around too many bytes in ways it was never designed. I'm not getting any response from a terminal/ssh/GUI so I can't kill Chromium. My Solution : pull the plug.

Well that's fine for me because I'm pretty sure I know what's going on (having suffered at the hands of a Mac mini for too long). But for the average kid learning programming or a none too techy adult punter who doesn't know his swap partition from his boot partition, it's frustration at best, 'bin' at worst.

I'm betting that wedging your pi will become a common thing. So my question is, are there any tools to prevent or recover from this sort of issue?

[Montala]

The title of this post rather intrigued me, as I had never heard the term 'wedging' before, and my first thoughts were not even electronically related!

On reading the OP however, I guess this is another way of saying 'it's 'hung''… isn't it?

[cheery]

Occassionally it happens to me as well that my linux box knocks itself down. When that happens the first thing for me to do is to abandon xorg. You should be able to jump into tty screen by pressing down CTRL+ALT+F1 combo, login, run program ´top´ and finally, kill the blunderer. Alternatively you can try CTRL+ALT+backspace, which restarts the desktop environment.

Didn't RPi hardware have soft/hard reset button as well somewhere down there? I haven't my RPi yet so I can only guess..

If you happen to recover the system the way I proposed, you can try CTRL+ALT+F7 to get back into xorg. Sometimes I use these tty screens just for fun/convenience as well!

[cDave]

Montala said:


The title of this post rather intrigued me, as I had never heard the term "wedging" before, and my first thoughts were not even electronically related!

On reading the OP however, I guess this is another way of saying 'it's "hung"'… isn"t it?


Not just me then who hadn't heard of this then.

cheery said:


Didn"t RPi hardware have soft/hard reset button as well somewhere down there? I haven"t my RPi yet so I can only guess..


There is no power button it's just pull the plug.

I think I've seen spec of the Pi has been described as that of a Pentium III by a few people recently, and thinking back to those days, my Windows PC would certainly lock up on occasion.

[jamesh]

Is the OP saying that the device has 'bricked' or just 'hung'? Use of the word wedge has me confused also.

For those who don't know - simplistically, bricked means won't restart after a power cycle, hung means it will.

[dom]

I believe ctrl-alt-del launches task manager on debian image which allows you to kill an errant process.

[quitequick]

Seems like not many people here have heard of 'wedged'. It has been common parlance in my circles.

To me, it generically means that the machine is swapping like crazy, so much so that it spends all of its time moving data in/out of swap and can do little else, like giving you an opportunity to kill the offending process(es). In this sense it is 'wedged' - think of crowds of people trying to get in/out of a single small doorway. To me, 'hanging' means the opposite - sitting there doing nothing for no good reason (including no disk activity) - although the two would exhibit a similar outward appearance.

Anyway, nomenclature aside, Cheery and Dom, thanks for the suggestions which sound useful when you have the desktop running.

The spec of the pi (mem, cpu, 'disk' speed) is a ways away from your 'average' modern desktop/laptop which many kids will be used to (but the pi still being fantastic value). Your 'average' kid could be forgiven for trying to use the pi in a similar way to a desktop/laptop. I just thought it would be a bit more user friendly if the o/s could handle overload situations a little more gracefully than just locking up.

I know this problem is not peculiar to the pi or it's o/s but I'm thinking that it may be encountered a lot more often on the pi, once in the hands of the target audience. And when we are trying to encourage kids to use this machine, the last thing we want is for it to frequently lock up on them for no apparent good reason.

So I was just wondering if were any bright ideas (probably o/s tweaks) that could help.

[bredman]

quitequick said:


So I was just wondering if were any bright ideas (probably o/s tweaks) that could help.


The obvious answer is to enlarge the swap partition or file. I'm not even sure which prebuilt images have swap enabled. I remember seeing mention that at least one image has no swap.

However, overuse of swap will quickly destroy an SD card. SD cards are definitely not suitable for swap, so maybe it is a bad habit to let people depend on swap too much.

By the way, this topic is wandering a bit outside the "Absolute Beginners" realm. I wouldn't be a bit surprised it an admin moved it. hint hint...

[nick.mccloud]

quitequick said:


Seems like not many people here have heard of 'wedged'. It has been common parlance in my circles.


The phrase we use is 'carroted'.

As in, hung like a baby carrot.

[jamesh]

With regard to moving posts, once the new forum comes in to place, we will be moving more stuff to more appropriate forums, because there will be more appropriate forums! And the new board makes it easier to leave a stub behind showing where the post went.

[SN]

nmcc said:


quitequick said:


Seems like not many people here have heard of 'wedged'. It has been common parlance in my circles.


The phrase we use is 'carroted'.

As in, hung like a baby carrot.


More variations on "its hung!"

"Put its head in the sand" (or up its arse)

"Puts its fingers in its ears"

"Bunged up"

[Joe Schmoe]

Re #8:

I was going to suggest turning swap off.  For 2 reasons:

1) Because then when it runs out of space, it will just crash, rather than wedging (thrashing).  Actually, I'm not 100% sure what does happen, but I think it is better than what happens when you do have swap (but not enough swap...)

2) I think it has been posted many times here that swapping to an SD card is Not Good, for several reasons.

I'm actually surprised that swap is on now - at least in the first Debian image (the only one I ever did anything with), it was not.

[jbeale]

Is there some way to automatically detect when some large fraction of the system bandwidth is going towards swapping, and proactively offer the user some option to remedy the situation?  Maybe power users wouldn't like it, but it could be helpful for others.

EDIT, hmm, I guess the user might be at a GUI window, or remote via SSH, or completely absent and the offending task was started by a script. Ok, maybe an interactive task isn't the right way.

[jamesh]

SN said:


nmcc said:


quitequick said:


Seems like not many people here have heard of 'wedged'. It has been common parlance in my circles.


The phrase we use is 'carroted'.

As in, hung like a baby carrot.


More variations on "its hung!"

"Put its head in the sand" (or up its arse)

"Puts its fingers in its ears"

"Bunged up"




I've always referred to it as thrashing. Remember having terrible performance at UEA in the mid 80's on a MicroVAX - too many people trying to do their first year assignments.

[jamesh]

jbeale said:


Is there some way to automatically detect when some large fraction of the system bandwidth is going towards swapping, and proactively offer the user some option to remedy the situation?  Maybe power users wouldn"t like it, but it could be helpful for others.

EDIT, hmm, I guess the user might be at a GUI window, or remote via SSH, or completely absent and the offending task was started by a script. Ok, maybe an interactive task isn't the right way.


Mainframes and big systems have suffered with this for years. According to Wikipedia, the solution is....more memory.

http://en.wikipedia.org/wiki/T.....r_science)

[quitequick]

jbeale said:


Is there some way to automatically detect when some large fraction of the system bandwidth is going towards swapping, and proactively offer the user some option to remedy the situation?  Maybe power users wouldn"t like it, but it could be helpful for others.


I think this is a good idea. The o/s is (obviously) ideally placed to determine if bad things are happening. To have the o/s sit by while the SD card slowly destroys itself and its data and the user tears their hair out doesn't sound very clever at all. If the user is absent, perhaps all the *more* reason for the o/s to take some pre-emptive action.

I have to keep reminding myself who the pi is aimed at. Kids learning programming are not likely to be running their pi as a headless server. They will be sat in front of their monitor (tv) writing python shoot-em-up games, while 'researching' some web-based miniclip games (or none Flash equivalents). I'm sure they will not appreciate the pi locking up on them for no apparent reason and losing all their work.

[Joe Schmoe]

I think the basic answer to the question being raised (is there some way the Pi could detect that it is doing too much swapping?) is "No.".  The nature of the virtual memory beast is that the application - and, for all practical purposes, the kernel as well - can't tell the difference between real and virtual memory.

So, as I mentioned before, the best advice is just not to do any swapping at all.  When you run out of real memory, you should crash.  I'm serious about this (I just realized that that last sentence could be taken as tongue-in-cheek; it is not intended as such).

As a previous poster noted, there is no point in grinding your SD card into bits trying to get out of out-of-memory situation by swapping yourself to death.

[mahjongg]

Well, I would wager that if you need swapping, you simply are trying to use a too memory hungry application!

I too am amazed that the latest distro turns on swapping, on a SD-card no less! No good can come from that!

In fact, I would propose that swapping is normally turned off, and if, and only if, a USB memory stick (or external HD) is detected, with a swap partition swapping is turned on "automagically".

[jojopi]

The command "free" will confirm that swap is not activated out of the box in the recommended 2012-04-19 image.  But this will not stop the system from thrashing if too many processes are started, nor will it mean that applications crash quickly rather than going very slow first.

With no swap space, the kernel cannot swap dirty pages out to disk.  But it can still discard clean pages with the hope of reloading them from disk later.  You are likely to reach the point of thrashing -- OK led mostly off and very poor response in the GUI -- well before the kernel has to kill anything due to lack of memory.

For many workloads, you will actually get much better performance at a given level of memory contention by enabling swap and increasing swapiness.  That allows the kernel to choose to swap potentially less active dirty pages instead of only clean ones.  But there are no magic bullets.  Ultimately you need to not run more than the Pi can cope with.

[jbeale]

...the application – and, for all practical purposes, the kernel as well – can't tell the difference between real and virtual memory.


I'm no OS guru but I'm surprised to hear that there is no way for the kernel to react to poor performance due to excessive swapping. Isn't the kernel the one doing the swapping? Can't you empirically determine that X MB/sec going in or out of swap is the threshold of usability, and have it take some action at that point?

[quitequick]

Joe Schmoe said:


So, as I mentioned before, the best advice is just not to do any swapping at all.  When you run out of real memory, you should crash.  I'm serious about this (I just realized that that last sentence could be taken as tongue-in-cheek; it is not intended as such).


Well, I'm firmly of the opinion that having the machine crash unexpectedly and without reporting a reason is as a bad thing.

Switching off the swap maybe a reasonable solution. But to have it crash when it runs out of memory, imo, is not. And I suspect, like all things in life, a little bit of swapping is no bad thing.

I also find it hard to believe the kernel doesn't know when swapping is happening. I would expect the kernel to be the entity that decides when and what to swap and then performs it. So as jbeale suggests, have a  threshold beyond which some swap reducing swap strategy is applied e.g. kill the worst offender - better to do this than have the system crash or worse, damage itself?

I'm sure a not-very-good solution could be a shell script running at max priority monitoring /proc/* and/or 'ps' output. But that's because I'm not a Linux kernel programmer. Any Linux kernel programmer care to comment?

[jamesh]

Thing is, this isn't a Raspi problem - it afflicts all devices that use virtual memory. And as far as I know, it's not something that has been 'fixed' on those devices. ie. Mainframes, PC's (I've had windows do exactly the same thing), all sorts. So, it's possible it's an intractable problem. There may be 'bodges' that try to ameliorate the effects, but the real solutions are  a) use less memory intensive programs, b) add more memory c) Run fewer apps.

[Joe Schmoe]

Re #22:

Certainly, I agree with what you say, but there are two aggravating factors in the case of the Pi:

1) The fact (this something we can all agree to) that the machine is slow (by modern standards) to begin with.  So, once it starts swapping, it goes from bearable to horrible.  (A modern desktop PC, when it starts swapping, goes from good to annoying)

2) The fact that it is (by default) swapping to an SD card.  This issue has been adequately covered in previous posts on this thread.

This is why I really think it would be better if the Pi was configured (at least by default) to not swap.  And, it seems Jojo has confirmed that that is, indeed, the out-of-the-box default setting.

[plugwash]

Joe Schmoe said:


1) Because then when it runs out of space, it will just crash, rather than wedging (thrashing).  Actually, I'm not 100% sure what does happen


Assuming a standard linux install with overcommit turned on (the default) when memory (and swap if enabled) run out the kernel will activate the "OOM killer" which will pick a process to kill.

If overcommit is off then memory allocations will fail.

[jojopi]

A good option for dealing with wedged Linux systems generally is to enable magic kernel.sysrq, especially on boxes used for development.  Then if the system becomes unresponsive you can use Alt+SysRq keystrokes to manually trigger the out-of-memory killer, kill all processes on the console, sync and reboot, or other useful actions.  All of which are certainly preferable to pulling to the power, especially during heavy I/O.

Unfortunately while kernel.sysrq support is available in regular debian kernels, it is not compiled into the 2012-04-19 image raspi kernel.