Memory leak causes SD card failure.

keith.b · Mon Aug 27, 2012 4:00 pm

Jojopi and RaTTuS both responded to my earlier queries (19-20th Aug) under the topic ‘No Swap Space?’, following which I used Task Manager to monitor memory usage.

I had previously thought that too many writes to my SD cards had caused their ultimate failure, and I was trying to minimize these writes by disabling the swap file.

In fact without the swap file, failure occurred earlier than before, with chromium-browser reporting ‘Either Chrome ran out of memory or the process was terminated for some other reason’. The SD card finally gave up the ghost after another re-boot.

On using my 3rd SD card, I finally found Task Manager, and have been monitoring memory use ever since, this time without disabling the swap file.

What I have found is that in either chromium or midori, when continuously looping between two local pages of .html, memory use builds up inexorably at the rate of approximately 1MB / hr. This time I have not allowed it to reach critical values near the installed limit, fearing the loss of another card.

It appears that objects created upon opening subsequent .html pages may not be disposed of gracefully at the next switch. I have tried both jquery’s ‘delayer’ function and HTML’s META REFRESH, to cycle between pages (each calling the other after a timeOut), and have used both chromium and midori as browsers. The result is the same every time.

Now that I am using Raspbian Wheezy, it may be that the swap file will kick in before the ‘kernel panic’ that I have previously observed, and save the day, before all activity is lost forever (cards won’t then reimage after a reformat). Nevertheless, surely the memory use should not be building up as described? HTML code is available to demo the problem.

If no solution can be offered, could this topic be escalated up to an OS bug report?

khulat · Mon Aug 27, 2012 8:08 pm

Have you tried to run your code on a normal Linux PC to see if you see the same behavior? If you cannot do that personally, you can publish your code and i can take a look at it if you want.

keith.b · Tue Aug 28, 2012 1:52 pm

khulat wrote: Have you tried to run your code on a normal Linux PC?

The pi is my first experience of a Linux system, so I don't have it anywhere else. Here is a very much condensed (and speeded up) example that demonstrates the observed behaviour:

first file (delayer[1].html):

<html><head><script type="text/javascript">
function delayer(){document.location = "delayer[2].html"}
</script></head>

<body onLoad="setTimeout('delayer()', 1000)">
<h1>This page is a time delayed redirect to delayer[2]!</h1>
</body></html>

2nd file (delayer[2].html):

<html><head><script type="text/javascript">
function delayer(){document.location = "delayer[1].html"}
</script></head>

<body onLoad="setTimeout('delayer()', 1000)">
<h2>This page is a time delayed redirect to delayer[1]!</h2>
</body></html>

Thanks for the offer to test!

fredjam · Wed Aug 29, 2012 12:02 am

Are you looking at your sd cards using gparted on another computer. To work an sd
card needs to have an msdos partition table and at least one good partition.
If your pi is running out of memory and crashing it may be destroying the partition
table. This will make it appear that the sd card is completely dead.

jojopi · Wed Aug 29, 2012 8:30 am

fredjam wrote:If your pi is running out of memory and crashing it may be destroying the partition
table. This will make it appear that the sd card is completely dead.

That is nonsense. If the Pi runs out of memory, the kernel will page to swap or kill processes. Linux knows the difference between swap space and partition tables.

If you measure used memory in a way that includes buffers and cache, then it is normal for memory use to increase over time. You generally want to look at the second row of figures in the output of "free", not the first.

If memory is leaking, then it may be useful to find out which process is responsible, such as by comparing the output of "ps axu |sort -nk5" at different times, and arrange to restart the problem process periodically.

Unless you have configured it not to, the browser may be writing cache, history, tab configurations, and other data at every page load. This is likely to be more significant than the potential swap io. "vmstat 1" running continuously on another terminal will tell you if swap or other io are happening.

keith.b · Wed Aug 29, 2012 2:54 pm

jojopi wrote:
If you measure used memory in a way that includes buffers and cache, then it is normal for memory use to increase over time . . .

Thanks, I had wondered which set of memory figures to look at; I had used the default (checked) 'show memory used by cache as free' (the precise meaning of this is not clear to me). When un-checking this, memory used increases to above 200MB out of 215MB. Both figures increase over time, although the higher one does seem to reach a limit, perhaps before being swapped out?
I had indeed expected that memory would be gracefully swapped and/or garbage collection would be enabled, but my experience had suggested otherwise.

My 2nd 'lost' card was running with the swap file disabled, and I did receive memory warnings from chromium-browser before final meltdown. However, all cards are now on 'Wheezy', so it may be that with the swap file once more enabled, the build-up may eventually be gracefully dealt with. The original lost card was running Squeeze, which may itself have had a problem.
The system is NOT configured to disable write cache, history &c, and I understand (now) that such writes would increase the memory use.

'vmstat' reports swpd 524; free 19264; buffer 39208; cache 80296.
'ps axu|sort -nk5 doesn't appear to change significantly, just reports the last process as 'midori'.
How do I disable writing cache, history &c?
Can I restart the browser automatically, say every 24 hr, as I need the project to run 24/7?

Thanks to your pointers, I am now able to monitor vital statistics more thoroughly, although not all are familiar to me!

I’ll soon have another supply of blank SD cards, which will allow me to extend the soak tests.

Despite you thoughts about fredjams comments, the only failed SD card I have left refuses to be imaged, reporting multiple sector ‘failure to read’ errors, suggesting that significant memory had been corrupted. The FAT partition, however, is still read/writable.

jojopi · Thu Aug 30, 2012 9:25 pm

You need to run "vmstat 1" (or some other number) and observe the output continuously during normal operation. Non-zero values flashing in the "so" and "bo" columns indicate blocks per second written as swap and non-swap, respectively. Plain "vmstat" just reports the average since boot.

Putting the browser in private browsing mode may reduce writes to disk. The package "eatmydata" may be helpful if something is making excessive calls to fsync. I would be tempted to run everything as a user whose home directory was a tmpfs, such as /tmp. These filesystems are not preserved over a reboot, so there is no requirement for the kernel to actually write to disk. It will still write tmpfs blocks to swap if short of memory, though.

If your cards cannot be re-imaged that definitely supports the theory that too many writes were made to them, especially if concentrated in specific areas to overwhelm the cards' limited wear-levelling abilities. It rules out any software bug simply damaging the partition table or boot files. That would be recoverable with a re-flash.

Incidentally, during boot the kernel says:

Code: Select all

Adding 102396k swap on /var/swap.  Priority:-1 extents:1 across:102396k SS

The "SS" here means that the kernel knows the swap is on a solid-state device. On rotational media it aims to reuse swap pages as soon as possible and keep the data bunched together to reduce seek times. On solid-state it aims to spread the writes around the whole area to reduce wear.

So actually it might be best to have a big swap file on a Pi, even though you should be trying not to use it. And blocks written to filesystems should perhaps be considered more damaging that blocks written to swap.

niftyhacking · Fri Aug 31, 2012 6:08 am

If one of the USB ports is free add an inexpensive USB stick
and make a swap file on it....

Also explore limits
$ ulimit -a

Since most limits are unlimited by default
it makes sense on a tiny system to set some.

Running your little abuse html files ... firefox is smaller on a x86_64 box!
29345 me 20 0 686m 98m 26m R 76.2 1.3 23:59.36 firefox
30660 me 20 0 1827m 75m 31m S 36.5 1.0 0:11.98 midori
On x86_64 midori also shows an apparent memory leak-- one min later.
29345 me 20 0 687m 100m 26m R 76.7 1.3 26:42.85 firefox
30660 me 20 0 1884m 131m 75m R 36.9 1.7 1:29.98 midori

So with an apparent mem leak limits (see ulimit) make a lot
of sense while the bug you file is being addressed.

keith.b · Fri Aug 31, 2012 11:11 am

jojopi wrote:You need to run "vmstat 1" (or some other number) and observe the output continuously during normal operation. Non-zero values flashing in the "so" and "bo" columns indicate blocks per second written as swap and non-swap, respectively. Plain "vmstat" just reports the average since boot.

Putting the browser in private browsing mode may reduce writes to disk. The package "eatmydata" may be helpful if something is making excessive calls to fsync. I would be tempted to run everything as a user whose home directory was a tmpfs, such as /tmp. These filesystems are not preserved over a reboot, so there is no requirement for the kernel to actually write to disk. It will still write tmpfs blocks to swap if short of memory, though . . .

I have now been able to merge my two problem html files into one, and ‘switch’ between pages by alternatively setting their display style to ‘none’ and ‘block’.

This has reduced the ‘swpd’ tally to 8 over a period of 42 hr. since reboot, together with zeros in si; so; bo: ‘bi’ reports 1, but does this every time vmstat is invoked, with or without continuous observation.

This is looking much more hopeful, and the 2nd row of task manager’s used memory approaches, but does not now exceed, 203MB of 215MB.

However, the 1st ‘memory’ row has steadily increased since boot, from 64MB (of 215MB) to 116MB and counting . . . I’m not sure how this will pan out over the next few days.

Your comments on the kernel’s boot message are re-assuring, and could perhaps lead me to using a second storage device, either a usb ssd or hard-drive as a last resort. Hopefully these may not now be needed.

I will also first increase the swap file size to maybe 512MB following your advice.

Not sure how to use a private browser session, or how to use a home directory such as /tmp. After creating an alternative (safer) directory named /tmpkb, alongside /tmp in ‘File System’, Leafpad couldn’t see it, even though it saw /tmp. I think maybe you meant me to use a different /tmp directory?

RaTTuS · Fri Aug 31, 2012 11:19 am

you can use a external usb HD as a device
[even put your root fs on it]
and a swap partition if you so desire

keith.b · Fri Aug 31, 2012 11:28 am

niftyhacking wrote:If one of the USB ports is free add an inexpensive USB stick
and make a swap file on it....

Also explore limits
$ ulimit -a

Since most limits are unlimited by default
it makes sense on a tiny system to set some . . .

Will consider putting the swap file on a USB stick if the problem persists, but have made significant improvements so far (see latest post).

Which limits are you recommending to restrict, and to what?

Not sure that Firefox is bug free under Raspbian yet, perhaps still in testing, but will monitor the situation.

Also unsure what your statistics under x86-64 refer to specifically, but a smaller browser footprint must help. Tried both Dillo and Netsurf, but neither support JQuery, which I need to fetch Twitter feeds.

fredjam · Sun Sep 02, 2012 1:39 am

jojopi wrote:
fredjam wrote:If your pi is running out of memory and crashing it may be destroying the partition
table. This will make it appear that the sd card is completely dead.

That is nonsense. If the Pi runs out of memory, the kernel will page to swap or kill processes. Linux knows the difference between swap space and partition tables.

If you measure used memory in a way that includes buffers and cache, then it is normal for memory use to increase over time. You generally want to look at the second row of figures in the output of "free", not the first.

In fact Linux knows nothing. If you are destroying or think you are destroying your memory cards
then people spinning you stories about swap space and other things rather than helping you to
recover your memory cards are wasting your time but you will have to sort that out for yourself.
Linux is not a perfect thing. The C compilers used to compile linux programs have memory leaks
and so do all the linux web browsers. It is all very well for people to spin you stories about what
should happen in theory but you have already thrown away two perfectly good memory cards.

jojopi · Sun Sep 02, 2012 9:44 pm

fredjam wrote:In fact Linux knows nothing. If you are destroying or think you are destroying your memory cards
then people spinning you stories about swap space and other things rather than helping you to
recover your memory cards are wasting your time but you will have to sort that out for yourself.
Linux is not a perfect thing. The C compilers used to compile linux programs have memory leaks
and so do all the linux web browsers. It is all very well for people to spin you stories about what
should happen in theory but you have already thrown away two perfectly good memory cards.

OP has reported that the cards cannot be re-imaged, and that one was returned to the supplier, who suggested it had been worn out by excessive writing. He also says that on one of the cards the FAT filesystem was still readable. So your theory about a software bug destroying the partition table simply does not fit the evidence. It is okay to be sceptical about whether the cards really are permanently damaged, I suppose. But you would need to suggest a test, or a means of resurrection, rather than just asserting that they are "perfectly good".

Sure, Linux is not perfect. But in a thousand host-years I have never once seen it trash a partition table. Your claim that it is likely to do so within weeks of installation on a Pi, and for no better reason than running out of memory, is laughable. Who would use such a system? Even the mechanism you seem to have in mind makes no sense. C compiler inserts memory leak into programs — by accidentally changing the arguments to malloc? — so far, so unlikely. Memory runs out. Now why does the kernel overwrite the partition table instead of swapping? Rubbish.

keith.b · Mon Sep 03, 2012 2:22 pm

OK guys, don't scrap between yourselves!

Thanks to your various inputs, I now have a stable system. After finally adding a page refresh once every 12 hours to my (now) dual purpose single page, I have a controlled system on which both 'memory used' lines stay within the system boundaries, without further creep.

I have learnt a lot about Linux stats and other, command line features; I'm much more confidant that my approaches are now consistent with better practices. I've also discovered through the images that I've taken along the way, that my first 2 cards had not been resized to make use of their additional memory, despite my suppliers assertions, which may have further muddied the waters. That has now been rectified.

Please consider the case SOLVED, I'm extremely grateful for everybodies contributions!

Memory leak causes SD card failure.

Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.

Re: Memory leak causes SD card failure.