free memory gets smaller and smaller

General programming chat and advice for beginners

8 posts
by Beach » Fri Dec 21, 2012 1:35 pm
Dear Forum

I am in the midst of a video transcoding project (hardware accelerated). To view the free memory whilest running I use the sysinfo structure to track the free memory available to hunt for memory leaks. I somehow fail to get those leaks, but it appears that the video.c example as well as OMXplayer suffer from the same.

Just do a ''free -m'' before and after using either of these programs.

Where does the memory go?
Posts: 47
Joined: Tue Jul 24, 2012 11:07 am
Location: Noordwijk The Netherlands
by jamesh » Fri Dec 21, 2012 4:45 pm
Interesting...

Sounds like a memory leak on the GPU side, since the Arm side stuff should be cleaned up after the process shuts down.
Raspberry Pi Engineer
Raspberry Pi Engineer
Posts: 10595
Joined: Sat Jul 30, 2011 7:41 pm
by gordon@drogon.net » Fri Dec 21, 2012 5:20 pm
Beach wrote:Dear Forum

I am in the midst of a video transcoding project (hardware accelerated). To view the free memory whilest running I use the sysinfo structure to track the free memory available to hunt for memory leaks. I somehow fail to get those leaks, but it appears that the video.c example as well as OMXplayer suffer from the same.

Just do a ''free -m'' before and after using either of these programs.

Where does the memory go?


Sure it's not the normal file/block buffers and cache?

Try running this:

sudo /sbin/sysctl vm.drop_caches=3

and see if it magically comes back...

If that's the case, then don't do anything - Linux is just doing its thing normally.

-Gordon
--
Gordons projects: https://projects.drogon.net/
User avatar
Posts: 1421
Joined: Tue Feb 07, 2012 2:14 pm
Location: Devon, UK
by MattHawkinsUK » Fri Dec 21, 2012 11:02 pm
Unless it is causing a problem, ignore it. I thought I had a memory leak when my free memory dropped to 20mb from 120mb over a few days. It is just Linux using it. It is usually available for use when something asks for it.
My Raspberry Pi blog and home of the BerryClip Add-on board : http://www.raspberrypi-spy.co.uk/
Follow me on Google+, Facebook and Twitter (@RPiSpy)
User avatar
Posts: 457
Joined: Tue Jan 10, 2012 8:48 pm
Location: UK
by Beach » Sat Dec 22, 2012 12:48 pm
Well the sysctl ... does actually give back memory. It gives even back more than I started with.

So let me rephrase the question:

How can I monitor memory leakage in a program (simple)

What I want is a ''print memory'' function to call at the start of a program and just before exit. The difference should be 0.

I tried it with the sysinfo structure but that does not work
Posts: 47
Joined: Tue Jul 24, 2012 11:07 am
Location: Noordwijk The Netherlands
by gordon@drogon.net » Sat Dec 22, 2012 2:37 pm
Beach wrote:Well the sysctl ... does actually give back memory. It gives even back more than I started with.

So let me rephrase the question:

How can I monitor memory leakage in a program (simple)

What I want is a ''print memory'' function to call at the start of a program and just before exit. The difference should be 0.

I tried it with the sysinfo structure but that does not work


tl;dr - Don't wory this is perfectly normal.

Using that sysctl command tells linux to flush all it's buffers and caches. Whenever you read a file (or write it), Linux keeps a copy in RAM - so the next time you read it, it will fetch it from RAM and not disk (or SD card in our case).

When memory becomes "tight" (ie. when your own program needs more RAM), Linux will automatically drop these buffers, then it will consider moving unused programs into swap, before moving used programs & data into swap before finally giving up... (OOM Error)

Back to your program... It's probably not losing memory - it's highly likely that all you are seeing is the normal things that Linux does on any system.

The best tool I've found for debugging memory allocations is 'valgrind', however I don't know if it's been ported to ARM architecture yet. If you can make your program run on an x86 Linux system then you could try it there.

However - given that you say you can't find the "leak" in your program, I'd be fairly confident that your program is not leaking at all, and what you are seeing is perfectly normal in the grand Linux scheme of things.

(Another thing to note, is that unless you're using shared memory, then when your program ends, it will free up all dynamically allocated memory anyway).

-Gordon
--
Gordons projects: https://projects.drogon.net/
User avatar
Posts: 1421
Joined: Tue Feb 07, 2012 2:14 pm
Location: Devon, UK
by gordon@drogon.net » Sat Dec 22, 2012 2:58 pm
Beach wrote:Well the sysctl ... does actually give back memory. It gives even back more than I started with.

So let me rephrase the question:

How can I monitor memory leakage in a program (simple)

What I want is a ''print memory'' function to call at the start of a program and just before exit. The difference should be 0.

I tried it with the sysinfo structure but that does not work


Realising I've not quite answered your question, then try this:

At the start of your program (very very start), have this:

#include <unistd.h>

and in main ():

{
void *brk = sbrk (0) ;

and immediately before the exit:

printf ("Memory not correctly unallocated: %d\n", sbrk (0) - brk) ;
}

However... Some libraries malloc/brk memory and don't free it, expecting the normal exit of a program to free it for them (which does happen - e.g. in the SDL libaries).

If you want to test this, write a trivial program:

Code: Select all
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main ()
{
  void *brk = sbrk (0) ;
  unsigned char *c ;

  printf ("%8d - ", brk) ;

  c = malloc (1024) ;

  printf ("%8d\n", sbrk (0) - brk) ;
}


Try it with and without the c = line...

-Gordon
--
Gordons projects: https://projects.drogon.net/
User avatar
Posts: 1421
Joined: Tue Feb 07, 2012 2:14 pm
Location: Devon, UK
by dcw803 » Sat Dec 22, 2012 4:24 pm
Following Gordon's clever sbrk() answer, which I like for it's sheer simplicity, here's another way of answering Beach's question:

So let me rephrase the question:

How can I monitor memory leakage in a program (simple)

What I want is a ''print memory'' function to call at the start of a program and just before exit. The difference should be 0.


If you take a look at my C Tools notes at:

http://www.doc.ic.ac.uk/~dcw/c-tools/

there's a whole section on memory debugging, especially the memlib library I use (contained in the tarball downloadable from the above page). Install memlib, then recompile all your code with memlib and then run your program. Every malloc(), calloc(), realloc(), strdup(), free() and exit() call is redefined, so that exit() automatically displays a list of all your allocated but never freed blocks - with the source file name and line no where each block was allocated. Makes memory debugging - in your own code - dead easy. However, if a library you're using leaks memory, that won't be detected unless you recompiled that library using memlib and then relinked your memlib-aware program with the memlib-aware library.

Give it a play, thoroughly recommended! --- duncan
Posts: 1
Joined: Sat Dec 22, 2012 4:13 pm