Debugging a c program?

[gmc]

Anyone got any ideas on how to debug a c program on the pi?

I've got a c program I wrote which runs fine but seems to crash at least once a week.

I've got another program that monitors the pid and restarts the program as soon as it detects the program has died. A bit of a cheap but it seems to keep the program running.

I really would like to get to the bottom of why it keeps crashing though.

Any programs I can look at? Can I get my c program to dump any useful info which will help me determine why it crashed?

[joan]

Compile it with the -g debugging option. Run it under gdb, i.e. if your program is called x use gdb x (then r to run it). When it crashes you should be able to see where.

[gmc]

Thanks - I'll look into gdb.

[joan]

Given the program runs a fair while it might be worth looking at the code for memory being allocated but not freed, any time-based events, and any rare events.

[jamesh]

Don't think Valgrind works on the Pi/ARM, but you might try mudflap, or e-fence to find memory problems. I used e-fence to find a memory problem in the raspicam apps, worked well.

[gmc]

I ran it through efence - no errors so far.

With gdb - since the program only crashes every week, is it possible to run gdb in the background so I can exit the ssh session, but then leave gdb running so when it crashes it spews out the info to a log file?

[Jim Chapman]

You can also run gdb on the core file, assuming the program dumps core when it crashes.

[topguy]

And if no core files are generated, you might have to enable them with the..

Code: Select all

ulimit -c xxxx

command. xxxx is the max size (in blocks) of the core file. Use a number like 10240 or more if your program uses a lot of memory.

[stephj]

There is always the good old fashioned method of making a debug version of your program that maintains some sort of event log as it runs. When the program starts generate a date and time stamped file name, and at strategic points, write out whatever the program is up to.

Be aware that the OS will buffer writes to a text file, so the last entry in the file may not be the last event that occurred. In the event of a crash the final buffer may not get flushed to the file.

To overcome this open the file as append, write the entry, and close the file immediately, on every event write. This will add quite an overhead to its execution but will give you more of an idea of where the fault is occurring.

If your program is auto-restarted then you should have a nice collection of log files to analyse afterwards which will hopefully point you in the right direction.

This is one of the oldest tricks in the book and works in any programming language.

[rurwin]

Code: Select all

$ man 3 fflush
$ man 2 sync

I have very rarely needed anything more than printf to debug an application. Failing that, the core file will tell you where it crashed, although not necessarily why. You're unlikely to get more information running gdb for a week, because you're not going to be watching it in the vital few minutes or seconds before it crashes.

My recommendation would be to enable the core file. Then when it crashes, run gdb on the core file and follow the stack frame back to find out exactly what it was doing. You might be able to work it out from there, but if not then add lots of printf's to a log file or the screen, flushing and syncing after any that might be close to the crash-site.

Things you should be looking for:

• Using memory after it has been free'd
• Multi-tasking that needs to be locked but isn't
• A slow memory leak (printf sbrk(0) every few minutes to test. See man sbrk.)
• A memory structure that grows continually
• A coincidence in timing that is ridiculously unlikely. (If it was likely, it wouldn't last a week.)

Ensure that you are checking the return value of every function call. malloc() sometimes fails and you want to know then, not when the code uses the null pointer half an hour later. Even if you have no idea what to do if an error happens, check every return value and at least printf a log of it.

[jojopi]

$ man 3 fflush
$ man 2 sync

Because I doubt it will be clear from the man pages alone:

fflush(3) flushes the C library buffers in your process. This is necessary if you want to read the output file in real time from another process, or if your process is liable to crash with a fatal signal and therefore not flush its buffers on exit.

fsync(2) or sync(2) flushes the kernel buffers to disk. This is not necessary unless you want to read the output file in real time from another kernel, or your process is somehow able to crash the whole machine.

Calling sync(2) unnecessarily will greatly harm performance.

[gmc]

Thanks for the tips. I'm logging a ton of info to a debug.log file and have also setup ulimit.

Now just waiting for it to crash.....come on......crash!!!

[topguy]

The gdb command "backtrace" (or "bt") should give you the callstack. It should show which function that calls vfprintf(), which function that calls the function that calls vfprintf() and so on. This will tell you more.

You can move "up" or "down" in the callstack and check out the code and variables for each function. If GDB can find the sourcecode you can list the code also.

Hey!! what happened to the post I thought I was replying to ?

[gmc]

Thanks - I remove it as soon after I posted it as I read about the backtrace

[IanH2]

With gdb - since the program only crashes every week, is it possible to run gdb in the background so I can exit the ssh session, but then leave gdb running so when it crashes it spews out the info to a log file?

Use GNU screen. In a nutshell:

Run 'screen' - this will start a shell; you can start your gdb session in it. When it's going, type Ctrl-A D and the session will disappear ("detach") into the background. You can now disconnect.

When you reconnect, run 'screen -r' and the gdb session will reappear. You can detach again, or Ctrl-A K will kill it off if you're done.

[gmc]

Thanks, that's very useful to know.