rzusman
Posts: 328
Joined: Fri Jan 01, 2016 10:27 pm

Valgrind

Thu Oct 04, 2018 10:48 pm

Based on a hint from a recent thread, I installed valgrind and ran it on my software. It found thousands of "Use of uninitialised value" errors - even before my software ran! Seems like the GCC pre-main code is full of "iffy" code. Anyway, it also found a whole bunch of these in my main function, even though I was pretty careful to initialize everything, and my compiler didn't catch any errors. But, I went through and explicitly initialized every variable I declared, which got rid of all those errors.

The problem is, I can't run the program much past the initial setup. It's just too slow. I know that valgrind is supposed to make your software run 30-50x slower, but this is more like 500-1000x. I have a real-time clock display, and it took over a half an hour to update the time.

Are there any tricks I can use to speed this up? I suspect that the problem may have to do with the fact that this software is very much "real time" - it's a scheduler, so it is always doing things based on wall-clock time. I wonder if valgrind is slowing it down so much that it is never able to finish some of it's tasks. Is there any way to query valgrind as to what function is executing at any given time?

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Fri Oct 05, 2018 8:10 am

rzusman wrote:
Thu Oct 04, 2018 10:48 pm
Seems like the GCC pre-main code is full of "iffy" code.
I get zero valgrind errors from crt0 on all my machines.
I use Valgrind-3.13.0 and GCC 8.2
A hint: compile your code for debug and any valgrind errors will have source line numbers.
rzusman wrote:
Thu Oct 04, 2018 10:48 pm
The problem is, I can't run the program much past the initial setup. It's just too slow. I know that valgrind is supposed to make your software run 30-50x slower, but this is more like 500-1000x.
Yes, valgrind is slow, it is basically an interpreter that checks every single instruction and some of the common library calls like memcpy.
rzusman wrote:
Thu Oct 04, 2018 10:48 pm
Are there any tricks I can use to speed this up?
You could speed up your code! (and/or get the new Pi3B+ model).
I have a real-time clock display, and it took over a half an hour to update the time.
This seems ridiculously slow even running valgrind. The call to ctime() or whatever should be near instantaneous even through valgrind.
Here is the entire "date" command which probably does more than your program:-

Code: Select all

[email protected]:~ $ time valgrind /bin/date
==14329== Memcheck, a memory error detector
==14329== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==14329== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==14329== Command: /bin/date
==14329== 
Fri Oct  5 09:22:51 BST 2018
==14329== 
==14329== HEAP SUMMARY:
==14329==     in use at exit: 64 bytes in 1 blocks
==14329==   total heap usage: 14 allocs, 13 frees, 6,990 bytes allocated
==14329== 
==14329== LEAK SUMMARY:
==14329==    definitely lost: 64 bytes in 1 blocks
==14329==    indirectly lost: 0 bytes in 0 blocks
==14329==      possibly lost: 0 bytes in 0 blocks
==14329==    still reachable: 0 bytes in 0 blocks
==14329==         suppressed: 0 bytes in 0 blocks
==14329== Rerun with --leak-check=full to see details of leaked memory
==14329== 
==14329== For counts of detected and suppressed errors, rerun with: -v
==14329== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 6 from 3)

real	0m2.867s
user	0m2.786s
sys	0m0.081s
You could try the sanitizers which are much faster, but more fiddly to use and don't check every instruction (but do have knowledge of the original C code).
See man gcc (search for the string "sanitize").
You may have to use the latest gcc, or clang.

I actually find valgrind useful for performance testing programs, if they are usable with valgrind then they will be vary fast otherwise.
An alternative is to downclock a Pi Zero to say 200Mhz or less and do your testing on that ...
Last edited by jahboater on Fri Oct 05, 2018 9:22 am, edited 2 times in total.

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Fri Oct 05, 2018 8:51 am

For interest, valgrind comprises several tools.
One tool that can be used for optimizing code is:-

valgrind --tool=exp-bbv --bb-out-file=/tmp/bbv --pc-out-file=/dev/null <your program>

which will provide an instruction count.
Here is the result for running it on /bin/date which you can see takes 240728 instructions (mostly in the C startup code I guess).

Code: Select all

[email protected]:~ $ valgrind --tool=exp-bbv --bb-out-file=/tmp/bbv --pc-out-file=/dev/null /bin/date
==14404== exp-bbv, a SimPoint basic block vector generator
==14404== NOTE: This is an Experimental-Class Valgrind Tool
==14404== Copyright (C) 2006-2017 Vince Weaver
==14404== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==14404== Command: /bin/date
==14404== 
Fri Oct  5 09:48:01 BST 2018
==14404== 
==14404== 
==14404== 
==14404== # Thread 1
==14404== #   Total intervals: 0 (Interval Size 100000000)
==14404== #   Total instructions: 240728
==14404== #   Total reps: 0
==14404== #   Unique reps: 0
==14404== #   Total fldcw instructions: 0
==14404== 
If you do an optimization and the number of instructions executed goes up, you have likely made a mistake :)

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Fri Oct 05, 2018 11:42 am

You might want to try

-O3 -Wuninitialized -Wmaybe-uninitialized

GCC options (note the -O3 which does a deeper analysis hopefully reducing any false positives).

Heater
Posts: 10013
Joined: Tue Jul 17, 2012 3:02 pm

Re: Valgrind

Fri Oct 05, 2018 2:05 pm

jahboater,

Do you have an example C file we can compile that shows GCC changing it's warnings with a change in optimization level?

Because I have never seen this happen nor read anything that implies that it might.

Certainly compilers work harder to find more optimizations when asked to bu I have never seen it produce more warnings.

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Fri Oct 05, 2018 2:25 pm

Heater wrote:
Fri Oct 05, 2018 2:05 pm
Because I have never seen this happen nor read anything that implies that it might.
I saw it in the GCC documentation long ago, I cant remember where. I have seen examples where use of an uninitialized variable was reported, and upping the optimization level removed the warning. I prefer not to write code with uninitialized variables in, so rarely see this warning (and I don't have any such code to hand - though I suppose I could try and artificially create a problem).
Heater wrote:
Fri Oct 05, 2018 2:05 pm
Certainly compilers work harder to find more optimizations when asked to but I have never seen it produce more warnings.
I have only ever seen -O3 produce less warnings. (that is, less false positives).
The doct said "greater accuracy" if I remember correctly.
It does make sense, it is a by-product of the more in-depth analysis -O3 does.

Edit:
The gcc man page just says this:
These warnings are only possible in optimizing compilation, because otherwise GCC does
not keep track of the state of variables.

rzusman
Posts: 328
Joined: Fri Jan 01, 2016 10:27 pm

Re: Valgrind

Fri Oct 05, 2018 2:49 pm

I'll look at some of the suggestions later, but as an example, valgrind is emitting thousands of errors like this:


==857== 2 errors in context 110 of 270:
==857== Use of uninitialised value of size 4
==857== at 0x401AB20: memcpy (memcpy.S:339)
==857== by 0x400BBE7: _dl_new_object (dl-object.c:87)
==857== by 0x4006067: _dl_map_object_from_fd (dl-load.c:1059)
==857== by 0x400939F: _dl_map_object (dl-load.c:2605)
==857== by 0x400DEAF: openaux (dl-deps.c:63)
==857== by 0x401049F: _dl_catch_error (dl-error.c:187)
==857== by 0x400E0A3: _dl_map_object_deps (dl-deps.c:254)
==857== by 0x4003A17: dl_main (rtld.c:1858)
==857== by 0x401713B: _dl_sysdep_start (dl-sysdep.c:249)
==857== by 0x400137B: _dl_start_final (rtld.c:424)
==857== by 0x400160B: _dl_start (rtld.c:652)
==857== by 0x4000CAF: ??? (in /lib/arm-linux-gnueabihf/ld-2.19.so)

Note that this is not "MY" code...

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Fri Oct 05, 2018 3:07 pm

rzusman wrote:
Fri Oct 05, 2018 2:49 pm
valgrind is emitting thousands of errors like this:
What version of valgrind are you using?
It was ported to ARM quite recently and IMO some of the early versions left something to be desired.
Version 3.13.0 on my Pi3B+ works fine in this respect.
[email protected]:~ $ dpkg -l | grep valgrind
ii valgrind 1:3.13.0-2~bpo9+1 armhf instrumentation framework for building dynamic analysis tools
ii valgrind-dbg 1:3.13.0-2~bpo9+1 armhf instrumentation framework for building dynamic analysis tools (debug)
[email protected]:~ $
If you have an Intel PC to hand, perhaps try your program on that (if its easily portable).
Valgrind on Intel x86 has been around for decades.
and its faster too :)

Heater
Posts: 10013
Joined: Tue Jul 17, 2012 3:02 pm

Re: Valgrind

Fri Oct 05, 2018 6:07 pm

jahboater,
The gcc man page just says this:

"These warnings are only possible in optimizing compilation, because otherwise GCC does
not keep track of the state of variables."
Where is there a GCC man page that says that? I cannot find one. Neither can google.

I'm interested to know what warnings it means exactly and what optimization levels. And of course what version of GCC does this.

We all want our variables to be initialized, implicitly or otherwise, else we are writing gibberish. Mistakes do happen.

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Fri Oct 05, 2018 6:12 pm

Heater wrote:
Fri Oct 05, 2018 6:07 pm
jahboater,
The gcc man page just says this:

"These warnings are only possible in optimizing compilation, because otherwise GCC does
not keep track of the state of variables."
Where is there a GCC man page that says that? I cannot find one. Neither can google.
Have you tried just typing "man gcc" on the Pi ?
Type "/-Wmaybe-un" to search for the option.

There are two relevant warning options: -Wuninitialized and -Wmaybe-uninitialized

I am using GCC 8.2 on all my machines, but I think the compiler has always done this.

I can post the two entries here if you like.

I usually use -Os, but I have a special target in the makefile called "lint" (in memory of the old system tool).
It turns on every warning option imaginable, adds -O3, does the compilation sending the output (with -o) to /dev/null

Heater
Posts: 10013
Joined: Tue Jul 17, 2012 3:02 pm

Re: Valgrind

Fri Oct 05, 2018 6:34 pm

Sadly I don't have a Pi to hand to try the man page with.

I tried man on a Debian and Ubuntu PC here. Could not find any such text or part of.

Tried asking google about GCC man pages as well. To no avail.

Not that I don't believe you. Just that I'd like to see, and try, for myself.

What I do find, as expected, is this statement in the GCC documentation:

"Unfortunately, detecting when the use of an uninitialized variable is equivalent, in the general case, to solving the halting problem."

That is to say that in general it is impossible for static analysis to determine if a variable is initialized before use or not.

For this reason it is better to use Valgrind or the sanitizers to detect these problems at run time.

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Fri Oct 05, 2018 6:45 pm

Code: Select all

       -Wmaybe-uninitialized
           For an automatic (i.e. local) variable, if there exists a path from the function entry
           to a use of the variable that is initialized, but there exist some other paths for
           which the variable is not initialized, the compiler emits a warning if it cannot prove
           the uninitialized paths are not executed at run time.

           These warnings are only possible in optimizing compilation, because otherwise GCC does
           not keep track of the state of variables.

           These warnings are made optional because GCC may not be able to determine when the
           code is correct in spite of appearing to have an error.  Here is one example of how
           this can happen:

                   {
                     int x;
                     switch (y)
                       {
                       case 1: x = 1;
                         break;
                       case 2: x = 4;
                         break;
                       case 3: x = 5;
                       }
                     foo (x);
                   }

           If the value of "y" is always 1, 2 or 3, then "x" is always initialized, but GCC
           doesn't know this. To suppress the warning, you need to provide a default case with
           assert(0) or similar code.

           This option also warns when a non-volatile automatic variable might be changed by a
           call to "longjmp".  The compiler sees only the calls to "setjmp".  It cannot know
           where "longjmp" will be called; in fact, a signal handler could call it at any point
           in the code.  As a result, you may get a warning even when there is in fact no problem
           because "longjmp" cannot in fact be called at the place that would cause a problem.

           Some spurious warnings can be avoided if you declare all the functions you use that
           never return as "noreturn".

           This warning is enabled by -Wall or -Wextra.

and

Code: Select all

       -Wuninitialized
           Warn if an automatic variable is used without first being initialized or if a variable
           may be clobbered by a "setjmp" call. In C++, warn if a non-static reference or non-
           static "const" member appears in a class without constructors.

           If you want to warn about code that uses the uninitialized value of the variable in
           its own initializer, use the -Winit-self option.

           These warnings occur for individual uninitialized or clobbered elements of structure,
           union or array variables as well as for variables that are uninitialized or clobbered
           as a whole.  They do not occur for variables or elements declared "volatile".  Because
           these warnings depend on optimization, the exact variables or elements for which there
           are warnings depends on the precise optimization options and version of GCC used.

           Note that there may be no warning about a variable that is used only to compute a
           value that itself is never used, because such computations may be deleted by data flow
           analysis before the warnings are printed.

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Fri Oct 05, 2018 6:55 pm

Heater wrote:
Fri Oct 05, 2018 6:34 pm
That is to say that in general it is impossible for static analysis to determine if a variable is initialized before use or not.
It is very hard indeed to prove it in complex deeply nested situations.

int n;
int m = n + 4;

is pretty simple though.
For this reason it is better to use Valgrind or the sanitizers to detect these problems at run time.
I use all three,

What should be avoided I think is simply initializing every single variable (as a coding standard I worked with mandated).

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Fri Oct 05, 2018 6:59 pm

Heater,
I saw this in the -Wuninitialized warning description.
Because these warnings depend on optimization, the exact variables or elements for which there
are warnings depends on the precise optimization options and version of GCC used.
Just use -O3 !!!

Heater
Posts: 10013
Joined: Tue Jul 17, 2012 3:02 pm

Re: Valgrind

Fri Oct 05, 2018 7:05 pm

Cool, thanks, I'm going to try that out a bit.

Thinking about it, this issue of uninitialized variables is even worse than solving the halting problem. The halting problem is self contained with no external input as the program runs.

What if the initial value of something is not even knowable at compile time? What if it is not possible to know it before run time?

For example: One might have a "length" variable that will hold the length of some incoming message. Perhaps extracted from the header of a received message.

Well, one could initialize "length" in it's declaration or elsewhere, to zero for example. That would make the compiler happy but it is wrong. Unless a zero length message comes in.

If that "length" is, by accident, not initialized from the incoming message header, and then checked for reasonableness as things proceed while processing the message, then all hell can break lose.

Such things are the cause of all manner of buffer overruns and security exploits in recent history.

In such cases not even tools like Valgrind or the sanitizers can help.

Heater
Posts: 10013
Joined: Tue Jul 17, 2012 3:02 pm

Re: Valgrind

Fri Oct 05, 2018 7:10 pm

jahboater,
What should be avoided I think is simply initializing every single variable (as a coding standard I worked with mandated).
Ah yes. I think we are on the same page. See my post above.

Mind you, such a blanket coding standard might at least get coders thinking about how and when their variables are initialized. And if it makes any sense.

rzusman
Posts: 328
Joined: Fri Jan 01, 2016 10:27 pm

Re: Valgrind

Fri Oct 05, 2018 9:43 pm

OK, so I upgraded from 3.7 to 3.13 (apt-get installs 3.7), and that stopped all the bazillion errors before main().
It's still really slow, but I can figure out how to deal with that.

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Sat Oct 06, 2018 1:59 am

rzusman wrote:
Fri Oct 05, 2018 9:43 pm
It's still really slow, but I can figure out how to deal with that.
What kind of Pi are you using?
I run valgrind on the new 3B+ and it is usable.

rzusman
Posts: 328
Joined: Fri Jan 01, 2016 10:27 pm

Re: Valgrind

Sat Oct 06, 2018 3:17 pm

I'm testing this on the Compute Module 3 lite.

bzt
Posts: 183
Joined: Sat Oct 14, 2017 9:57 pm

Re: Valgrind

Wed Oct 10, 2018 1:52 pm

The others gave really good advices, I just want to reflect on the slow clock issue.

Make sure your clock update routine does not allocate any memory. Preallocate everything, or even better, use static stack variables only. Also minimize library function calls, for example don't call localtime() and print the time in every iteration, just when the clock really changes, something like:

Code: Select all

time(&t);
if (t!=old) {
  old=t;
  ts=localtime(&t);
  printf(clockformat, ts->tm_hour, ts->tm_min, ts->tm_sec);
}
Furthermore, I suggest to fork a new process only to update the clock, that way your main process won't have to care about that, and valgrind will also start a new thread to follow it (meaning your main code will run and will be valgrinded instantly).

Cheers,
bzt

rzusman
Posts: 328
Joined: Fri Jan 01, 2016 10:27 pm

Re: Valgrind

Thu Oct 11, 2018 12:59 am

Thanks for the suggestions.
The code is quite complicated - it’s a scheduler for controlling industrial lighting, and it communicates via RS-485 and cellular modems. Maybe 20,000 lines of C.

Anyway, I’ll spend some time trying to figure out what is slowing down valgrind so much when I get a chance.

jahboater
Posts: 3054
Joined: Wed Feb 04, 2015 6:38 pm

Re: Valgrind

Thu Oct 11, 2018 6:42 am

rzusman wrote:
Thu Oct 11, 2018 12:59 am
The code is quite complicated - it’s a scheduler for controlling industrial lighting, and it communicates via RS-485 and cellular modems. Maybe 20,000 lines of C.

Anyway, I’ll spend some time trying to figure out what is slowing down valgrind so much when I get a chance.
Have you looked at "gprof" (man gprof) ?

Return to “C/C++”