Debugging and profiling tools

We have several tools available to us when trying to improve the quality of our programs. In general, debuggers enable us to search for/localize bugs and observe program behaviour, while profilers allow us to examine the program's overall use of system resources - focused primarily on memory use and cpu consumption.

While our primary debugger is gdb (along with its graphical interface ddd, and our primary profiling tool is gprof, there is another tool, valgrind that provides additional information both for profiling and for debugging.

Short summaries of the three tools are provided below.


valgrind analysis tool: quick reference

valgrind attempts to analyze your program for potential flaws,
   and produces plain text output describing them
   (although the "plain text" does assume you have a solid
    understanding of how things work in C/C++)

First, as with gdb, you must compile your program with the -g option,
then you can run your program through valgrind, e.g.
   valgrind myprog arg1 arg2 arg3 etc

valgrind can also provide information/analysis with respect to
   potential memory leaks if run with the option --leak-check=yes
      valgrind  --leak-check=yes myprog arg1 arg2 arg3 etc

valgrind can also perform an analysis of heap (dynamic memory)
   utilization if run as a two step process:
(1) valgrind --tool=massif myprog arg1 arg2 arg3 etc
    this produces an extra file named massif.out.NNN
       where NNN is the process id your program ran under
(2) run the analysis using the command
       ms_print massif.out.NNN
    (using the NNN from step 1)


gprof profiling tool: quick reference

Allows you to perform an analysis of cpu utilization for
   the functions in a program.

(1) compile the program with the -pg option
(2) run the program normally,
    it will create an extra file, named gmon.out
    (each run of the program overwrites this file)
(3) run gprof progname
    (where progname is the name of your program)

This last step produces the text analysis of cpu utilization 
   in your program.  The report contains two tables and a lot of
   explanatory text, we're interested in the two tables.

The first table summarizes the cpu time spent in each of the
   functions called during the run of the program.  This includes
   the total cpu time, the number of times the function was called,
   the average time per call, and the percent of overall cpu time
   that was spent in the function in question.

The second table breaks this down further, identifying how much
   time was spent in the function and, of that, how much time
   was spent in the functions it called.  E.g. if function f called
   functions g and h, it would specify how much time was spent in f
   (including the time for g and h) plus how much of that time was
   spent in g and how much of that time was spent in h.

Additional options:

 -z: gprof will also list functions that were never called during
     that run of the program

 -r: gprof will suggest an ordering for your functions to try to minimize
     page faults during execution

 -R: gprof will suggest an ordering for linking .o files to try to minimize
     page faults during execution


gdb debugger: quick reference

Debugging approach:

   notice a problem (bug report)

   iterate through the following (not necessarily in order):

      establish/refine theories as to what the source of the
         problem might be

      try to isolate exactly what the problem is:
         conduct widening/narrowing cases to find the scope
         of the bug

      use the test cases and debugger to isolate the region
         wherein the error occurs and/or becomes observable

   once you're sure of the source of the problem, try and
      develop a fix - running it through the test cases to
      see if it solves the problem or reveals the need for
      more testing

   once the problem seems fixed, conduct regression tests
      to be sure you didn't break something else


=======================================================================

GDB Notes
---------

good for C or C++ or Java
must compile with -g option

special note: compiling with both -g and -O (optimize) can have
   funky effects, so don't do it (the optimizer from -O rearranges 
   your code, the debugger shows you what is really there,
   which won't match what it looks like in your source code)

you can run gdb with executable name as the command line parameter, eg
    gdb myprogram

if you're running without full GUI support (e.g. just a terminal
   window from home) use the -nw (no windows) option
   (this is the default for our installation)

if you want gui support there is also ddd,
   which is a GUI interface to gdb

crash support
   since gdb keeps the program in memory even after a crash,
   it allows us to explore what state the program was in at
   the point of crash

to see time and memory stats after each command use the
   -statistics option

to quit use either quit or q


Logging output
--------------

To turn on/off recording of all output (goes to gdb.txt) use
   set logging on/off

To change the logfile use
   set logging file YOURFILENAME

You can also redirect output at the run command, e.g.
   run > YOURFILENAME

Getting help
------------
General: help or h
Specific: help command/topic
Search: apropos keyword

Running your program within the debugger
----------------------------------------

set up any command line arguments using
    set args val1 val2 etc

run progname 
   or 
r progname
   or
r progname arg1 arg2 etc

alternative is to use the 'start' command, 
   which effectively puts a breakpoint in the
   main routine then issues the run command

Examining the source code
--------------------------

The list command allows you to display source code (in case
   you don't have it open in another window someplace)

There are many options for list, two of the most common are:
   list functionname
   list linenumber


Interrupting execution
----------------------

To see what the code is doing when it runs, we need to be able
   to interrupt the program in mid-execution to have a look around.

There are three ways of specifying when a program should stop execution:

 - Breakpoints stop whenever a specific spot in the code is reached

 - Watchpoints stop whenever the value of an expression changes

 - Catchpoints stop whenever a specific kind of event takes place 
   (e.g. exceptions)

You can set up these points, then enable/disable them temporarily

Setting up breakpoint:
   break FUNCTIONNAME
   break LINENUMBER
   break FILENAME:FUNCTIONNAME
   break FILENAME:LINENUMBER

Optionally you can add a condition under which a breakpoint should be used,
   e.g.  break FUNCTIONNAME if (SOMECONDITION)

You set a temporary breakpoint, i.e. one that is only used once,
   using tbreak instead of break

You can clear (remove) breakpoints using
    clear 
    clear FUNCTIONNAME
    clear LINENUMBER
    etc

You can enable/disable breakpoints using
    enable BREAKPOINT
    disable BREAKPOINT

You can set up commands to run when a breakpoint is hit as follows:
    break foo if x>0
    commands
    printf "x is %d \n",x
    end

You can also add the command silent if you want it to ONLY print the
stuff you tell it,
    break foo
    commands
    silent
    ....
    end

You can set a watchpoint using
    watch EXPRESSION
    where the expression can be as simple as a variable name
    *** but when you set the watchpoint on one or more variables you must
        currently be somewhere in the program execution where all those
        variables are visible/in scope - this may mean setting a breakpoint
        in a pertinent function, running the program to the breakpoint,
        THEN setting the watchpoint ***

You can set a catchpoint using 
    catch EVENT
   where event can be any of throw, catch, exec, fork, load, unload

You can get a list of the current break/watchpoints using
   info breakpoints
     or
   info watchpoints


Resuming execution
------------------
To resume an interrupted run use
   continue or c

To continue just to the next source line use
   step

To continue just to the next machine code instruction
   (remember there may be multiple lines of machine code
    per line of source code) use stepi or nexti

To continue for n source lines use
   step N

To continue but count function calls as single lines use
   next
   next N

To continue until a source line is reached use
   until LINE
   or
   advance LINE

Looking at the call history on the stack
----------------------------------------
We can look at the sequence of (still active) function calls
   that lead to where we are now using backtrace, or bt, eg
   backtrace

For just the last N calls use
   backtrace N

We can get info on the current frame using
   info frame

We can look at specific frames (i.e. for specific calls) using
      frame N
   where the current frame is 0, the one that invoked it is frame 1,
   the one that invoked that is frame 2, etc

You can move up/down the call stack (up moving to the
   caller's stack frame) using the commands
      up
      down


Looking at memory during an interuption
---------------------------------------

We can display the value of variables or expressions at the point
   of interruption using the print or p statement, eg
     print x+y

To specify which function or file contains a variable of interest, 
   use ::, eg print FUNCTIONNAME::VARIABLENAME

Note it will only recognize variables that are normally visible at
   the current point in the program (i.e. scoping rules apply)

You can choose the format of the display using print/f where f is a
format character
  - x for hex
  - d for decimal
  - u for unsigned
  - o for octal
  - t for binary
  - a for address
  - c for character
  - f for float 

If you want to refer to a previously printed value within an expression
use $$N to refer to it ($ is the current value, $$ or $$1 is the one
before it, etc)

Suppose we just printed the address of a struct, and want to print a field:
   print *$.next
   (* for dereferencing the pointer specified in $, then .next is the field)

We can also examine memory addresses using the x (examine) command:
  x ADDRESS

Command completion
------------------
TAB causes gdb to attempt command completion,
    RETURN cycles through the possibilities

Declaring temporary variables within gdb
----------------------------------------
You can set variables of convenience within the debugger using 
   set $thevarname = somevalue

You can then use/refer to the variable using $thevarname

To see the list of variables of convenience use
   show convenience

Configuring gdb
---------------
There are a number of environment changes you can make to set gdb up the
way you like:
   set prompt NEWPROMPT
   set height Hpixels
   set width  Wpixels
   set verbose on/off
   set confirm on/off
and a great many others

If these are specified in your .gdbinit file then they will automatically
be loaded by gdb each time it starts up.

Defining your own commands and command sequences
------------------------------------------------
If there is a command or command sequence you find yourself using 
frequently during debugging, you can define it as a new command:
   define mycmd
      print/a
   end

You can supply up to 10 command line arguments, which can
   be referenced within the definition as $arg0, $arg1, etc

Such definitions can also be placed in your .gdbinit file