Week 03

Week 03

1/67

Things to Note ...

• Quiz 1 in Week-5 lab (lab exercises week 2 to 4 are a good preparation)

In This Lecture ...

• Testing, Debugging, Performance tuning (Sec.5.7,Slides)

Coming Up ...

• Structured data types (Ch.8)

Nerdy Things You Should Know

2/67

Consider the following scenario ...

• you're sitting in a lab
• you're looking at some code like   '/^s?[0-9]{7}$/'
• you want to ask a question about the code
• but you're not sure how to refer to the ^ char
• and you don't want to sound clueless

From
blog.codinghorror.com/ascii-pronunciation-rules-for-programmers/

... Nerdy Things You Should Know

3/67

Software Development Process

4/67

Reminder of how software development runs ...

• specification   (via requirements analysis)
• design   (data structures, algorithms)
• implementation   (C code)
• testing, debugging   (code analysis)
• user testing   (if has a user interface)
• performance tuning   (if required)

... Software Development Process

5/67

Tools available to assist each stage ...

• specification   (english, formal languages)
• design   (your brain ... and CS knowledge)
• implementation   (editors, IDEs, compilers)
• testing   (testing frameworks, (e.g. Check))
• debugging   debuggers (e.g. gdb))
• user testing   (usability testing methods)
• performance tuning   (profilers (e.g. prof))

Testing

Testing

7/67

A systematic process of determining whether a program

• has mistakes (bugs) in it
• handles bad inputs "reasonably"

• the program specification or detailed requirements
• an executable version of the program
• sample input data and corresponding output data
  (or a tool that applies validation rules to the output)

... Testing

8/67

Testing happens at different stages in development:

• Unit tests on behaviour of components
• System tests on overall input/output behaviour
• System integration tests on interaction of components
• User acceptance tests to find out what they don't like

A useful approach for unit testing:

• start from lowest level functions   (don't call other functions)
• test each function as it is completed   (and "tick it off")
• use only tested functions in testing higher-level functions

... Testing

9/67

Testing alone cannot establish that a program is correct.

Why not?

To show that a program is correct, we need to show

• for all possible inputs, it produces the correct output

• we can only feasibly test a small subset of possible inputs

Testing increases our confidence that the program is ok.

Well-chosen tests significantly increase our confidence.

Exercise: Testing a function

10/67

Consider the notion that an array int A[N] is ordered:

ordered(A[N]):  forall i:1..N-1,  A[i] >= A[i-1]

and some code to implement this test

ordered = 1; // assume that it *is* ordered
for (i = 1; i < N; i++) {
    if (A[i] >= A[i-1])
        /* ok ... still ordered */;
    else
        ordered = 0; // found counter-example
}
// ordered is set appropriately

Put this code in a function and write a driver to test it.

Print "Yes" if ordered, or print "No" otherwise.

Testing Strategy

11/67

Testing only increases our confidence that the program works.

Unfortunately, we tend to make the big assumption:

The only way to be absolutely sure that this is true:

• feed in every possible valid input value   (exhaustive testing)
• if each input produces the expected output, it's correct

• e.g. can't test all int arrays of size 100

... Testing Strategy

12/67

Realistic testing:

• determine classes/partitions of input data set
• choose representative input values from each class
• determine expected output for each input
• execute program using all representative inputs

• we have increased our confidence that the program works OK
• but we have not demonstrated that the program is correct

Developing Test Cases

13/67

Examples:

... Developing Test Cases

14/67

Years of (painful) experience have yielded some common bugs:

• iterating one time too many or one time too few through a loop
• using < rather than <=, or vice versa
• forgetting to handle the null/empty/zero case
• assuming that other parts of the program supply valid data
• assuming that library functions like malloc (see later) always succeed
• etc. etc. etc.

Requires you to understand the "limit points" of your program.

Making Programs Fail

15/67

Some techniques that you can use to exercise potential bugs:

• make array sizes tiny   (#define SIZE 2)
• write a special malloc that fails at random
• initialize data structures with a value other than zero
• test all possible combinations of parameter settings
• supply empty input   (empty file, no argv values)
• test on different compilers/machines/operating systems

Summary: Testing Strategies

16/67

• "Big Bang" approach
  • you write the entire program
  • then you design and run some test cases
  • generally a bad idea
• "As You Go" Testing
  • you write a small piece of code, then you test it
  • integrate it with other tested pieces and test again
  • repeat iteratively until the entire program has been constructed
• Regression Testing
  • re-run all testing after any changes to the system

Debugging

Debugging

18/67

Debugging = process of removing errors from software.

Required when observed output != expected output.

Typically ...

• not due to the software being full of errors
• is due to a small incorrect fragment of code

... Debugging

19/67

Consequences of bugs:

• compiler gives syntax/semantic error    (if you're very lucky)
• program halts with run-time error    (if you're lucky**)
• program never halts    (not lucky, but at least you know)
• program completes, but gives incorrect results    (if you're unlucky)

** but if the runtime error is due to pointer mismanagement, you're very unlucky

... Debugging

20/67

Debugging has three aspects:

• find the code that's causing the problem
• understand why it's causing the problem
• modify the code to eliminate the problem

• understanding a bug is (usually) easy once you find it
• fixing a bug is (usually) easy once you find/understand it

To fix: re-examine spec, modify code to satisfy spec.

... Debugging

21/67

The easiest bugs to find:

• the ones the compiler tells you about

• ones that are not reproducible (random)
• those involving pointers/dynamically-allocated memory

Corollary: an onlooker will find the bug quicker than you.

Debugging Strategies

22/67

The following will assist in the task of finding bugs:

• make the bug reproducible
• search for patterns in the failure
• divide and conquer (isolate the buggy region)
• write self-checking code (e.g. assert)
• write a log file (execution trace)
• draw a picture (esp. for pointer bugs)

Debuggers

23/67

Debuggers: tools to assist in finding bugs

Typically provide facilities to

• control execution of program
      (step-by-step execution, breakpoints)
• view intermediate state of program
      (values stored in data structures, control stack)

• gdb, lldb ... command-line debugger, useful for quick checks
• ddd ... visual debugger, can display dynamic data
• valgrind ... execution "harness" for pointer bugs

Exercise: Buggy Program

24/67

Spot the bugs in the following simple program:

int main(int argc, char *argv[]) {
	int i, sum;
	int a[] = {7, 4, 3};

	for (i = 1; i <= 3; i++) {
		sum += a[i];
	}
	printf(sum);
	return EXIT_SUCCESS;
}

What will we observe when it's compiled/executed?

The Debugging Process

The Debugging Process

26/67

Debugging requires a detailed understanding of program state.

The state of a program comprises:

• the location where it's current executing
• names/values of all active variables on the stack
• names/values of all data in the global/heap regions

• "at this point in the program, x==3, y==7, and z==-2"

... The Debugging Process

27/67

The real difficulty of debugging is locating the bug.

Since a bug is "code with unintended action", you need to know:

• in detail, what the code should do
• in detail, what the code actually does

The trick to effective debugging is narrowing the focus of attention.

That is, employ a search strategy that enables you to zoom in on the bug.

... The Debugging Process

28/67

When you run a buggy program, initially everything is ok.

At some point, the buggy statement is executed ...

• the program state changes from valid to incorrect
• but the program won't necessarily crash at this point

The goal: find the point at which the state becomes "corrupted"

• initially you know broadly where the problem is
• then you move to the function where the problem occurs
• the you find the precise statement with the bug

Locating the Bug

29/67

A simple search strategy for debugging is as follows:

• initially the whole program is "suspect"
• put a print statement in the middle of the suspect part of the program to display values of variables at that point
• if the values are correct, then the bug must be in the second part of the execution
• if the values are incorrect, the bug is in the first half
• now restrict your attention to just the relevant half of the program (it becomes the new "suspect part of the program")
• repeat the above steps

... Locating the Bug

30/67

At each stage you have eliminated half of the program from suspicion.

In not too many steps, you will have identified a specific buggy statement.

The problems:

• which variables to print? all? what if too many?
• how to work out where the "half-way execution point" is?

E.g.     x:int { y = x+x; } y==x2,   when intially   x==2

... Locating the Bug

31/67

A slightly smarter strategy, relying on the typical structure of programs:

• display all major data structures just after they have been initialised
  • typically requires to implement a print function for each data structure (e.g. 2d array)
• display major data structures at "strategic points" during the program's execution
• display major data structures at the end of program execution

• after the first, second, middle iterations of the main program loop
• after the keystroke that causes an interactive program to crash
• after the point where the last output from the program occurred

Examining Program State

32/67

A vital tool for debugging is a mechanism to display state.

One method: diagnostic printf statements of "suspect" variables.

Problems with this approach:

• it changes the program   (so you're not debugging quite the same thing)
• you may guess wrong about what the suspect variables are
• generally too much is printed   (e.g. printing to trace execution of a loop)

... Examining Program State

33/67

An alternative for obtaining access to program state:

• a tool that allows you to stop program execution at a certain point
• and then allows you to inspect the state (preferably selectively)

Debuggers also allow you to inspect the state of a program that has crashed due to a run-time error.

Often, this takes you straight to the point where the bug occurred.

C Program Execution

34/67

Under Unix, a C program executes either:

• to completion, producing results   (correct?)
• until the program detects an error and calls exit()
• until a run-time error halts it   (e.g. Segmentation violation)

Normal C execution environment:

... C Program Execution

35/67

C execution environment with a debugger:

Debuggers

36/67

A debugger gives control of program execution:

• normal execution   (run, cont)
• stop at a certain point   (break)
• one statement at a time   (step, next)
• examine program state   (print)

There are GUI front-ends available (e.g. xxgdb, ddd, ...).

• provide facilities such as graphical display of data structures

... Debuggers

37/67

For gdb, programs must be compiled with the gcc -g flag.

gdb takes two command line arguments:

$ gdb executable core

E.g.

$ gdb  a.out  core
$ gdb  myprog

The core argument is optional.

gdb Sessions

38/67

gdb is like a shell to control and monitor an executing C program.

Example session:

$ gcc -g -Wall -Werror -o prog prog.c
$ gdb prog
Copyright (C) 2014 Free Software Foundation, Inc...
(gdb) break 9
Breakpoint 1 at 0x100f03: file prog.c, line 9.
(gdb) run
/Users/comp1921 Starting program: ..../prog 

Breakpoint 1, main (argc=1, argv=0x7ffbc8) at prog.c:9
9               for (i = 1; i <= 3; i++ { 
(gdb) next
10              sum += a[i];
(gdb) print sum
$1 = 0
(gdb) print a[i]
$2 = 4
(gdb) print i
$3 = 1
(gdb) print a@1
$4 = {{7, 4, 3}}
(gdb) cont
...

Basic gdb Commands

39/67

quit -- quits from gdb

help [CMD] -- on-line help (gives information about CMD command)

run ARGS -- run the program

• ARGS are whatever you normally use, e.g.

  $ xyz  <  data
  

  is achieved by:

  (gdb) run  <  data
  

gdb Status Commands

40/67

where -- find which function the program was executing when it crashed.

list [LINE] -- display five lines either side of current statement.

print EXPR -- display expression values

• EXPR may use (current values of) variables.
• Special expression a@1 shows all of the array a

gdb Execution Commands

41/67

break [PROC|LINE] -- set break-point

On entry to procedure PROC (or reaching line LINE), stop execution and return control to gdb.

next -- single step (over procedures)

Execute next statement; if statement is a procedure call, execute entire procedure body.

step -- single step (into procedures)

Execute next statement; if statement is a procedure call, go to first statement in procedure body.

For more details see gdb's on-line help.

Using a Debugger

42/67

The most common time to invoke a debugger is after a run-time error.

If this produces a core file, start gdb ...

• it typically shows you the line of code causing the crash
• use where to find out who called the current function
• pay attention to parameter values in the stack
• use list to see the current function code
• display values of local variables

... Using a Debugger

43/67

Once you have an idea where the bug might be:

• set breakpoints slightly earlier in the code
• run the program again (supplying the same data)
• single-step through the suspect region of code
• check the values of suspect variables after each step

... Using a Debugger

44/67

Once you find that the value of a given variable (e.g. x) is wrong, the next step is to determine why it is wrong.

There are two possibilities:

• the statement that assigned a value to x is wrong
• the values of other variables used by that statement are wrong

if (c > 0) { x = a+b; }

If we know that

• x is wrong after this statement
• the condition and the expression correctly implement their specs

Laws of Debugging

45/67

Courtesy of Zoltan Somogyi, Melbourne University

Before you can fix it, you must be able to break it (consistently).
(non-reproducible bugs ... Heisenbugs ... are extremely difficult to deal with)

If you can't find a bug where you're looking, you're looking in the wrong place.
(taking a break and resuming the debugging task later is generally a good idea)

It takes two people to find a subtle bug, but only one of them needs to know the program.
(the second person simply asks questions to challenge the debugger's assumptions)

(In fact, sometimes the second person doesn't have to do or say anything! The process of explaining the problem is often enough to trigger a Eureka event.)

Possibly Untrue Assumptions

46/67

Debugging can be extremely frustrating when you make assumptions about the problem which turn out to be wrong.

Some things to be wary of:

• the executable comes from the source code you're reading
• the problem must be in this source file
• the problem cannot be in this source file
• code that calls a function never provides unexpected arguments
  (e.g. "this pointer will never be NULL; why would anyone pass a NULL?")
• library functions never return an error status
  (e.g. "malloc will always give the memory I ask for")

Performance Tuning

Software Development Process

48/67

Reminder of how software development runs ...

• specification   (via requirements analysis)
• design   (data structures, algorithms)
• implementation   (C code)
• testing, debugging   (code analysis)
• user testing   (if has a user interface)
• performance tuning   (if required)

Performance

49/67

Why do we care about performance?

Good performance → less hardware, happy users.

Bad performance → more hardware, unhappy users.

Generally:   performance = execution time

Other measures: memory/disk space, network traffic, disk i/o.

Execution time can be measured in two ways:

• cpu ... time your program spends in the processor
• elapsed ... wall-clock time between start and finish

... Performance

50/67

In the past, performance was a significant problem.

• much programming effort was spent on efficiency "tricks".

• execution efficiency achieved by "tweaking" code
• the understandability of the code

Knuth: "Premature optimization is the root of all evil"

Development Strategy

51/67

A pragmatic approach to efficiency:

• first, make the program simple, clear, robust and correct
• then, worry about efficiency ... if it's a problem at all

• good design is always critical
  (at design time, make sensible choice of data structures, algorithms)
• can handle efficiency at system level
  (e.g. buy a bigger machine, use compiler optimisation, ...)

... Development Strategy

52/67

Strategy for developing efficient programs:

1. Design the program well
2. Implement the program well
3. Test the program well
4. Only after you're sure it's working, measure performance
5. If (and only if) performance is inadequate, find the "hot spots"
6. Tune the code to fix these
7. Repeat measure-analyse-tune cycle until performance ok

Exercise: Prime Number Tester

53/67

Consider a function to test numbers for primality.

An integer n is prime if it has no divisors except 1 and n

Straightforward, literal, C implementation:

int is_prime(int n) {
	int i, ndiv = 0;
	for (i = 1; i <= n; i++) {
		if (n % i == 0) {
			ndiv++;
		}
	}
	return (ndiv == 2);
}

Implement, test, and examine performance. Tune, if necessary.

When to Tune?

54/67

We should only consider tuning the performance of a program:

• after testing shows program is correct,
• and only if the program is going to be used frequently.

Performance Analysis

55/67

Before we can tune the performance of a program

• we need to measure its performance
• on a range of inputs, including very large inputs
• and then analyse the reasons for its behaviour

• coarse-grained ... get an overview of performance characteristics
• fine-grained ... find detailed explanations for performance

• time -- cpu/elapsed time   (coarse measure of performance)

Benchmarks

56/67

A benchmark is

• a standard collection of sample data (inputs)
• useful for analysing overall performance of a program
• useful for comparing alternative implementations

E.g. sorting benchmark

Could potentially use an extension of the cases developed for testing the program.

... Benchmarks

57/67

Benchmark caveats:

• must compare programs in similar (identical) environments
  • eliminate system effects ... system load, caching
• avoid data sample bias ... use likely/varied sample data
• benchmarks give a coarse view of program performance

Benchmarks are not useful as a basis for performance tuning

• they provide no (direct) information to help analyse inefficiency

The time Command

58/67

The time command:

• monitors the execution of a program on some data
• gives an overview of resource usage for that execution

What resources it measures:

• user cpu time
  time spent in executing code for your program and the libraries it uses
• system cpu time
  time spent in executing system calls (e.g. open) on behalf of the program

... The time Command

59/67

What other resources it measures:

• elapsed time
  difference in wall-clock time between when the program started and finished
• % machine share
  what percentage of processor time was dedicated to the process during execution
• memory usage
  maximum amount of memory space occupied by the process
• i/o statistics
  how much disk traffic the program generated

... The time Command

60/67

Things to note when interpreting time output:

• cpu times are measured by sampling
  • may be different for each execution of the same program
  • to overcome sampling variation - run several trials and average
• elapsed time depends on machine share
  • if program gets greater share of the CPU, it finishes quicker
• different shells/Unixes have different format of time output

Exercise: Word Frequencies

61/67

Consider a program wfreq to process text files (via stdin):

• treat the input file as a sequence of words
• count the number of times each word occurs
• print a list of words and their occurrence frequencies

• on small, medium and "large" text files

Program Execution

62/67

Observation shows that most programs behave as follows:

• most execution time is spent in a small part of the code

This means that

• most of the code has little impact on overall performance
• small parts of the code account for most execution time

(Sometimes this require us to change the code that invokes the heavily-used code)

Performance Improvement

63/67

Once you have identified which region of code is "hot", can improve the performance of this code:

• change the algorithm and/or data-structures
  • may give orders-of-magnitude better performance
  • but it is extremely costly to rebuild the system
• use simple efficiency tricks to reduce costs
  • may improve performance by one order-of-magnitude
• use the compiler's optimization switches   (gcc -O)
  • may improve performance by one order-of-magnitude

Efficiency Tricks

64/67

Avoid unnecessary repeated evaluation ...

Compilers detect straight-forward examples but may not handle some examples obvious to humans.

for (i = 1; i <= N; i++) {
	x += f(y);
}

becomes

res = f(y);
for (i = 1; i <= N; i++) {
	x += res;
}

... Efficiency Tricks

65/67

Use local variables instead of global variables ...

May enable compiler to optimize some operations.

i = k + 1;
i = i - j;
x = a[i];

If i is local then compiler may generate:

x = a[k+1-j];

... Efficiency Tricks

66/67

Caching

• remember earlier results instead of recomputing
• store computed value with the data used to compute it

• perform input/output in chunks to minimise disk traffic

• can help to make loops uniform, thus reducing tests

• implement a function via a lookup table, rather than computing

Tips for Quiz 1 (during Week-5 lab)

67/67

• Understand and use argc, *argv[]. E.g.

  prompt$ ./myprog 123
  

  • type and value of argc?
  • type and value of argv[1][0]?
  • type and value of *argv[0]?
• Be able to
  • prompt for, and process, user input (e.g. ints, floats)
  • print error messages and return error status
  • use a (simple) given file.h, file.c
  • write a (simple) Makefile

Produced: 11 Aug 2016