TOPIC 3.4 PROGRAM TESTING

• Often it is not immediately obvious that there are logic errors in a program.
• Indeed some programs may function correctly for months or even years before a particular circumstance or piece of input data causes a problem or error to surface.
• It is the job of the programmer to test as far as possible that all parts of the program work correctly.
• It should be realised that complete testing is not possible except in the case of the most trivial program; once can never be completely certain that all errors have been removed, but sufficient tests can be performed to give a reasonable measure of confidence in the program.
• Consider the simple pseudocode segment given below which looks up a table and prints out the entry for 'OLIVER'.

OBJECTIVES OF TESTING

The objectives of testing can be stated in two basic questions:

1. Does the logic work properly?

• Does the program work as intended?
• Can it be made to 'crash'?

2. Is all the necessary logic present?

• Are there any function missing?
• Does the program or module do everything specified?

TYPES OF TESTING

(1) FUNCTIONAL TESTING; and

(2) LOGICAL (STRUCTURAL) TESTING

FUNCTIONAL TESTING

• Functional testing is carried out independently of the code used in the program.
• It involves looking at the program specification and creating a set of test data that covers all the inputs and outputs and program functions.
• This type of testing is also known as 'black box testing'.
• For example, to test the program which calculates check digits, we could draw up the following test plan:

• These tests seem to cover the problem quite well but nevertheless they do not completely test the program.

LOGICAL (STRUCTURAL) TESTING

• Logical testing ('white-box testing') is dependent on the code logic and derives from the program structure rather than its function.
• In other words, we study the program code and try to test each possible path in the program at least once.
• The problem with logical testing is that it will not detect missing functions; you cannot test what is not there.
• One method of devising a test plan is to start with a set of functional test cases and then add additional tests to exercise each statement in the program at least once, making sure that each decision is tested for all outcomes.

TESTING STRATEGIES

The two most common testing strategies are:

1) BOTTOM-UP TESTING

• A testing where individual modules are tested in a stand-alone fashion. Then individual modules are combined together into larger units and tested together and finally a system test is performed on the complete system.

2) TOP-DOWN TESTING

• A testing where the skeleton of the complete system is tested, with individual modules being replaced by 'stubs' which may for example simply display a message to say that a certain procedure has been executed. As individual modules are completed they are included in subsequent tests.

DESIGNING A TEST PLAN

Good testing requires:

• a thorough knowledge and understanding of what the program is supposed to do;
• thinking out what ought to be tested;
• working out expected results for each of the test cases;
• writing down the test plan.

Note:

• Since we cannot test everything, each test must be carefully planned to provide more information about the program.
• A major benefit of preparing a comprehensive test plan with expected results is that it forces the programmer to think carefully about the program and often errors are spotted even before running the test.
• Test data should test the program to its limits and should include: -

PROGRAM ERRORS

A program may have any or all of four types error:

(1) Syntax error

• A statement in the program violates a rule of the language; this could be as simple misspelling of a keyword, a missed punctuation mark, or a wrongly formed statement.
• For example;                       

        Writteln('This statement contains 2 syntax errors);

         (misspelt keyword and missing quote-mark)

(2) Semantic error

• Violating rules of the language.
• For example by speicifying the wrong number or type of parameters in a subroutine call, or trying to assign a real value (one with a decimal point) or a string of characters to a variable which ahs been declared as an integer, e.g.

        Num1 := 'John Robert Sharpe' {Num1 being an integer variable}

(3) Logic error

• The program runs but gives the wrong answer or performs wrongly in some way.
• For example, the programmer writes

        Net := Gross + Tax;

       when the statement should have been written below:

       Net := Gross - Tax;

(4) Run-time error

• The program has no logic or syntax errors (though possibly weaknesses in not anticipating exceptional circumstances and catering for them), but 'crashes' during execution.
• This may be caused, for example, by attempting to open a file on a floppy disk which is not in the drive, or division by zero caused by the user entering 0 when the program expected a non-zero number, or entering a character when a number was expected.

NOTE:

• The first two types of error will be detected by the compiler, which will usually pinpoint the error and give a suitable (thought sometimes puzzling) error message.
• Logic errors and run-time errors are detected by rigorous testing with data for which the expected answers have been calculated.
• A test plan should test all unexpected occurrences.

DESIGNING TEST DATA/TEST CASES

• Program testing is best performed at three different levels:

        (1) Components (subprograms) need to be individually tested to ensure that they conform to specifications.

        (2) When combined to form a program, the components must not act in unexpected ways, so each program module must be tested.

        (3) The suite of programs as a hole must be tested to ensure that al programs integrate correctly.

• Although program testing in theory should be exhaustive with every possible route being tested, with a large program this is impractical as it would mean performing perhaps millions of tests. At a minimum, the test cases should be selected to ensure that:

            - Every statement in the program is executted at least once.

            - The effectiveness of all sections of codde which detect invalid input is tested.

            - The accuracy of all processing is verifiied.

            - The program operates according to its orriginal design specifications.

• There are three general categories of test data:

            1. NORMAL DATA - data that the program is designed to handle;

            2. EXTREME VALUES - valid data at the limits of acceptability;

            3. EXCEPTIONAL OR INVALID DATA - the programmer must assume that users sooner or later will attempt to enter invalid data, and the program must not as a consequence 'crash' or give wrong results.

RESOURCES:

1) P M Heatcote, [A level Computing, 3rd Edition], Ashford Colour Press Ltd, 1996.

2) P Heatcote & K Bond, [A Level Computing], Letts Educational Ltd, 1997.