CEN 5070 – Software V&V Automation for Program Testing © 2001-2008, E.L. Jones

CEN 5070 – Software V&V Automation for Program Testing

© 2001-2008, E.L. Jones

October 2008 Program Test Automation 2

Purpose

This lecture introduces design patterns to automate the testing of executable programs (or applications). The testing lifecycle phases involved are implementation, execution and evaluation.


Agenda

• Revising the Test Script

• The Need for Shell Scripts

• Putting it All Together

• Invoking program under test

• Set-up: required files

• Insertion of stored test cases• Capture and formatting of test results

• Test execution instructions

• Verification of test results


Shell Scripts

Shell scripts automate the use of operating system commands and utilities for program execution, and management of test data and results. We begin with the test environment.


Application Test Environment

Application Driver

Test data

Application Under Test

Test History Index‘thx_xxx.txt’

Verification

Actual results

Test results‘utr_xxx.txt’

Test History Index‘thx_xxx.txt’

Test Case Data‘tdi_xxx.txt’

Test Script‘uts_xxx.txt’

Test Log‘utl_xxx.txt’

1

3

4

25

6

7

8

9


0. Application Driver Specification

Application Test Driver

(1) tdi_ filename

(2) thx_ filename

(3) utr_ filename

thx_ file

utr_ file

tdi_ file

thx_ file

Application .run file


1. Automated Invocation of Interactive Application

echo 5 4 | ./Add.run

echo “5 4” | ./Add.run

Pipe inputs into stdin stream. Add.run reads inputs from stdin.

WARNING: Piping as a quoted string does not work properly.


2. Automated Insertion of Stored Test Cases

set testcase=(`cat tdi_Add.txt`)

set num1=$testcase[1]set num2=$testcase[2]set expected=$testcase[3]

echo $num1 $num2 | ./Add.run

shift testcaseshift testcaseshift testcase

Create list of all test cases in in file. A list is an array $testcase[1..$#testcase]

Extract test case into application specific variables.

Pipe first two stimui as input to stdin and invoke Add.run

Each shift command removes one (used) value from front of the list.


3. Automated Capture of Execution Results

set res = `echo $num1 $num2 | ./Add.run`

Set sum=`echo $res | awk ‘{print $5}’`

Variable $res contains application output (prompts and results).

Based on the FORMAT of output, “parse for” the results using awk..

Example: x=## y=## SUM = 14

The answer (sum) is the 5th token in the output.


@ tcnum++echo –n “$runid $tcnum num1=$num1 num2=$num2 “echo “SUM=< $sum $expected >” >> utrFile

NOTES: The $runid must be managed (see next slide). The name of the utr_ file is an argument.After the testcase data have been used, remove them in preparation for the next test case.

Create utr record showing stimuli and results. Record unique by testrun# and testcase#.

4. Store Formatted Test Result Record


5. Set the Test Run#

if (! –e thx_File) then #-| generate header lines.endif

set runid=`tail -1 thxFile | awk –F# ‘{print $1}’`

#-| The current runid is 1 +previous runid.@ runid++

If the thx_ file does not exist, create the file with the required header lines.NOTE: The thx_ file name is the second argument.

the last line in the thx_ file contains the runid.


6. Update Test History Index

#-| Write runid to thx_ file.echo “#$runid $user `date`“ >> thx_File

#-| Write runid to thx_ file.echo “#$runid $user `date`“ >> utr_File

Append a line to the thx_ file to record the fact that this test run is beginning.

Also append a line to the utr_ file to create a “section heading” for the results from the test run.


7. Update Test Script

SETUP Section

TEST RUN Section

B. Unit Execution

C. Test Log Preparation

D. Test Sequence

Include run_ shell script, test log, and the files used as inputs,

Modify to reflect automated steps.

Include invocation of run_ script.

Log is required if run_ script does not contain an oracle.

Remove individual test cases: these will now be read from the tdi_ file.


7. Update Test Script (cont’d)

CLOSE-OUT Section

A. Test Log

B. Test Documentation Files

C. Saving Artifacts to Repository

This is the EVALUATION and ARCHIVAL activity.

Explain how to update test log from trs_ and thx_ files, if oracle not implemented. MUST describe each discrepancy and propose probable cause of defect.

Identify each artifact and how it is to be used/handled. Some may have to be modified. All go to TS repository.file.


8. Update Test Log

Test Log MUST include: a) Verification Summary;

b) Description of each defect/ hypothesis of root cause.

c) Describe any “fixes” made to enable the test to be run again.

Count all defects (unexpected results/inability to perform steps)

Description should be specific and general (if a pattern is seen).Hypothesis is an educated guess as to the root cause. Program is wrong Test case is wrong Test automation is wrong

Fixes to test cases or various scripts are the responsibility of the tester.


9. Implement Test Oracle

#-| Case 1: One exected result.if ($res == $expected) then echo “ PASSED” >> utr_Fileelse echo “ FAILED” >> utr_Fileendif

echo –n “#$runid $user `date` “ >> utr_Fileecho “$passed PASSED $failed FAILED“ >> utr_File

Compare actual result to expected result, and append “PASSED” or “FAILED” to test record.

NOTE: Include COUNTERS for passes and failures. These can be included in FOOTER records in both utr_ and thx_ files.


9. Implement Test Oracle (cont’d)

#-| Case 2: Multiple expected #-| results (e.g. 3).

@ match1 = ($res1 == $exp1)@ match2 = ($res2 == $exp2)@ match3 = ($res3 == $exp3)@ matched = $match1 + $match2 \ $match3If ($matched == 3) then echo “ PASSED” >> utr_Fileelse echo “ FAILED” >> utr_Fileendif

Note that the result from each comparison is 1 (true) or 0(false). ALL must be true to PASS.

NOTE: Include COUNTERS for passes and failures.


Shell Scripts for Testing Programs -- Summary

Unix provides a powerful set of primitive operating system and scripting facilities for automating the testing of programs. This lecture has shown that automation can be applied to support most of the SPRAE principles. The design patterns shown can be extended to programs of moderate complexity.

Documents

CEN 5070 – Software V&V Automation for Program Testing © 2001-2008, E.L. Jones