The Challenge of Teaching Software TestingThe Challenge of Teaching Software TestingEarlier into Design CoursesEarlier into Design Courses
Ellen Francine Barbosa 1Richard LeBlanc 2
Mark Guzdial 2
José Carlos Maldonado 1
1 University of São Paulo (Brazil)2 College of Computing, Georgia Institute of Technology (USA)
Workshop on the Teaching of Software Testing
Agenda
IntroductionCS2340 Course
Goals and audiencePrerequisitesDevelopment process
Testing Approach for CS2340 CourseEquivalence Partitioning Criterion
Students’ EvaluationTesting earlyUsing testing criteriaWriting test plans
Conclusions and Further Work
Introduction
Teaching Software Testing…
Waterfall Model» Analysis, Design, Coding, Testing, Code Release
Students’ perception about Sofware Testing» Not a popular discipline
They have to do to show that their programming assignments work as required
Software Testing has traditionally been taught according to the classical approaches to software development.
Something that happens at the end of the development process.
Introduction
Teaching Software Testing…Different approach
eXtreme Programming (XP)» Write tests even before writing code» Development setting
Enhance the system’s qualityReduce development costs
Introduce testing practices during all phases of thedevelopment process, starting from analysis and going through
design and implementation phases.
“Test first, code later”.
Introduction
Objective
Investigate different learning opportunitiesProvide extra motivation
» Ideas for modern methodologies (like XP)
CS2340 Course» Encourage the students on thinking about Software
Testing earlier in their projectsProvide a more pragmatic view of testing
» Evaluate the students’ attitude
Explore the impact of introducing testing practices as early as possible, throughout all phases of the development process.
CS2340 – Objects and Design
Main Goal
Object-oriented paradigm
AudienceSecond-year undergraduate students
» Fall 2002: 160 students
StructureOne semester
» 2 lecture classes per week» 2 exams, 1 development project
Explore higher-level issues of analysis and design, with someemphasis on user interface development.
CS2340 – Objects and Design
Pre-requisitesIntroduction to Computing
» Design, construction and analysis of algorithmsObject-Oriented Programming
» JavaLanguages and Translation
» Issues of language implementation, tokenizing and parsing
Software Practicum (sometimes concurrently)» Basics of Software Engineering
General view on Software Testing
CS2340 – Objects and Design
ProjectsTeam-orientedDevelopment of a small-to-medium sized OO systemFall 2002: Genealogy Information
» Create a system to...Collect genealogical informationNote inconsistencies and flaws in the databaseProvide some graphical representationsSupport a standard genealogical information format
CS2340 – Objects and Design
Development ProcessTraditional phases of the object-oriented development
» OO AnalysisDevelopment of scenarios and CRC(Class-Responsibility-Collaborator) cards
» OO DesignDevelopment of UML class diagrams
» OO ProgrammingUse of Squeak language (variant of Smalltalk)
– Open source and highly portable language– Multimedia support– Good infrastructure for complex projects
Squeak: supports the XP’s testing-driven practices– S-Unit testing framework
CS2340 – Objects and Design
Development ProcessRegarding Software Testing...
» Fall 2002: first time that testing-related activities were explored in the context of the CS2340 classes
Decide what testing criteria to use– Develop lecture materials
Figure out how to integrate testing into student assignments
– How they should be evaluated and gradedPut together an assessment to see how well it worked
No systematic method to test the systems had been adopted in theprevious terms of the CS2340 course.
Testing Approach for CS2340 Course
Equivalence Partitioning CriterionStarting point to fit testing practices into the scope of the course
» Key criterion for functional testing» Seems to be easily understood even by students with
little experience on testingEducational setting
» Small programsNumeric input values
» Genealogy systemInput domain involves more complex types of elements
– How to apply the criterion?
Testing Approach for CS2340 Course
Equivalence Partitioning CriterionSet of “directions”
» Help the students on using the criterion in their projects
OO AnalysisCRC cards and scenarios +
OO DesignUML class diagrams +
OO ImplementationSqueak +
Definition of input conditionsIdentification of equivalence classes
Definition of test casesDevelopment of test plans
Execution of test cases (S-Unit framework)
ATM Simulation System
Specification (simplified)The customer is required to enter the account number and the PIN.
» There is no need of an ATM card.The ATM must provide the following transactions to the customer:
» Cash Withdrawals, Deposits, Tranfers and Balance Inquiries.» Only one transaction is allowed in each session.
The ATM communicates the transaction to the bank and obtains verification that it was allowed by the bank.
» If the bank determines that account number or PIN is invalid, the transaction is canceled.
The ATM has an operator panel that allows an operator to start and stop the servicing of customers.
» When the machine is shut down, the operator may remove deposit envelopes and reload the machine with cash.
» The operator is required to enter the total cash on hand before starting the system from this panel.
ATM Simulation System
Classes...ATMCashDispenserEnvelopeAcceptorOperatorPanelSessionTransactionWithdrawalTransactionDepositTransactionTransferTransactionInquiryTransactionBank
Some Scenarios...System StartupSessionCash Withdrawal TransactionDeposit TransactionTransfer TransactionBalance Inquiry
CRC cards...
1. Analyze each CRC card, looking for “suggestions” on the input data items provided by the user.
(a) Consider the responsibilities in a macroscopic way. Focus on what each class really has to do.
(b) Analyze the responsibilities of the collaborators too.
(c) Write down all the input data items related to a specific class.
Defining Input ConditionsInput conditions are closely related with
input data provided by the user.
Input conditions are closely related with
input data provided by the user.
Although the CRC cards do not explicitly deal with the input data items, analyzing
their responsibilities under amacroscopic perspective can provide
some insight into that.
Although the CRC cards do not explicitly deal with the input data items, analyzing
their responsibilities under amacroscopic perspective can provide
some insight into that.
Since each class can interact with others, the collaborators should also be
investigated in order to find the right class where a specific input item is being treated.
Since each class can interact with others, the collaborators should also be
investigated in order to find the right class where a specific input item is being treated.
Defining Input Conditions
Responsibility CollaboratorStart up system operation (initial cash) OperatorPanelStart a session for each customer SessionShut down on operator request OperatorPanelGet PIN from the customerGet transaction choice from the customerGet account from the customerGet amount entry from the customer (Class xxxTransaction)Verify the availability of cash for withdrawal CashDispenserDispense cash CashDispenserAccept deposit envelope EnvelopeAcceptor
Responsibility CollaboratorStart up system operation (initial cash) OperatorPanelStart a session for each customer SessionShut down on operator request OperatorPanelGet PIN from the customerGet transaction choice from the customerGet account from the customerGet amount entry from the customer (Class xxxTransaction)Verify the availability of cash for withdrawal CashDispenserDispense cash CashDispenserAccept deposit envelope EnvelopeAcceptor
Class ATMClass ATM
Responsibility CollaboratorSet initial cash on hand at startup Operator PanelReport cash availableDispense cash
Responsibility CollaboratorSet initial cash on hand at startup Operator PanelReport cash availableDispense cash
Class CashDispenserClass CashDispenser
Responsibility CollaboratorAccept deposit envelope
Responsibility CollaboratorAccept deposit envelope
Class EnvelopeAcceptorClass EnvelopeAcceptor
Responsibility CollaboratorGet initial cash on hand from operator
Responsibility CollaboratorGet initial cash on hand from operator
Class OperatorPanelClass OperatorPanel
Defining Input Conditions
Responsibility CollaboratorPerform session use case ATM, TransactionPerform invalid data ATM, BankFurnish account to TransactionFurnish PIN to Transaction
Responsibility CollaboratorPerform session use case ATM, TransactionPerform invalid data ATM, BankFurnish account to TransactionFurnish PIN to Transaction
Class SessionClass Session
Responsibility CollaboratorPerform a particular transaction use case WithdrawalTransaction,
DepositTransaction,TransferTransaction,InquiryTransaction,Session, Bank
Responsibility CollaboratorPerform a particular transaction use case WithdrawalTransaction,
DepositTransaction,TransferTransaction,InquiryTransaction,Session, Bank
Class TransactionClass Transaction
Defining Input Conditions
Responsibility CollaboratorGet specifics from customer ATMSend to bank Session, BankDispense cash and notify bank when complete ATM, Bank
Responsibility CollaboratorGet specifics from customer ATMSend to bank Session, BankDispense cash and notify bank when complete ATM, Bank
Class WithdrawalTransactionClass WithdrawalTransaction
Responsibility CollaboratorGet specifics from customer ATMSend to bank Session, BankAccept envelope and notify bank when complete ATM, Bank
Responsibility CollaboratorGet specifics from customer ATMSend to bank Session, BankAccept envelope and notify bank when complete ATM, Bank
Class DepositTransactionClass DepositTransaction
Responsibility CollaboratorGet specifics from customer ATMSend to bank Session, Bank
Responsibility CollaboratorGet specifics from customer ATMSend to bank Session, Bank
Class TransferTransactionClass TransferTransaction
Defining Input Conditions
Responsibility CollaboratorGet specifics from customer ATMSend to bank Session, Bank
Responsibility CollaboratorGet specifics from customer ATMSend to bank Session, Bank
Class InquiryTransactionClass InquiryTransaction
Responsibility CollaboratorInitiate withdrawalFinish withdrawalInitiate depositFinish depositDo transferDo inquiry
Responsibility CollaboratorInitiate withdrawalFinish withdrawalInitiate depositFinish depositDo transferDo inquiry
Class BankClass Bank
Defining Input Conditions
Defining Input Conditions
Input Data Items
accountPIN
transaction
accountPIN
transaction
Class ATMClass ATMinitial cashinitial cash
Class OperatorPanelClass OperatorPanel
amountamount
Class WithdrawalTransactionClass WithdrawalTransaction
amountamount
Class DepositTransactionClass DepositTransaction
receiver accountamount
receiver accountamount
Class TransferTransactionClass TransferTransaction
2. Identify the main characteristics of each input data item.
3. Identify the interactions among the input data items.Think about how a specific input entry can be related to the other input data of the system.Focus on the system’s operation (scenarios can help on this).
4. Write down the characteristics and interactions of all input data items related to all classes of the system.
Defining Input Conditions
Each input data item has specific characteristics related to it, which can directly
affect the system’s behavior.
Each input data item has specific characteristics related to it, which can directly
affect the system’s behavior.
Input data items can interact with each other, also resulting on different values
in the output domain.
Input data items can interact with each other, also resulting on different values
in the output domain.
The set of characteristics and interactions related to all input data items can be seen as the
input conditions for the system.
The set of characteristics and interactions related to all input data items can be seen as the
input conditions for the system.
Input Data Item Characteristic / Interactionaccount matching with bankPIN matching with accounttransaction type
Input Data Item Characteristic / Interactionaccount matching with bankPIN matching with accounttransaction type
Class ATMClass ATM
Input Data Item Characteristic / Interactioninitial cash valid characters
availability of money in cash dispenser
Input Data Item Characteristic / Interactioninitial cash valid characters
availability of money in cash dispenser
Class OperatorPanelClass OperatorPanel
Defining Input Conditions
Input Data Item Characteristic / Interactionamount valid characters
availability of money in the account for withdrawal
Input Data Item Characteristic / Interactionamount valid characters
availability of money in the account for withdrawal
Class WithdrawalTransactionClass WithdrawalTransaction
Input Data Item Characteristic / Interactionamount valid characters
Input Data Item Characteristic / Interactionamount valid characters
Class DepositTransactionClass DepositTransaction
Input Data Item Characteristic / Interactionreceiver account matching with bankamount valid values
availability of money in the sender account for transfer
Input Data Item Characteristic / Interactionreceiver account matching with bankamount valid values
availability of money in the sender account for transfer
Class TransferTransactionClass TransferTransaction
Defining Input Conditions
1. Analyze the set of input conditions for the system.
(a) Look for related conditions. Try to join them.
(b) Look for broad conditions. Try to refine them.
2. Analyze each input condition separately.
(a) Think about the valid values to satisfy the input condition. They will correspond to the elements of a valid equivalence class.
(b) Think about other possible values associated to the input condition, i.e., the invalid values. They will correspond to the elements of a invalid equivalence class.
3. Enumerate the equivalence classes, assigning a unique number to each class.
Identifying Equivalence ClassesIf two or more conditions are closely related to
each other, it can make more sense to join them into a single condition.
If two or more conditions are closely related to each other, it can make more sense to join them
into a single condition.If a condition is too broad, it can be
necessary to refine it by creating two or more specific ones.
If a condition is too broad, it can be necessary to refine it by creating two
or more specific ones.
The valid and invalid equivalence classes are established by analyzing the input conditions, in terms of the
correct and incorrect values needed to cover them.
The valid and invalid equivalence classes are established by analyzing the input conditions, in terms of the
correct and incorrect values needed to cover them.
Input Condition Valid Invalidaccount matches with bank
size (a) of account a = 8 (1) a > 8 (2)0 ≤ a < 8 (3)
valid characters for account numerical (4) alpha, special (5)
PIN matches with accountsize (p) of PIN p = 4 (6) p > 4 (7)
0 ≤ p < 4 (8)valid characters for PIN numerical (9) alpha, special (10)
type of transaction withdrawal (11) none (“blank choice”) (15)deposit (12)transfer (13)inquiry (14)
Input Condition Valid Invalidaccount matches with bank
size (a) of account a = 8 (1) a > 8 (2)0 ≤ a < 8 (3)
valid characters for account numerical (4) alpha, special (5)
PIN matches with accountsize (p) of PIN p = 4 (6) p > 4 (7)
0 ≤ p < 4 (8)valid characters for PIN numerical (9) alpha, special (10)
type of transaction withdrawal (11) none (“blank choice”) (15)deposit (12)transfer (13)inquiry (14)
Class ATMClass ATM
Identifying Equivalence Classes
Input Condition Valid Invalid
valid characters for amount numerical (1) alpha, special (2)
availability of money balance - amount ≥ 0 balance - amount < 0in the account for withdrawal (3) (4)
Input Condition Valid Invalid
valid characters for amount numerical (1) alpha, special (2)
availability of money balance - amount ≥ 0 balance - amount < 0in the account for withdrawal (3) (4)
Class WithdrawalTransactionClass WithdrawalTransaction
Identifying Equivalence Classes
1. Derive test cases associated with valid classes to cover all of them. A test case can cover a number of valid equivalence classes, as large a set as possible.
2. Derive test cases associated with invalid classes to cover all of them. A test case should be written for each invalid equivalence class.
3. Write a test plan providing information on the test cases.
Defining Test Cases
The test cases are defined at the same time as the development of UML class diagrams,when the students
have to deal with more detailed issues of the design (such as instance variables and methods for the classes).
The test cases are defined at the same time as the development of UML class diagrams,when the students
have to deal with more detailed issues of the design (such as instance variables and methods for the classes).
Students have to provide information on:- purpose of the test case- system’s class it tests
- equivalence class it covers- expected result, obtained result.
Students have to provide information on:- purpose of the test case- system’s class it tests
- equivalence class it covers- expected result, obtained result.
At the end of the OOD phase, a number of test cases have already been written, even
before the students have started coding.
At the end of the OOD phase, a number of test cases have already been written, even
before the students have started coding.
The definition of test cases to cover the equivalence classes requires a more detailed knowledge on the
classes (and objects) in terms of structure, attributes, and services that need to be provided.
The definition of test cases to cover the equivalence classes requires a more detailed knowledge on the
classes (and objects) in terms of structure, attributes, and services that need to be provided.
Test CaseEquivalence Class #
Expected Result
Actual Result
Class
Balance: US$ 100,00
Amount: US$ 90,00
(3) Transaction OK
Balance: US$ 10,00
Transaction Canceled
WithdrawalTransaction
Error: Debugging
Defining Test Cases
Test plan
Actual Result: fill in after the test case execution
Executing Test Cases
S-UnitUnit testing framework
» Squeak’s environmentStructure, describe the context of test cases and run them automaticallyDevelopment of a well-documented test suiteMechanism to motivate regression tests
None specific structural testing criterion was adopted
» Instead… some “general” guidelines…Test the base classTest each subclass in conjunction with the base class
OO Implementation» Design structural test cases
Test plans» Execute the structural test cases against the system
S-Unit framework
Students were encouraged to keep defining test cases while coding the system, based on its implementation details.
Testing Approach for CS2340 Course
Relevant PointsCRC cards can be helpful to define input conditions and…Input conditions can be useful in checking the consistency of CRC cards
» Responsibilities and interactions among classes
Defining input conditions and equivalence classes during the OOA phase can provide a better understanding of the system’s requirements
Writing test cases even before writing code (OOD phase)» Encourage the students to think about the functionality they
are designing
Testing Approach for CS2340 Course
SurveyApplied at the end of the course
» After students had finished their projectsVoluntary
» 105 studentsQuestions consisting in some statements
» Students were required to choose all the statementsthey agreed with
Evaluate the student’s attitude toward…» XP idea of starting testing early» Using testing criteria» Writing test plans
Students’ Evaluation
Students’ Evaluation
XP testing ideas
I prefer to deal with testing after coding the system,at the end of the development process. 12I prefer to deal with testing after coding the system,at the end of the development process. 12
I prefer to apply testing practices through all thephases of the development process. 53I prefer to apply testing practices through all thephases of the development process. 53
Statement StudentsStatement Students
Start thinking about testing in the analysis phase ishelpful to better understand the system’s functionality. 41Start thinking about testing in the analysis phase ishelpful to better understand the system’s functionality. 41
I feel it helpful to design a set of test cases at leastbefore implementing the system. 23I feel it helpful to design a set of test cases at leastbefore implementing the system. 23
Positive Attitude!!!Positive
Attitude!!!
Further attempts should be made to integrate the idea of “testing early” into
CS courses as a way improvestudents’ attitude toward testing and their
ability to do it effectively.
Further attempts should be made to integrate the idea of “testing early” into
CS courses as a way improvestudents’ attitude toward testing and their
ability to do it effectively.
Students’ Evaluation
Using testing criteria
I prefer to test the system by my “own way”,without applying a specific testing criterion. 46
I prefer to test the system by my “own way”,without applying a specific testing criterion. 46
I feel more confident about the test cases when working with some testing criterion. 21
I feel more confident about the test cases when working with some testing criterion. 21
Statement StudentsStatement Students
I feel more confident about the system’s qualitywhen working with some testing criterion. 36
I feel more confident about the system’s qualitywhen working with some testing criterion. 36
I feel motivated to think about common errors thatcould be committed during the development when working with some testing criterion. 29
I feel motivated to think about common errors thatcould be committed during the development when working with some testing criterion. 29
I feel that working with some testing criterionI can reduce the total development effort. 10
I feel that working with some testing criterionI can reduce the total development effort. 10
35 students were strictly “against” the idea of working
with testing criteria.
35 students were strictly “against” the idea of working
with testing criteria.
70 students agreed on the relevance of applying testing criteria for some aspect of the
development process.
70 students agreed on the relevance of applying testing criteria for some aspect of the
development process.
Using testing criteriaMoreover…
» Negative attitude on using Equivalence Partitioning
Keep refining/improving our directions for applying the criterion in the next offerings of the courseInvestigate the application of other testing criteria
Students’ Evaluation
Teaching the criterion for testing systems whose input valuesare not numerical requires more investigation.
Writing testing plansStudents’ opinion on having to do a test plan
Completely worthless / worthless 62Completely worthless / worthless 62
Neutral position 31Neutral position 31
Attitude StudentsAttitude Students
Valuable / really valuable 12Valuable / really valuable 12
Negative Attitude!!!Negative
Attitude!!!
Students’ Evaluation
I believe it is more useful to elaborate test plans forcomplex systems than for small/medium systems. 54
I believe it is more useful to elaborate test plans forcomplex systems than for small/medium systems. 54
Students’ Evaluation
Writing testing plansSome statements about test plans
Statement StudentsStatement Students
I believe it is valuable to elaborate test plans aspart of any development process. 23
I believe it is valuable to elaborate test plans aspart of any development process. 23
Test plans are valuable as a wayto keep track of test cases. 13
Test plans are valuable as a wayto keep track of test cases. 13
Test plans are valuable when changesin the system are required. 16
Test plans are valuable when changesin the system are required. 16
One reason for the “high rejection” of creating test plans can be related to the kind of system
(small-to-medium size) the students had to develop in the CS2340 projects.
One reason for the “high rejection” of creating test plans can be related to the kind of system
(small-to-medium size) the students had to develop in the CS2340 projects.
37 students (from the 62 who had a negative attitude toward test plans)15 students (from the 31 who presented a neutral position)02 students (from the 12 who had a positive attitude)
37 students (from the 62 who had a negative attitude toward test plans)15 students (from the 31 who presented a neutral position)02 students (from the 12 who had a positive attitude)
Some of these students were the same who had a negative attitude
toward test plans.
Although this specific group of students had been “against” the
test plans idea, they were able to recognize its usefulness for some
aspects of the development process.
Some of these students were the same who had a negative attitude
toward test plans.
Although this specific group of students had been “against” the
test plans idea, they were able to recognize its usefulness for some
aspects of the development process.
Conclusions
Positive attitudes towards…Starting testing activities as early as possibleS-Unit
Negative attitudes towards…Equivalence PartitioningTest plans
XP’s testing-driven practices can be seen as aninteresting and challenging mechanism to create new
Software Testing learning opportunities.
Further Work
CS2340 CourseKeep investigating ways to introduce testing practices into the context of the course
» Introduce the use of S-Unit earlier in the development processKeep applying the Equivalence Partitioning
» Investigate better ways for teaching the criterion» Refine/improve our directions
Explore the use of other testing criteria» Functional and structural
Explore the idea of “testing early” into the context of other courses
Software TestingSoftware Engineering