©2007 S. T. Frezza

‘Real World Problems’ as Assessment of Software

Engineering

Stephen T. Frezza, Ph.D., C.S.D.P.

Erie, Pennsylvania

©2007 S. T. Frezza

1) CSDP preparation

2) Organized around SE topics

3) Question + answer format

4) Style: Multiple-choice, standardized-examination

Real-World Software Real-World Software Engineering ProblemsEngineering Problems

Useful for classroom assessment?Useful for classroom assessment?

©2007 S. T. Frezza

ExperimentExperimentApproachApproach

1) Pose one ‘real-world’ question related to recent course materials.

2) Use Turning Point™ (PRS)

3) Allow anonymous answers

4) Record individual scores for assessment.

5) Present: Statistics & Correct response rationale

Goals/expectationsGoals/expectations Will help stimulate

classroom interaction and student performance.

Time investment will reinforce student learning, rather than detract from it.

Sample QuestionSample Question

©2007 S. T. Frezza

Which requirements defects can be identified in Jo’s Which requirements defects can be identified in Jo’s requirements list?requirements list?

Question Question 3D3D: :

Jo is gathering the requirements for a software-controlled Jo is gathering the requirements for a software-controlled furnace. After interviewing several users, Jo obtained the furnace. After interviewing several users, Jo obtained the following requirements list:following requirements list:

I.I. AmbiguousAmbiguousII.II. Design Design

dependentdependentIII.III. IncompleteIncompleteIV.IV. UnverifiableUnverifiable

R1.R1. Gas inlet valves should be always open when furnace is Gas inlet valves should be always open when furnace is heating.heating.

R2.R2. Heating stops when furnace temperature reaches 150 ºC.Heating stops when furnace temperature reaches 150 ºC.

R3.R3. Furnace temperature should increase gradually when Furnace temperature should increase gradually when heating.heating.

R4. R4. The gas inlet valves should be closed when the temperature The gas inlet valves should be closed when the temperature goes above 200ºC.goes above 200ºC. 40% 40%

20%

0%

a.I only

b.I and II only

c.II and III only

d.I, III and IV only

©2007 S. T. Frezza

Correct Response: dCorrect Response: dExplanationThe word “gradually” makes R3 ambiguous and unverifiable. An unambiguous, verifiable expression for R3 would be, “Furnace

temperature should increase no more than 1ºC/second when heating.” Even better would be, “Furnace temperature should increase between

1ºC/second and 2ºC/second when heating.”

The set of requirements is also incomplete, because they do not explain what happens when temperature is over 150ºC (heating done) and 200ºC (valve closing). Consider the implications of R2:

We suppose the valves remain open, but we cannot be sure about it: R4 may be a safety requirement, and unrelated to R1 or R2.

In part, the incompleteness is caused because R1 and R2 interact in a very subtle (and undefined) way.

©2007 S. T. Frezza

Which of the following is NOT a Which of the following is NOT a characteristic of the application of use characteristic of the application of use cases as a modeling tool?cases as a modeling tool?

Your organization employs Use Cases as its requirements modeling toolYour organization employs Use Cases as its requirements modeling tool

Question Question 20B20B::

You are a requirements engineer on a project that will replace the current training You are a requirements engineer on a project that will replace the current training system for a large company's training department. The new orientation process system for a large company's training department. The new orientation process will be online, with a trainer and a group of new employees. While some courses will be online, with a trainer and a group of new employees. While some courses will be offered online, others will be offered in a traditional mode, complemented will be offered online, others will be offered in a traditional mode, complemented with computer-based activities. The course registration and billing systems will be with computer-based activities. The course registration and billing systems will be automated, e.g., the employee's department will be automatically billed for courses.automated, e.g., the employee's department will be automatically billed for courses.

40%

0%

20%

40%

1. Use cases present the functionality of the system in a stepwise fashion.

2. Use cases present the system from the user's point of view.

3. Each use case step can represent more than one requirement.

4. Use cases present the system from the system's point of view.

©2007 S. T. Frezza

Correct Response: dCorrect Response: d(1) Use cases used to model the interactions between external entities,

called “actors”, and the system being modeled. The end user would be one of the actors in these models, so the user’s point of view (1) is presented.

(2) The use case notation explicitly embodies the concept of sequence of interactions, and so presents functionality in a step-wise fashion (2).

(3) The analyst may further elaborate the use case by adding scenarios for normal interaction as well as for exception conditions, or may represent further detail in a sequence diagram. Thus, A use case may represent more than one requirement (3).

(4) The system point of view might be documented as an architectural model. Other facets of the system point of view might be supported by a data flow model or an entity-relationship model. Use cases do not, however, present the system from a system point of view.

©2007 S. T. Frezza

Issues with classroom useIssues with classroom use

1. Questions are long – significant reading time

2. Answers are long – significant reading time

3. Delivery mode (Turning Point™) does not allow both Q & A displayed

©2007 S. T. Frezza

Assessment ResultsAssessment Results

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

4A 2A 6D 17E 1B 20B 19C 3D 22D

Text Question

% R

esp

on

den

ts C

orr

ect

©2007 S. T. Frezza

Assessment ResultsAssessment Results

Best: 5/9Worst: 1/9Mode: 2/9

11

22 33

4455

ConclusionConclusionNot the best classroom Not the best classroom

assessment toolassessment tool

Did not appear to stimulate Did not appear to stimulate student performancestudent performance

©2007 S. T. Frezza

Stimulating DiscussionStimulating Discussion“Comment on how the questions were challenging in a

positive way…”They helped you think outside the "norm".think outside the "norm". Interesting, yet

hard questions. It was good as the RE tools w[ere] applied in different RE tools w[ere] applied in different

situationssituations at various instances of the organizationsYes, it made us to listen classes carefully to try to answer the

questions correctlyQuestions did make you think about multiple view pointsmake you think about multiple view points, and

consider things more in-depth.Overall not difficult, and can provide thought and discussioncan provide thought and discussion.

©2007 S. T. Frezza

Stimulating DiscussionStimulating Discussion“Comment on how the questions were challenging in a

negative way…”Very hard for someone [just] learning [the] subject matter Some questions were ambiguous and subjective. (2x)Usually questions had some vagueness to them that often led

to missing an answer due to little nit-picky parts. It takes time to think and answer to the question. Mostly we

will click wrong answers within such short period of time, which impact[ed] our scores

Getting a "clicker" every day was tedious. Allow more time questions and discussion.Topic of the question was completely unrelated to the topic

©2007 S. T. Frezza

ConclusionConclusionCan lead to useful discussion

Application and various viewpoints interestingNeed good question-topic alignment

In-class use and discussion qualitatively appears to support student learning.

May not fit PRS-style in-class assessment useReserved Q&A sessions probably more effectiveMechanics costly