Experimentation in Computer Science (Part 1)

Outline

Empirical Strategies Measurement Experiment Process

Outline

Empirical Strategies Measurement Experiment Process

Empirical Strategies: Research Paradigms

Qualitative -- study objects in their natural setting, interpret phenomena based on people’s explanations, discover causes noticed by subjects

Quantitative -- compare 2 or more groups or treatments, identify or quantify cause-effect relationships

Qualitative and quantitative are complementary Quantitative research considers measures to assess effects of treatment, whereas qualitative research considers beliefs/understanding to explain why treatments differ.

Empirical Strategies:Types of Investigations

Survey – often retrospective, through interviews or questionnaires

Case Study - observational, often in-situ for an ongoing project

Experiment - controlled study, laboratory, manipulation of variables

Quasi-Experiment – experiment, but lacking randomization

Experiments are quantitative; surveys and case studies can be quantitative or qualitative

Empirical Strategies:Surveys

Purpose: descriptive

(assert characteristics)

explanatory (assess why)

exploratory (pre-study)

Process select variables select sample collect data analyze and

generalize

Example: survey 10% of web application users in a community on their opinions of a new web technology, to infer overall opinion across all developers in the community, and understand why they use or do not use this technology.

Empirical Strategies:Surveys

Data collection methods: Questionnaires

o Cheaper distribution and execution

Interviewso Higher response rateo Removes possible ambiguity

Resource: E. Babbie, Survey Research Methods, Wadsworth, 1990

Empirical Strategies:Case Studies

Study a phenomenon in a specific time/space Applicable to dynamic or larger studies:

Long term evaluations Industrial evaluations

Can be used to compare approaches (e.g. on a project and sister project); not an experiment because not randomly selected

Easier to plan than controlled experiments

Resource: R. K. Yin, Case Study Research Design and Methods, Sage Publications, 1994

Empirical Strategies:Case Studies

Harder to control, hence less useful for asserting causality

Example: observe the application of a web application development method within an industrial context, using two different development methodologies, over a long portion of the lifecycles of two development projects

Empirical Strategies:Experiments

Most controlled form of study Manipulate independent variables View affects on dependent variables Controlled environment Reproducible Randomization over subjects and objects Often involve a baseline (control group)

Resource: Wohlin et al., Experimentation in Software Engineering, Kluwer, 2nd Ed.

Empirical Strategies:Experiments

Uses: Confirm / reject theories Confirm conventional wisdom, test pre-conceptions Explore relationships and interactions Evaluate accuracy of models Validate measures

Example: randomly assign a set of web app testers from company C to two groups: one using testing technique A and one using testing technique B, and ask them to apply these techniques to a randomly selected set of C’s web apps containing seeded faults, and measure fault-detection effectiveness

Empirical Strategies:Quasi-Experiments

The same as an experiment, except with a lack of randomization over subjects or objects

Drawback: reduced generality

Example: a large set of test suites created by a specific technique, and a set of equivalently sized randomly generated test suites, are applied to a single large web appthat your research group has developed.

Empirical Strategies:Comparison

Factor Survey Case Study Experiment

Execution Control

Low Low High

Measurement Control

Low High High

Investigation Cost

Low Medium High

Ease of Replication

High Low High

Empirical Strategies:Factors in Selecting a Strategy

Type of research question Level of control required Time and space of target of study Resources available

All types of studies are valuable.

Experimentation in Software Engineering --- Outline

Empirical Strategies Measurement Experiment Process

Measurement:Terminology

Measurement: a mapping from the empirical world to the formal, relational world

Measure: a number or symbol assigned to an entity (object in the world) by this mapping to characterize and manipulate an attribute

Valid measure: a measure that captures necessary properties of the attribute, and properly characterizes it mathematically.

Measurement:Failing to Capture Properties

Goal: measure programmer productivity Usual: code size or coding effort

o Size = LOCo Effort = person_months

Problems with LOCo LOC only reflects part of the output o We want to measure production (output) as delivered benefit

Problems with efforto Equivalence among employers or even developerso Full / part time

An objective measure involves no subjective judgment about the measurement value

A subjective measure does involve subjective judgment, depends on both the object and the viewpoint from which the measurement is taken

Examples: Objective: lines of code, delivery date Subjective: personnel skill, usability

Measurement: Objective vs Subjective

A direct measure of an attribute does not involve measurement of other attributes

An indirect (derived) measure does involve measurement of other attributes.

Often, only indirect measures are available.

Examples: Direct: number of defects Indirect: programmer productivity = LOC/effort

Measurement: Direct vs Indirect

Experimentation in Software Engineering --- Outline

Empirical Strategies Measurement Experiment Process

Experiment Process:Motivation

Properly designed experiments provide: Control of subjects, objects, and instrumentation,

allowing us to draw more general conclusions or conclusions about causality

Ability to use statistical analyses (hypothesis testing) Support for replication

Achieving proper design requires us to define a process for experimentation

Experiment Process:Overview

Causeconstruct

Effectconstruct

Treatment Outcome

Theory(hypothesis)

Observation

cause-effect construct

treatment-outcomeconstruct

Independent variable Dependent variable

Experiment objective

Experiment operation

Experiment Process:Variables, Factors

Variable: Entity that can change and take on different values Independent variable(s): variables that are manipulated and controlled

– also known as factors Dependent variable(s): variables we want to see the effects of changes

in independent variables on Factor: variables other than those chosen as independent and

dependent; these must be carefully controlled for

S.E. Process Dependentvariables

Independentvariables (Factors)

Experiment Process:Treatments

Treatment: one value of an independent variable (the variable manipulated)

S.E. Process

Dependentvariables

Independentvariables

Exp. Design

Experiment

Independent variables at fixed levels

Treatment

Experiment Process:Objects and Subjects

Treatments are assigned to subjects / objects Subjects: people to whom treatments are applied, or

more often, who apply treatments Objects: artifacts that treatments are applied to, or

that are manipulated by people

Tests (or trials): a combination of treatments with subjects and/or objects