Fundamentals and

Technische Universität München

Case Studies in Industry

Daniel Méndez !!

Technische Universität München Germany

!IASESE 2014 Torino, Italy

@mendezfe

Based on material from a joint work with: Andreas Jedlitschka (Fraunhofer Institute for Experimental Software Engineering) Stefan Wagner (University of Stuttgart) !Feedback from Antonio Vetrò (Technische Universität München)

What is a case study?

Empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident. [Yin 2002]

In quest for a case study definition We opt for realistic environments

Theoretical space (methods, models)

Theory

Environment made irrelevant

Artificial (controlled) environment

Realism

Reality

In quest for a case study definition We rely on a mix of qualitative & quantitative data

Quantitative Qualitative

Case studies

Surveys and interviews

Controlled experiments

Quasi experiments

Action research

Grounded Theory

Data collection procedure

In quest for a case study definition We rely on a mix of empirical methods

Case studies

Action research

Grounded Theory

Data analysis procedure

Surveys and interviews

* Example

**

!!

Most* case studies:

• are conducted in socio-economic contexts

• are of confirmatory nature (“Let’s see if my method is better than yours”)

• rely on qualitative data (e.g. as part of interview transcripts)

• rely on a mix of various research methods, often (technical) action research

In quest for a case study definition Summary of some important characteristics

Cas

e st

udy

rese

arch

Data type

Setting

Purpose

Mixed w/

•Confirmatory •Exploratory

•Qualitative •Quantitative

•Realistic environments •Realistic models

* Unsupported claim, don’t sue me.

•Action research •Interview research •…

Agenda

1. Case study research: Why and how? 2. Examples 3. Challenges in case study research 4. Time for questions

Agenda

1.Case study research: Why and how? 2. Examples 3. Challenges in case study research 4. Time for questions

Learn from realistic (socio-economic) environments with realistic conditions: • subjects with: opinions, experiences, expectations, fears, beliefs, … • cases with: realistic models, realistic complexity, realistic decisions, … … at the cost of control (and internal validity) !

» Ideally: Explore the unpredictable (e.g. when testing a SE method)

Why case studies? Learn from real life environments

Disclaimer Cases studies do not imply more “validity” or “trustworthiness” than experiments - they complement each other!

Why case studies? Case studies support scaling up to practice

Lab credibility

Street credibility

Simple model

Realistic model

Small sample Large sample

Based on: WieringaEmpirical Research Methods for Technology Validation: Scaling Up to Practice

Focus of case studies

(Focus of field studies and replications)

Scaling u

p to prac

tice

Focus of (lab) experiments

What is the lifecycle of a case study?

Case study design

Preparation for data collection

Data collection Data analysis Reporting

• Research objectives • Research goals • …

• Schedules • Protocols

• Document analyses • Interviews, Surveys • Action research • …

• Descriptive statistics • Grounded theory • …

• Packaging • (Publishing)

Further reading

Agenda

1. Case study research: Why and how?

2.Examples 3. Challenges in case study research 4. Time for questions

!

!

Example #1

Model-based process enactment

Develop and evaluate an approach to dynamically test the conformance of software project (instances) and software process models

Problem analysis Evaluation

Development (TAR*)

Case study research

InterviewsDocument

analysis

* Technical Action Research

Results

• Moving targets detected when designing the instruments for the interviews

• New goals had no scientific relevance (very low theoretical and practical impact)

Problem analysis Evaluation

Development (TAR)

Case study research

InterviewsDocument

analysis

!

!

Example #2

Develop and evaluate a model-based requirements engineering approach for embedded reactive systems in the automotive domain

Problem analysis Evaluation

Development (TAR)

Case study research

InterviewsDocument

analysis

Model-based requirements engineering

Results

No access to software project environments (“The ivory tower”)

Problem analysis Evaluation

Development (TAR)

Case study research

InterviewsDocument

analysis

!

!

Example #3

Evaluate a model-based requirements engineering approach for business information systems in comparison to previously used approach

Model-based requirements engineering (2nd try)

Case study

Assessment(Interview)

TAR workshops

Case study

Assessment(Interview)

TAR workshops

Results

The new model-based approach supported:

• more flexibility in process

• higher productivity

• higher syntactic quality in artefacts

!Conclusion: The new model-based approach better supports a flexible process with high quality artefacts.

See also: Mendez Fernandez et al. A Case Study on the Application of an Artefact-based Requirements Engineering Approach !

Conclusion: The new model-based approach better supports a flexible process with syntactically consistent results.

What problems do you see in this case study (and conclusions)?

• Action research workshops inherently threaten the internal validity

• High degree of subjectivity

• Results sensitive to their context (generalisation?)

• …

!» Each “successful” case study still offers plethora of problems and challenges!

Agenda

1. Case study research: Why and how? 2. Examples

3.Challenges in case study research 4. Time for questions

Industrial research cooperations with

• up to 3 years duration

• 3-30 participants

• multiple case studies

We distinguish three classes of challenges:

1. Approaching a case study

2. Conducting a cases study

3. Transversal challenges

(…)

Challenges in approaching a case study

» Integrate studies into ongoing / planned research projects

» Actively approach

– alumni of your university

– companies in your area

– practitioners at (local) events

Challenges in approaching a case study #1: Finding contacts

Image source: https://c1.staticflickr.com/1/74/196191428_7dc71b7bf7_z.jpg?zz=1 !

Challenges in approaching a case study #2: Convincing contacts to participate

» Find appropriate terminology (technology transfer)

» Find appropriate incentives for project partners

» Respond to their local needs

» Provide early feedback / results

» In funded projects, probability is higher that practitioners collaborate

Challenges in approaching a case study #3: Opportunistic case and subject selection

» Welcome to case study research!

Image source: xkcd !

You need to: • schedule meetings / workshops • allocate resources • (formally) define deliverables • clarify responsibilities • clarify NDA-related issues • …(the list can get long)

Challenges in approaching a case study #4: Planning a case study

!» Plan a case study like you’d plan a software development project:

– Time plan and resource allocation

– Quality assurance and acceptance of protocols and reports

– …

Image source: https://c1.staticflickr.com/1/186/467087455_e90b880c92.jpg !

Challenges in approaching a case study #5: Uncertainty

• What will the stakeholder characteristics be? • Availability • Skills • Motivation to participate • Commitment to goals • Representativeness • …

• What will the data look like? • Quality of the data • Quantity of the data

» Take early samples and test them!

» Test instruments and data quality via pilots

» Get to know your subjects early

» Be flexible, always ask yourself (and others): “What is the potential of the data I’m getting?”

Challenges in conducting a case study

Context variables (and phenomena) • hard to determine (or unavailable) • hard to measure

» Context often hard to describe (hence, hard to reproduce by others)

Variables describedin reportChallenges in conducting a case study

#1: Properly characterising the context

» Orient yourself on

» other (similar) studies or

» on classification schemes from the area of software process models (“Tailoring”)

» Focus on whole context, not only on cases for which you want to draw conclusions

» Ask yourself: “What information would I need to understand and replicate the study?”

Variables you should measure

Variables youcan measure

Possible problems:

• No clear communication of case study in organisation

• Case study delegated to s.o. not interested

• No clear communication lines and reporting lines

» Define clear communication/reporting/escalation lines at the beginning. Take no shortcut!

» When communicating with subjects:

» Communicate purpose and goals of case study

» Find a supporter at project level (with own interests)

» Don’t call them subjects ;-)

Challenges in conducting a case study #2: Communication

Image source: https://c1.staticflickr.com/1/32/62630024_10a8fa36ce_z.jpg?zz=1 !

Challenges in conducting a case study #3: Defining the proper instruments

» Pre-tests your design, e.g.

» Pilot your surveys

» Check if planned data analysis techniques apply

» A way to calibrate the instruments is to conduct a small experiment with students first

Image source: How to lie with statistics !

Possible problems:

• Wrong assumptions about population

• (Research) questions not thought through

Challenges in conducting a case study #4: The data collection might take long…

» Be patient

» Never push!

Image source: https://c1.staticflickr.com/9/8185/8375141385_df9bd4f283_z.jpg !

Challenges in conducting a case study #5: Working with sensitive data

» Clarify the rules:

» what you need for the case study

» what can be stored and reported (internally and externally)

» Plan review and acceptance cycles for reports and publications

» The “easier” way (sometimes): Joint publication

Possible distortions in data collection: • biased researchers

(trying to sell own PhD topic) • weak moderation skills • hidden agendas of respondents

(missing trust!) • (…)

Challenges in conducting a case study #6: (Social) skills

» Don’t be biased!

» Involve other researchers in:

» Design of instruments

» Data collection (e.g. interviews)

» Continuously practice your

» Moderation skills

» Rhetoric and listening skills

» Build trust by showing loyalty and respect

Challenges in conducting a case study #7: Analysing qualitative data

Qualitative data • is inherently subjective • is difficult to interpret

(we all have different mental models!)

» When analysing qualitative data

» Work in teams (researcher triangulation)

» When making interpretations, document the rationale

» Validate the results with the subjects (if possible)

» Be careful when drawing conclusions

Image source: How to lie with statistics !

• Moving targets are something natural • The difficult part:

• Detecting moving targets • Reacting to them

Challenges in conducting a case study #8: Moving targets

» Continuously provide and get feedback

» When planning your study, agree on

» key goals and questions

» open space left for new research questions (flexible part)

» Goals should always meet the interest of both industry and academia

Challenges in conducting a case study #9: Tool support

Unfortunately, still the best tools we have…

Transversal challenges

Possible problem: Support by management…only by management.

» No backup in projects (organisational culture)

» Make clear study plan including needed subjects & cases (and formally agree on it)

» Talk early to subjects

» Find active supporter (not sponsor) with own interests in project (= their own incentives)

Transversal challenges #1: The ivory tower

» „[Even if] one swallow does not make a summer, one swallow does...

» prove the existence of swallows

» provide a great deal of reliable information about swallow anatomy“*

» Do not be that kind of reviewer that asks for both realistic models and large sample sizes

Transversal challenges #2: Sometimes, size seems to matter…

* Y. Anzai and H.A. Simon. The theory of learning by doing (1979) / H. Sharp. Keynote (EASE 2012)

Transversal challenges #3: Drawing proper conclusions

» Knowledge is more than statistical significance

» When analysing the results, be accurate and critical!

» Stay on the ground

» You may interpret, as long as the interpretation is clearly stated as such

» Rely on other case studies that might explain your observations

Image source: xkcd !

Transversal challenges #4: The quest for universal truth (“generalisation”)

1. Software engineering is about human beings (Subjectivity) 2. Reality cannot be captured in complete system of variables !» “Truth” will always be something relative to its context

» Aim for trustworthiness and transparency rather than for int./…/ext. validity

» Support scaling up to practice

» Appreciate the value of replication studies

» Align your study with similar ones

» Make clear relation to existing evidence

» Disclose your data (e.g. via promisedata)

See also: Mendez et al. In Quest for Requirements Engineering Oracles: Dependent Measurements and Variables for (good) RE

Case study data should be disclosed • reliability and trustworthiness? • reproducibility and replicability?

» The success of empirical SE relies on a collaborating community

Transversal challenges #5: Openness and transparency

» If possible, make your (anonymised) data accessible to others

Transversal challenges #6: Properly reporting the results

Case study results need to be disseminated • to practice • to academia

» How to properly report?

» Report results according to target group:

» Small presentations to management w/ focus on results

» Peer-reviewed publication w/ (selected) research scope

» (Technical) Reports with full data (and analysis)

» …

» Rely on established guidelines for reporting on case studies

!

KEY TAKE AWAYS

My personal manifesto for case study research

6. Be unbiased

7. Be accurate and critical

8. Be honest

9. Be patient

10. Be flexible (and pragmatic!)

1. Carefully plan your study like a project

2. Make clear goals, purpose, context, threats

3. Train your empirical and social skills

4. Disclose your data (reliability & replicability)

5. Collaborate

Agenda

1. Case study research: Why and how? 2. Examples 3. Challenges in case study research

4.Time for questions

Don’t ask ask us!,

!!!

Thank you!Daniel Méndez

Daniel.Mendez@tum.de

@mendezfeFurther reading

• Slides will be made available (and probably tweeted)

• Ask me if you need material (studies, templates, …)