Upload
david-melton
View
216
Download
1
Embed Size (px)
Citation preview
Mapping Studies – Why and How
Andy Burn
Resources• The idea of employing evidence-based practices in
software engineering was proposed in (Kitchenham et al., 2004).
• Some experiences of our own are embodied in the paper (Brereton et al., 2007) and a good example of a published systematic literature review is given in (Jørgensen & Shepperd, 2007).
• A useful book is: Mark Petticrew & Helen Roberts (2006). Systematic
Reviews in the Social Sciences: A Practical Guide, Blackwell Publishing
• www.ebse.org.uk
Mapping Studies – Why and How 2
Overview• Why conduct a mapping study?
Literature reviews The evidence-based paradigm Secondary studies Mapping studies
• How to conduct a mapping study Protocol design Data collection Data aggregation Literature review
Mapping Studies – Why and How 3
Literature Reviews• In many domains there are well-established journals
that publish ‘review papers’, usually by an expert in a field, where they review the literature and draw any appropriate conclusions.
• In computing, we have the ACM Computing Surveys journal, which more or less does that.
• However, such reviews are essentially ‘expert reviews’ and hence it is possible that two different experts might select different papers and draw separate conclusions!
• This leads us to...
Mapping Studies – Why and How 4
The Evidence-Based Paradigm• The evidence-based paradigm originated in clinical
medicine (and relatively recently) following concerns over the quality of research evidence being used to inform practice/teaching.
• Evidence-Based Medicine (EBM) seeks to employ secondary studies to find, judge and synthesise the outcomes of all relevant empirical studies to draw conclusions about particular treatments. It has had a major impact upon clinical practice and upon healthcare in general.
Mapping Studies – Why and How 5
Secondary Studies• Seek to aggregate the outcomes of many primary
studies in an objective and unbiased manner, using either quantitative or qualitative forms of analysis
• Major tool is the systematic literature review for which a protocol will define the question of interest, how it can be categorised in keywords, and how and where the search for source material will be conducted
• Sometime use the first steps of systematic literature review to perform a mapping study, which identifies the set of papers addressing a topic but is mainly concerned with formulating a research question
Mapping Studies – Why and How 6
Primary vs Secondary Studies
Mapping Studies – Why and How 7
EBSE• Evidence-Based Software Engineering
1. Convert need for information (about a technique, method, etc.) into an answerable question
2. Find the best evidence with which to answer the question
3. Critically appraise the evidence for its validity (closeness to the truth), its impact (size of effect) and its applicability (usefulness)
4. Integrate the critical appraisal with SE expertise and with stakeholders’ values and circumstances
5. Evaluate the effectiveness and efficiency of the process in steps 1--4 and seek ways to improve them
• Steps 1--3 are essentially the process of systematic literature review
Mapping Studies – Why and How 8
Systematic Literature Review
Mapping Studies – Why and How 9
1. Specify Research Questions
2. Develop Review Protocol
3. Validate Review Protocol
4. Identify Relevant Research
5. Select Primary Studies
6. Assess Study Quality
7. Extract Required Data
8. Synthesise Data
9 & 10. Write & Validate Report
Phase 1Plan Review
Phase 2Conduct Review
Phase 3Document Review
Mapping Studies
Mapping Studies – Why and How 10
1. Specify Research Questions
2. Develop Review Protocol
3. Validate Review Protocol
4. Identify Relevant Research
5. Select Primary Studies
6. Assess Study Quality
7. Extract Required Data
8. Synthesise Data
9 & 10. Write & Validate Report
Phase 1Plan Review
Phase 2Conduct Review
Phase 3Document Review
Conducting a Mapping Study
• Develop your protocol
• Identify studies and papers
• Aggregate and analyse the results
Mapping Studies – Why and How 11
Develop Study Protocol• A document describing the design of a study
Create before the study Try it out with a dry run – change if necessary Record all divergences from the protocol in the final study
• The protocol should describe Keywords Sources
Search engines, such as ACM, IEEEXplore, Science Direct, Google Scholar...
Manual search, e.g. prior knowledge, asking an expert, snowballing...
Inclusion criteriaGrey literature?Date restrictions?
Mapping Studies – Why and How 12
Identifying Studies• Selection is based on titles and abstracts
Fast and simple, but often inaccurate
• Papers and studies have a many-to-many relationship One study may be reported in several papers One paper may report several studies Don’t confuse ‘papers’ and ‘studies’ in your documents
• Data extraction: Title, author, abstract, year, reference Source – search engine, search string Topic or category
• Validation Second opinion Expert search
Mapping Studies – Why and How 13
Notes on Search Engines• Each engine has a different syntax• Different engines cover a subset of sources and
overlap each other• Results are sometimes inconsistent, even within the
same engine E.g. Changing the order of the search terms can change the
results
• Anecdotally, snowballing adds about 10% over automated searching
• Spelling Modeling vs modelling Visualisation vs visualization
Mapping Studies – Why and How 14
Data Aggregation• Remember
The aim of a mapping study is to find the limits, shape and nature of the research on a topic
• Aggregation creates an overview of your results• This overview lets you
Make statements about the current state of research on a topic
Find gaps and clusters Create research questions based on these gaps and
clusters
Mapping Studies – Why and How 15
Experiences• Many areas of Computer Science and Software
Engineering have a very weak empirical basis• Titles and abstracts are often poor
Not descriptive of the research Frequent misuse of the terms ‘experiment’ and ‘case study’
• Managing large sets of results is difficult and time-consuming Several tools exist, none of them are ideal
Mapping Studies – Why and How 16
Final Notes• A mapping study doesn’t have to search for empirical
research Might be looking for tools or frameworks
• A mapping study provides a solid basis for a strong literature review Ideally the study will be extended into a complete systematic
literature review In a postgraduate time-scale, more likely to be used to
Shape your research question Increase confidence in the set of studies used in your literature
review
Mapping Studies – Why and How 17
For More Information• www.ebse.org.uk
Templates Guidelines Glossary Bibliography Previous studies and protocols
Mapping Studies – Why and How 18