Quality in qualitative research the role of the software’s in quality assurance

Computer-Aided Qualitative Research Europe

7 & 8 Oct 2010, Lisbon

For more information about our events, please visit:

http://www.merlien.org

Quality in Qualitative Research: The role of the software’s in Quality

Assurance

SÍLVIA SILVA

SARA RAMOS

ISCTE-IUL

3rd European workshop on COMPUTER-AIDED QUALITATIVE RESEARCH 2010

OVERVIEW

Main points of our presentation:

• Aims

• Background

• Literature review and results

• Quality and CAQDAS

Additionally :

- Distribution of Some Examples

- Distribution of List of Main References

AIM

To present a review of quality criteria in Qualitative Research and explore and conclude about the potential roles that software packages may play in quality assurance.

Specific goals:

(1) Identify the criteria used

(2) Identify the most common/consensual criteria;

(3) Link quality to CAQDAS

(4) Contribute for debate and consensus (?)

Background

• In the last thirty years several authors proposed criteria for considering when approaching the issue of Quality in Qualitative Research (QR).

• It is somehow recognized that we are far from having shared quality assumptions

Background: Examples on well-known criteria presented in books

• Guba and Lincoln (1985) Trustworthiness :

• Transferability ; • Credibility ; • Dependability; • Confirmability

• Bauer & Gaskell (2000) Accountability:

• Confidence • Relevance

• Flick (2007/9) • Transparency, • Documentation • Writing

Research accountability (Gaskell & Bauer, 2000)

• Confidence (c)

(results represent the reality studied)

• Relevance (r)

(utility and importance)

� Triangulation and Reflexivity (c)

� Transparency and procedural clarity (c)

� Corpus construction (c,r)

� Thick description (c,r)

� Local surprise (r)

� Communicative validation (r)

Background: Why to focus on Quality

• The relevance of quality criteria: epistemological issues and

practical issues • A well-written description of the rigour in research analysis

should convince readers that the study findings are credible and trustworthy (Belgrave et al., 2002).

Moreover: • QR Internal needs (development and proliferation)

(Flick,2007) • QR external challenges (e.g. competition: publication;

funding; teaching and curriculum planning) (Flick,2007)

The big question:

One size fits all?

Literature Review: Search Approach (I)

• Covering last 20 years

• Databases:

- Psycharticles;

- ABI/INFORM;

- ISI

- Google academics

• Search in the Abstracts and Titles

Literature Review: Search Approach (II)

• Papers focusing Quality Issues/Criteria in Qualitative Research

• Keywords used:

Quality+ Qualitative Research/Methods;

Validity+ Qualitative Research/Methods;

Rigor/our + Qualitative Research/Methods; Trustworthiness +Qualitative Research/Methods;

all the above with Computer Programs, Software and CAQDAS

• Results: > 100 papers ; >30 specifically focusing quality as the main topic

Literature Review: Main Characteristics

• Huge progressive increase of papers about Quality Issues in QR or QS in the last 20 years

• Journals: Either general (Methods in General; Methods in QR); either in specific fields (mainly: health; business; education; communication)

Literature Review: Main Characteristics

Therefore:

Papers focusing QR Quality in specific research fields VS generic criteria and recommendations

• Papers departing from validity & reliability and framing them in the QR VS papers against that assumptions

Literature Review: Main Findings

Main types of papers focus:

• Criteria used by Journals and Referees • Specific Criteria for specific QR • Generic Criteria for all QR • Techniques for assuring quality: Audit;

triangulation; … • Quality and CAQDAS

Most cited Authors: • Guba and Lincoln (1985)

LR Results – Journals and Referees criteria

• Papers focusing specific Criteria used by Journals and Referees to sustain Qualitative Papers Review

Examples:

• Academy of Management Journal Editors (2002, 2004 & 2009)

• Savall et al. (2008)

• Crescentini & Mainard (2009)

Academy of Management Journal 2009 Editors: Tips for Writing Qualitative Papers

1. Make sure your paper includes “the basics”:

• Discuss why this research is needed

• Are you building a new theory or elaborating existing theory?

• Why did you choose this context and this “unit of analysis”?

• How did you get from your data to your findings?

2. Show data in a smart fashion

3. Think about using/organizing figures

4. Think about telling a story

5. Consider “modeling” someone whose style you like who

consistently publishes qualitative

10 (generic) Criteria used by Reviewers in an European

Management Journal

Savall et al. (2008)

• Rigor • Formulation • Coherency • Originality • Relevance • Explication • Positioning • Contribution • Rationale • Delimitation

LR Results – Specific Criteria

• Papers focusing specific Criteria for specific Qualitative Approaches: epistemological adequate criteria

• For instance: adequate to the the type of analysis: Content Analysis (e.g., Lombard et al., 2002); Grounded Theory (e.g., Chiovitti & Piran, 2002; Elliot et al., 2005); Discourse Analysis (e.g., Nixon, 2007 )

LR Results – Specific Criteria

• For instance:

Characterizing the Philosophy and Politics of Quality in QR. Distinguishing: review of quality indicators attached to

- Foundational;

- Quasi-Foundational;

- Non-Foundational QR

(Amis & Silk, 2008 )

LR Results - Generic Criteria or Recomendations I

• Papers focusing specific Criteria for generic/transversal Qualitative Approaches: criteria that applies to all approaches

LR Results - Generic Criteria or Recomendations II

Authors Criteria

Akkerman et al. (2008)

•Visibility •Compreehensibility •Acceptability

•AUDITING

Morrow (2005) •Social validity •Subjectivity and Reflexivity •Adequacy of data •Adequacy of interpretation

Guidelines for Writing QR

Shank & Villela(2004)

•Investigative depth •Interpretative adequacy •Illuminative fertility •Participatory accountability

Whitemore et al., (2001)

•Primary Criteria (for all QR) •Secondary Criteria •Tecnhiques

Primary criteria: Credibility; Authenticity; Integrity and Criticality Techniques: Design consideration; Data generating; Analytic; Presentation

Meyrick (2008): Good Qualitative Research

Rolfe (2006)

“the commonly perceived quantitative–qualitative dichotomy is in fact a continuum which requires a continuum of quality criteria, or to recognize that each study is individual and unique, and that the task of producing frameworks and predetermined criteria for assessing the quality of research studies is futile.”

“individual judgments of individual studies”

Quality Objects

• Theory, Method and Epistemological Coherence

• Design and Report of the Research

• Data Collection

• Data Analysis

• Adequate methodological approach considering the analytic grounding

General:

• Transparency

• Reflexivity

LR Results – Quality and CAQDAS

• Papers focusing the role of CAQDAS on assuring Rigor and Quality

LR Results – Quality and CAQDAS Authors Focus

Lu & Shulman (2008) CAQDAS and Rigour and Flexibility and the use of CAT (Coding Analysis Toolkit)

Rambaree (2007) Rigour in Qualitative Social Research: The Use of a CAQDAS Atlas.ti example

Sin (2007)

CAQDAS for achieving Transparency Illustration with NVivo

Sinkovics et al. (2008)

CAQDAS for achieving trustworthiness of QR in Business Research

Westphal (2000)

N4 and Trustworthiness: Searching for negative evidence – Easier to Find Inconsistency Linking Data and conclusions and theory Conducting Coding Checks Audit Trails and Conducting Audits Detailed reporting

What Software Do (Gibbs, Lewins & Silver, 2005;

Lewins & Silver,2009) • Structure work (access and organization of all project elements)

• 'Closeness to data' interactivity (quick access to source data files)

• Explore data

• Code and Retrieve Functionality

• Project Management and Data Organisation

• Search and interrogating the database

• Writing tools - Memos, comments and annotations,

• Output - Reports to view a hard copy or export to another package.

Advantages (Gibbs, Lewins & Silver, 2005; Lewins &

Silver,2009)

• Organised and controllable data set

• Support for coding

• Searching Text and codes

• Support for comparative analysis

• Models, networks and diagrams

• Interface with quantitative data

Qualitative Softwares – Possible contributions for quality (I)

•Data and Coding:

More Easy to Be sure that all data are analysed (assure that

participants perspectives are covered) and code and retrieve

approach more complete and rigorous

Reporting:

More adequate support for reporting results and allowing

simple and complex results

Qualitative Softwares – Possible contributions for quality (II)

•Good support for assisting a systematic analysis approach:

allows and simplifies the use of “equivalent” procedures to all

the data

•TRANSPARENCE and Reflexivity :

Allowing a more easy access to the data analysis (e.g., Codes;

Quotations; Memos); methodological analytic decisions, and

reflexions (Memos, Comments)

•Facilitation of TRIANGULATION and AUDIT:

Simplifies the combination of different sources of data (or

other type of triangulation); team work; …

Qualitative Softwares – Possible contributions for quality (III)

Besides:

allows the researchers in assuring quality when

following most of the specific recommendations

But:

it can also be misused and give “overconfidence”

Conclusions (I)

• There is not a single criteria solution adequate for all “one best way”

• Quality issues cover both theoretical issues and technical problems

• There is an ethical obligation to demonstrate “Rigor” and Integrity of research

• Quality Reflexion is needed but we are still far from consensus: We suggest some debate on this

Conclusions (II)

Anyway:

• Debate and establishing “criteria” about Quality of QR contributes for a bigger awareness of methodological decisions during QR (Seale, 1999)

• CAQDAS may play an important role but it always depend on it is used

Conclusions (III)

Myth 94:

qualitative researchers will agree about

validity

Sparkes (2001)

Our Future work on this subject

• First Paper in Preparation

• Project about the Quality of Qualitative Research and CAQDAS(without funding at the moment) in the beginning; looking forward possible cooperation/network for proposal to be submitted to the Portuguese Science Foundation

References

A list with the Main References will be distributed

Computer-Aided Qualitative Research Europe

7 & 8 Oct 2010, Lisbon

For more information about our events, please visit:

http://www.merlien.org