Quality issues in mixed methods research (with an emphasis on teaching) - Alan Bryman

Quality issues in mixed methods research

(with an emphasis on teaching) Alan Bryman

Talk to Higher Education Academy event – Teaching Research Methods in Business and Management, 10 January 2014

©Alan Bryman

Is quality appraisal important for students?

Growing focus on quality criteria

Need for consistent standards in assessments of research

Interest among qualitative researchers has increased

Use of systematic reviews

Relevance for mixed methods research?

Important for students to be familiar with aspects of quality that are especially valued

May increase skill levels

Provides vantage point for critical assessment of research

May improve literature reviews for dissertations and theses

In the case of mixed methods research, important to realise that it is more that just 2 separate components

But might creativity and innovation be stifled?

Creswell’s skill-based mixed methods

seminar

Learning how to identify a mixed methods study

Learning appropriate search terms for locating mixed methods studies

Writing a rationale for mixing quantitative and qualitative research

Writing research questions for mixed methods study

Summarizing types of data collected in mixed methods research

Transforming quantitative data into qualitative data and vice versa

Combining quantitative and qualitative data to arrive at conclusions

Creswell’s skill-based mixed methods

seminar (contd.)

Identifying procedures used in software for analysis of quantitative and qualitative data

Drawing visual model of mixed methods research design

Identifying distinguishing features of mixed methods designs

Comparing strengths and weaknesses of different mixed methods designs

Evaluating a mixed methods study in terms of criteria

Applying steps in design of a mixed methods study to a research proposal

J. Creswell et al. (2003) ‘Teaching mixed methods research: practices, dilemmas and challenges’, in A. Tashakkori and C. Teddlie (eds), Handbook of Mixed Methods in Social & Behavioral Research, Thousand Oaks, CA: Sage.

Do we expect too much of quality criteria?

Two prominent roles:

1. Appraisal role.

Consideration of adequacy of research from the point of view

of assessing research quality (e.g. for inclusion/exclusion in a

systematic review) or as part of the refereeing function for

research proposals and journal articles.

With qualitative research, has led to long lists partly associated

with lack of agreement on criteria (cf. Cassell & Symon, 2011)

Difficult to argue for the removal of criteria.

In the case of qualitative research, has led to long lists…

C. Cassell & G. Symon (2011) ‘Assessing “good” qualitative research in the work psychology field: a narrative

analysis’, Journal of Occupational and Organizational Psychology, 84: 633-50.

Encyclopaedic approach to quality criteria for qualitative research?

1. How credible are the findings?

2. Has knowledge/understanding been extended by the research?

3. How well does the evaluation address its original aims and purposes?

4. Scope for drawing wider influences—how well is this explained?

5. How clear is the basis of the evaluative appraisal?

6. How defensible is the research design?

7. How well defended is the sample design/target selection of cases/documents?

8. Sample composition/case inclusion—how well is the eventual coverage described?

9. How well was the data collection carried out?

10. How well has the approach to, and formulation of, the analysis been conveyed?

11. Contexts of data sources—how well are they retained and portrayed?

12. How well has diversity of perspective and content been explored?

13. How well has detail, depth and complexity (richness?) of the data been conveyed?

14. How clear are the links between data, interpretation and conclusions—i.e. how well can the route to any conclusions be seen?

15. How clear and coherent is the reporting?

16. How clear are the assumptions/theoretical perspectives/values that have shaped the form and output of the evaluation?

17. What evidence is there of attention to ethical issues?

18. How adequately has the research process been documented?

Spencer, L., et al. (2003), Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence www.civilservice.gov.uk/wp-content/uploads/2011/09/a_quality_framework_tcm6-38740.pdf

Tracy’s 8 ‘must have’ criteria

1. Worthy topic—relevant, interesting, significant, etc.

2. Rich rigour—rich data supplied in abundance and appropriately

3. Sincerity—the researcher is reflexive about values and biases and is transparent in

approach

4. Credibility—implements practices such as thick descriptions, triangulation, and

respondent validation

5. Resonance—has an affecting impact on readers

6. Significant contribution—makes an impact in terms of such outcomes as theory,

practice, and morality

7. Ethical—considers and engages in ethical practices

8. Meaningful coherence—addresses what it claims to address, uses appropriate

methods, and links research questions, literature, findings and interpretations.

S. J. Tracy (2010). ‘Qualitative Quality: Eight “Big Tent” Criteria for Excellent Qualitative Research’, Qualitative

Inquiry, 16: 837–51.

Do we expect too much of quality criteria?

2. Teaching role

Cohorts of students introduced to quantitative research

quality criteria (internal, external, construct, conclusion, and

ecological validity). Advantage of small number of issues around

which criteria are grouped.

From a teaching point of view, long lists may be unhelpful.

Lack of differentiation and therefore guidance concerning

importance/significance.

Potentially off-putting.

Unrealisable.

Therefore, for teaching purposes, shorter lists may be more

helpful to students (and maybe lecturers).

Trends in discussions of mixed methods

criteria

Discussions of mixed methods research criteria may be

treading a similar path to discussions of qualitative

research criteria

Mixed methods criteria

The comprehensive approach (O’Cathain, 2010) – 6 domains + 2

Domain 1: Planning quality – 4

Domain 2: Design quality – 4

Domain 3: Data quality – 5

Domain 4: Interpretive rigour – 8

Domain 5: Inference transferability – 4

Domain 6: Reporting quality – 3

28 + 2 = 30 quality criteria

A. O’Cathain (2010). ‘Assessing the quality of mixed methods research: toward a comprehensive framework’. In

A. Tashakkori and C. Teddlie (Eds.), SAGE handbook of mixed methods in social and behavioral research (2nd

edition, pp. 531-55). Los Angeles: Sage.

Some recurring themes: Alan’s ‘must haves’

Need for quantitative and qualitative components to be

appropriately implemented

Need for transparency

Use of mixed methods to be linked to research questions

Need to be explicit about the mixed methods design and its

appropriateness to research questions

Importance of a rationale for the use of mixed methods

research

Importance of integration

To appear in a forthcoming article which has been accepted for publication









research



Recent review of health services research

‘Key methodological components that cut across qualitative and quantitative methodologies were often missing from mixed methods publications. Descriptions of sample selection and sampling procedures, the study context, and data-gathering procedures are essential aspects of interpreting study findings, and mixed methods studies should not be exempt from these basic research requirements. Many mixed methods studies did not include the level of detail that would likely be required for a qualitative or quantitative paper to be accepted in these high-ranking journals’.

J.P. Wisdon et al. (2012) ‘Methodological reporting in qualitative, quantitative and mixed methods health services research articles’, Health Services Research, 47(2): 721-45.









research



Two common approaches to mixed methods

research questions

1. Separate quantitative and qualitative research questions.

2. Overarching mixed research question; then expanded

or broken down into quantitative and qualitative sub-

questions.

From: Teddlie, C. and Tashakkori, A. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative

Approaches in the Social and Behavioral Sciences, Los Angeles: Sage, 2009, p. 133.









research



Types of mixed methods research design









research



Structure of the article: Parmelee et al. Introduction

Young people, voting, cynicism, and advertising

Theoretical and mixed methods perspectives

Method

Focus groups

Content analysis

Results

Short introduction

Media-based frames and the failure to engage college-age voters

Negative ads, audience-based frames, and cynical voters

Students’ recommendations for how to build more engaging political ads

Discussion

Summary and implications

Limitations and future research

J.H. Parmelee, et al. (2007) ‘ “What about people our age?”: Applying qualitative and quantitative methods to uncover how political ads alienate college students’, Journal of Mixed methods research, 1(2): 183-99

May be more useful for research quality

appraisal too

Lessons from qualitative research

Pratt (2008) – reviewers of qualitative management research articles for North American journals believe qualitative standards are: difficult to understand

lack consensus/consistent standards

lack clarity

also, articles often evaluated against inappropriate (usually quantitative research) criteria

Savall et al. (2008) – reviews of qualitative management research articles for a European journal: Quality criteria proposed by literature not identified in a content analysis

of reviews (9 criteria)

Quality criteria used by reviewers but rarely mentioned in the literature (6 criteria)

M.G. Pratt (2008) ‘Fitting oval pegs into round holes: tensions in evaluating and publishing qualitative research in top-tier North American journals’, Organizational Research Methods, 11: 481-509.

H. Savall. et al. (2008) ‘The emergence of implicit criteria actually used by reviewers of qualitative research articles: case of a European journal’, Organizational Research Methods, 11: 510-40.

May be more useful for research quality

appraisal too

Focusing on a core set of criteria rather than a

proliferation may mitigate some of these practices.

For mixed methods research, concentrating on small

number of core issues may help to focus attention on

quality at a relatively early stage of its evolution.

Importance of examples of mixed methods research

Key points

Need to distinguish between

quality criteria as a screening or evaluation tool; and

the possible pedagogical role of quality criteria

In the context of mixed methods research, I’ve tried to

suggest that a small number of criteria that capture

widely-held positions may be more helpful in a teaching

context.