Exploring the RolesExploring the Rolesof Faculty Supervision: of Faculty Supervision:

Improving Qualitative Doctoral Improving Qualitative Doctoral Dissertation MethodologyDissertation Methodology

Dan Kaczynski, PhDDan Kaczynski, PhDdkaczyns@uwf.edudkaczyns@uwf.edu

Research ProblemResearch Problem

Increasing emphasison quality

Understand assessment practicesStrengthen qualitative research skills

Develop future researchers

Morequalitative

dissertations

Shiftingsupervisory

roles

Open DiscussionOpen Discussion

Do you use technology Do you use technology in your research?in your research?

Open DiscussionOpen Discussion

How should we explore the tensions How should we explore the tensions within and between:within and between:

Assessing QualityAssessing Quality Adoption of QDASAdoption of QDAS

Qualitative SoftwareQualitative SoftwareNVivo, MAXqda, Atlas ti, QDA Miner, Qualrus, TransanaNVivo, MAXqda, Atlas ti, QDA Miner, Qualrus, Transana

Kaczynski (2004) http://www.aare.edu.au/04pap/kac041065.pdf

What is Good Qualitative Research?What is Good Qualitative Research?

What does good work look like?What does good work look like?

Identify indicators of Identify indicators of

quality in a thesis:quality in a thesis:

Identify common errors Identify common errors in a thesis:in a thesis:

What is Quality?What is Quality?

The researchers logic of The researchers logic of justificationjustification

“ “Flaws in the logic of justification can potentially Flaws in the logic of justification can potentially occur anywhere in the inquiry process. The occur anywhere in the inquiry process. The nature of such flaws and where they occur can nature of such flaws and where they occur can jeopardize the soundness of a study in one or jeopardize the soundness of a study in one or more ways.”more ways.”

((Piantanida & GarmanPiantanida & Garman 1999, p. 147) 1999, p. 147)

Types of Quality CriteriaTypes of Quality Criteria

Philosophical Criteria Philosophical Criteria (Lincoln, 1995; Lincoln & Guba, 1985)(Lincoln, 1995; Lincoln & Guba, 1985)

Procedural CriteriaProcedural Criteria(Creswell, 1998)(Creswell, 1998)

Philosophical CriteriaPhilosophical Criteria (Lincoln & Guba, 1985)(Lincoln & Guba, 1985)

CredibilityCredibility Is the work authentic?Is the work authentic?

TransferabilityTransferability Will the work fit outside this situation?Will the work fit outside this situation?

DependabilityDependability Is the researcher consistent?Is the researcher consistent?

ConfirmabilityConfirmability Are interpretations defensible?Are interpretations defensible?

Procedural CriteriaProcedural Criteria

Quality of methodsQuality of methods(open-ended interviews)(open-ended interviews)

Quality of dataQuality of data(verbatim long transcripts)(verbatim long transcripts)

Quality of data analysisQuality of data analysis(comprehensive data treatment)(comprehensive data treatment)

(Silverman, 2004 [Sacks, 1984])(Silverman, 2004 [Sacks, 1984])

Standardized Procedural CriteriaStandardized Procedural Criteria(controversial checklists or guidelines)(controversial checklists or guidelines)

Does the title reflect the study focus?Does the title reflect the study focus? Is the problem socially important?Is the problem socially important? Is the literature review comprehensive?Is the literature review comprehensive? Has study conformed to ethics standards?Has study conformed to ethics standards? Are issues of sampling discussed?Are issues of sampling discussed? Did findings answer the questions?Did findings answer the questions? Was study written convincingly? Was study written convincingly? Are issues of trustworthiness addressed?Are issues of trustworthiness addressed?

Quantitative and Qualitative CriteriaQuantitative and Qualitative Criteriafor Assessing Research Quality and Rigorfor Assessing Research Quality and Rigor

Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002) p. 30Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002) p. 30

Quantitative Quantitative term term

Qualitative Qualitative termterm

Strategy employedStrategy employed

Internal validityInternal validity CredibilityCredibility Prolonged engagement in fieldProlonged engagement in fieldUse of peer debriefingUse of peer debriefingTriangulationTriangulationMember checksMember checksTime samplingTime sampling

External validityExternal validity TransferabilityTransferability Provide thick descriptionProvide thick descriptionPurposive samplingPurposive sampling

ReliabilityReliability DependabilityDependability Create an audit trailCreate an audit trailCode-recode strategyCode-recode strategyTriangulationTriangulationPeer examinationPeer examination

ObjectivityObjectivity ConfirmabilityConfirmability TriangulationTriangulationPractice reflexivityPractice reflexivity

Transparent AssessmentTransparent Assessment

Explore rich diversity of meaningsExplore rich diversity of meanings Sensitized appreciation of worthSensitized appreciation of worth Deeper assessment of analysisDeeper assessment of analysis Multiple paths to look inside Multiple paths to look inside Transparency strengthens credibilityTransparency strengthens credibility

Data CollectionData Collection

Stage 1Stage 1 Stage 2Stage 2 Stage 3Stage 3

Survey Survey ResearchResearch

Interviews/Interviews/ Documents Documents

Document Document ReviewReview

Supervisory Supervisory AbilityAbility

Quality Factors Quality Factors TechnologyTechnology

ResourcesResources

Professional Professional DevelopmentDevelopment

PerceptionsPerceptions

ProcessesProcesses

Assessment Assessment PracticesPractices

Findings: Knowledge,Findings: Knowledge,Ability, and ConfidenceAbility, and Confidence

Satisfactory Satisfactory or Higheror Higher

MM SDSD

SupervisingSupervising 73.9%73.9% 3.483.48 1.281.28

Serving on a Serving on a committeecommittee

91.3%91.3% 4.174.17 .94.94

Judging Judging qualityquality

91.3%91.3% 4.094.09 .95.95

Findings: TechnologyFindings: TechnologyTools Used in AssessmentTools Used in Assessment

0

2

4

6

8

10

12

14

NO NA AT HR IR MQ NV QM QU SS TR OT

Technology Tools

No

. of

Res

po

nd

ents

52% (None)30% (Not applicable)17% NVivo4% InfoRapid

Findings: ResourcesFindings: ResourcesConsulted for ExpertiseConsulted for Expertise

0

2

4

6

8

10

12

14

16

18

20

NA CF SP CE WA WC

Resources Used

Nu

mb

er o

f R

esp

on

den

ts83% Others70% Publications30% Conference

workshops22% Campus

workshops13% Continuing

education

Findings: ConceptualizationsFindings: Conceptualizationsof Quality (cont.)of Quality (cont.)

AIAI: Alternative interpretationsAlternative interpretationsCE: Consideration of ethical CE: Consideration of ethical issuesissuesAG: Ability to generalize findingsAG: Ability to generalize findingsHC: Hierarchical code structureHC: Hierarchical code structureMC: Member-checkingMC: Member-checkingMSMS: MemosMemosAT: Methodological audit trailAT: Methodological audit trailPD: PD: Peer debriefing Peer debriefing PF: Prolonged field engagementPF: Prolonged field engagementAS: Qualitative data analysis AS: Qualitative data analysis softwaresoftwareRORO: Researcher objectivityResearcher objectivitySSSS: Sampling strategiesSampling strategiesSDSD: Self-disclosureSelf-disclosureSC: Social context SC: Social context TOTO: Theoretical orientationTheoretical orientationTN: Triangulation TN: Triangulation VYVY: ValidityValidity

Stage 2 Findings: Critical NeedsStage 2 Findings: Critical Needs

Building knowledge and skillsBuilding knowledge and skills– Moving beyond superficial assessmentMoving beyond superficial assessment– Significance of researcher transparencySignificance of researcher transparency– Teaching students to self-assessTeaching students to self-assess

Building a community of practiceBuilding a community of practice– Strengthening qualitative research skillsStrengthening qualitative research skills– Sharing assessment strategiesSharing assessment strategies– Engaging in professional developmentEngaging in professional development

Stage 3 Findings:Stage 3 Findings:Role of TechnologyRole of Technology

0

10

20

30

40

50

60

70

80

90

Total Dissertations Analysis Software

(Not identified) Mixed Methods Qualitative Quantitative

Study FindingsStudy Findings

Highly favorable attitudes toward Highly favorable attitudes toward qualitative researchqualitative research

Diverse conceptualizations of qualityDiverse conceptualizations of quality Need for alternative assessment Need for alternative assessment

frameworksframeworks Need and desire to strengthen Need and desire to strengthen

knowledge and skillsknowledge and skills Need for professional developmentNeed for professional development

Future ChallengesFuture Challenges

Progressing research methodsProgressing research methods

Mixed Mixed →→ Blended Blended →→ Integrated Integrated

Emerging research innovationsEmerging research innovationsmainstream adoption of QDASmainstream adoption of QDAS

Positioning quality research Positioning quality research standardsstandards

Future Research QuestionsFuture Research Questions

What does it mean to disclose or conceal the role What does it mean to disclose or conceal the role of technology?of technology?

Does nondisclosure of analysis software imply the Does nondisclosure of analysis software imply the presumption that the use of technology is presumption that the use of technology is ubiquitous and commonly accepted?ubiquitous and commonly accepted?

Does nondisclosure of QDAS suggest a student’s Does nondisclosure of QDAS suggest a student’s fear of the supervisor’s acceptance or fear of the supervisor’s acceptance or sanctioning?sanctioning?

In what ways and under what conditions does a In what ways and under what conditions does a technological tool become a barrier to the technological tool become a barrier to the learning process for the teacher and the learner? learning process for the teacher and the learner?