Using Mixed Method Approaches in Institutional Research Rick Kroc University of Arizona Richard...

Using Mixed Method Approaches in Institutional Research

Rick Kroc

University of Arizona

Richard Howard

University of Minnesota

Arizona Association for Institutional ResearchAnnual Meeting

Tucson, Arizona March 2007

Purpose of Decision Support

is

to reduce the risk to the decision maker

Structure and Subjectivity

Research Process

Ways of Knowing

Two Classic Paradigms

Positivist Constructivist

there are multiple realities and that truth is ever changing, dependent on context and the individual (subject and researcher)

there is a single reality or truth across time and contexts, and that this truth can be understood through the objective study of independent variables

Conceptual Model of the Constructivist Research Process

After a literature review and pilot study, define the Study:

ProblemPurposeQuestion

Data Collection & Management

Data Restructuring/Coding & Analyzing/Read-Theme

Data Verification with Respondents:

FACTUAL ERROR

Analysis Verification with Reviewers:

Interpretation Error

COMMUNICATENew Theory/Model

TEST Theory/Model

Iterative Process

Delimited by PP&Q

Conceptual Model of the Positivist Research Process

After a literature review and pilot study define the project

ProblemPurposeQuestion

Operationalize theory or model constructs

Data Collection and Management

Data restructuring and analysis

(Interpretation of Results)

Communication of new

knowledge

For internal decision support- test meaning of the findings

For external contexts- test/replicate for generalization

RESEARCH PARADIGMS

APPROACHES

Constructivist Positivist

QuantitativeQualitative

Consumer Generalizes Researcher Generalizes

Mixed Methods

Combines the collection, analysis, and interpretation of qualitative and

quantitative data.

Often used in studies that require an understanding of not only “what” is

observed but also “why” the observed behavior occurred.

Such as in assessment, evaluation, action research, and other decision support studies.

Use of Mixed Method Approaches in Institutional Research

Provides answers to two questions:

What - What happened and what was the outcome? Answers tend to be

arrived at using quantitative methods

Why - Why did people behave the way they did? Answers tend to be arrived at using qualitative methods

What - goals are - how measured – Supports Accountability (External)

Why – goals were met or not – people issues – Often the focus of Assessment/Evaluation (Internal)

So What?

Ways of Knowing

A Third Paradigm

Positivist Constructivistthere are multiple realities and that truth is ever changing, dependent on context and the individual (subject and researcher)

there is a single reality or truth across time and contexts, and that this truth can be understood through the objective study of independent variables

PragmatismUse what ever approach and methodology that will best answer the question.

Charge from President Likins“As Chair of the NCAA Division I Presidential Task Force on the Future of Intercollegiate Athletics, I have both the opportunity and the responsibility to examine the culture of higher education athletics programs in America. My responsibilities begin at home. My task force experience as well as events at colleges and universities around the country led me to conclude that it is appropriate to examine the environment defined by the culture of high-level athletic competition here at The University of Arizona. At the national level a serious movement to strengthen the academic experience of student-athletes is well underway. We must examine the presence of any cultural influences here that have the potential of distorting the shared values of academics and athletics in higher education.”

Focus of the Study

The study was divided into two phases. Phase I, mostly quantitative in nature, was designed to look somewhat broadly at the two fundamental study areas:– Student athletes’ academic experience; and– Student athletes’ culture and environment.

Using qualitative methods, Phase II was designed to dig more deeply into the critical issues identified from Phase I. Detailed, targeted recommendations would be developed from these two phases.

Domain of the Study

Athlete’s academic experience

• Academic preparation• Progress to degree• Student engagement• Academic support• Post-eligibility academic

experience• Other academic issues

Athlete’s culture and environment

• Alcohol and drugs• Dishonesty and cheating• Gambling• Racism• Sexism• Sexual behavior• Violence• Weapons

Phase 1—Quantitative • Board of Regents and NCAA reports on athletes’ grades

and graduation rates (ten year history).• Institutional student records data.• Health and Wellness Survey, administered annually to

assess student alcohol, drug, and sexual behaviors (seven year history).

• Dean of Students Code of Conduct and Code of Academic Integrity violations (five year history).

• Coalition on Intercollegiate Athletics (COIA) report.• Annual University Survey of Graduating Seniors, assessing

student engagement, satisfaction, and other academic issues (five year history).

• NCAA GOALS survey (student-athletes and student non-athletes).

• Additional UA-specific survey questions appended to GOALS survey (student-athletes and student non-athletes).

Phase II—Qualitative

• Focus groups• Interviews• Follow-up surveys• Populations:

– Students (athletes and non-athletes)– Coaches– Faculty– Staff– Advisors

The Value of a Mixed Methods Approach

• Athletic culture spans a spectrum of areas, some of which (alcohol use, for example) have become fairly well understood over the years from quantitative methods, others of which (physical violence, weapons use) need the deeper exploration and understanding provided by qualititative methods.

The Value of a Mixed Methods Approach

• The formative and summative aspects of this study, which is more like program evaluation in many ways than like a traditional research study, lend themselves to a mixed methods approach.

The Value of a Mixed Methods Approach

• Audiences and stakeholders consist of some people who respond best to systematic, inferentially rigorous, quantitative data, but also others who want to hear the complex, richly detailed stories derived from the individuals who are immersed in the culture. Different audiences resonate to different approaches.

The Value of a Mixed Methods Approach

• Mixed methods also means that multiple methods have been employed, which is essential to using triangulation as a means to establish greater validity of the findings. Any single approach used in the messy world of social science research will be flawed. If a wide array of methods leads to consistent conclusions, then analysts and stakeholders feel more confident about conclusions and recommendations.

Questions??