Student Life Assessment Project

Module 4: Analyzing & Reporting Assessment Results

© 2013 Christie Cruise-Harper, PhD All Rights Reserved

The purpose of this module is to assist you with analyzing the data you gathered for the program/service you assessed in the 2013 – 2014 academic year.

Student Life Assessment Project

Department Program

Office of Multicultural Programs Multicultural Scholars Program/Dean’s Award Program

Personal Counseling Mandated Substance Abuse Assessment Program

Health and Wellness HEROs Program

Campus Ministry and Community Service

KLILV Sophomore Colloquium

Student Involvement CAB/MSG/CSI

Athletics SAAC and Champs

Residence Life Resident Assistant Program

Student Life/Associate Dean of Students

Habitat for Humanity

Student Life Assessment Project

In module 1 you developed learning outcomes for your program/service using Bloom’s Taxonomy as a guide.

Module 2 allowed you to strengthen those learning outcomes and guided you through the process of choosing learning activities.

Module 3 assisted you with choosing appropriate assessment tools/methods for the learning outcomes and learning activities you established.

This module will help you with analyzing and reporting the assessment data.

Student Life Assessment Project

Learning Outcome

What should your students be able

to do?

Learning Activity

What activity will help your

students achieve the learning

outcome?

AssessmentHow will you know whether students have achieved the learning outcome?

AnalysisWhat will you do

with all the information you collected from

your assessment plan?

Module 1 Module 2 Module 3 Module 4

Student Life Assessment Project

Assessment results must be analyzed to determine if student learning outcomes were met.

Data is analyzed for context, understanding and to draw conclusions.

Analysis of data gives the information meaning.

Taken from Academic Program Assessment: Tools & Techniques for Program Improvement

Analyzing Assessment Data

Determining how to organize, synthesize, interrelate, compare and present the assessment results are all part of analyzing the data.

Assessment data can be compared to findings from previous assessments, baseline data and existing criteria.

Taken from Academic Program Assessment: Tools & Techniques for Program Improvement

Analyzing Assessment Data

Quantitative◦ Also known as “empirical research”◦ Refers to any research based on something that

can be accurately and precisely measured.◦ Collects numerical data in order to explain or

predict a particular phenomena.

Taken from University of Wisconsin – Madison’s Ebling Library & R. Ouyang’s Basic Inquiry of Quantitative Research

For more information visit: http://researchguides.ebling.library.wisc.edu/content.php?pid=325126&sid=2940225

Types of Data

Qualitative◦ Refers to any research based on something that

cannot be accurately and precisely measured.◦ Collects narrative data to gain insights into a

particular phenomena.

Taken from University of Wisconsin – Madison’s Ebling Library & R. Ouyang’s Basic Inquiry of Quantitative Research

For more information visit: http://researchguides.ebling.library.wisc.edu/content.php?pid=325126&sid=2940225

Types of Data

Analyzing Quantitative Data

It’s all about the numbers!

There are four types of quantitative research methods:◦ Descriptive: collecting data for hypotheses testing*◦ Correlational: determining whether and to what degree a

relationship exists◦ Cause-Comparative: establishing the cause-effect

relationship◦ Experimental: establishes the cause-effect relationship,

but manipulates the cause

See notes section for more detailed information.

Taken from R. Ouyang’s Basic Inquiry of Quantitative Research

Types of Quantitative Research

The following are common methods of data collection in quantitative research:◦ Surveys and Questionnaires◦ Structured Interviews◦ Observation or Interaction Analysis◦ Secondary Data or Content Analysis◦ Experiments

See notes section for more detailed information.

Methods of Data Collection

In quantitative research there are two ways in which data are analyzed:

Descriptive Statistics◦ Procedures used to describe a given collection of

data.◦ The purpose is to describe the sample at hand-the

collection of cases that we have examined. Inferential Statistics

◦ Procedures that let us generalize our findings beyond the particular sample at hand to the larger population represented by that sample.

Taken from Diekhoff, G.M.(1996). Basic Statistics for the Social and Behavioral Sciences.

Data Analysis

Most Student Life assessment projects do not seek to generalize its findings to the entire Maryville University student body. Because our goal is to learn about the sample at hand, descriptive statistics will be the focus of the quantitative data analysis for module 4.

There are three types of descriptive statistics to provide you with an overview of your data:◦ Central Tendency Measures*◦ Variability Measures◦ Frequency and Percentages*

Descriptive Statistics

More on descriptive statistics later. Before beginning any data analysis, you

must first identify the level of measurement associated with your quantitative data. There are four levels of measurement:◦ Nominal◦ Ordinal◦ Interval◦ Ratio (Scale)

Levels of Measurement

Nominal: basic classification data; do not have meaningful numbers attached to them, but are broader categories

Ordinal: have numbers attached to them and the numbers are in a certain order, but there are not equal intervals between the numbers

Interval: have equal intervals between the numbers; the distance between attributes have meaning

Ratio: have equal intervals between the numbers; there is an absolute zero that is meaningful

Taken from: http://www.uni.edu/commstudies/researchmethods/chapterfour1.html

Levels of Measurement

Once you have decided on your data collection method, decided on the level of measurement for your variables, and collected the data, you are ready to begin analyzing the data. There are two software programs I recommend and they are available on campus:

Qualtrics SPSS (Statistical Package for the Social

Sciences)

Data Analysis and Software

The following quantitative data analysis procedures are used to describe the data and can be done in Qualtrics and SPSS:◦ Data Tabulation (Frequency Distributions, Percentiles)

◦ Descriptive Data (Central Tendency)

◦ Data Disaggregation

◦ Moderate and Advanced Analytical Methods

For a detailed description of these analyses visit:http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/ ORChapters 1-3 in Diekhoff’ s Basic Statistics for the Social and Behavioral Sciences

Data Analysis and Software

Surveys and questionnaires can be developed in Qualtrics and the data can be analyzed within the software. There are tutorials to assist you with analyzing your data.

Crosstabs: http://qualtrics.com/university/researchsuite/reporting/cross-tabs/about-cross-tabulations/

Understanding Statistics: http://qualtrics.com/university/researchsuite/reporting/cross-tabs/understanding-statistics/

Qualtrics

SPSS◦ Below are several links to resources to assist you

with using SPSS. http://www.slideshare.net/sspink/seminar-on-spss http://www.youtube.com/watch?v=eTHvlEzS7qQ http://

www.youtube.com/watch?v=HT0Skh2UP1U&feature=related

http://calcnet.mth.cmich.edu/org/spss/toc.htm

Data Analysis and Software

In addition to a narrative about your data analysis, quantitative results are presented in the following ways:◦ Charts◦ Graphs◦ Tables

The following websites provide examples, in APA, of how quantitative results are presented:

https://owl.english.purdue.edu/owl/resource/560/19/ https://owl.english.purdue.edu/owl/resource/560/20/

Reporting Quantitative Results

Tables and graphs can be created in Qualtrics. For more information visit: ◦ http

://qualtrics.com/university/researchsuite/reporting/reporting-beta/tables/#AboutTables

◦ http://qualtrics.com/university/researchsuite/reporting/reporting-beta/graphs/

Graphs can also be created in SPSS. For directions visit:◦ http://julius.csscr.washington.edu/pdf/spss.pdf◦ http://

academic.udayton.edu/gregelvers/psy216/spss/graphs.htm◦ http://

www.ats.ucla.edu/stat/spss/seminars/SPSSGraphics/spssgraph.htm

Reporting Quantitative Results

Analyzing Qualitative Data

I want to know how you feel!

It is important to achieve empathic understanding to comprehend the participant’s experience with a minimum of distortion or bias;

The researcher must attempt to recognize their own personal prejudices, stereotypes, myths, assumptions and other thoughts or feelings that may cloud the perception of other people’s experiences;

Knowledge of other’s experience cannot be assumed regardless of familiarity with their subcultural landscape; and

Do not expect participants to hold the same values as you.

M. Ely, M. Anzul, T. Friedman, D. Garner, A.M. Steinmetz in Doing Qualitative Research: Circles within Circles

Before Analyzing Qualitative Data…

Qualitative Research Methodologies◦ Basic Interpretive Qualitative Study*◦ Grounded Theory◦ Phenomenology◦ Case Study◦ Ethnography◦ Postmodern Research◦ Critical Qualitative Research◦ Narrative Analysis

For more information visit: http://www.fctl.ucf.edu/researchandscholarship/sotl/creatingsotlprojects/implementingmanaging/qualitativeresearchtypes.php

Types of Qualitative Research

Participant Observation Interviewing (formal & informal) Focus Groups Document Analysis Logs (notes/reflections from observations

and interviews) Audio and videotaping On-going data analysis

M. Ely, M. Anzul, T. Friedman, D. Garner, A.M. Steinmetz in Doing Qualitative Research: Circles within Circles

Methods of Data Collection

Coding is the process of organizing data into chunks or segments before making meaning of the information.

Coding involves taking text data or pictures gathered during data collection, segmenting it into categories, and labeling those categories with a term, often a term used by the actual participant.

Taken from: Creswell, J.W.(2009). Qualitative, Quantitative and Mixed Methods Approach, chapter 9, Qualitative Procedures.

Coding and Categorizing

Begin the coding process by first reviewing your learning outcomes as a reminder of what you are assessing. Your coding scheme will be based on your learning outcomes.◦ For example: Undergraduate students who

participate in the Multicultural Scholars Program will be able to describe their talents, strengths and social group memberships.

Coding involves assigning a word, phrase, number or symbol to each coding category.◦ For example: describe talents and strengths

Coding Qualitative Data

Codes can be pre-set or emergent. You should have both.◦ Pre-set: A list of codes created in advance by the

researcher based on the research question, learning outcomes, or conceptual framework.

◦ Emergent: Ideas, concepts, actions and meanings that come up from reading and analyzing the data that are not in the pre-set codes.

Coding will serve as a system to help you to organize your data.

For more information and an example of coding visit: http://programeval.ucdavis.edu/documents/Tips_Tools_18_2012.pdf

Coding Qualitative Data

Once you have gone through all documents and coded them, they can now be gathered into families of codes or categories.

Materials are sorted by these categories, identifying similar phrases, patterns and relationships.

As you code and categorize the data, look for the interrelationships among categories.

Sorted materials are examined to isolate meaningful patterns.

Identified patterns are used to create themes.

Creating Categories

It is best to start any report of qualitative results with an overview of how data were processed and coded.

The results are presented as “findings”. The findings are organized by themes with

substantial evidence that links to the themes included in the findings (i.e. quotes from participants).

Reporting Qualitative Results

Using the DataSo now what?

Closing the loop - using the assessment results for program improvement.

Closing the Loop

Assessment Plan

Data Collection

Data Analysis

Assessment Report

Identify Changes

Implement Changes

The findings from your quantitative and/or qualitative research will yield rich information that will be included in your assessment report.

These findings will provide you with insights into what’s working with your program/service and areas of improvement.

From these findings you will make recommendations for improvement of your program/service.

Closing the Loop

Recommendation:

Action step(s) What action steps must be completed to implement the recommendation?

Estimated implementation date When does the program expect to begin to implement the action steps?

Estimated completion date When does the program expect the recommendation to be fully implemented and/or achieved?

Person(s) responsible Who will take responsibility for seeing that the actions steps are implemented?

Expected outcome What is the expected impact/outcome the recommendation will have on the program if it is implemented?

Estimated cost(s) What is the estimated cost of implementing the recommendation?

Status update What progress has been made towards achieving the recommendation?

Program ImprovementYou may find it helpful to create a program improvement plan to transform the recommendations made into actions for improvement.

Accreditation Planning and budgeting Maryville University requirements Student Life requirements Program promotion/marketing Recruitment/retention initiatives Publications Conference presentations Student development opportunities Professional development opportunities Grant applications

Taken from: Academic Program Assessment: Tools & Techniques for Program Improvement

Other Uses of Assessment Results

Academic Program Assessment: Tools & Techniques for Program Improvement.(2013). SUNY.

Analyzing Assessment Data.(2013). http://www.sunyorange.edu/assessmentapa/docs/AnalyzingandUtilizingAssessmentData.pdf. State University of New York (SUNY) Orange County Community College.

Basic Inquiry of Quantitative Research.(n.d.). http://ksumail.kennesaw.edu/~rouyang/ED-research/details.htm. Kennesaw State University.

Center for Evaluation and Research.(2012). http://programeval.ucdavis.edu/documents/Tips_Tools_18_2012.pdf. University of California at Davis.

References

Creswell, J.W.(2009). Qualitative, Quantitative and Mixed Methods Approach, chapter 9, Qualitative Procedures.

Diekhoff, G.M.(1996). Basic Statistics for the Social and Behavioral Sciences.

Differences Between Quantitative and Qualitative Research. (2013). http://researchguides.ebling.library.wisc.edu/content.php?pid=325126&sid=2940225. University of Wisconsin-Madison-Health Sciences Ebling Library

Ely, M., Anzul, M., Friedman, T., Garner, D., Steinmetz, A.M.(1991). Doing qualitative research: Circles within circles. RoutledgeFalmer: New York.

References

Faculty Center for Teaching and Learning.(2013). Types of qualitative research: Explained within a SOTL framework. http://www.fctl.ucf.edu/researchandscholarship/sotl/creatingsotlprojects/implementingmanaging/qualitativeresearchtypes.php. University of Central Florida.

Gathering Data and Assessing Results.(2013). http://nnlm.gov/evaluation/guide/stage 5.pdf. National Network of Libraries of Medicine.

The Pell Institute and Pathways to College Network.(2013). Evaluation Toolkit. http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/

University of Northern Iowa.(2013). Communication Studies – Research Methods Chapter Four. http://www.uni.edu/commstudies/researchmethods/chapterfour1.html

References