2/5/2014

1

Analysis of Open‐Ended Survey Responses –Where to start?

Yao Zhang Hill, Ph.D.

Institutional ResearcherKapi‘olani Community College

In Support of Assessment Office’s Mission

Improve student learning through 

program assessment

2/5/2014

2

Workshop OutcomesYou will be able to:

1

• List the basic steps involved in analyzing open‐ended survey responses

2

• Carry out preliminary open‐ended survey responses analysis

3

• Know how to report the results

4• Use results

Program Assessment: It’s a Cycle

Student Learning Outcomes

Opportunities to Learn

Collection & Analysis of Evidence

Interpretation of Results

Improvement Plan

2/5/2014

3

Steps to take

1: Careful Reading

2: Unitization

3: Categorization

4: Coding

5: Finding patterns and trends

6: Reporting and Using

(D) Classes are small and fun. Many teachers care about their students.

Step 1 Careful Reading

2/5/2014

4

• Break responses into thought units/sense units/units of meaning

• Each unit communicates only one idea

• Can be several sentences, one sentence, text chunks, or one work.

Step 2 Unitization: Break it Apart

How about:  , “?” or “…”

Example

(G) Class size and teachers were engaged with the students.

(G) Class size

(G) Teachers were engaged with the students.

2/5/2014

5

Example

(A) Small class size and access to resources, like STEM

(A) Small class size

(A)Access to resources, like STEM

• How much material to use?– All if possible. If not, try a balanced random sub‐sample

• How to go about it?– Deductive‐inductive procedure– Skip difficult ones first– Select a sample (30) and check your code against your colleagues’.

• How detailed should it be? – Balance between verifiability and consistency

• Aggregate or break it done?

Step 3 Categorization

2/5/2014

6

Example Aspects of Categories

General

• People

• Perspective

• Activities

• Processes

• Relationship and social structure

Learning outcomes

• Attitudes

• Feelings

• Knowledge

• Skills

Accreditation Standards

• Instructional programs

• Student support services

• Library and learning support services

• Physical resources

• Technology resources

Example: Generating categories

Class Size

Instructional Faculty

Counselor/Advisor

Campus Resource

Faculty & Staff

2/5/2014

7

Hard to deal with categories:

• Other

• N/A

• Not relevant

• None

• Label each comment unit with the category.

• Decide to collapse categories or break it down more 

• Check your work 

– with another colleague (reproducibility)

– against yourself at a different time (stability)

Step 4 Coding

Step 3

2/5/2014

8

Example: CodingComment Category

(A) Small class size

(A) Access to resources, like STEM

(B) I really liked how the teachers…were.

(B) I really liked how the …counselors were.

(C) Instructor … are very easy to talk to and they are 

professional

(C) …Advisors are very easy to talk to and they are very 

professional

(D) Classes are small and fun.

(D) Many teachers care about their students

(E) Supportive faculty…

(E) Supportive … counselors

(F) Great resources

(G) Class size

(G) Teachers were engaged with the students

Class Size

Campus Resource

Example: CodingComment Category

(A) Small class size Class Size

(A) Access to resources, like STEM Campus Resource

(B) I really liked how the teachers…were. Instructional Faculty

(B) I really liked how the …counselors were. Counselors/Advisors

(C) Instructor … are very easy to talk to and they are 

professional

Instructional Faculty

(C) …Advisors are very easy to talk to and they are very 

professional

Counselors/Advisors

(D) Classes are small and fun. Class size

(D) Many teachers care about their students Instructional Faculty

(E) Supportive faculty… Instructional Faculty

(E) Supportive … counselors Counselors/Advisors

(F) Great resources Campus Resource

(G) Class size Class size

(G) Teachers were engaged with the students Instructional Faculty

2/5/2014

9

Comment Category(B) I really liked how the teachers…were. Instructional Faculty

(C) Instructor … are very easy to talk to and they are professional

Instructional Faculty

(D) Many teachers care about their students Instructional Faculty

(E) Supportive faculty… Instructional Faculty

(G) Teachers were engaged with the students Instructional Faculty

(B) I really liked how the …counselors were. Counselors/Advisors

(C) …Advisors are very easy to talk to and they are very professional

Counselors/Advisors

(E) Supportive … counselors Counselors/Advisors

(D) Classes are small and fun. Class size

(A) Small class size Class Size

(G) Class size Class size

(A) Access to resources, like STEM Campus Resource

(F) Great resources Campus Resource

Example: Coding Sentiment/Effect

Comment Category Sentiment

(B) I really liked how the teachers…were.Instructional Faculty

(C) Instructor … are very easy to talk to and they are professional

Instructional Faculty

(D) Many teachers care about their studentsInstructional Faculty

(E) Supportive faculty…Instructional Faculty

(G) Teachers were engaged with the studentsInstructional Faculty

(F) Some teachers always come to class late. I have a math teacher who came to class one hour late one time.

Instructional Faculty

… … …

Example: Coding Sentiment

2/5/2014

10

Comment Category Effect(B) My instructor’s presentation is clear. It helped me understand the coursecontent.

Instructional Faculty

(C) The advise that my instructor gave me helped me through the college.

Instructional Faculty

(D) The two courses that I took from Prof. XXX is so useful. The skills that I learned from these courses helped me find a job in the field of assessment. It changed my life!!!

Instructional Faculty

Example: Coding Effect

Coding Scheme1: Small Effect2: Medium Effect3: Large Effect

1

2

3

• Make the obvious obvious

• Make the obvious dubious

• Making the hidden obvious 

(Schlechty & Noblit, 1982)

Step 5 Identify the Patterns & Trends

2/5/2014

11

Techniques (1)

• Identify dominant themes

– “The greatest strength of the XXX department recognized by the respondents is [faculty]!”

– “... is another common theme raised by the students to demonstrate the strength of the department”

– “The main issues mentioned are…”

– “The most prevalent theme/factors are…”

– “To a lesser extent, … were mentioned.”

Techniques (2) 

• Identify relationships (advanced)

– Native Hawaiian students tend to believe…

– Tenure track faculty are more likely to raise the issue of …

– Part‐time students are in a more dire need of … 

2/5/2014

12

The report: To Write or Not to Write

“I find in my own work that final reports frequently have less impact than the direct, face‐to‐face interactions I have with primary evaluation users to provide them with feedback about evaluation findings and to share with them the nature of the data.” (Pattern, M., 2002, p. 510)

• Use executive summary for long reports

• Focus on the patterns and trends and the unexpected

• Use quotes to bring it to life

• Use bullet points, underlines, italics to make it easy to read

• Summarize the findings in tables and charts

If you must write a report…

2/5/2014

13

Step 6 Use – To Celebrate

Finding:

85% the respondents demonstrated gain in civil knowledge and sense of civil responsibility after participating in the Service‐Learning Program

Step 6 Use – To Improve the Program

Finding:An overwhelming complaint emerged is about the transcript review process being too long (e.g., over a year).

Diagnose 1: Under‐Staffed

Strategy 1: Expedite hiring 

Diagnose 2: Manual work

Strategy 2: Automatic Processing

2/5/2014

14

• Can you think of option categories for the closed‐ended question:

What are the important factors that helped you through the college? (Choose all that apply)

Instructors/Professors

Counselors/Advisors

Campus resources (e.g., library, STEM Center)

Class size

…

Step 6 Use – To Improve the Instrument

Summary of the steps

Reading

Unitization

Categorization

Coding

Patterns & Trends

Report & Use

2/5/2014

15

Questions?

Analyzing Open-Ended Responses Workshop at UHM 9/14/12 Yao Zhang Hill yaozhang@hawaii.edu

1

Sample Coding from the 2010 KapCC Professional Development Day Evaluation Survey Responses

Question: Please provide any comments on the organization of the event.

Category Sentiment

1 I was also signed up for clean-up, but since the lunch was served later than planned, I wasn't able to help because I had another meeting later. I felt badly so I stopped by again but they did not seem to need my help.

Clean-up Irrelevant

2 This was the best Faculty Development day we've ever had at K.C.C..I felt a sense of human warmth. I also got so much more out of 'playing' and 'learning with clay' and learning from Kauka de Silva than any meeting about something like assessment or some other pedagogical workshop. The reason this was the best one we've ever had was because it was rooted in spirit and identity of our namesake, Queen Kapi'olani. I was so happy and honored to learn about the roots of the Native Hawaiian culture and how it gives meaning to everything we do here. Please make next semester or next year's seminar similar..(copied to suggestions).the food was great!

General (best)Collegial connections Culture LearningFood

+++++

3 very different from all previous PD days. not necessarily better, just different.

General (different) Neutral

4 The food was excellent, the taste, the presentation and the traditional preparation.

Food +

5 I didn't get word until the last minute about what time the event began. The Hawaiian food was great, but once again, we were unsure what time lunch would begin, and got hungry waiting for lunch service to begin. Working in the garden was relaxing, fun and educational!

Info/communicationFoodActivities

-++

6 It was difficult to find specific information on the options. I was denied entry to the website on laulima when I tried to log in. The event at the Arboretum was fine.

Info/communicationRegistrationActivities

--+

… … … …

Sample Reports Lande, M., Parikh, S., Sheppard, S., Toye, G., Chen, H., & Donaldson, K. (2009). More to say: Analyzing

open-ended student responses to the Academic Pathways of People Learning Engineering Survey. In American society for engineering education conference proceedings. Retrieved from http://www.engr.washington.edu/caee/CAEE_Briefs_PDFs/MoretoSay_Lande_ASEE09.pdf

Hunter, A. (2006). Report on the qualitative analysis of five open-ended questions from the Carleton College off-campus studies alumni survey of the Marine Biology Seminar. (Report to Carleton College) Boulder, CO: University of Colorado at Boulder, Ethnography & Evaluation Research. Retrieved from http://www.colorado.edu/eer/downloads/CarletonMarineBiologyExecutiveSummary2006.pdf

Analyzing Open-Ended Responses Workshop at UHM 9/14/12 Yao Zhang Hill yaozhang@hawaii.edu

2

Selected Sample Report from the 2010 KapCC Professional Development Day Evaluation Survey Results

Summary of the Comments on the Evaluation of the Organization of the Event (Total comment units=45)25 out of 54 (46%) of

the participants provided feedback. In general they seemed to be happy with the organization of the event. The organization of the event received positive comments mostly in general organization, food, team activities, team leaders, and the feeling of collegial connection.

The most common negative comments were about the lack of information on the schedule and the location of the activities, not getting the word out enough, attendance, and registration process.

See Appendix 1 for the full responses and their coding.

Quotes:Food:-“The food was excellent, the taste, the presentation and the traditional preparation.”-" The Hawaiian food was great."-“ …lunch was very good.”- “Best Hawaiian food I have ever had!”…Collegial connection: -“Loved the bonding that occurred”-“ I feel enriched by this whole experience and a new fondness for my colleagues.”…Info/Communication:

“… the beginning was kinda bumpy getting information about the sessions, and the times”“Never found where lunch was being served. Went to cafeteria, but it was not there. Couldn't find any

reference to lunch location in WILD day flyers, emails, etc. Finally gave up with no lunch.”“I didn't get word until the last minute about what time the event began. “

Advanced Reading with Practical Guidance and ExamplesLeCompte, M. D. (2000). Analyzing qualitative data. Theory into practice, 39(3), 146-154. Patton, M. Q. (2002). Qualitative Research & Evaluation Methods (3 ed.). Thousand Oaks, CA: Sage. Ruona, W. E. A. ( 2005). Chapter 14: Analyzing qualitative data. In R. A. Wanson, & E. F. Holton III,

(Eds). Research in organizations: Foundations and methods of inquiry (pp. 233-263). San Francisco, CA: Berrett-Koehler Publishers, Inc. Retrieved from http://203.128.31.71/articles/157675314X.pdf#page=252

Thoms, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American Journal of Evaluation, 27(2), 237-246. doi: 10.1177/1098214005283748