Design and Analysis of Open Response Surveys: Lessons Learned

Design and AnalysisDesign and Analysis of Open Response of Open Response Surveys: Lessons Surveys: Lessons LearnedLearned

Dr. Joan Burtner Associate Professor of Industrial Engineering Mercer University

Design and Analysis of Design and Analysis of Open Response SurveysOpen Response Surveys

Designing Survey

Administering Survey

Conducting Survey Analysis

Reporting Survey Results

IIE/SHS FEB 2012

Mercer University School of Engineering Slide 2

Open Response SurveysOpen Response SurveysAsk participants to respond to a question such

as “What can we do to improve patient care?

Excellent method for collecting “Voice of the

Customer” data within a Quality Mgt. System

Allows analysts to interpret qualitative

characteristics (tone, frustration, pride)

Opportunity to discover unanticipated

responses

IIE/SHS FEB 2012


Designing and Administering Designing and Administering the Surveythe Survey

Purpose of survey

Delivery method (paper, electronic, phone, in-

person, etc.)

Length of survey, allowable response time

Incentives for participation

Anonymity, confidentiality

Who will collate and/or analyze data?

How will data be analyzed?IIE/SHS FEB

2012Mercer University School of Engineering Slide 4

Design of Survey Design of Survey QuestionsQuestions

Structure, specificity of written questions

◦ What can we do to improve the process?

◦ What is the one thing that will make your job easier?

◦ What is the “one idea” that will improve the

registration process?

Text-based questionnaires vs. oral interviews

◦ Limited, prescribed, uniform questions

◦ Non-uniform follow on questions in addition to

prescribed questions

IIE/SHS FEB 2012


Example 1: On-line Survey Example 1: On-line Survey Question Question Please describe challenges Please describe challenges that make it difficult to deliver patient that make it difficult to deliver patient

care. care. Over 700 responses submitted

anonymously over a two week period

Data downloaded into an Excel file

Many rows of data contained no text,

just time and date of submission

Two 2-person teams reviewed the data

and developed preliminary codesIIE/SHS FEB


Example 1: Initial CodingExample 1: Initial CodingCoding done electronically by labeling

each response in Excel

Initial decision to allow only one major

code for each entry to simplify analysis.

Twenty-four categories emerged between

the two teams.

Overlap and clarification of labels resulted

in the elimination of four categories.IIE/SHS FEB


Example 1: Revised Example 1: Revised CodingCoding

Difficult to gain consensus among all four

coders on the “one” main concept for

each response

Realization that multiple codes increased

validity of work

Excel spreadsheet expanded to four

columns to accommodate multiple codes

for each responseIIE/SHS FEB


Code - Process Change?Code - Process Change?Patients should not be transferred to med surg

during change of shift. There needs to be a

window of time that patients cannot be

transferred. It is very unfair to the patient to

be transferred at that time as going off nurses

are trying to get caught up and give report;

and coming on nurses are trying to get report.

For example; no transfers between 6:00 and

07:15pm IIE/SHS FEB


Code - Work Load? Policy?Code - Work Load? Policy? If a patient is computer literate, why do we not

have some of the admission forms on line so that

patients can fill out the questions at home? They

would certainly have access to their medications &

we might have a more exact list of what they take.

They would be in a more comfortable & less

stressful environment when filling out the forms &

the older folks would most likely have a family

member to assist in remembering when they had

previous procedures, etc. IIE/SHS FEB

2012Mercer University School of Engineering

Slide 10

Audience Participation Audience Participation ExerciseExercise

Form teams of two or three

Consider the following responses

(handout)

How would you code the data?

Do multiple codes increase

validity?

IIE/SHS FEB 2012

Mercer University School of Engineering

Slide 11

Example 2: Nursing Example 2: Nursing InterruptionsInterruptions

Research agreement with a local

hospital

Students observed nurses at work

Data collected orally and transcribed by

observers

Anecdotal reports from nurses collected

periodically as time permitted

IIE/SHS FEB 2012


Slide 12

Example 2: Nursing Example 2: Nursing Interruptions Data Interruptions Data Excerpt 1Excerpt 1

April 15th

Comments by Nurse - Interruptions

Helping with code lift or code response

Family

Telephone calls

Doctors

When technician needs help

Comments by Observer 1

Tech was using chart station when Nurse was trying to chart

Having room in another section causes a lot of extra walking

This could be a system failure interruptionIIE/SHS FEB


Slide 13

Example 2: Nursing Example 2: Nursing Interruptions Data Interruptions Data Excerpt 2Excerpt 2

April 16th

Comments by Nurses - Interruptions

Family

Lab Calls asking about results or talking about results

Doctors calling

Patients – getting called by other patients and helping other patients

Unexpected events – Patients getting out of bed, codes

Helping nurses, covering nurses when they are out for lunch

Comments by Observer 2

Quiet day with not many interruptions

Nurse didn’t have much to doIIE/SHS FEB


Slide 14

Coding Nursing Coding Nursing InterruptionsInterruptions

Nature of the interruption

◦Phone call , call light, patient’s

family, healthcare professional, etc.

Urgency of the interruption

◦Avoidable, justifiable, etc.

Consequence of the interruption

◦Delay, error, etc.IIE/SHS FEB


Slide 15

Example 3: On-line Survey Example 3: On-line Survey Question Question Please list and describe Please list and describe

ideas that will contribute to cost efficiency ideas that will contribute to cost efficiency within our system.within our system.

Over one hundred responses submitted

Data downloaded into an Excel file

Single person reviewed the data and

categorized each response

Fifty-eight usable responses

Data organized into a pivot table

IIE/SHS FEB 2012


Slide 16

Reporting Analysis ResultsReporting Analysis ResultsExample 1

◦ Ten-page written report◦ Power-Point presentation to five-person

committee◦ Follow-up submission: frequency distribution

models overall and departmentalizedExample 2

◦ In-person feedback to nurse manager◦ Brief report with Pareto diagrams

Example 3◦ Electronic feedback to client◦ Pivot table demonstration

IIE/SHS FEB 2012


Slide 17

Code Frequency - All Code Frequency - All LocationsLocations

IIE/SHS FEB 2012


Slide 18

Frequency - AberdeenFrequency - Aberdeen

IIE/SHS FEB 2012


Slide 19

Frequency - DallasFrequency - Dallas

IIE/SHS FEB 2012


Slide 20

Frequency - GreenvilleFrequency - Greenville

IIE/SHS FEB 2012


Slide 21

Example 3: Pivot Table 1Example 3: Pivot Table 1

IIE/SHS FEB 2012


Slide 22


IIE/SHS FEB 2012


Slide 23


IIE/SHS FEB 2012


Slide 24

Lessons Learned 1Lessons Learned 1Nurses are willing to spend time

completing surveys.

Many are eager to have their ideas

heard.

It is important to give feedback, when

possible, with respect to how the survey

results may change the process.

IIE/SHS FEB 2012


Slide 25

Lessons Learned 2Lessons Learned 2The process of analyzing and quantifying

open-ended responses is very time-

consuming.

It is very difficult to gain consensus on some

responses.

System-wide data should also be reported at

the unit level if possible.

In the case of “one best idea” it is difficult to

prioritize.IIE/SHS FEB


Slide 26

Lessons Learned 3Lessons Learned 3Due to the somewhat subjective nature of

the coding and frequency tabulation, it

may be difficult to use open-ended survey

responses to measure the effectiveness of

process changes.

However, changes in the culture and

attitudes of the respondents may be

apparent.IIE/SHS FEB


Slide 27

Questions / Contact Questions / Contact InformationInformation

Dr. Joan Burtner [email protected]

Associate Professor of Industrial

Engineering and Industrial Management

(478) 301- 4127


1400 Coleman Avenue

Macon, GA 31207

IIE/SHS FEB 2012


Slide 28

Documents

Design and Analysis of Open Response Surveys: Lessons Learned