Surveys and Such

Presented by Dr. Brandi Elliott

and T.N. Vaught

Road Map

• Addressing Survey Fatigue

• Survey Construction

• Survey Analysis

Survey Fatigue

• Survey Fatigue Lowers Response Rates

• This results from over-surveying

• When an individual is inundated with invitations to

complete many surveys, they feel tired, or

“fatigued” when it comes to taking surveys

• This feeling can have adverse affects on response

rates

A Few SurveysType Pro Con Example

Paper/Pencil • High response

• Cheap

• Data entry

required

• Environment

• Ally Training

Curriculum Series

Email • Easy analysis (survey

engines)

• Larger group

• Incentives

• Cost varies

• Low response

• Campus Climate

Survey

• Welcome Weekend

Social Media • Free marketing

• High interaction

w/participants

• Easy to embed

• Response varies

• Engagement

• Incentive

• LGBTQ Meet and

Greet 2015

#LGBTQGreet15

• Drag Shows

Post Its/Index Cards • High response

• Cheap

• Easy to embed

• Data entry

required

• Organize

• Emerging Ethnic

Leadership Retreat

• RAPP Retreat

PRS/Response Software • High response

• Easy analysis

• Easy to embed

• Expensive

• Cumbersome

• Student

Government/Classroom

Survey Construction

• Purpose

– The why behind program/event

• Likert Scale

– Unipolar

– Bipolar

• Qualitative Questions

– Only ask the question if you are going to utilize it

Tips for ConstructionSurvey building is as much an art as it is science

• Keep it simple

• Use scales whenever possible

• Keep coded values consistent

• Explain why

• Speak your respondents’ language

• Follow a logical order

• Take your survey for a test drive (Also incentives)

Avoid in Survey Five Things to Avoid

• Don’t lose sight of what you want to know

• Don’t just ask, measure

• Don’t create your own survey format

– Most popular: Likert Scale

• Don’t be afraid of validity

– Measuring the correct variable

• Don’t ignore reliability

– Consistent, predictable, and fairly accurate ways

Reliability vs. Validity

Examples

Safe Zone Training Evaluation Group: ________________________________________ Date: _____________ Please use the following scale to evaluate today's training:

1 = Poor 2 = Fair 3 = Adequate 4 = Good 5 = Excellent

Please rate the presenter(s) overall skill. For example i.e., style, pace delivery,

instruction skills, etc.

1 2 3 4 5

Please explain:

Please rate the training materials/handouts in this presentation?

1 2 3 4 5

Please rate the facilities and location where this training was held (space and layout of

room, equipment, adequacy of parking, personal comfort or room?

1 2 3 4 5

Please Explain:

Overall, how would you rate this training?

1 2 3 4 5

Would you recommend this training to a colleague?

Yes No

Please explain:

Did this program meet your expectations

Yes No

Please take a few moments to answer the following questions in detail:

Please take a few moments to answer the following questions in detail:

What did you enjoy most about Safe Zone 101 Training?

What would you change about Safe Zone 101 Training?

I leave this workshop feeling……

What additional feedback would you like to give to the presenter(s)?

Thank you for your time

Safe Zone 101 Training Evaluation Group: ________________________________________ Date: _____________ Please use the following scale to evaluate today's training:

1 = Poor 2 = Fair 3 = Adequate 4 = Good 5 = Excellent

Please rate the presenter(s) overall skill. For example i.e., style, pace delivery,

instruction skills, etc.

1 2 3 4 5

Please explain:

Please rate the usefulness of training materials/powerpoint?

1 2 3 4 5

Please rate your comfort level around using inclusive language?

1 2 3 4 5

Please Explain:

Please rate your comfort level around being a Safe Zone Ally?

1 2 3 4 5

Overall, how would you rate this training?

1 2 3 4 5

Please explain:

Will you continue to update your ally status and complete our series?

Yes No

Please Explain:

Please take a few moments to answer the following questions in detail:

What do you feel you learned most from Safe Zone 101 Training?

What did you enjoy most about Safe Zone 101 Training?

What would you change about Safe Zone 101 Training?

I leave this workshop feeling……

What additional feedback would you like to give to the presenter(s)?

Thank you for your time

Safe Zone 101 Pre-Test

Group: ________________________________ Date: _____________

Primary Affiliation: ☐ Undergraduate Student ☐ Graduate Student ☐ UC Faculty ☐ UC Staff

Please use the following scale to evaluate your current knowledge base about LGBTQ

communities:

1 = Poor 2 = Fair 3= Adequate 4 = Good 5 = Excellent

1. Please rate your comfort in using appropriate language when discussing LGBTQ topics.

1 2 3 4 5

Please explain:

2. Please rate your ability to distinguish between sexual orientation and gender identity.

1 2 3 4 5

Please explain:

3. Please rate your ability to have a dialogue around issues in the LGBTQ community.

1 2 3 4 5

Please explain:

4. Please rate your awareness of issues pertaining to LGBTQ-identified students at UC.

1 2 3 4 5

Please explain:

5. Please rate your confidence in being an ally for the LGBTQ community.

1 2 3 4 5

Please explain:

1. Please rate your knowledge of resources on campus and in the community for LGBTQ students.

1 2 3 4 5

Please explain:

2. How did you hear about Safe Zone Training?

☐ Campus Advertising (emails, flyers, etc.) ☐ Class Announcement ☐ Referred by a Friend

☐ Other: ________________

3. Why are you participating in Safe Zone Training?

☐ Personal Interest ☐ I have a friend/classmate/family member who identifies as a part of the LGBTQ

community. ☐ Required for Class ☐ Other: _______________________

4. What do you expect to learn in this Safe Zone training?

5. What questions do you have about the LGBTQ community?

Thank you for your willingness to engage!

Safe Zone 101 Post-Test

Group: ________________________________ Date: _____________

Primary Affiliation: ☐ Undergraduate Student ☐ Graduate Student ☐ UC Faculty ☐ UC Staff

Please use the following scale to evaluate your current knowledge base about LGBTQ

communities:

1 = Poor 2 = Fair 3= Adequate 4 = Good 5 = Excellent

1. Please rate your comfort in using appropriate language when discussing LGBTQ topics.

1 2 3 4 5

Please explain:

2. Please rate your ability to distinguish between sexual orientation and gender identity.

1 2 3 4 5

Please explain:

3. Please rate your ability to have a dialogue around issues in the LGBTQ community.

1 2 3 4 5

Please explain:

4. Please rate your awareness of issues pertaining to LGBTQ-identified students at UC.

1 2 3 4 5

Please explain:

5. Please rate your confidence in being an ally for the LGBTQ community.

1 2 3 4 5

Please explain:

1. Please rate your knowledge of resources on campus and in the community for LGBTQ students.

1 2 3 4

Please explain:

2. Please rate the presenter(s) overall skill. For example i.e., style, pace delivery, instruction skills, etc.

1 2 3 4 5

Please explain:

3. Please rate the usefulness of training materials/powerpoint.

1 2 3 4 5

4. Will you continue to update your ally status and complete our Ally Training Curriculum Series?

Yes No

5. Overall, how would you rate the training?

1 2 3 4 5

Please explain:

6. What do you feel you learned most from Safe Zone 101 Training?

7. I leave this workshop feeling….

8. What additional feedback would you like to give the presenter(s)?

Thank you for your willingness to engage!

Think-Pair-Share

Question Construction

• Purpose of Survey/Data

– Programmatic changes, policy change, validate

current programming

• Learning Objectives

– Questions should align with LOs

• Keep it simple

• Language of target population

• Identify appropriate scaling as needed

Collecting and Analyzing

Data

Data Collection

• Collecting data means putting your design

for a measurement system into operation.

• Collecting data involves gathering

information through observation,

interviews, testing, surveys, and/or other

means; recording it in appropriate ways;

and organizing it so that it’s easier to

work with.

Research Variables

• An independent variable (the

intervention) is a condition implemented

by the researcher or community to see if it

will create change and improvement.

– This could be a program, method, system, or

other action.

Research Variables

• A dependent variable is what may change

as a result of the independent variable or

intervention.

• A dependent variable could be a behavior,

outcome, or other condition.

– Independent variable on-campus versus off-

campus housing. Dependent variable would

be sense of belongingness.

Data Types

• Quantitative data refer to the information

that is collected as, or can be translated into

numbers, which can then be displayed and

analyzed mathematically.

Data Types

• Qualitative data are collected as descriptions,

anecdotes, opinions, quotes, interpretations,

etc., and are generally either not able to be

reduced to numbers, or are considered more

valuable or informative if left as narratives.

Analyzing data

• Analyzing data involves examining the information

you’ve collected in ways that reveal the relationships,

patterns, trends, etc. that can be found within it.

• Data can be quantitative (collected as numbers) or

qualitative (collected as narrative information, records,

journal notes, etc.)

• Quantitative data is usually analyzed by subjecting it to

one or more graphical displays or statistical operations

that demonstrate the significance of relationships

among variables.

• Data are also usually analyzed logically, by looking for

patterns and relationships revealed within them.

• Qualitative data can sometimes be turned into quantitative

data by, for instance, counting (e.g., the number of times a

behavior occurs in various circumstances) or by rating on a

number scale such dimensions as importance, satisfaction,

or quality (e.g., the quality of housing or quality of life in

neighborhoods).

• A combination of quantitative and qualitative data often

yields the best overall picture.

Analyzing data

Collecting and analyzing data

for your evaluation

• This can show whether or not there was actually any

significant change in the dependent variable(s) you hoped to

influence.

• This can show connections between or among various

factors that may have an effect on the results of your

evaluation.

• This can imply or show the reasons that your work was

effective or ineffective.

• This can provide you with credible evidence to show the

university, stakeholders, and the community that your program

is successful, or that you’ve uncovered, and are fixing, the

elements that are barriers to success.

• This can show that you’re serious about evaluation and about

improving your work.

• This can show the field that what you’re doing works well, and

paving the way for others to use similar methods and

approaches as best practices

Collecting and analyzing data for

your evaluation

How do you collect and analyze data?

• Implement the observational system you’ve

planned.

• Organize the data you’ve collected.

• Conduct data graphing, visual inspection, statistical

analysis, or other operations on the data as

appropriate.

• Take note of any significant or interesting results.

• Interpret the results.

Recap

• Be Creative

• Organization is Key

• Keep It Relevant

• Make Your Questions Short and Sweet

• Write Straightforward Questions

• Use Simple, Precise Language

• Be Careful with Your Response Options

• Use Clearly Defined Answer Labels

• Use Open-Ended Questions Sparingly and Intelligently

Increasing Response Rate

• Time Your Survey Smartly

• Get to the Point

• Tell a Story

• Motivate Response (Incentives)

• Limit Survey Frequency

• Optimize and Test

Questions