MELJUN CORTES Thesis Questionnaire

Developing a Questionnaire

MELJUN P. CORTES MELJUN P. CORTES

MELJUN CORTESMELJUN CORTES


Types of Questions

Open-ended high validity, low manipulative quality

Closed-ended low validity, high manipulative quality


Open-ended

An open-ended question is one in which you do not provide any standard answers to choose from.

1.How old are you? ______ years.

2.What do you like best about your job?




Closed-ended

A closed-ended question is one in which you provide the response categories, and the respondent just chooses one:

•

What do you like best about your job?(a) The people(b) The diversity of skills you need to do it(c) The pay and/or benefits(d) Other: ______________________________





Dichotomous Questions

Dichotomous Question: a question that has two possible responses only

Could beYes/NoTrue/FalseAgree/Disagree


Interval Level Attempt to measure on an interval level Likert response scale: ask an opinion question

on a 1-to-5, 1-to-7, etc. bipolar scale Bipolar: has a neutral point and scale ends are at

opposite positions of the opinion

Semantic differential: an object is assessed by the respondent on a set of bipolar adjective pairs

Guttman scale: respondent checks each item with which they agree; constructed as cumulative, so if you agree to one, you probably agree to all of the ones above it in the list



.14

Strongly Disagree

Strongly Agree

1 2 3 4 5

RESPONSE OPTIONS:

Neutral

Strongly Disagree

Strongly Agree

1 2 3 4

New Four–Point Likert Scale

Disagree Agree



Questions based on Level of Measurement

Use a nominal question to measure a variable Assign a number next to each response that

has no meaning; simply a placeholder.

Use an ordinal question to measure a variableRank order preferencesMore than 5 – 10 items is difficultDoes not measure intensity


Filter/Contingency Questions To determine if a respondent is ‘qualified’ to

answer questions, might need a filter or contingency question (also known as knowledge)

Limit # of jumps–

If only two levels, use graphic to jump

If you can't fit the response to a filter on a single page, it's probably best to be send them to a page, rather than a question #


How many steps in the response scale?

Statistical reliability of the data increases sharply with the number of scale steps up to about 7 steps

After 7, it increases slowly, leveling off around 11

After 20, it decreases sharply


Should there be a middle category?

Does it make sense to offer it?

Should not be used as the “don’t know or no opinion” option.

•

The middle option is usually placed between the positive and negative responses.

Sometimes it’s last in an interview.


Direct Magnitude Scaling Method of obtaining ratio-scaled data

Idea is to give respondents an anchor point, and then ask them to answer questions relative to that

Example:Suppose you are interested in the severity

of crimes. Begin by assigning a number to one

crime and then have respondents assign numbers to the others based upon a ratio.


Filtering "Don't Know" Standard format

No "don't know" option is presented to the respondent, but is recorded if the respondent volunteers it.

Quasi filter A "don't know" option is included among the possible

responses.

Full filter First the respondent is asked if they have an opinion.

If yes, the question is asked.


Question Placement It's a good idea to put difficult, embarrassing or

threatening questions towards the end

More likely to answer.

If they get mad and quit, at least you've gotten most of your questions asked!

Put related questions together to avoid giving the impression of lack of meticulousness

Watch out for questions that influence the answers to other questions.


Wording of Questions Direction of Statements

Response bias Socially desirable

Always and never Avoid this Better to phrase as ‘most’, ‘infrequently’

Language Reflect educational level and reading ability Need for various languages


Frequency and Quantity

Consider both frequency and quantity

Consider number of times

Consider duration of times


Mutually Exclusive and Exhaustive

Mutually exclusive: not possible to select more than one category/value

Exhaustive: providing all possible categories/values


Forced Choice

Choose between 2 choices

Might not be relevant

Other choices exist (or at least possible)

Lesser of two evils


Recalling Behavior

Can be difficult to remember

Ask questions that can be answered

Choose time frames that are reasonable

Pilot test for time frame issues


Response Bias

Exaggerating the truth

Socially desirable answers

Consider using ‘trap’ questionsPossibly fictional choice


Sensitive Items

More comfortable answering in categories

Minimize missing data

Might loose statistical power


Evaluating Questions

Pre-testing Cognitive interviewing Behavior coding Peer review

Peer review has shown to be the best method but it’s the least used.


Validity and Reliability of Questions Evaluative strategies:

Analysis of data to evaluate the strength of predictable relationships among answers and with other characteristics of respondents.

Comparisons of data from alternatively worded questions asked of comparable samples.

Comparison of answers against records.

Measuring the consistency of answers of the same respondents at two points in time.


Coding the Questionnaire

Create a codebook: reference guide for the data set

Code: assigning a value to a response categoryOften numeric codePre-coding makes it easierContent analysis on open-ended itemsYes/No often coded as present or not (0 or 1)


Missing Responses

Why blank? Missed them Refusal to answer Didn’t feel it applied Didn’t know the answer

To code or not Analyze the difference If know why, might consider


Piloting the Questionnaire

Test it on yourselfPossibly other experts

Test on people similar to sampleDon’t reuse (some exceptions)

Discuss the survey with individualsDuring completion or After


Finding Respondents

Best Methods of Selection

Even with a good survey, poorly chosen sample leads to poor results



Technology

MELJUN CORTES Thesis Questionnaire