Survey Designs

Basic Elements of Questionnaires and

Survey Research

Questions in the Survey

Variable Variable TypeDependent/

Independent

Data TypeNominal/ Ordinal/

Interval/ Ratio

Age Independent Ratio

Undergraduate(Where?)

Independent Nominal

First degree(What field?)

Independent Nominal

Current degree(What field?)

Independent Nominal

Questions:Demographic & Descriptive

Typically the first few questions in a survey help to describe the respondent in some way that establishes what subgroup of the population he/she represents

Population variables may include things like age, income, years of education completedthese are ratio data—they can be averagedplace of birth, highest degree, ethnicitythese are nominal data—they name categories

Population variables are typically independent variables

5. How much to you agree or disagree with this statement:

“Most days I am enthusiastic about being a student.” Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree

Opinion questions may be independent or dependent, depending on how they are used in the survey

5. How much to you agree or disagree with this statement:

“Most days I am enthusiastic about being a student.” Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree

The scale used here is a “5-point Likert scale”

Likert Scales: Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree

Usually 5 point, but some prefer 7 point or 9 point

Typically a Likert scale provides ordinal data May be reduced to nominal by combining all

“Agree” categories and all “Disagree” categories

Summary Idea

When using surveys, it is important to consider

-variable type (dependent or independent)

-data type (nominal, ordinal, interval, ratio)

These features will affect the analysis plan, which is the procedures that can be used to analyze the data.

What research questions?

1. Age 6. Groups you belong to

(gender, student, ethnicity)

2. Place of first degree 7. Academic activities

3. Field of first degree 8. Opinion about Malaysia Government—English

4. Field of current degree 9. How the use of English has affected you

5. Enthusiastic student 10. Hours in a week spent in activities

Survey Designs

Longitudinal—changes within a population over time

Cohort Studies—a longitudinal study of a subgroup in a population (over time)

Panel Study—a cohort study of the same group of people (over time)

Survey Designs

Cross Sectional—data are collected at one

point in time Can measure current attitudes, beliefs opinions Can measure needs Can be used to evaluate a program May be used to study one group, or compare groups

Study Designs

Time of Data Collection

Study Over Time Study at a Point in Time

Longitudinal Cross-sectional

Population

Trends

Subgroup Changes

Same Group

Changes

Attitudes and

Practices

Community Needs

Program Evaluation

Trend Cohort Panel Single group

or Comparisons between groups

Who Did I Survey?

UKM Students Graduate Students

Students Taking Research Class Students in Prof. Subahan’s Class

Students who have e-mail access Students who have Internet access Students willing to do the survey

Levels of selection

Population (Who do I want to talk about?)

Group of individuals that have one characteristic (independent variable) in common

Target Population (Who can be chosen?)

List of individuals that qualify for study (they share one or more an independent variables of importance) (also known as Sampling Frame)

Sample (Who is actually chosen?)

The set of individuals selected to be surveyed Respondents (Who actually takes part?)

The group of individuals who actually take part

Levels of Selection

Population—UKM Graduate students Target Population—Graduate students in

Prof. Subahan’s class Sample—Members of Prof. Subahan’s class

who have e-mail and Internet access. Respondents—Member’s of Prof. Subahan’s

class in the sample who responded (n=30+)

Inferences From Data

For n=30 Masters Student 83.3% Doctoral Student 20.0% Age = 35.9 years

For n=10 (first 10 respondents) Masters Student 67.2% Doctoral Student 35.0% Age = 34.9 years

Limitations of Information

Where did you do your undergraduate degree?  Malaysia. 83.9% Asian country other than Malaysia. 3.2% Europe. 3.2% North America. 0.0% Africa 0.0% Other 9.7%

Inaccuracies in data

Tell which of the following groups you consider yourself to belong to. If more than one applies to you, check all that apply:  

Male 53.3%Female 60.0%

In this case, all respondents fit one category or the other.

The total number of responses should be 100%.

Forms of Data Collection

Mailed questionnaires Electronic questionnaires One-on-one interviews Telephone interviews Focus group interviews

Focus group—a small group of respondents assembled to respond to survey topics

Open-ended Responses

Describe how the teaching of mathematics and science in English has affected you.

Thematic analysisresearcher reads responses and looks for themes that help to describe the situation, using an external theoretical framework to inform analysis and conclusions

Targeted analysis (Grounded Theory)researcher draws categories from data and looks for statements that relate—building theory that informs further analysis and conclusions

Thematic AnalysisDescribe how the teaching of mathematics and science in English has affected you. I am more confident and dare to talk to people using English. The quality of my teaching seems like it has decreased! I cannot explain some points very clearly and sometimes I cannot give examples spontaneously. However, I like to teach science in English. From this system, I get to know the weaknesses of my English and I have the chance to force myself to improve and practice my English.

Thematic AnalysisEffect on the person (outside of classroom) I am more confident and dare to talk to people using English. I get to know the weaknesses of my English and I have the

chance to force myself to improve and practice my English.

Effect on teaching The quality of my teaching seems like it has decreased! I

cannot explain some points very clearly and sometimes I cannot give examples spontaneously.

Affective (attitude) response However, I like to teach science in English. I am more confident and dare to talk to people using English.

Designing A Good Survey

Clear questions What would you like to do with research?

Single questionsHave you seen people pushing other on trains and not waiting until people have disembarked?

Positive wordingDo you think that people should not get on trains until others disembark?

Designing A Good Survey

Keep language simple and commonHow can a concerns-based adoption model explain responses to teaching in English?

Watch out for overlapping responsesWould you rather have rice for lunch or fish?

Make sure questions include all in sample, or use branching to clarify.Do you consider riding the Rapid KL to be pleasant?

Pilot Testing

Describe how the teaching of mathematics and science in English has affected you.

“None. I am not a teacher.” “I find the lesson more interesting and I think it is one

way to improve on my language proficiency as well.”“At first I found it hard to teach in English. By now I am

more comfortable with it.”

How might I reword the survey item?

Response Rate

137 students in the Sample30 Respondents

21.89% Response Rate (What % actually respond?)

Ways to improve response rate Direct contact Use of incentives (but not coercion) Follow up procedures Avoid sensitive questions

Sensitive Questions

What is your age?

Of 30 overall respondents, 30 responded

Describe your opinion of the decision by the Malaysian government to require that English be used to teach mathematics and science.

Of 30 overall respondents, 24 responded.

Standards of Practice

Ethical practices in research

Human subjects review Anonymity vs. Confidentiality Fairness in analysis Knowledge of data uses Opportunity to refuse Member checking of results (when possible)

Standards of Practice

Avoid bias within questions, especially in face-to-face interviews.

Train interviewers, especially if there are to be follow-up questions that are unscripted.

Be honest regarding the purposes of the questionnaire, the extent to which the results will be shared, and the level of anonymity/confidentiality.

An Abstract: KBSM Study

The focus of the study is to determine teachers’ attitudes towards the implementation of the KBSM geography curriculum, and its relationship with age, academic qualification, teaching experience and level taught. This study was carried out on a sample of 125 trained geography teachers. The same group of teachers completed a questionnaire three times; at the beginning, middle and end of the two-year implementation period.

Discuss the Study Attributes

What is the population?What is the target population?What are the variables? Which variables are dependent and independent?Is the study longitudinal or cross-sectional?What is a Likert scale question that could be used?What is one possible open-ended question?What is a variable that could provide ratio data?What is a variable that could be sensitive?What issues might the research face?

An Abstract: MARA Study

The objective of this study was to identify the differences in personality factors, family environments, locus of control, socioeconomic status and prior achievement between high and low achievers, and between male and female students. The subjects of the study were 356 high and low achieving fifth formers from eight MARA Junior Science Colleges. The personality traits were measured using the Cattell’s Sixteen Personality Trait Questionnaire, and the locus of control was measured using the Rotter’s Internal-External scale.

Discuss the Study Attributes

What is the population?What is the target population?What are the variables? Which variables are dependent and independent?Is the study longitudinal or cross-sectional?What is a Likert scale question that could be used?What is one possible open-ended question?What is a variable that could provide ratio data?What is a variable that could be sensitive?What issues might the research face?

An Abstract: Islamic EducationThe aim of this study is to explore the effectiveness of Islamic Education among the Muslim Students of UTM. In the questionnaire, 52 questions were asked. There were 50 questions with predetermined variables and two open-ended questions. The following predetermined variables were used.

(i) Improvement of knowledge about Islam(ii) Attitude of student towards Islamic Education(iii) Moral values in the student’s daily life(iv) The obedience to Islamic teaching among students

The variables were viewed with regard to sex difference, duration of exposure to Islamic Education, and achievement.

Discuss the Study Attributes

What is the population?What is the target population?What are the variables? Which variables are dependent and independent?Is the study longitudinal or cross-sectional?What is a Likert scale question that could be used?What is one possible open-ended question?What is a variable that could provide ratio data?What is a variable that could be sensitive?What issues might the research face?

Let’s Summarize

What are five terms you should remember?

What are three things that can affect a survey?

What are three ways that surveys can be used in research?

What are three things that can be problems?