EXPERIMENTAL RESEARCH

A very common way of research design A good design to establish cause and effect

relationships among variables

It is the type of research that directly attempts to influence a particular

variable tests hypotheses about cause-and-effect

relationships

MAJOR CHARACTERISTICS OF EXPERIMENTAL RESEARCH

The researcher looks at the effect(s) of at least one independent variable (experimental or treatment variable) on one or more dependent variables (criterion or outcome variable)

The researcher manipulates the independent variable, which is not the case in other research designs

Aim of the researcher is to see whether the treatment/manipulation has made a difference.

Some independent variables: Methods of instruction Types of assignment Learning materials Types of questions asked by teachers…

Some dependent variables: Achievement Interest Motivation Attitudes…

COMPARISON OF GROUPS

Experimental research involves two groups of subjects Experimental group Control / comparison group

Experimental group receives the treatment; control group receives no treatment / comparison group receives a different treatment

Comparison group rather than control group in educational research

MANIPULATION OF THE INDEPENDENT VARIABLE

The researcher actively manipulates the independent variable: s/he decides the nature of the treatment, to whom and to what extent it will be applied, and when, where and how it will be done

Variables that can be manipulated: teaching method, learning activities, materials, etc

Variables that cannot be manipulated: gender, ethnicity, age, etc

WAYS OF ESTABLISHING INDEPENDENT VARIABLES

A) one form of the variable versus another E.g. comparing the inquiry method vs the lecture

method in teaching listening

B) presence versus absence of a particular form E.g. comparing the use of videos vs no videos in

teaching speaking

C) varying degrees of the same form Comparing the effects of different amount of

homework on students’ motivation

HYPOTHESES

A hypothesis is a prediction of the possible outcomes of a study.

E.g.

RQ: Will students who are taught English by a teacher of the same gender like the subject more than students taught by a teacher of a different gender:

Hypothesis: Students taught English by a teacher of the same gender will like the subject more than students taught English by a teacher of a different gender.

NULL HYPOTHESES

A hypothesis is a prediction of the possible outcomes of a study.

It states our expectations in a positive sense. There will be a difference between the groups.

A null hypothesis states our expectations in a negative sense. There will be no difference between the groups

DIRECTIONAL VS NON-DIRECTIONAL HYPOTHESES

A directional hypothesis shows a specific direction (higher, lower, more…) that a researcher expects to emerge in a relationship.

Second-graders like school less than first-graders but more than third-graders.

A non-directional hypothesis does not make a specific prediction about the direction of the outcome of the study.

First-, second-, and third-graders will feel differently toward school.

EXAMPLES

Non Directional Null Hypothesis There is no difference between two groups on variable

x. There is no difference among three or more groups on

variable x. There is no relationship between variable x and

variable y.

Directional Null Hypothesis Group A will not have a higher mean score than Group

B. Group A will not have a higher mean score than Group

B and Group C. There is no positive relationship between variable x

and variable y.

Now, turn the following research question into hypothesis:

RQ: How do teachers feel about special classes for the educationally handicapped?

Hypothesis: Teachers in XYZ School District believe that students attending special classes for the educationally handicapped will be stigmatized.

OR

Teachers in XYZ School District believe that special classes for the educationally handicapped will help such students improve their academic skills.

ADVANTAGES OF STATING HYPOTHESES

It makes us think more deeply and specifically about the possible outcomes of the study.

It involves a philosophy of science as it enables the researcher to make specific predictions based on prior evidence or theoretical argument.

It helps us see if we are investigating a relationship.

DISADVANTAGES OF STATING HYPOTHESES

It may lead to a bias on the part of the researcher: when the researcher states a hypothesis, s/he may manipulate the data to obtain the desired outcome.

It may sometimes be unnecessary/inappropriate, such as in descriptive surveys, ethnographic studies, etc.

Focusing on the hypotheses may prevent the researcher from noticing other phenomena which might be important to study.

GROUP DESIGNS IN EXPERIMENTAL RESEARCH:WEAK EXPERIMENTAL DESIGNS

The One-Shot Case Study: A single group is exposed to a treatment and a

dependent variable is observed to assess the effect of the treatment

Why weak? Absence of control: the researcher does not know

whether the result is due to the treatment Not possible to compare so the researcher

doesn’t know whether the treatment had any effect at all

X Otreatment observation

The One-Group Pretest-Posttest Design: A single group is observed before and after treatment

Better than one-shot case study (we know whether change occurred), but it is still weak. Why?

O X O

observation(pre-test)

treatment observation(post-test)

The Static-Group Comparison Design Two already existing groups (non-equivalent) are

used

Although it provides better results, it might still have problems. Why?

X Otreatment observation

  O  observation

TRUE EXPERIMENTAL DESIGNS The Randomized Posttest-Only Control

Group Design Involves two groups which are formed by random

assignment: one receives the treatment; the other does not

treatment group 

R X O

 random treatment observation

control/comparison group

 

R C O

random control/comparison

observation

The Randomized Pretest-Posttest Control Group The difference is in using both pre and post test for both

groups

This design helps the researcher see if the random assignment actually succeeded (pretest).

This is particularly important if the number in each group is small If the amount of change over time is to be assessed

treatment group 

R O X O random observation

(pre-test)treatment observation

(post-test)control/

comparison groupR O C O

random observation(pre-test)

control/ comparison

observation(post-test)

The Randomized Solomon Four-Group Design Used to eliminate the possible effect of a pretest

The first group is like pretest-posttest control group; the second is posttest only control group

Best control of threats Weakness: requires a large sample (subjects are

assigned to four groups); requires a lot of energy and effort

Treatment group 

R O X O random observation treatment observation

Control group 

R O C Orandom observation control observation

Treatment group 

R   X O random   treatment observation

Control group 

R   C Orandom   control observation

QUASI-EXPERIMENTAL DESIGN

It is like experimental design but it doesn’t use random assignment.

It uses other techniques to control threats

There are many techniques used in this design but the most commonly used one in social research is nonequivalent groups design

Nonequivalent groups design It is like pretest-posttest randomized experiment but

does not use random assignment. Instead similar groups are used as the treatment and

the control groups E.g. you can pick up two comparable classrooms or

schools. O X Oobservation(pre-test)

treatment observation(post-test)

O C Oobservation(pre-test)

control/comparison

observation(post-test

CONCLUSION

In any type of experimental design, we create two groups that are “equivalent” to each other.

One group gets the treatment; the other does not. In all other respects the groups are treated the same. If we observe differences in the outcome between

these groups, then we believe that the difference is due to the treatment.

However, in all these designs, we are setting up an artificial situation, so we have to consider all the situation.

If the situation is right, then an experiment can be a very strong design to use.

INTERNAL VALIDITY

Possible threats to internal validity A) Selection Bias (subject characteristics)

Might be differences in groups (age, gender, intelligence…)

B) Loss of Subjects (mortality) Losing some of the subjects (e.g. some subjects

may not return the questionnaires)

C) Location Place in which data are collected or an

implementation is conducted may affect the study

D) Instrumentation instrument decay (e.g. essay scoring causes

fatigue and can be scored differently) data collector characteristics (age, gender,

experience of data collector) data collector bias (e.g. if the collector knows the

hypotheses, s/he might unconsciously distract the data)

E) Testing Subjects can remember the test if the same is

used in pre and post test at a short interval

F) History personal experiences during the course of the

study

G) Maturation E.g. young learners will change due to

maturation

H) Attitude of Subjects The way subjects view the study (e.g.

experimental group might perform better because they know they are having a special treatment)

I) Regression e.g. if you include all good students in one group,

they will improve no matter what you do

J) Implementation The researcher can give the experimental group

an advantage

EXERCISES

Exercise 1. A researcher uses his morning and afternoon history classes to test a new instructional strategy. In the morning class the students are given a pretest to assess their knowledge of WWI and then they are given the opportunity to learn about this topic using a computerized program. In the afternoon class the students are also given a pretest; they learn about WWI through discussion. At the end of six weeks, a final exam is given to both classes.

1. a) How is the research designed?

Quasi-experimental (non-equivalent pre-test/post-test)Gr. A (NR) O X1 OGr. B (NR) O X2 O

A researcher uses morning and afternoon history classes to test a new instructional strategy. In the morning class the students are given a pretest to assess their knowledge of WWI and then they are given the opportunity to learn about this topic using a program. In the afternoon class the studecomputerized nts are also given a pretest; they learn about WWI through discussion. At the end of six weeks, a final exam is given to both classes.

1. b) How can you analyze the data? t-test (comparison of the means in pre- and post-

tests for both groups)

How can we present this data?

    N mean s.d. t

Pre-test Experimental  31 7.23 .88 3.45*

Control 38 7.10 .81  

           

Post-test Experimental 31 9.78 .96 3.88*

Control 38 7.88 .89  

* p< .05 

Table 1Students’performance results 

Exercise 2. A social studies teacher is interested in determining the effects of a new textbook series, which has a multicultural thematic approach. The teacher will teach a unit on colonization in America. After the unit, a multicultural attitude scale will be administered to students.

2. a) How is the research designed? one-shot case study

A social studies teacher is interested in determining the effects of a new textbook series, which has a multicultural thematic approach. The teacher will teach a unit on colonization in America. After the unit, a multicultural attitude scale will be administered to students.

2. b) How can you analyze the data? Descriptive statistics

Exercise 3 A school administration wants to field-test a Methodology curriculum in the ELT course that is offered in the summer. Two classes are to be offered during the summer session. Students in both classes will be given a pretest to measure their knowledge of Methodology. One class will study Methodology using traditional textbook materials with a mostly lecture-style of delivery. Another class, an experimental class, will study Methodology using films and contemporary materials along with presentations by guest speakers. The final exam for the course, a parallel form of the pretest, will be given to both groups.

3. a) How is the research designed? Quasi-experimental (pre-test/post-test nonequivalent

comparison groups)

A school administration wants to field-test a Methodology curriculum in the ELT course that is offered in the summer. Two classes are to be offered during the summer session. Students in both classes will be given a pretest to measure their knowledge of Methodology. One class will study Methodology using traditional textbook materials with a mostly lecture-style of delivery. Another class, an experimental class, will study Methodology using films and contemporary materials along with presentations by guest speakers. The final exam for the course, a parallel form of the pretest, will be given to both groups.

3. b) How can you analyze the data? t-test between and within groups

For the following exercises, work in pairs and describe the research design answering following questions:

1. How will the subjects be assigned to groups?

2. Identify relevant variables 3. Describe the treatment design 4. Identify the way the data will be analyzed.

Exercise 4 With the current emphasis on parental involvement in children’s academic endeavors, a researcher is interested in assessing the confidence level of parents who will read orally to children in schools. He also wants to know if there is a difference in the confidence level of parents of different education levels. From a list of volunteers who are interested in reading orally to students, parents will be randomly selected and assigned to one of two conditions—training or no training. Following training of parents in the experimental condition, all parent volunteers will complete a confidence scale. Describe this study using a true experimental posttest-only control group design.

Random assignment for two groups: give a number to each and draw out a number for Group A and then Group B until you finish all the numbers.

Indep variable: training parents Dep variable: parents’ confidence level Group A will get training whereas Group B will

not. Parents in both groups will read orally to children and then they will be given a confidence scale. T-test for the mean scores obtained from this scale for each group will be found out to see the effect of training on confidence level.

Please read a few article which used experimental research method!!