1 RESEARCH DESIGN. 2 –What--What was studied? –What about--What aspects of the subject were studied? the subject were studied? –What for--What is/was

11

RESEARCH DESIGN

22

– WhatWhat--What was studied?--What was studied?– What aboutWhat about--What aspects of --What aspects of the subject were studied? the subject were studied? – What forWhat for--What is/was the--What is/was the significance of the study?significance of the study?

– What did prior lit./research say?What did prior lit./research say?

– What was doneWhat was done--How was the--How was the study conducted?study conducted?

– What was found?What was found?– So what?So what?– What now?What now?

1. Introduction, Research Problems/ Objectives, & Justification

2. Literature Review

3. Methodology (Research sample, data collection, measurement, data analysis)

4. Results & Discussion 5. Implications6. Conclusions and

Recommendations for Future Research

PROCESS OF DESIGNING AND CONDUCTING A PROCESS OF DESIGNING AND CONDUCTING A RESEARCH PROJECT:RESEARCH PROJECT:

33

RESEARCH DESIGNRESEARCH DESIGN

RESEARCH DESIGNRESEARCH DESIGN refers to the plan, structure, and strategy of research-- refers to the plan, structure, and strategy of research--the the blueprintblueprint that will that will guide the research processguide the research process. .

Developing ResearchHypotheses

Intriguing Observation,Intellectual Curiosity

Defining ResearchProblem & Objectives

Testing Hypo.:Data Analysis &Interpretation

Sampling Design

Refinement of theory(Inductive Reasoning)

Data Coding,And

Editing

Developing OperationalDefinitions for

Research Variables

Building the Theoretical Framework and the

Research Model

Data Collection

More Careful Studyingof the Phenomenon

THE PROCESS OF

EMPIRICAL RESEARCH

44


CONCLUSION VALIDITYCONCLUSION VALIDITY refers to the extent of researcher’s refers to the extent of researcher’s ability to ability to draw accurate conclusionsdraw accurate conclusions from the research. That from the research. That is, the degree of a study’s: is, the degree of a study’s:

a)a) Internal ValidityInternal Validity——correctness of conclusionscorrectness of conclusions regarding the regarding the relationships among variablerelationships among variables examined s examined

Whether the research findings accurately reflect how the research variables are Whether the research findings accurately reflect how the research variables are really connected to each other.really connected to each other.

b)b) External ValidityExternal Validity – –Generalizability of the Generalizability of the findings to the findings to the intended/appropriate population/settinintended/appropriate population/settingg

Whether Whether appropriate subjectsappropriate subjects were selected for conducting the study were selected for conducting the study

RESEARCH DESIGNRESEARCH DESIGN:: The The blueprint/roadmapblueprint/roadmap that will guide the that will guide the research. research. The test for the The test for the quality of a study’s research design quality of a study’s research design is theis the study’s study’s conclusion validityconclusion validity..

55


• Variance of the INDEPENDENT & DEPENDENT Variance of the INDEPENDENT & DEPENDENT variables (variables (Systematic VarianceSystematic Variance))

• Variability of potential NUISANCE/EXTRANEOUS/ Variability of potential NUISANCE/EXTRANEOUS/ CONFOUNDING variables (CONFOUNDING variables (Confounding VarianceConfounding Variance))

• Variance attributable to ERROR IN MEASUREMENT Variance attributable to ERROR IN MEASUREMENT ((Error VarianceError Variance).).

How?How?

How do you achieveHow do you achieve internal and external validity internal and external validity (i.e., (i.e., conclusion validity)?conclusion validity)?

By effectively By effectively controlling 3 types of controlling 3 types of variancesvariances::

66

Effective Research Design Effective Research Design

– MAXMAXimize Systematic Varianceimize Systematic Variance– MINMINimize Error Varianceimize Error Variance– CONCONtrol Variance of Nuisance/Extraneous/ trol Variance of Nuisance/Extraneous/

Exogenous/Confounding variablesExogenous/Confounding variables

Guiding principleGuiding principle for effective control of for effective control of variances (and, thus, effective research variances (and, thus, effective research design) is: design) is:

The The MAXMINCONMAXMINCON Principle Principle

77

Effective Research Design Effective Research Design

IN EXPERIMENTS?IN EXPERIMENTS? (where the researcher actually (where the researcher actually manipulates the independentmanipulates the independent

variable and measures its impact on the dependent variable):variable and measures its impact on the dependent variable):– Proper manipulation of experimental conditionsProper manipulation of experimental conditions

to ensure high variability in indep. var.to ensure high variability in indep. var.

IN NON-EXPERIMENTAL STUDIES?IN NON-EXPERIMENTAL STUDIES? (where independent and dependent variables are measured (where independent and dependent variables are measured

simultaneously and the relationship between them are simultaneously and the relationship between them are examined):examined):– Appropriate subject selectionAppropriate subject selection (selecting subjects (selecting subjects

that are sufficiently different with respect to the that are sufficiently different with respect to the study’s main var.)--avoid study’s main var.)--avoid Range RestrictionRange Restriction

MAXMAXimizing Systematic Variance: imizing Systematic Variance:

Widening the range of valuesWidening the range of values of research variables. of research variables.

88

Effective Research DesignEffective Research Design

Sources of error variance:Sources of error variance:– Poorly designed measurement instrumentsPoorly designed measurement instruments

((instrumentation errorinstrumentation error))

– Error emanating from study subjects (e.g., Error emanating from study subjects (e.g., response errorresponse error))

– Contextual factorsContextual factors that reduce a sound/accurate that reduce a sound/accurate measurement instrument’s capacity to measure measurement instrument’s capacity to measure accurately.accurately.

How to Minimize Error Variance?How to Minimize Error Variance?– Increase Increase validityvalidity and and reliabilityreliability of of

measurement instruments.measurement instruments.– Measure variables under as Measure variables under as ideal ideal

conditionsconditions as possible. as possible.

MINMINimizing Error Variance imizing Error Variance (measurement error):(measurement error): MinimizingMinimizing the part of variability in scores that is the part of variability in scores that is caused by caused by error in measurementerror in measurement..

1. Historical data on pollution and longevity

2. Relationship between likelihood of hearing problems and high blood pressure

3. Recent stat. show in-vitro kids are 5 times more likely to develop eye tumors (Culprit: in-vitro fathers’ older age)

4. Significantly more armed store robberies during the cold winter days. 99


May or may not be of primary interest to the researcher,May or may not be of primary interest to the researcher, But, can produce But, can produce undesirable variationundesirable variation in the study's in the study's

dependent variable, and cause dependent variable, and cause misleadingmisleading or or weirdweird results results Thus, if not controlled, Thus, if not controlled, can contaminate/distort the true can contaminate/distort the true

relationship(s)relationship(s) between the independent and dependent between the independent and dependent variable(s) of interestvariable(s) of interest

• i.e., confounding var. can result in a i.e., confounding var. can result in a spuriousspurious-- as opposed to -- as opposed to substantivesubstantive--correlation --correlation between IV and DVbetween IV and DV. . Example?Example?

Hearing Blood Problem Pressure

CONCONtrolling Variance of Confounding/Nuisance Variables:trolling Variance of Confounding/Nuisance Variables:

FIRST, what are FIRST, what are Nuisance/ConfoundingNuisance/Confounding Variables? Variables?

Age

1010


– Conducting the experiment in a Conducting the experiment in a controlled environment controlled environment (e.g., (e.g., laboratory), where we can laboratory), where we can hold valueshold values of potential confounding of potential confounding variables variables constantconstant..

– Subject selectionSubject selection (e.g., matching subjects in experiments) (e.g., matching subjects in experiments)– Random assignmentRandom assignment of subjects (variations of confounding variables of subjects (variations of confounding variables

are evenly distributed between the experimental and control groups)are evenly distributed between the experimental and control groups)

In Survey Research:In Survey Research:– Sample selectionSample selection (e.g., including only subjects with appropriate (e.g., including only subjects with appropriate

characteristics—using characteristics—using malemale college graduatescollege graduates as subjects will control as subjects will control for potential confounding effects of gender and education)for potential confounding effects of gender and education)

– Statistical ControlStatistical Control---anticipating, measuring, and -anticipating, measuring, and statistically statistically controlling for confounding variables’ effectscontrolling for confounding variables’ effects (i.e., hold them (i.e., hold them statistically constantstatistically constant, or statistically removing their effects)., or statistically removing their effects).

HOW TO CONTROL FOR CONFOUNDING/HOW TO CONTROL FOR CONFOUNDING/ NUISANCE VARIABLES? NUISANCE VARIABLES? In Experimental Settings In Experimental Settings (e.g., Fertilizer Amount Rate of Plant Growth)(e.g., Fertilizer Amount Rate of Plant Growth) ::

Some Potential Confounding Variables?Some Potential Confounding Variables?

1111


Adequate (full range of) variability in values of Adequate (full range of) variability in values of research variables,research variables,

Precise and accurate measurement,Precise and accurate measurement, Identifying and controlling the effects of Identifying and controlling the effects of

confounding variables, andconfounding variables, and Appropriate subject selectionAppropriate subject selection

RECAP:RECAP:Effective research design is a function of ?Effective research design is a function of ?

1212

BASIC DESIGNSBASIC DESIGNS

Experimental Designs:Experimental Designs:– True Experimental StudiesTrue Experimental Studies– Pre-experimental StudiesPre-experimental Studies– Quasi-Experimental StudiesQuasi-Experimental Studies

Non-Experimental Designs:Non-Experimental Designs:– Expost Facto/Correlational StudiesExpost Facto/Correlational Studies

SPECIFIC TYPES OF RESEARCH DESIGN

BASIC RESEARCH DESIGNS:

1313

EXPERIMENTAL DESIGNSEXPERIMENTAL DESIGNS

RESULT:RESULT: Significant Improvement from O1 to O2 Significant Improvement from O1 to O2 (i.e., sig. O2 - O1 difference)(i.e., sig. O2 - O1 difference)

QUESTION:QUESTION: Did X (the drug) cause the Did X (the drug) cause the improvement?improvement?

One of the simplest experimental designs is the ONE GROUP PRETEST-POSTTEST DESIGN--EXAMPLE?

One way to examine Efficacy of a Drug:

O1 X O2 Measure DRUG Measure

Patients’ Condition Experimental Patients’ Condition

(Pretest) Condition/ (Posttest) intervention

1414


– Have Have only shownonly shown “X” is a SUFFICIENT“X” is a SUFFICIENT condition condition for the change for the change “Y”“Y” (i.e., presence of X is (i.e., presence of X is associated with a change in Y).associated with a change in Y).

But, But, is “X” also a NECESSARYis “X” also a NECESSARY condition for condition for Y?Y?

– How do you verify the latter?How do you verify the latter? By showing that the change would not have By showing that the change would not have

happened in the absence of X—using ahappened in the absence of X—using a CONTROL GROUP.CONTROL GROUP.

David Hume would have been tempted to say “YES.” He was a positivist and wanted to infer causality basedon high correlations between events.

But such an inference could be seriously flawed.

Why?

David Hume, 18th Century Scottish

Philosopher

1515


CONTROL GROUPCONTROL GROUP simulates absence of X simulates absence of X– Origin of using Control Groups (A tale from Origin of using Control Groups (A tale from ancient Egypt)ancient Egypt)

Pretest Post-Test Pretest Post-Test Control GroupControl Group Design-- Design--Suppose random Suppose random assignment (assignment (RR) was used to control confounding variables:) was used to control confounding variables:

R R Exp. Group O1E X O2EExp. Group O1E X O2ERR Ctrl Group O1C O2C Ctrl Group O1C O2C

RESULT:RESULT: O2E > O1E & O2C Not> O1C O2E > O1E & O2C Not> O1CQUESTION:QUESTION: Did X cause the improvement in Exp. Did X cause the improvement in Exp.

Group?Group?

1616


– Need Need proper form of controlproper form of control—e.g., —e.g., PlaceboPlacebo..

RR Exp. Group O1E X O2E Exp. Group O1E X O2E

RR Ctrl Group O1C Placebo O2C Ctrl Group O1C Placebo O2C QUESTION:QUESTION: Can we now conclude X caused the improvement Can we now conclude X caused the improvement

in Exp. Group?in Exp. Group?

NOT NECESSARILY! Why not?

• Power of suggestibility (The Hawthorne Effect)CONCLUSION?

• Maybe, but be aware of the Experimenter Effect (it tends to prejudice the results especially in medical research).• SOLUTION: Double Blind Experiments (neither the subjects nor the experimenter knows who is getting the placebo/drug).

1717


Experimental studies Experimental studies need to need to controlcontrol for for potentialpotentialconfounding factorsconfounding factors that may threaten internal validity that may threaten internal validityof the experiment:of the experiment:

– Hawthorne Effect Hawthorne Effect is only one potential confounding factor is only one potential confounding factor in experimental studies. in experimental studies.

Other such factors areOther such factors are::– History?History?

Biasing events that occur between pretest and post-testBiasing events that occur between pretest and post-test

– Maturation?Maturation? Physical/biological/psychological changes in the subjects Physical/biological/psychological changes in the subjects

– Testing?Testing? Exposure to pretest influences scores on post-test Exposure to pretest influences scores on post-test

– Instrumentation?Instrumentation? Flaws in measurement instrument/procedureFlaws in measurement instrument/procedure

1818


Experimental studiesExperimental studies need to need to controlcontrol for for potentialpotentialconfounding factorsconfounding factors that may threaten internal validity that may threaten internal validityof the experiment (Continued):of the experiment (Continued):

– Selection?Selection? Subjects in experimental & control groups different from the startSubjects in experimental & control groups different from the start

– Statistical Regression (regression toward the mean)?Statistical Regression (regression toward the mean)? Subjects selected based on extreme pretest valuesSubjects selected based on extreme pretest values Discovered by Discovered by Francis GaltonFrancis Galton in 1877 in 1877

– Experimental Mortality?Experimental Mortality? Differential drop-out of subjects from experimental and control groups Differential drop-out of subjects from experimental and control groups

during the studyduring the study– EtcEtc..

Experimental designs mostly used in Experimental designs mostly used in natural and physical natural and physical sciences sciences..

Generally, higher internal validity, lower externalGenerally, higher internal validity, lower external validity validity

1919

2020

2121

CORRELATIONAL DESIGNSCORRELATIONAL DESIGNS

The design of choice in The design of choice in social sciencessocial sciences since the since the phenomenonphenomenon under study is usually under study is usually not reproducible in a laboratory settingnot reproducible in a laboratory setting

Researcher has Researcher has little or no controllittle or no control over study’s indep., dep. over study’s indep., dep. and the and the numerous potential numerous potential confounding variablesconfounding variables,,

Often the researcher Often the researcher concomitantly measuresconcomitantly measures all the study all the study variables (e.g., independent, dependant, etc.),variables (e.g., independent, dependant, etc.),

Then examines the following types of relationships:Then examines the following types of relationships:– correlations among variablescorrelations among variables or or– differences among groupsdifferences among groups,,

Inability to controlInability to control for effects of for effects of confounding variables confounding variables makes makes causal inferencescausal inferences regarding relationships among variables regarding relationships among variables more difficultmore difficult and, thus: and, thus:

Generally, higher external validity, lower internal validityGenerally, higher external validity, lower internal validity

NON-EXPERIMENTAL/CORRELATIONAL DESIGNSNON-EXPERIMENTAL/CORRELATIONAL DESIGNS

2222

CORRELATIONAL DESIGNSCORRELATIONAL DESIGNS

NOT NECESSARILY! EXAMPLES:NOT NECESSARILY! EXAMPLES:– Water Fluoridation and AIDSWater Fluoridation and AIDS

((San Francisco ChronicleSan Francisco Chronicle, Sep. 6, 1984), Sep. 6, 1984)– Armed store robberies and cold weatherArmed store robberies and cold weather– Longevity and PollutionLongevity and Pollution– In-vitro birth and likelihood of developing eye tumorsIn-vitro birth and likelihood of developing eye tumors– Hearing problem and blood pressure Hearing problem and blood pressure

What can a significant correlation mean then?What can a significant correlation mean then?

Non-experimental designs rely on correlational evidence.

QUESTION: Does a significant correlation between two variables in a non-experimental study necessarily represent a causal relationship between those variables?

2323

CORRELATIONAL STUDIESCORRELATIONAL STUDIES

a.a. Both variables are effects of a common cause (or Both variables are effects of a common cause (or bothbothcorrelated with a third variable), i.e., spurious correlationcorrelated with a third variable), i.e., spurious correlation(e.g., air pollution and life expectancy, hearing problem & (e.g., air pollution and life expectancy, hearing problem & blood pressure, country’s annual ice cream sales and blood pressure, country’s annual ice cream sales and frequency of hospital admissions for heat stroke)frequency of hospital admissions for heat stroke)

b.b. Both var. Both var. alternative indicatorsalternative indicators of same concept of same concept(e.g., Church attend. & Freq. of Praying--religiosity).(e.g., Church attend. & Freq. of Praying--religiosity).

c.c. Both parts of Both parts of a common "system" or "complex;" tend to a common "system" or "complex;" tend to come as a packagecome as a package(e.g., martini drinking and opera attendance--life style)(e.g., martini drinking and opera attendance--life style)

d.d. Fortuitous--Fortuitous--Coincidental correlationCoincidental correlation, no logical relationship, no logical relationship(e.g., Outcome of super bowl games and movement of stock (e.g., Outcome of super bowl games and movement of stock market)market)

AT LEAST FOUR OTHER POSSIBLE INTERPRETATIONS/REASONSFOR CORRELATIONS BETWEEN TWO VARIABLES:

2424

CORRELATIONAL STUDIESCORRELATIONAL STUDIES

Covariation RuleCovariation Rule (X and Y must be (X and Y must be correlated)--Necessary but not sufficient condition.correlated)--Necessary but not sufficient condition.

Temporal Precedence RuleTemporal Precedence Rule (If X is the cause, Y (If X is the cause, Y should not occur until after X).should not occur until after X).

Internal Validity RuleInternal Validity Rule (Alternative plausible (Alternative plausible explanations of Y and X-Y relationships should be explanations of Y and X-Y relationships should be ruled out (i.e., eliminate other possible causes).ruled out (i.e., eliminate other possible causes).– In practice, this means exercising caution by In practice, this means exercising caution by

identifying potential identifying potential confounding variables and confounding variables and controlling for their effectscontrolling for their effects).).

WHEN IS IT SAFER TO INFER CAUSALLINKAGES FROM STRONG CORRELATIONS?

John Stuart Mill’s Rules for Inferring Causal Links:John Stuart Mill

1806-1873

2525

Questions or Comments

?

Documents

1 RESEARCH DESIGN. 2 –What--What was studied? –What about--What aspects of the subject were studied? the subject were studied? –What for--What is/was