Research SystemTheory

Hypotheses

Data

Verification

Theory buildingHypothesis generation

Measurement issuesResearch designSampling issues

Statistical analysis

Interpretation

Presentation of results

Generalization

CulminationModification

Research Strategies

“All research strategies are seriously flawed…”

McGrath, J. E. Martin, J., & Kulka, R. A. (1982). Judgment Calls in Research. Beverly Hills, CA: SAGE Publications Inc.

A Three-Horned DilemmaDifferent methods have different strengths:

• Rigor• Relevance• Generalizability

Every research strategy either avoids two of the horns byan uneasy compromise but gets impaled, to the hilt, onthe third horn; or it grabs the dilemma boldly by onehorn, maximizing on it, but at the same time “sittingdown” (with some pain) on the other two horns.

(McGrath, 1982: 74)

Dilemmatics: McGrath TaxonomyThe study of research choices and tradeoffsThere is no perfect study. Research involves

tradeoffs Research looks at Actors emitting Behaviors

in a ContextWe want

Generalizability from Actors populationPrecise measure and control of Behavior Realistic Contexts for observation of actor

behavior

J

II

I

IV

III

Judgment Task

Sample Survey

Formal Theory

ComputerSimulation

Field Study

Field Experiment

Lab Experiment

ExperimentalSimulation

RELEVANCE

GENERALIZABILITY

RIGOR

Q I: Field StrategiesField study– No deliberate manipulation; everything is

measured– Naturally occurring setting– Example: Survey of QWL in 100 organizations

Field experiment– Deliberate manipulation of one or more variables– Naturally occurring setting– Example: Hawthorne studies; Greenberg’s work

RELEVANCE

Q II: Experimental StrategiesLaboratory experiment– Deliberate manipulation of variables– Contrived setting– Example: Effects of communication channels on

team performance, effects of feedback and goal-setting on individual performance

Experimental simulation– Deliberate manipulation of variables– Contrived realistic setting– Example: Center for Creative Leadership Looking

Glass simulation; Zimbardo prison experiment – http://www.youtube.com/watch?v=JxGEmfNl-xM

RIGOR

Q III: Respondent StrategiesJudgment tasks– Emphasis on task/judgment selection, often

with a limited number of participants– All variables are measured– Example: “policy capturing” studies, creation

of the competing values frameworkSample Survey– Emphasis on sample selection– All variables measured– Example: National Survey of Organizations

GENERALIZABILITY

Q IV: Theoretical StrategiesFormal theory/literature reviews– No actual research participants– Summarize the literature to create new

models for testing (inductive process)Computer Simulations– No actual research participants– All outcomes are computer- generated– Example: garbage can decision-making

processes, monte carlo studies

GENERALIZABILITY

Integrative & Hybrid StrategiesBecause every methodological choice is

flawedor incomplete, you can decrease the effects of the trade-offs by:Using different methods across studiesUsing multiple methods within a single studyPackaging different studies with different

methods together

Alternatively, Three Study TypesExperimentalQuasi-ExperimentalCorrelational or passive observational

study (field)Single subject (case study)

Important Concepts: The Building Blocks of Research MethodsIndependent VariableDependent VariableExtraneous VariableHypothesisExperiment

Causal InferenceConditions needed for causality

Covariation of cause and effectTemporal precedence (cause must come before

effect in time)Control to rule out alternative interpretations

True experiments are best suited to infer causality because the include the greatest degree of control

Apply MAXIMINICON principleMaximize relevant systematic varianceMinimize irrelevant systematic varianceControl extraneous sources of variance

What is a TRUE experiment?There must be manipulation

Manipulation of a cause results in an effectThere must be random assignment to

experimental conditionsThere must be control of extraneous

variable

ExperimentAdvantages

high degree of controlstrong inference of causalitymeasurement of behavior is preciseoften laboratory experiments can be replicated

easilyDisadvantages

low realismlow external validity in generalsome phenomena cannot be analyzed in a laboratorysome variables may have a weaker (or stronger)

impact in the lab than they would in a natural environment

Quasi-ExperimentAdvantages

high realismgreater external validitymoderate degree of controlmoderate to high inference of causality

Disadvantagesinternal validity may be compromisedexternal validity may be compromisedmeasurement may be impreciseit may be difficult to get people to agree to

participateit is often difficult to get access to many field

settings

Correlational Field StudyAdvantages

realisticdata on a large number of variables can be

collectedthe researcher's impact on the study is often

lowerallows exploration of contextual effects

Disadvantagescausality is difficult to assessinternal validity may be compromisedexternal validity may be compromisedorganizations may not agree to participatemeasurement of variables less precise than lablow response rate common

Single Subject/Case StudyAdvantages

good for low base rate occurrencesprovides source of rich and descriptive datagood for generating new ideas

Disadvantageslow internal and external validity

Threats to ValidityValidity = the confidence we can have that

our findings from any study are “true”All research has threats to validity – that is,

things that minimize the degree to which we can embrace a particular finding as “true”

Many sources of validity threats but two common ones: Research participantsResearchers themselves

Threats due to ParticipantsRoles

Good subjectFaithful subjectNegativistic subjectApprehensive subject

Role Multiplicity and ConflictAttributes of ParticipantsComprehension artifact

They misunderstand

Threats due to Researcher Attributes of researchers Expectancies Designer, observer, and interpreter effects Data analyst Tester

Poor measurement decisions

Week 2 AssignmentDescribe a research topic that you are

interested inWrite three hypothesis statements about

relationships you might expectIdentify what the IVs and DVs are in these

relationshipsIf there are mediators or moderators in your

hypothesis statements, identify what these areIndentify the type of research strategy you

would use to study this research questionUse the article you identified last week as

your reference