Common Designs and Quality Issues in Quantitative Research

Research Methods and Statistics

Intended Learning Outcomes

To familiarise yourself with the different types of quantitative research designs commonly used in occupational psychology research

To understand the concepts of validity and reliability and why these are important to consider when designing research studies

What is Research Design

“A design specifies the logical structure of a research project and the plan that will be followed in the execution. It determines whether a study is capable of obtaining an answer to the research question in a manner consistent with the appropriate research methodology and the theoretical and philosophical perspectives underlying the study.”

(Sim & Wright, 2000: 27)

Elements of Research Designs phenomena/variables to be researched

how will these phenomena/variables be measured? (what method/technique?)

who/where will the data be collected from?

when will the data be collected?

what type of data will I have as a result?

what will be the consequences of this for data analysis?

Elements of Research Designs phenomena/variables to be researched

how will these phenomena/variables be measured? (what method/technique?)

who/where will the data be collected from?

when will the data be collected?

what type of data will I have as a result?

what will be the consequences of this for data analysis?

Common Designs

Group differences

Relationships between variables: correlations regression models

Surveys / questionnaires

Time series

Other designs

Group Differences

INTERVENTIONPRE

CONTROLPRE

INTERVENTIONPOST

CONTROLPOST

e.g. to determine the effect of a training intervention on scores

0

10

20

30

40

50

60

PRE POST

Ou

tco

me

Mea

sure

INTERVENTION

CONTROL

Group Differences Designs - Variations

No control group

More than two groups

More than one outcome measure

No time element

More than two time points

Etc.

Relationships between Variables

Bivariate relationships each participant is measured on two or

more variables (either both are categorical or both are ordinal or above)

Regression models based on linear correlations various predictor variables and one

outcome variable

Bivariate Relationships – Categorical Data

PUBLIC SCHOOL

PRIVATE SCHOOL

READING DIFFICULTIES

8 2

NO READING DIFFICULTIES

24 22

e.g. to find out whether the proportion of pupils with reading difficulties varies from public to private schools

Bivariate Relationships – Ordinal, Interval or Ratio Data

e.g. to find out how the amount of TV viewing is correlated with academic performance

0

1

2

3

4

5

6

0 5 10 15 20

Hours of Weekly TV Viewing

En

gli

sh

SA

T S

co

re

Regression

e.g. which are the best predictors of academic performance?

e.g. which are the best predictors for whether a child will get a statement of educational needs?

PREVIOUS SAT SCORE GENDER

FREE MEALS

TV VIEWING

ATTENDANCE RECORD

ACADEMIC PERFORMANCE

Surveys / Questionnaires

May be used: as an outcome measure (evaluation) to describe (the attitudes of) a

particular group – SURVEY

Surveys can be used to check for: differences between groups relationships between variables

Time Series multiple data points (50+) – recorded data

useful for evaluation when trend and/or seasonality are existent

0

5

10

15

20

25

30

35

Se

p-0

2

De

c-0

2

Ma

r-0

3

Jun

-03

Se

p-0

3

De

c-0

3

Ma

r-0

4

Jun

-04

Se

p-0

4

De

c-0

4

Ma

r-0

5

Jun

-05

Se

p-0

5

De

c-0

5

Ma

r-0

6

Jun

-06

Se

p-0

6

De

c-0

6

Ma

r-0

7

Nu

mb

er o

f R

efer

rals

Other Designs

Single case designs

0

10

20

30

40

50

60

70

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Days

Ou

tco

me

Mea

sure

A A

B B

Which One to Choose…???Your choice of study design needs to take into account:

your research question

data available / feasible tests available

other details: trends / seasonality existent?

Making Sure your Study is a Good Quality One

Just two thing to worry about…

High internal and external validity

Validity and reliability of instruments

Internal and External Validity

Interval validity refers to the lack of confounding variables (related to design)(e.g. can we really conclude the children’s reading performance has improved because of our IV – intervention we introduced?)

External validity refers to whether we can generalise our results to our target population (related to sampling)

Threats to Internal Validity

REGRESSION TO THE MEAN

MORTALITY

COMPENSATORY RIVALRY

EXPERIMENTER BIAS

DIFFUSION OF BENEFIT

MATURATION

External Validity

Can we generalise our findings to other people/places/settings/conditions/etc.?

Related to: artificiality

does the experimental situation resemble the real world?

sample selection is our sample different from the

population you want to apply our findings to?

High Quality Instruments Validity: Does your test measure what it claims to? Reliability: Does it measure it consistently?

Not reliable therefore not

valid

Reliable but not valid

Both reliable and valid

Reproduced from Trochim (2002) on http://www.socialresearchmethods.net/kb/reliability.htm

Relationship between Validity and Reliability

RELIABILITY

VALIDITY

random error

systematic error

When Is Quality Compromised?

Ethics Practical issues

THINK ABOUT… How do validity and ethics relate to

one another? Is it ethical to sacrifice validity in a

study to make it more ethical?