Assignment 2- Slides

P S Y C H O L O G Y 3 8 0 0 , L A B 0 0 2

In Today’s Lab…

!  refresher on populations, hypotheses, error

!  Monte Carlo methods and the t-distribution

!  exploring assumption violations using Monte Carlo

!  this week’s assignment (assignment #2)

Populations versus Samples

population: includes all people or items with the characteristic one wishes to understand

sample: a subset of the desired population, which is (ideally) representative of the population and obtained through random sampling

In statistics, we test samples, and use the results from these samples to make inferences about the populations from which they were drawn.

Null Hypothesis

Alternative Hypothesis (Claim Being Tested)

•  null hypothesis suggests that there is no difference between the two populations of interest, or that the effect is in the opposite direction than expected

•  example: study looking at average self-reported hunger of males versus females after watching the Food Network (H0: µfemales = µmales)

•  reflects the hypothesized effect (opposite of null hypothesis)

•  example: males and females report significantly different levels of hunger (HA: µfemales ! µmales)

Hypotheses in the t-Test

Note that hypotheses are stated in terms of population parameters (µ), but are tested using sample statistics (e.g., )

!

x

The results of our analysis will tell us whether or not we reject our null hypothesis using the significance (p-value; probability value).

If p < .05 (including p < .01, p < .001) ! reject H0 (go with the effect specified in HA) ! conclude that significant effect exists

If p > .05 (i.e., ns) ! do not reject H0 (go with the effect specified in H0) ! conclude that no significant effect exists

Hypothesis Testing in the t-Test

Hypothesis Testing in the t-Test

Mistakes happen…

Type I error !  reject the null hypothesis (H0) when it is in fact true !  conclude populations differ when they are essentially the same

Type II error !  do not reject the null hypothesis (H0) when it is actually false !  conclude population do not differ when significant differences exist

Both types of errors are bad, and you don’t want to make either!

Types of Errors

t-distribution

F-distribution

!2-distribution

sampling distributions of various statistics…

Sampling Distributions

In your t-test outputs from last week, you were seeing and reporting t-statistics (i.e., the t-obtained value for a given t-test):

Each individual t-statistic represents a measure of the difference between two means, given the variance and size of the samples:

!

t =xa " xbs2

na+s2

nb

Sampling Distributions: t-Distribution

!  a distribution is derived by taking an infinite number of samples from any given pairs of populations with specified characteristics and calculating t-values for the pairs of samples

!  plotting t-values gives us the t-distribution

t-distribution


Population 1 Population 2

Take a given sample of scores from both populations (e.g., n = 5).

Calculate a t-statistic for that comparison using the standard formula (e.g., t = 3.46).


These might be the samples for pair 2

(t = -2.97)



And these might be the samples for the pair 3 t-statistic calculation

(t = 0.21)

…And it would go on from here to generate a number of t-values that form a t-distribution…



When populations are the same, this should result in t-obtained values near 0 on our null hypothesis (H0) distribution

! smaller values indicate less dramatic differences between our two means ! with small t-values, we would not reject H0 (conclude that our means do not differ significantly)


95%

If two population means are equal, then a t-statistic comparing sample means will fall between these two dotted lines (the non-rejection region) 95% of the time (at " = .05 in a two-tailed test)

! 5% chance of concluding that means differ when populations are, in fact, essentially the same (Type I error)

2.5% 2.5%


adds to 100% (accounting for all possible outcomes)

99%

If two population means are equal, then a t-statistic comparing sample means will fall between these two dotted lines (the non-rejection region) 99% of the time (at " = .01 in a two-tailed test)

! 1% chance of concluding that means differ when populations are, in fact, essentially the same (Type I error)

0.5% 0.5%


Researcher sets the level of alpha based on Type I error tolerance.


When populations are different, this should result in larger t-obtained values that fall at the extremes (positive or negative) of our our null hypothesis (H0) distribution

! larger values indicate more dramatic differences between our two means ! with large t-values, we would reject H0 (conclude that our means differ significantly)


If our population means do indeed differ, then rejecting our null hypothesis after obtaining large t-values is the correct decision (power).

If our population means do not differ, then rejecting our null hypothesis after obtaining large t-values is the incorrect decision (Type I error).

In practical research settings, we do not know whether the population means truly differ or not… As a result, we can never know whether we have made the right conclusion or whether we have committed an error.

! importance of good research design and addressing the assumptions of a test properly

H0 is true H0 is false

do not reject H0 (conclude H0 is true)

Type II error (#)

reject H0 (conclude HA is true)

Type I error (") w

hat w

e co

nclu

de

(kno

wn

afte

r we

carr

y ou

t tes

t)

what is true (unknown to us)

Types of Errors

We cannot commit both types of errors in the same analysis, because they hinge on two different “realities”.

Relationship exists between Type I and Type II error: if we decrease the probability of one, the probability of the other increases

•  with very small alpha values !  low Type I error rate !  high Type II error rate: becomes very difficult to obtain t-values large enough to reject H0 (could erroneously conclude that H0 is true when in reality it is not)

Relating Type I and Type II Error

balancing act between the two errors

The t-Distribution and Monte Carlo

Generating such a distribution on our own would be overwhelming!

•  we can use the Monte Carlo statistical package to investigate these sampling distributions

! does not use SPSS ! involves running computer simulations of statistical tests (e.g., t-tests) using artificial data sets and huge numbers of repetitions

The t-Distribution and Monte Carlo

•  if we can generate distributions using Monte Carlo, we can also use this programs to examine what happens to these distributions when certain assumptions underlying tests are violated

Assumptions Underlying the t-Distribution (p. 58 in textbook)

(1)  independent random sampling

(2) normality

(3) homogeneity of variance

violating these assumptions can affect Type I and Type II error rates

N(0,1)5 vs. N(0,1)5

shape of population (N = normal)

mean (µ)

variance ("2)

sample size (n)

1) define our two populations

2) MONTE obtains random samples of specified sample size (via simulations, up to 1000)

3) MONTE computes and outputs statistics pertaining to simulations

4) we compare obtained t-distribution values to known t-distribution values to assess how our specified parameters affected our results

Monte Carlo: Overview of Steps

Accessing MONTE on SSC Network

Start ! My Computer


double-click to select your “L” drive (under “Network Drives”)


double-click to select the “Course Library” folder

double-click to select the “Psychology” folder


double-click to select the “MONTE.exe” program

specify that you want to “Run” the program


MONTE Example 1

"  What is the sample size for 1? 5

"  What is the sample size for 2? 5 "  What is the mean for population 1? 0

"  What is the mean for population 2? 0 "  What is the standard dev. for population 1? 1 (!1=1) "  What is the standard dev. for population 2? 1 (!1=1) "  What is the shape of population 1 (N, E, or R)? N

"  What is the shape of population 2 (N, E, or R)? N "  What is the value of t required for significance for at the .05 level? 2.306 (from tables) "  What is the value of t required for significance for at the .01 level? 3.355 (from tables) "  Number of samplings? 1000 "  Do you only want the summary output? Y "  Do you want the frequency distribution for the populations? N

N(0,1)5 vs. N(0,1)5 …no assumptions violated

Obtaining the Results

MONTE Example 1

…navigate to your U: drive

MONTE Example 1

results from all simulations will be stored in one file (double-click to open)

click “Open” to run the file


MONTE Example 1


MONTE Example 1

Note: given that the results are derived from simulation of random samples, all output will vary (but general trends will remain)

MONTE Example 1


this is just a repeat of the info that you provided to MONTE (lets you know which results are which in the output document)

MONTE Example 1


Results for the first 3 simulations (of 1000):

! for each: t-obtained value, mean of each population, variance of each population (two populations being compared)

! these values represent sample values

!  if running multiple simulations, can look to see how these vary

MONTE Example 1


At the .05 level: 4.4% significant t-values

At the .01 level: 0.9% significant t-values !

positive t's + negative t'snumber of simulations

=24 + 201000

= .044

!


=2 + 71000

= .009

MONTE Example 1

!  As expected, with no violations of our assumptions… "  just under 5% of t-values are significant when # = .05 "  just under 1% of t-values are significant when # = .01 "  these significant t-values are Type I errors (because, in fact, our two

population means as provided to MONTE were equal)

!  What we might want to know is: “If we violate one of our assumptions, will this Type I error rate go up?”

MONTE Example 2





N(1,1)10 vs. N(1, 9)10 …equal variances assumption violated

MONTE Example 2




=25 + 311000

= .056

!


=7 + 71000

= .014

MONTE Example 2

!  recall: we expect about 5% of the t-values to be significant at ! = .05, and 1% at ! = .01

!  by violating the assumption of homogeneity of variance, we found slightly more significant t-values than expected at ! = .05 (5.6%), and at ! = .01 (1.4%)

!  the significant t-values are still considered to be error here because our two true means are, in fact, equal

What do the results look like when the population means differ?

MONTE Example 3





N(0,1)10 vs. N(2,1)10 …means actually differ in the populations

MONTE Example 3



positive t's +negative t'snumber of simulations

=990 + 01000

= .990

!

positive t's +negative t'snumber of simulations

=941+ 01000

= .941

MONTE Example 3

•  MONTE program tells us that with our particular populations and samples, 99 of 1000 t-obtained values were significant

! that is, 99% of the time, we will catch the significant difference between our two populations at the .05 level ! this value is no longer error… it is “power”

power: the probability that if a significant difference exists in real life between the populations, our statistical test will show that difference

Can we also obtain an understanding of error from this information?

MONTE Example 3 (Power/Beta)

1 – POWER = Type II Error Rate

! also: 1 – Type II Error Rate = POWER ! also: POWER + Type II Error Rate = 1

Type II error (" or beta) do not reject the null hypothesis (H0) when it is actually false (i.e., you conclude no differences between means when in fact a significant difference exists)

•  if MONTE tells us that 99% of the time, we will catch the significant difference between our two populations at the .05 level

! 1% of the time we will conclude that there the populations mean do not differ despite there being an effect of our manipulation (commit Type II error)

Assignment Format

!  not to be written like an APA-style results section as we are not running statistical tests to assess certain hypotheses (but adhere to APA formatting)

!  instead: more like a position paper, where you support your conclusion with evidence

!  because MONTE takes random samples, everyone will have different results (don’t panic)

!  define Type I error and power

!  describe the following trends, providing concrete examples from your output, and explaining why they occur: "  effect of sample size on power "  effects of alpha level on power "  effects of sample size on Type I error "  effects of alpha level on Type I error

!  keep the discussion to two, double-spaced pages

!  summarize your results in an APA-style table (extra page)

!  submit your MONTE output (neat, labeled, headings where needed, can delete frequency info)

What to Include…

What to Include…

Note: This is an example figure. It is not specific to your assignment question and should not be copied.

Simulation Distribution Observed % error at 5% level 1% level

1 N(1,1)12 – N(0,1)12 5.3 .90 2 N(0,1)25 – N(0,1)25 4.0 .80 3 N(1,1)5 – N(0,1)15 4.0 .60

table title (italicized, proper

capitalization)

table number (no italics or

period)

only horizontal lines

Note, if necessary

double-spaced if possible

Tables: APA Formatting

* when in doubt, check out APA sources

only first word capitalized in table

headings

Next Week

!  single-factor analysis of variance (a.k.a. one-way ANOVA)

!  hand in Monte Carlo assignment at the beginning of lab

!  I will return marked t-test assignments

!  for those who want extra practice: check out the weekly practice assignments posted on Sakai (Practice Labs folder)

Simulations to Run in MONTE

Simulations Critical values

" = .05 " = .01

N(1, 2)5; N(0, 2)5 2.306 3.355

N(1, 2)10; N(0, 2)10 2.101 2.878

N(1, 2)20; N(0, 2)20 2.025 2.713

N(1, 2)40; N(0, 2)40 1.990 2.641

N(1, 2)100; N(0, 2)100 1.980 2.617

N(1, 2)5; N(1, 2)5 2.306 3.355

N(1, 2)10; N(1, 2)10 2.101 2.878

N(1, 2)20; N(1, 2)20 2.025 2.713

N(1, 2)40; N(1, 2)40 1.990 2.641

N(1, 2)100; N(1, 2)100 1.980 2.617

power

Type I error

Important: the second value in the parentheses represents the standard deviation ($) rather than the variance ($2) … can be entered into MONTE as is

Education

Assignment 2- Slides