Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal

Replacement Cases Framework

Conclusion

Correlational Frameworkoverview

Thresholds for inference and % bias to invalidateThe counterfactual paradigmInternal validity example: kindergarten retention External validity exampleOpen Court curriculumExtensions of the framework

Correlational Framework

Impact of a Confounding variable

Internal validity

Example: Effect of kindergarten retention

External validity

example: effect of Open Court curriculum

What would it take to Change your Inference? Quantifying the Discourse about Causal Inferences in the Social Sciences

Kenneth A. Frank Help from Yun-jia Lo and Mike Seltzer

AERA workshop April 19, 2015 (AERA on-line video – cost is $95)Motivation Statistical inferences are often challenged because of uncontrolled bias. There may be bias due to uncontrolled confounding variables non-random selection into a sample. We will answer the question about what it would take to change an inference by formalizing the sources of bias and quantifying the discourse about causal inferences in terms of those sources. For example, we will transform challenges such as “But the inference of a treatment effect might not be valid because of pre-existing differences between the treatment groups” to questions such as “How much bias must there have been due to uncontrolled pre-existing differences to make the inference invalid?”

ApproachesIn part I we will use Rubin’s causal model to interpret how much bias there must be to invalidate an inference in terms of replacing observed cases with counterfactual cases or cases from an unsampled population. In part II, we will quantify the robustness of causal inferences in terms of correlations associated with unobserved variables or in unsampled populations. Calculations for bivariate and multivariate analysis will be presented in the spreadsheet for calculating indices [KonFound-it!] with some links to SPSS, SAS, and Stata. FormatThe format will be a mixture of presentation, individual exploration, and group work. Participants may include graduate students and professors, although all must be comfortable with basic regression and multiple regression. Participants should bring their own laptop, or be willing to work with another student who has a laptop. Participants may choose to bring to the course an example of an inference from a published study or their own work, as well as data analyses they are currently conducting.

Materials to download: spreadsheet for calculating indices [KonFound-it!] powerpoint with examples and calculations

http://eo2.commpartners.com/users/aera/session.php?id=12854

https://www.msu.edu/~kenfrank/KonFound-it!.xlsx








https://www.msu.edu/~kenfrank/quantifying%20the%20robustness%20of%20causal%20inferences.pptx

https://www.msu.edu/~kenfrank/quantifying%20the%20robustness%20of%20causal%20inferences.pptx


Conclusion





Internal validity


External validity


overviewReplacement Cases Framework (40 minutes to reflection)

Thresholds for inference and % bias to invalidate and inference

The counterfactual paradigm

Application to concerns about non-random assignment to treatments

Application to concerns about non-random sample

Reflection (10 minutes)

Examples of replacement framework

Internal validity example: Effect of kindergarten retention on achievement

(40 minutes to break)

External validity example: effect of Open Court curriculum on achievement

Review and Reflection

Extensions

Extensions of the framework

Exercise and break (20 minutes: Yun-jia Lo, Mike Seltzer support)

Correlational Framework (25 minutes to exercise)

How regression works


Internal validity: Impact necessary to invalidate an inference

Example: Effect of kindergarten retention on achievement

Exercise (25 minutes: Mike Seltzer supports on-line)

External validity (30 minutes)

combining estimates from different populations

example: effect of Open Court curriculum on achievement

Conclusion (10 minutes)


Conclusion





Internal validity


External validity


https://www.msu.edu/~kenfrank/research.htm#causal

Materials for course




Conclusion





Internal validity


External validity


Quick Survey

Can you make a causal inference from an observational study?


Conclusion





Internal validity


External validity


Answer: Quantifying the Discourse

Can you make a causal inference from an observational study?

Of course you can. You just might be wrong. It’s causal inference, not determinism.

But what would it take for the inference to be wrong?


Conclusion





Internal validity


External validity


I: Replacement of Cases Framework

How much bias must there be to invalidate an inference?

Concerns about Internal Validity• What percentage of cases would you have

to replace with counterfactual cases (with zero effect) to invalidate the inference?

Concerns about External Validity• What percentage of cases would you have

to replace with cases from an unsampled population (with zero effect) to invalidate the inference?


Conclusion





Internal validity


External validity


What Would It Take to Change an Inference? Using Rubin’s Causal Model to

Interpret the Robustness of Causal Inferences

Abstract

We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubin’s causal model (RCM) to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse.

Keywords: causal inference; Rubin’s causal model; sensitivity analysis; observational studies Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. 2013. What would it take to Change an Inference?: Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences. Education, Evaluation and Policy Analysis. Vol 35: 437-460.

http://epa.sagepub.com/content/early/recent

https://www.msu.edu/~kenfrank/What%20would%20it%20take%20to%20Change%20an%20Inference%20published.docx







http://epa.sagepub.com/content/early/recent


Conclusion





Internal validity


External validity


0

1

2

3

4

5

6

7

8

9

A B

Study

Es

tim

ate

d E

ffe

ct

above threshold

below threshold

Figure 1 Estimated Treatment Effects in Hypothetical Studies A and

B Relative to a Threshold for Inference

Threshold

{}

% bias necessary

to invalidate

the inference


Conclusion





Internal validity


External validity


Quantifying the Discourse: Formalizing

Bias Necessary to Invalidate an Inference δ =a population effect, =the estimated effect, andδ# =the threshold for making an inference

An inference is invalid if: (1)

An inference is invalid if the estimate is greater than the threshold while the population value is less than the threshold.

Defining bias as -δ, (1) implies an estimate is invalid if and only if:

Expressed as a proportion of the estimate, inference invalid if:

# #ˆ( )ˆ% ( ) 1ˆ ˆ

bias

#ˆ ˆ( ) (2)bias

#


Conclusion





Internal validity


External validity


Algebra for % Bias to Invalidate#

#

#

#

#

#

inference invalid if

ˆ ,

ˆsubtract from all parts

ˆ ˆ ˆ ˆ

ˆ ˆ0

multiply by -1

ˆ ˆ0

ˆsubstitute bias=

ˆ0 bias

ˆbias > > 0

ˆas % of estimate, divide by

ˆbias >

ˆ

#

#

> 0ˆ

%bias >1 > 0ˆ


Conclusion





Internal validity


External validity


0

1

2

3

4

5

6

7

8

9

A B

Study

Es

tim

ate

d E

ffe

ct

above threshold

below threshold



Threshold

δ#

{}

% bias necessary

to invalidate

the inference

# #ˆ( ) 4 1ˆ% ( ) to invalidate= 1 1 33%ˆ ˆ 6 3

bias


Conclusion





Internal validity


External validity


Interpretation of % Bias to Invalidate an Inference

% Bias is intuitive

Relates to how we think about statistical significance

Better than “highly significant” or “barely significant”

But need a framework for interpreting


Conclusion





Internal validity


External validity


Framework for Interpreting % Bias to Invalidate an Inference: Rubin’s Causal

Model and the Counterfactual

1) I have a headache

2) I take an aspirin (treatment)

3) My headache goes away (outcome)

Q) Is it because I took the aspirin?

A) We’ll never know – it is counterfactual – for the individual

This is the Fundamental Problem of Causal Inference


Conclusion





Internal validity


External validity


Definition of Replacement Cases as Counterfactual: Potential Outcomes

t ci i iY Y Definition of treatment effect for individual i:

value on outcome if unit received treatment

value on outcome if unit received control

ti

ci

Y

Y

Fundamental problem of causal inference is that we cannot simultaneously observe

and t ci iY Y


Conclusion





Internal validity


External validity


Fundamental Problem of Inference and Approximating the Counterfactual with

Observed Data (Internal Validity)

345

6?6?6?

But how well does the observed data approximate the counterfactual?

91011


Conclusion





Internal validity


External validity


Symbolic: Fundamental Problem of Inference and Approximating the

Counterfactual with Observed Data (Internal Validity)

6?6?6?

But how well does the observed data approximate the counterfactual?

Yt|X=t Yc|X=t

Yc|X=cYt|X=c


Conclusion





Internal validity


External validity


Approximating the Counterfactual with Observed Data

345

But how well does the observed data approximate the counterfactual?Difference between counterfactual values and observed values for the control implies the treatment effect of 1

8910

111

6

is overestimated as 6 using observed control cases with mean of 4

9


Conclusion





Internal validity


External validity


Using the Counterfactual to Interpret % Bias to Invalidate the Inference

How many cases would you have to replace with zero effect counterfactuals to change the inference?Assume threshold is 4 (δ# =4):1- δ# /

=1-4/6=.33 =(1/3)

666

6.00

The inference would be invalid if you replaced 33% (or 1 case) with counterfactuals for which there was no treatment effect. New estimate=(1-% replaced) +%replaced(no effect)=(1-%replaced) =(1-.33)6=.66(6)=4

000

64

345

1011

9


Conclusion





Internal validity


External validity


Which Cases to Replace

• Think of it as an expectation: if you randomly replaced 1 case, and repeated 1,000 times, on average the new estimate would be 4

• Assumes constant treatment effect• conditioning on covariates and

interactions already in the model


Conclusion





Internal validity


External validity


0

1

2

3

4

5

6

7

8

9

A B

Study

Es

tim

ate

d E

ffe

ct

above threshold

below threshold



Threshold

δ#

{}

% bias necessary

to invalidate

the inference


bias

To invalidate the inference, replace 33% of cases with counterfactual data with zero effect


Conclusion





Internal validity


External validity


Fundamental Problem of Inference and Approximating an Unsampled Population

with Observed Data (External Validity)

910 Yt|Z=p1134 Yc|Z=p5

64

How many cases from p would you have to replace with cases with zero effect from p` to change the inference?Assume threshold is: δ# =4:1- δ# /

=1-4/6=.33 =(1/3)

6 Yt|Z=p´

6 Yc|Z=p´

0

66 6 6


Conclusion





Internal validity


External validity


0

1

2

3

4

5

6

7

8

9

A B

Study

Es

tim

ate

d E

ffe

ct

above threshold

below threshold



Threshold

δ#

{}

% bias necessary

to invalidate

the inference


bias

To invalidate the inference, replace 33% of cases with cases from unsampled population data with zero effect


Conclusion





Internal validity


External validity


Review & Reflection

Review of Framework

Pragmatism thresholds

How much does an estimate exceed the threshold % bias to invalidate the inference

Interpretation: Rubin’s causal model• internal validity: % bias to invalidate

number of cases that must be replaced with counterfactual cases (for which there is no effect)

• external validity: % bias to invalidate number of cases in sample population p that must be replaced with unobserved population p` (for which there is no effect)

Reflect

Which part is most confusing to you?

Is there more than one interpretation?

Discuss with a partner or two


Conclusion





Internal validity


External validity


Example of Internal Validity from Observational Study : The Effect of Kindergarten Retention on Reading and Math Achievement

(Hong and Raudenbush 2005)

1. What is the average effect of kindergarten retention policy? (Example used here)

Should we expect to see a change in children’s average learning outcomes if a school changes its retention policy?

Propensity based questions (not explored here)

2. What is the average impact of a school’s retention policy on children who would be promoted if the policy were adopted?

Use principal stratification.

Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics.

Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205–224

http://www.psych.umass.edu/uploads/people/79/Hong__Raudenbush_2005.pdf




Conclusion





Internal validity


External validity


Data

• Early Childhood Longitudinal Study Kindergarten cohort (ECLSK)– US National Center for Education Statistics (NCES).

• Nationally representative• Kindergarten and 1st grade

– observed Fall 1998, Spring 1998, Spring 1999 • Student

– background and educational experiences– Math and reading achievement (dependent variable)– experience in class

• Parenting information and style• Teacher assessment of student• School conditions• Analytic sample (1,080 schools that do retain some children)

– 471 kindergarten retainees – 10,255 promoted students


Conclusion





Internal validity


External validity


Effect of Retention on Reading Scores(Hong and Raudenbush)


Conclusion





Internal validity


External validity


Possible Confounding Variables(note they controlled for these)

• Gender• Two Parent Household• Poverty• Mother’s level of Education (especially relevant for reading

achievement)• Extensive pretests

– measured in the Spring of 1999 (at the beginning of the second year of school)

– standardized measures of reading ability, math ability, and general knowledge;

– indirect assessments of literature, math and general knowledge that include aspects of a child’s process as well as product;

– teacher’s rating of the child’s skills in language, math, and science


Conclusion





Internal validity


External validity


Calculating the % Bias to Invalidate the Inference:Obtain spreadsheet

From https://www.msu.edu/~kenfrank/research.htm#causalChoose spreadsheet for calculating indices

Access spreadsheet

spreadsheet for calculating indices [KonFound-it!]

https://www.msu.edu/~kenfrank/research.htm





Conclusion





Internal validity


External validity


Kon-Found-it: Basics


Conclusion





Internal validity


External validity


Obtain t critical, estimated effect and standard error

Estimated effect( ) = -9.01 Standard

error=.68n=7168+471=7639;df > 500,

t critical=-1.96

From: Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics. Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205–224


Conclusion





Internal validity


External validity


covariates

Page 215

Df=207+14+2=223


Conclusion





Internal validity


External validity


Kon-Found-it: Basics% Bias to Invalidate

Estimated effect( ) = -9.01

Standard error=.68

n=7168+471=7639 Covariates=223


Conclusion





Internal validity


External validity


Calculating the % Bias to Invalidate the Inference:Inside the Calculations

δ# =the threshold for making an inference =

se x tcritical, df>230 =.68 x -1.96=-1.33

[user can specify alternative threshold]

% Bias necessary to invalidate inference = 1-δ#/ =1-1.33/-9.01=85%

85% of the estimate must be due to bias to invalidate the inference.

}


Conclusion





Internal validity


External validity


Using the Counterfactual to Interpret % Bias to Invalidate the Inference


=1-4/6=.33 =(1/3)

666

6.00


000

64

345

1011

9


Conclusion





Internal validity


External validity


Original cases that were not replacedReplacement counterfactual cases with zero effect

Original distribution

Retained Promoted

Example Replacement of Cases with Counterfactual Data to Invalidate Inference of an Effect of Kindergarten Retention

Counterfactual:No effect


Conclusion





Internal validity


External validity


Interpretation1) Consider test scores of a set of children who were retained that are considerably lower (9 points) than others who were candidates for retention but who were in fact promoted. No doubt some of the difference is due to advantages the comparable others had before being promoted. But now to believe that retention did not have an effect one must believe that 85% of those comparable others would have enjoyed most (7.2) of their advantages whether or not they had been retained.

This is even after controlling for differences on pretests, mother’s education, etc.

2) The replacement cases would come from the counterfactual condition for the observed outcomes. That is, 85% of the observed potential outcomes must be unexchangeable with the unobserved counterfactual potential outcomes such that it is necessary to replace those 85% with the counterfactual potential outcomes to make an inference in this sample. Note that this replacement must occur even after observed cases have been conditioned on background characteristics, school membership, and pretests used to define comparable groups.

3) Could 85% of the children have manifest an effect because of unadjusted differences (even after controlling for prior achievement, motivation and background) rather than retention itself?


Conclusion





Internal validity


External validity


Evaluation of % Bias Necessary to Invalidate Inference

Compare Bias Necessary to Invalidate Inference with Bias Accounted for by Background Characteristics

1% of estimated effect accounted for by background characteristics (including mother’s education), once controlling for pretests

More than 85 times more unmeasured bias necessary to invalidate the inference

Compare with % Bias necessary to invalidate inference in other studies

Use correlation metric• Adjusts for differences in scale


Conclusion





Internal validity


External validity


% Bias Necessary to Invalidate Inference based on Correlationto Compare across Studies

2 2

t 13.25r .152

(n q 1) t (7639 223 1) (13.25)

t taken from HLM: =-9.01/.68=-13.25n is the sample size q is the number of parameters estimated

# critical

2 2critical

t 1.96threshold= r .023

(n q 1) t (7639 223 1) 1.96

Where t is critical value for df>200

% bias to invalidate inference=1-.023/.152=85%

Accounts for changes in regression coefficient and standard errorBecause t(r)=t(β)


Conclusion





Internal validity


External validity


% Bias Necessary to Invalidate Inference based on Correlationto Compare across Studies

2 2

t 13.25r .152

(n q 1) t (7639 238 1) (13.25)

# critical

2 2critical


(n q 1) t (7639 238 1) 1.96

% bias to invalidate inference=1-.023/.155=85%


Conclusion





Internal validity


External validity


Compare with Bias other Observational Studies


Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity


% Bias to Invalidate Inference for observational studieson-line EEPA July 24-Nov 15 2012

Kindergarten retention effect


Conclusion





Internal validity


External validity


Exercise 1 : % Bias necessary to Invalidate an Inference

• Take an example from an observational study in your own data or an article

• Calculate the % bias necessary to invalidate the inference– Interpret the % bias in terms of sample

replacement– What are the possible sources of bias?– Would they all work in the same

direction?• Debate your inference with a partner


Conclusion





Internal validity


External validity

example: effect of Open Court curriculum 44

Application to Randomized Experiment: Effect of

Open Court Curriculum on Reading Achievement

• Open Court “scripted” curriculum versus business as usual

• 917 elementary students in 49 classrooms• Comparisons within grade and school• Outcome Measure: Terra Nova

comprehensive reading score Borman, G. D., Dowling, N. M., and Schneck

, C. (2008). A multi-site cluster randomized field trial of Open Court Reading. Educational Evaluation and Policy Analysis, 30(4), 389-407.

http://www.eranet.co.za/zenex/files/264_Borman,%20Dowling_Schneck,%202008.pdf






Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity


Value of Randomization

Few differences between groups

But done at classroom levelTeachers might talk to each other

School level is expensive (Slavin, 2008)Slavin, R. E. (2008). Perspectives on evidence-based research in education-what works? Issues in synthesizing educational program evaluations. Educational Researcher, 37, 5-14.


Conclusion





Internal validity


External validity

example: effect of Open Court curriculum n=27+22=49


Conclusion





Internal validity


External validity

example: effect of Open Court curriculum Anadjusted difference of 10


Conclusion





Internal validity


External validity


Obtaining # parameters estimated, t critical, estimated effect and standard error

Estimated effect( ) = 7.95

Standard error=1.83

3 parameters estimated,Df=n of classrooms-# of parameters estimated=49-3=46.t critical = t.05, df=46=2.014


Conclusion





Internal validity


External validity


Quantifying the Discourse for Borman et al:What would it take to change the inference?

δ = population effect, =the estimated effect = 7.95, andδ # =the threshold for making an inference = se x tcritical, df=46 =1.83 x 2.014=3.69

% Bias necessary to invalidate inference = 1- δ #/ =1-3.69/7.95=54%

54% of the estimate must be due to bias to invalidate the inference


Conclusion





Internal validity


External validity



=the estimated effect = 7.95standard error =1.83t critical= 2.014

% Bias necessary to invalidate inference = 1-d#/d =1-3.68/7.95=54%


Conclusion





Internal validity


External validity



δ# =the threshold for making an inference = se x tcritical,

df=46 =1.83 x 2.014=3.686[user can override to specify hreshold]

% Bias necessary to invalidate inference = 1-d#/d =1-3.686/7.95=54%


Conclusion





Internal validity


External validity


OCR0

1

2

3

4

5

6

7

8

9

below threshold above threshold

Est

imat

ed E

ffec

t% Exceeding Threshold for Open Court

Estimated Effect

ˆ 7.95

δ# =3.68

54 % above threshold=1-3.68/7.95=.54}54% of the estimate must be due to bias to invalidate the inference


Conclusion





Internal validity


External validity



Observed Data (External Validity)

91011345

66 6 6

64

How many cases would you have to replace with cases with zero effect to change the inference?Assume threshold is: δ# =4:1- δ# /

=1-4/6=.33 =(1/3)

6

6

0


Conclusion





Internal validity


External validity


Interpretation of Amount of Bias Necessary to Invalidate the Inference: Sample

RepresentativenessTo invalidate the inference:

54% of the estimate must be due to sampling bias to invalidate Borman et al.’s inference

You would have to replace 54% of Borman’s cases (about 30 classes) with classes in which Open Court had no effect to invalidate the inference

Are 54% of Borman et al.’s cases irrelevant for non-volunteer schools?We have quantified the discourse about the concern of validity


Conclusion





Internal validity


External validity


Example Replacement of Cases from Non-Volunteer Schools to Invalidate Inference of an Effect of the Open Court Curriculum

Open Court

Original volunteer cases that were not replacedReplacement cases from non-volunteer schools with no treatment effectOriginal distribution for all volunteer cases

Business as Usual Unsampled Population:No effect


Conclusion





Internal validity


External validity


The Fundamental Problem of External Validity

Before a randomized experiment:

People believe they do not “know” what generally works

People choose treatments based on idiosyncratic conditions -- what they believe will work for them (Heckman, Urzua and Vytlacil, 2006)

After a randomized experiment:

People believe they know what generally works

People are more inclined to choose a treatment shown to generally work in a study because they believe “it works”

The population is fundamentally changed by the experimenter (Ben-David; Kuhn)

The fundamental problem of external validity

the more influential a study the more different the pre and post populations, the less the results apply to the post experimental population

All the more so if it is due to the design (Burtless, 1995)


Conclusion





Internal validity


External validity


Restatement of the Fundamental Problem of

External Validity

A researcher cannot simultaneously change the behavior in a population and claim that the pre-study population fully represents the post-study population.


Conclusion





Internal validity


External validity


Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative

Criminology, 26(4), 489-500. page 495


Conclusion





Internal validity


External validity



Criminology, 26(4), 489-500. page 496


Conclusion





Internal validity


External validity


Comparisons across Randomized Experiments (correlation metric)


Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity


Distribution of % Bias to Invalidate Inference for Randomized Studies EEPA: On-line Jul 24-Nov 5 2012


Conclusion





Internal validity


External validity


Review & Reflection

Review of applications

Concern about internal validity: Kindergarten retention (Hong and Raudenbush)

• 85% of cases must be replaced counterfactual data (with no effect) to invalidate the inference of a negative effect of retention on reading achievement

– Comparison with other observational studies

Concern about external validity: Open Court Curriculum

• 54% of cases must be replaced with data from unobserved population to invalidate the inference of a positive effect of Open Court on reading achievement in non-volunteer schools

– Comparison with other randomized experiments

Reflect

Which part is most confusing to you?

Is there more than one interpretation?

Discuss with a partner or two


Conclusion





Internal validity


External validity


Exercise 2 : % Bias necessary to Invalidate an Inference

Take an example of a randomized experiment in your own data or an article

Calculate the % bias necessary to invalidate the inference

Interpret the % bias in terms of sample replacement

What are the possible sources of bias?

Would they all work in the same direction?

Debate your inference with a new partner


Conclusion





Internal validity


External validity


Extensions of the Framework

Ordered thresholds for decision-making

Alternative hypotheses and scenarios

Relationship to confidence intervals

Related techniques


Conclusion





Internal validity


External validity


Ordered Thresholds Relative to Transaction Costs

1. Changing beliefs, without a corresponding change in action.

2. Changing action for an individual (or family)

3. Increasing investments in an existing program.

4. Initial investment in a pilot program where none exists.

5. Dismantling an existing program and replacing it with a new program.

Definition of threshold: the point at which evidence from a study would make one indifferent to policy choices


Conclusion





Internal validity


External validity


http://www.nytimes.com/2015/01/04/opinion/sunday/playing-dumb-on-climate-change.html?hp&action=click&pgtype=Homepage&module=c-column-top-span-region&region=c-column-top-span-region&WT.nav=c-column-top-span-region&_r=0

How District Leaders Really Make Decisions, and About Whatfrom Design-Based Implementation Research: A Strategic Approach to Scaling and Sustaining Educational Innovation: William R. Penuel

University of ColoradoPhilip Bell

University of Washington

Justify an already-made decision

Choose between programs

Mobilize support for adopted program

Challenge an adopted program

Improve district policy implementation

Find concepts to inform improvement efforts

To select standards focus

Improve existing programs

Improve PD

1 2 3 4 5

3.62

3.72

3.76

3.83

3.90

3.90

3.93

3.97

4.17

Use of Research Evidence


Criminology, 26(4), 489-500.


Conclusion





Internal validity


External validity


Alternative Hypotheses: Non Zero Null Hypothesis

Non-zero null hypotheses (for kindergarten retention)

H0:δ> −6.

se x tcritical, df=7639=.68 x (−1.645)= −1.12 (one tailed test).


Conclusion





Internal validity


External validity


Alternative Hypotheses: Non Zero Null Hypothesis

δ# = −6−1.12=−7.12

1− δ #/ =1− (−7.12/−9)=.21.

21% of estimated effect would have to be due to bias to invalidate inference for H0:δ> −6.


Conclusion





Internal validity


External validity


Alternative Scenario: Failure to Reject Null when null is False

Assume the data are comprised of two subsamples, one with an effect at the

threshold (d#) and the other (d ) with 0 effect, and define as the proportion of the sample that has 0 effect.

dxy =(1-)d# + dxy .

*Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors.

https://www.msu.edu/~kenfrank/papers/INDICES%20OF%20ROBUSTNESS%20TO%20CONCERNS%20REGARDING%20THE%20REPRESENTATIVENESS%20OF%20A%20SAMPLE.doc








Conclusion





Internal validity


External validity


Sample Replacement: Estimate Does not Exceed the Threshold

Set dxy=0 and solve for : =1- dxy /d# =1-.5/1.183=.577=58%If of the cases have 0 effect then if you replace

them (with cases at the threshold) then the overall estimate will be at the threshold.


Conclusion





Internal validity


External validity


Dehejia, Rajeev H., and Sadek Wahba. "Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs." Journal of the American statistical Association 94, no. 448 (1999): 1053-1062.

http://isites.harvard.edu/fs/docs/icb.topic1311568.files/dehejia_wahba%201999.pdf


Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity


Alternative Scenario: Alternative Thresholds for Inference

Use δ# = −4

To Invalidate Hong and Raudenbush’s inference, 56% of the estimate would have to be due to bias if the threshold for inference is -4.


Conclusion





Internal validity


External validity


Alternative Scenario: Non-zero Effect in the Replacement (e.g., non-volunteer) Population

Inference invalid if 1-πp<(δp − δ#)/(δ p − δ p´ ).

If δ p´ = −2, and δ#=3.68 and δ p =7.95 (both as in the initial example for Open Court). Inference is invalid if 1-πp<(7.95 – 3.68)/(7.95

− −2 ) =.43

inference invalid if more than 43% of the sample were replaced with cases for which the effect of OCR was −2 .


Conclusion





Internal validity


External validity


}0 1 2 3 4 5 6 7 8 9 1 1 1 1 0 1 2 3

Confidence Interval

Relationship between the Confidence Interval and % Bias Necessary to Invalidate the Inference of an Effect of Open Court on Comprehensive Reading Score

δ #

Lower bound of confidence interval “far from 0” estimate exceeds threshold by large amount

0 1 2 3 4 5 6 7 8 9 1 1 1 1

0 1 2 3} }

δ #

}}


Conclusion





Internal validity


External validity


Related Techniques

Bounding (e.g., Altonji et, Elder & Tabor, 2005; Imbens 2003; Manski)

lower bound: “if unobserved factors are as strong as observed factors, how small could the estimate be?”

• Focus on estimate

% robustness: “how strong would unobserved factors have to be to invalidate inference?”

• Focus on inference, policy & behavior

External validity based on propensity to be in a study (Hedges and O’Muircheartaigh )

They focus on estimate

We focus on comparison with a threshold

Other sensitivity (e.g., Rosenbaum or Robins)

Characteristics of variables needed to change inference

We focus on how sample must change.• Can be applied to observational study or RCT

Other Sources of Bias

Violations of SUTVA• Agent based models?

Measurement error• Just another source of bias (minor concern for examples here)

Differential treatment effects• Use propensity scores to differentiate, then apply indices


Conclusion





Internal validity


External validity


II: Correlation FrameworkConcerns about internal validity

At what levels must an omitted variable be correlated with a predictor and outcome to change an inference about the predictor?

Frank, K. 2000. "Impact of a Confounding Variable on the Inference of a Regression Coefficient." Sociological Methods and Research, 29(2), 147-194

Concerns about external validity

What must be the correlation between predictor and outcome in the unsampled population to invalidate an inference about an effect in a population that includes the unsampled population?


Combined application (with propensity scores)

Frank, K.A., Gary Sykes, Dorothea Anagnostopoulos, Marisa Cannata, Linda Chard, Ann Krause, Raven McCrory. 2008. Extended Influence: National Board Certified Teachers as Help Providers. Education, Evaluation, and Policy Analysis. Vol 30(1): 3-30.

https://www.msu.edu/~kenfrank/papers/impact%20of%20a%20confounding%20variable.pdf









https://www.msu.edu/~kenfrank/papers/Does%20NBPTS%20Certification%20Affect%20the%20Number%20of%20Colleagues%20a%20Teacher%20Helps%20with%20Instructional%20Matters%20acceptance%20version%202.doc








Conclusion





Internal validity


External validity


Making an Inference from an Observational Study

83

•Gather data on people as they naturally choose “treatments”

•Measure covariates, especially pretests•Randomly sample from a larger population – can

do because not assigning to treatments•Examples

•Coleman et al., Catholic schools effect•Hong and Raudenbush: effects of kindergarten

retention


Conclusion





Internal validity


External validity


Linear Adjustment to Invalidate Inference


=1-4/6=.33 =(1/3)

666

6.00


000

64

345

91011

9

+3=6+2=6+1=6

+6=9

Linear model distributeschange over all cases


Conclusion





Internal validity


External validity


Adjusting for Confound to get Correct Effect

666

To replace observed control cases to get estimated effect of 4instead of uncontrolled mean difference of 6, use confound: control group disadvantaged from the start

0 1 2

6 4 x 1 x

y s confound

y s confound

Replacing observed cases with linear counterfactuals

But how do we obtain these estimates?


Conclusion





Internal validity


External validity


Control for Confound ↔Adjusting the Outcome

666

0 1 2

6 4 x 1 x

example: 3=6 4 x 0 1 x -3

to get to 6 on the left, add 3 to each side

3+3=6 4 x 0

control for conound adjust the outcome

y s confound

y s confound

Replacing observed cases with linear counterfactuals


Conclusion





Internal validity


External validity


Regression!SAS Syntax for reading in toy counterfactual data

DATA one;input y s confound;cards;9 1 010 1 011 1 03 0 -34 0 -25 0 -1run;

proc reg data=one;model y=s ;run;

proc reg data=one;model y=s confound; run;

0 1 2

6 4 x 1 x

y s confound

y s confound

Biased estimated

Unbiased estimate


Conclusion





Internal validity


External validity


Regression!: SPSS Syntax for reading in toy counterfactual dataDATA LIST FREE / y s confound.

Begin DATA .9 1 010 1 011 1 03 0 -34 0 -25 0 -1End DATA .

REGRESSION /MISSING LISTWISE /STATISTICS COEFF OUTS R ANOVA /CRITERIA=PIN(.05) POUT(.10) /NOORIGIN /DEPENDENT y /METHOD=ENTER s.

REGRESSION /MISSING LISTWISE /STATISTICS COEFF OUTS R ANOVA /CRITERIA=PIN(.05) POUT(.10) /NOORIGIN /DEPENDENT y /METHOD=ENTER s confound.

0 1 2

6 4 x 1 x

y s confound

y s confound

Biased estimated

Unbiased estimate


Conclusion





Internal validity


External validity


How Regression Works: Partial Correlation

s· s· ·s· | 2 2 2 2

· s·

.965 .866 .928.866

1 1 1 .866 1 .928

y cv y cvy cv

y cv cv

r r rr

r r

Partial Correlation: correlation between s and y, where s and y have been controlled for the confounding variable

2 2

2 2

( | ) ( ) 1 3.406 1 .928 1.265

(s | ) (s) 1 .547 1 .866 .274

y c

s c

sd y cv sd y r

sd cv sd r

1 s· |

( | ) 1.265ˆ .866 4( | ) .274y cv

sd y cvr

sd x cv

Spss syntaxCORRELATIONS /VARIABLES=y s confound /PRINT=TWOTAIL NOSIG

Sas syntaxtproc corr data=one;var y s confound;run;


Conclusion





Internal validity


External validity


Derivation of Partial from Matrix Algebra (Normal Equations)

1 11

2 2

1 1

( ) ( ) ( ) ( ) /covariance of x and yˆ

variance of x( ) ( ) /

n n

i i i ii i

n n

i ii i

x x y y x x y y n

x x x x n

1 2 1 2

1 2

1 2

1 22 2

11 2 1

1 2 1 2 22

1 2

1 2 1 2 s· s·11 2 2

1

1 11 ' , ' , X'Y=

1 1 1 1

for , ' X'Y1 1

x x x x

x x

x x s cv

x x

x yx x

x x x x x y

x x

x y x y x x s y cv y s c y cv y

r

r r rrX X X X

r r r

r r

r r r r r r v r r rX X

r r

2·

2 2 2· s·

( ) 1

1 1 (s) 1

y cvcv

y cv cv s cv

sd y r

r r sd r

Multivariate: Β=[X’X]-1X’Y For 2 variables that are standardized:


Conclusion





Internal validity


External validity


s· s· ·1 2

s·

( )

1 (s)y cv y cv

cv

r r r sd y

r sd

Defining the Impact of a Confounding Variable

s· s· ·s· | 2 2

· s·1 1

y cv y cvy cv

y cv cv

r r rr

r r

s· s· ·

The "impact" is the product:

impact (of cv on ) y cv y cvr r r


Conclusion





Internal validity


External validity


rsy=.18

rscv=.17

rycv=.07rscv×rycv

CVEnhancement ofteaching through

leadership

SBoard

Certification YHelp

Provided

The Impact of a Enhancement of Teaching through leadership on Correlation Between

Board Certification and Help Provided

rsy|cv=.866

How Regression Works:Impact of a Confounding Variable on a Regression Coefficient

rs∙y=.964

rcv∙y=.928.866x.928

s y

rcv∙s=.866

rc∙sxrc∙y

cv

Impact weights the relationship between cv and y by the relationship between c and s: the stronger the relationship between cv and y, the more important the relationship between cv and s.


Conclusion





Internal validity


External validity


Alternative Conceptualization (overlapping variance)

(rx2∙y|x1)2


Conclusion





Internal validity


External validity


Everything old is new again


Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity


Regression works…

As long as you have the right confounding variables

Linear relationships assumed

Does this work in practice?


Conclusion





Internal validity


External validity


Regression in Practice: Observational studies with controls for pretests work better than you think

Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the

American Statistical Association, 103(484), 1334-1344.

Quantify how much biased removed by statistical

control using pretests in a given setting

Sample: Volunteer undergraduates

Outcome: Math and vocabulary tests

Treatment: • basic didactic, • showing transparencies• defining math concepts

OLS Regression with pretests removes 84% to 94% of bias relative to RCT!! Propensity by strata not quite as good

See also Concato et al., 2000 for a comparable example in medical research


Conclusion





Internal validity


External validity


Supporting findingsBerk R (2005) Randomized experiments as the bronze standard. J Exp Criminol 1:417–433

Berk R, Barnes G, Ahlman L, Kurtz E (2010) When second best is good enough: a comparison between a

true experiment and a regression discontinuity quasi-experiment. J Exp Criminol 6:191–208. ‘‘the results

from the two approaches are effectively identical’’ page 191.

Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009). Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis, 31(4), 463-479.

Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine, 342(25), 1887-1889.

Cook, T. D., Shadish, S., & Wong, V. A. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy and Management. 27 (4), 724–750.

Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association, 103(484), 1334-1344.

Steiner, Peter M., Thomas D. Cook & William R. Shadish (in press). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics.

Steiner, Peter M., Thomas D. Cook, William R. Shadish & M.H. Clark (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods. Volume 15, Issue 3. Pages 250-267. more than just pretest.

Kane, T., & Staiger, D. (2008). Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation. NBER working paper 14607.

35. Kane, T., & Staiger, D. (2008).

Bifulco, Robert . "Can Nonexperimental Estimates Replicate Estimates Based on Random Assignment in Evaluations of School Choice? A Within‐Study Comparison." Journal of Policy Analysis and Management 31, no. 3 (2012): 729-751.

Reports 64% to 96% reduced with pre-test

http://www.ncbi.nlm.nih.gov/sites/entrez?Db=pubmed&Cmd=Search&Term=%22Concato%20J%22%5BAuthor%5D&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_DiscoveryPanel.Pubmed_RVAbstractPlus


















http://www.ncbi.nlm.nih.gov/sites/entrez?Db=pubmed&Cmd=Search&Term=%22Shah%20N%22%5BAuthor%5D&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_DiscoveryPanel.Pubmed_RVAbstractPlus

http://www.ncbi.nlm.nih.gov/sites/entrez?Db=pubmed&Cmd=Search&Term=%22Horwitz%20RI%22%5BAuthor%5D&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_DiscoveryPanel.Pubmed_RVAbstractPlus

http://www.ncbi.nlm.nih.gov/sites/entrez?Db=pubmed&Cmd=Search&Term=%22Horwitz%20RI%22%5BAuthor%5D&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_DiscoveryPanel.Pubmed_RVAbstractPlus

http://www.sciencedirect.com/science?_ob=PublicationURL&_tockey=


Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity


New paper on Multiple Approaches (from Tom Dietz)

http://arxiv.org/pdf/1412.3773v1.pdf

http://arxiv.org/pdf/1412.3773v1.pdf


Conclusion





Internal validity


External validity


Reflection

What part if most confusing to you?Why?

More than one interpretation?

Talk with one other, share

Find new partner and problems and solutions


Conclusion





Internal validity


External validity


When Regression Doesn’t Work

Instrumental variables?• Alternative assumptions, may be not better

– Exclusion restriction– Strong instrument– Large n

Propensity scores?No better than the covariates that go into it

• Heckman, 2005; Morgan & Harding, 2006, page 40; Rosenbaum, 2002, page 297; Shadish et al., 2002, page 164);

• How could they be better than covariates?– Propensity=f(covariates).

Paul Holland says:

PAUL ROSENBAUM

http://www-stat.wharton.upenn.edu/~rosenbap/index.html


Conclusion





Internal validity


External validity


Sensitivity Analysis:What Must be the Impact of an Unmeasured

Confounding variable invalidate the Inference?

Step 1: Establish Correlation Between predictor of interest and outcomeStep 2: Define a Threshold for InferenceStep 3: Calculate the Threshold for the Impact Necessary to Invalidate the InferenceStep 4: Multivariate Extension, with other Covariates


Conclusion





Internal validity


External validity


Obtain t critical, estimated effect and standard error

Estimated effect( ) = -9.01 Standard

error=.68n=7168+471=7639;df > 500,

t critical=-1.96

From: Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics. Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205–224


Conclusion





Internal validity


External validity


Establish Correlation and Threshold for Kindergarten Retention on Achievement

2 2

t 13.25r .150

(n q 1) t (7639 2 1) (13.25)

t taken from HLM: =-9.01/.68=-13.25n is the sample size =7639q is the number of parameters estimated ( predictor+omitted variable=2)

# critical

2 2critical


(n q 1) t (7639 2 1) 1.96

Where t is critical value for df>200

Step 1: calculate treatment effect as correlation

Step 2: calculate threshold for inference (e.g., statistical significance)

r# can also be defined in terms of effect sizes


Conclusion





Internal validity


External validity


Step 3a: Calculate the Threshold for the Impact Necessary to Invalidate the Inference

.150 .022.130

1 | 022 |TICV

#·

#

r

1 | r |x yrTICV

Set rx∙y|cv =r# and solve for k to find the threshold for the impact of a confounding variable (TICV). Sometimes I say ITCV.

· · · ·· | 2 2

· ·11 1

x y x cv y cv x yx ycv

y cv x cv

r r r r impactr

impactr r

Assume rx∙cv =ry∙cv (which maximizes the impact of the confounding variable – Frank, 2000). Then impact= rx∙cv x ry∙cv = rx∙cv x rx∙cv = ry∙cv x ry∙cv , and

Magnitude of impact of an unmeasured confound > .130 → inference invalidMagnitude of impact of an unmeasured confound < .130 → inference valid.


Conclusion





Internal validity


External validity


Maxim

izing impact


Conclusion





Internal validity


External validity


Step 3b: Component Correlations

2r .130 r= .130 .361

The magnitude of each correlation (rx∙cv , ry∙cv ) must be greater than .36 to change inference. The correlations must have opposite signs.

If rx∙cv = ry∙cv =r, then impact= rx∙cv x ry∙cv =r2 .

Component correlations for threshold

.150 .022.130

1 .022TICV


Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity


Calculating the % Bias to Invalidate the Inference:Obtain spreadsheet

From https://www.msu.edu/~kenfrank/research.htm#causalChoose spreadsheet for calculating indices

Access spreadsheet

spreadsheet for calculating indices [KonFound-it!]

https://www.msu.edu/~kenfrank/research.htm





Conclusion





Internal validity


External validity


KonFound-it! SpreadsheetUser enters values in yellow


Conclusion





Internal validity


External validity


Inside the Calculations


Conclusion





Internal validity


External validity


Inside the Calculations

2 2

t 13.25r .150

(n q 1) t (7639 3) (13.25)

# critical

2 2critical


(n q 1) t (7639 3) 1.96

Step 1: calculate treatment effect as correlation

Step 2: calculate threshold for inference (e.g., statistical significance)


Conclusion





Internal validity


External validity


Inside the Calculations: Step 3a

.150 .022.130

1 | 022 |TICV

#·

#

r

1 | r |x yrTICV

2r .130 r= .130 .361 Step 3b


Conclusion





Internal validity


External validity


Reflection

• What part if most confusing to you?– Why?– More than one interpretation?

• Talk with one other, share• Find new partner and problems and

solutions


Conclusion





Internal validity


External validity


Inside the Calculations: Multivariate

2 2

t 13.25r .152

(n q 1) t (7639 3 238) (13.25)

# critical

2 2critical


(n q 1) t (7639 3 238) 1.96

Step 1M: calculate treatment effect as correlation with DF correction

Step 2M: calculate threshold for inference (e.g., statistical significance)

DF correction


Conclusion





Internal validity


External validity


Step 3M: Multivariate Extension, with Covariates

#· | |

#|

·cv| y·cv|

r|

1 | r |

and = = |

x y z z

z

x z z

rITCV z

r r ITCV z

2 2 2 2· · | · cv· · cv·

2 2 2 2x· x· | x· cv· x· cv·

1 1

1 1

y cv y cv z y z z y z z

cv cv z z z z z

r r R R R R

r r R R R R

k=rx ∙cv|z× ry ∙ cv|z

Following bivariate approach: maximizing the impact with covariates z in the model implies

And components


Conclusion





Internal validity


External validity


Assumption: Omitted Confound Independent of Observed

Covariates

2 2 2 2 2· · | · cv· · cv· · | ·

2 2 2 2 2x· x· | x· cv· x· cv· x· | x·

1 1 1

1 1 1

y cv y cv z y z z y z z y cv z y z

cv cv z z z z z cv z z

r r R R R R r R

r r R R R R r R

2cv· 0 by assumption (can be altered in spreadsheet)zR

Gives maximal credence to skeptic challenging inference


Conclusion





Internal validity


External validity


Step 3M: Multivariate Extension, with Covariates

#· | |

#|

·cv| y·cv|

r . .152 .023| .132,

1 | r | 1 | .023 |

and = = | =. .132=.364

x y z z

z

x z z

rITCV z

r r ITCV z

2· · | ·

2x· x· | x·

1 .364 1 .201 .325

1 .364 1 .275 .310

y cv y cv z y z

cv cv z z

r r R

r r R

Following bivariate approach: maximizing the impact with covariates z in the model implies

And

2cv· 0 by assumption (can be altered in spreadsheet)zR


Conclusion





Internal validity


External validity


Obtaining R2 YZ

2 22 2|Z . ,. ,2 2

|Z 2 2|Z1 1

YX Y X ZY X Z YZYX YZ

YZ YX

r RR Rr R

r r

2 2 2

|Z . ,2Z 2 2

|Z

.152 .22.201

1 .152 1YX Y X Z

YYX

r RR

r

:


Conclusion





Internal validity


External validity


Inside Calculations: Obtaining R2 YZ

2 2 2|Z . ,2

Z 2 2|Z

.152 .22.201

1 .152 1YX Y X Z

YYX

r RR

r

:


Conclusion





Internal validity


External validity


Inside Calculations: Obtaining R2 xz

xZ:


Conclusion





Internal validity


External validity


Extra Sufficient Statistics for Obtaining R2xZ

number of covariates

% retained= 471/7639=.061Var(x)=.061*.94=.057, std(x)=.24Page 208

Page 210

Residual variance=.55+.88=143Initial variance=13.532=183R2 =1-143/183=.22

or

page 216:page 215:

std(y),

page 217

Already have Se(β1)


Conclusion





Internal validity


External validity


Multivariate Calculations in SpreadsheetUser entersStd(y)Std(x)R2


Conclusion





Internal validity


External validity


Obtaining R2 xz

:

2 2 2. ,2

2 2221

ˆ 1 13.53 1 .221 1 .275

ˆ .24 (7639 241) .68ˆ Se( )

y Y X Z

XZ

x

RR

df


Conclusion





Internal validity


External validity



Conclusion





Internal validity


External validity


What must be the Impact of an Unmeasured Confound to Invalidate the Inference?

If |k| > .132 (or .130 without covariates) then the inference is invalid

If |r x cv|z |= |ry cv|z|, then each would have to be greater than k1/2 =.36 to invalidate the inference.

*Because effect is negative, components take opposite signs

*Correlations must be partialled for covariates z.Multivariate adjustment, to invalidate the inference:

|ry cv| > .325 and |r x cv| >.310


Conclusion





Internal validity


External validity


Links to sas and stata code

sas program for calculating indices and related measures

stata code for calculating indices (beta version – thanks to Yun-jia Lo)

https://www.msu.edu/~kenfrank/papers/itcv_2005_version_1d.sas



https://www.msu.edu/~kenfrank/stata%20code%20for%20itcv.zip






Conclusion





Internal validity


External validity


Alternative: Regression Coefficient and Standard Error


Conclusion





Internal validity


External validity


Maximizing Expression


Conclusion





Internal validity


External validity


Comparing Regression vs Correlation ITCV (example from Frank, 2000)

#· | |

#|

r .1809 .0579| .1305

1 | r | 1 | .0579 |x y z z

z

rITCV z

Regression (page 155)

Correlation (pages 160,182; Frank, Sykes et al., 2008)


Conclusion





Internal validity


External validity


Exercise 3: Impact Threshold1)Identify a statistical inference in an article you are

interested in.2) Describe possible confounds/alternative explanations

that could bias the estimate3) Note the sample size and t-ratio

recall t=estimate/[se(estimate)]4) Calculate robustness of inference usinghttp://www.msu.edu/~kenfrank/papers/calculating%20indices%203.xls5) If you have the data calculate the multivariate ITCV6) Discuss with a partner

http://www.msu.edu/~kenfrank/papers/calculating%20indices%203.xls



Conclusion





Internal validity


External validity


Applications of Impact Threshold in Education and Sociology (also some in accounting)Frank, 2000. ITCV=.228. Frank, K.A., Zhao, Y., Penuel, W.R., Ellefson, N.C., and Porter, S. 2011. Focus, Fiddle and Friends: Sources of Knowledge to Perform the Complex Task of Teaching. Sociology of Education, Vol 84(2): 137-156.Frank, K.A., Gary Sykes, Dorothea Anagnostopoulos, Marisa Cannata, Linda Chard, Ann Krause, Raven McCrory. 2008. Extended Influence: National Board Certified Teachers as Help Providers. Education, Evaluation, and Policy Analysis. Vol 30(1): 3-30. Frisco, Michelle, Muller, C. and Frank, K.A. 2007. Using propensity scores to study changing family structure and academic achievement. Journal of Marriage and Family. Vol 69(3): 721–741 *Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors.Frank, K. 2000. "Impact of a Confounding Variable on the Inference of a Regression Coefficient." Sociological Methods and Research, 29(2), 147-194

Crosnoe, Robert and Carey E. Cooper. 2010. “Economically Disadvantaged Children’s Transitions into Elementary School: Linking Family Processes, School Contexts, and Educational Policy.” American Educational Research Journal 47: 258-291. ITCV =.32Crosnoe, Robert. 2009. “Low-Income Students and the Socioeconomic Composition of Public High Schools.” American Sociological Review 74: 709-730. ITCV=.03Augustine, Jennifer March, Shannon Cavanagh, and Robert Crosnoe. 2009. “Maternal Education, Early Child Care, and the Reproduction of Advantage.” Social Forces 88: 1-30. ITCV=.4,.23

Maroulis, S. & Gomez, L. (2008). “Does ‘Connectedness’ Matter? Evidence from a Social Network Analysis within a Small School Reform.” Teachers College Record, Vol. 110, Issue 9.

Cheng, Simon, Regina E. Werum, and Leslie Martin. 2007. “Adult Social Capital: How Family and Community Ties Shape Track Placement of Ethnic Groups in Germany.” American Journal of Education 114: 41-74.

William Carbonaro1 Elizabeth Covay1 School Sector and Student Achievement in the Era of Standards Based Reforms. Sociology of eductaion vol. 83 no. 2 160-182 .Chen, Xiaodong, Frank, Kenneth A. Dietz, Thomas, and Jianguo Liu. 2012. Weak Ties, Labor Migration, and Environmental Impacts: Toward a Sociology of Sustainability. Organization & Environment. Vol. 25 no. 1 3-24.

see alsoPan, W., and Frank, K.A. (2004). "An Approximation to the Distribution of the Product of Two Dependent Correlation Coefficients." Journal of Statistical Computation and Simulation, 74, 419-443

Pan, W., and Frank, K.A., 2004. "A probability index of the robustness of a causal inference," Journal of Educational and Behavioral Statistics, 28, 315-337.









http://www.wiley.com/bw/journal.asp?ref=0022-2445










http://soe.sagepub.com/search?author1=William+Carbonaro&sortspec=date&submit=Submit

http://soe.sagepub.com/content/83/2/160.abstract

http://soe.sagepub.com/search?author1=Elizabeth+Covay&sortspec=date&submit=Submit

http://soe.sagepub.com/content/83/2/160.abstract

https://www.msu.edu/~kenfrank/papers/Final%20version%20of%20paper%20%20CSCS%20

https://www.msu.edu/~kenfrank/papers/Final%20version%20of%20paper%20%20CSCS%20

https://www.msu.edu/~kenfrank/papers/jebs%20with%20wei.doc

https://www.msu.edu/~kenfrank/papers/jebs%20with%20wei.doc


Conclusion





Internal validity


External validity


Extensions LogisticSimulation:

• Kelcey, B. 2009. Improving and Assessing Propensity Score Based Causal Inferences in Multilevel and Nonlinear Settings. Unpublished doctoral dissertation, University of Michigan

• Ichino, Andrea, Fabrizia Mealli, and Tommaso Nannicini. 2008. “From temporary help jobs to permanent employment: what can we learn from matching estimators and their sensitivity? ” Journal of Applied Econometrics 23:305–327.

• Nannicini, Tommaso. 2007. “Simulation–based sensitivity analysis for matching estimators.” Stata Journal 7:334–350.

Table• David J. Harding. 2003. “Counterfactual Models of Neighborhood

Effects: The Effect of Neighborhood Poverty on High School Dropout and Teenage Pregnancy.” American Journal of Sociology 109(3): 676-719.


Conclusion





Internal validity


External validity


Extensions: Multilevel

If omitted confound is at level 1, no concerns. Covariates include level 2 fixed or random effectsIf omitted confound at level 2:Seltzer, M. H., Frank, K. A., & Kim, J. (2007). Studying the Sensitivity of Inferences to Possible Unmeasured Confounding Variables in Multisite Evaluations. Paper presented at invited session at AERA 2007.


Conclusion





Internal validity


External validity


Extensions: Impact Not Maximized and Impact Curve

Impacts above the blue line invalidate the inference

Smallest impact to invalidate inference: rxcv=rycv=.364


Conclusion





Internal validity


External validity


Extensions: Impacts Before and After Pretests

If k > .109 (or .130 without covariates) then the inference is invalid

Impact of strongest measured covariate (student approaches to learning) is -.00126 (sign indicates controlling for it reduces estimated effect of retention)

Impact of unmeasured confound would have to be about 100 times greater than the impact of the strongest observed covariate to invalidate the inference. Hmmm….


Conclusion





Internal validity


External validity


Effect of Retention on Achievement After Adding each Covariate

Controls Est Se t

School -21.24 .63 -33.49

School+Pre2 (spring Kindergarten) -12.01 .45 -26.48

School+Pre2+(Pre2-Pre1)+ -12.10 .47 -26.28

School+Pre2+(Pre2-Pre1)+Momed

-12.00 .47 -26.26

School+Pre2+(Pre2-Pre1)+Female

-12.07 .46 -25.18

School+Pre2+(Pre2-Pre1)+2parent

-12.01 .46 -26.27

School+Pre2+(Pre2-Pre1)+poverty

-12.04 .46 -26.16

Hong and Raudenbush (model based) -9.01 .68 -13.27

n=10,065, R2 =.40Note: 1 year’s growth is about 10 points, so retention effect > 1 year growth


Conclusion





Internal validity


External validity


Consider Alternate Sample(External Validity)

Causal Inference concern: We cannot assert cause if the effect of is not constant across contexts.

Statistical Translation:Would the inference be valid if the sample included more of some population (e.g. non-volunteer schools) for which the effect was not as strong?

Rephrased for robustness: what must be the conditions in the alternative sample to invalidate the inference?


Conclusion





Internal validity


External validity


Consider Alternate Sample(External Validity)

Define as the proportion of the sample that is replaced with an alternate sample.

r is correlation in unobserved data

R is combined correlation for observed and unobserved data:

Rxy=(1-)rxy + rxy .










Conclusion





Internal validity


External validity


Thresholds for Sample Replacement

Set R=r# and solve for rxy:

If half the sample is replaced (=.5), original

inference is invalid if rxy < 2r#-rxy

Therefore, 2r#-rxy defines the threshold for replacement: TR(=.5)

If rxy =0, inference is altered if π> 1-r#/rxy . Therefore 1-r#/rxy defines the threshold for

replacement: TR(rxy=0)Assumes means and variances are constant across samples, alternative calculations available.


Conclusion





Internal validity


External validity


Thresholds for Sample Replacement: Toy example




Therefore, 2r#-rxy defines the threshold for replacement: TR(=.5):

If threshold = r#=.4 and rxy =.6 then inference invalid

if rxy = 2r#-rxy =2x.4-.6=.2.


Conclusion





Internal validity


External validity


Fundamental Problem of Inference to an Unsampled Population (External Validity)

But how well does the observed data represent both populations?

91011345

888666

64


Conclusion





Internal validity


External validity


Thresholds for Sample Replacement

If rxy =0, inference is altered if π> 1-r#/rxy . Therefore 1-r#/rxy defines the threshold for

replacement: TR(rxy=0)

If threshold = r#=.4 and rxy =.6 then inference invalid if % replaced = π =1-r#/rxy =1-.4/.6=1/3.


Conclusion





Internal validity


External validity



Observed Data (External Validity)

91011345

66 6 6

64

How many cases would you have to replace with cases with zero effect to change the inference?Assume threshold is: δ# =4:1- δ# /

=1-4/6=.33 =(1/3)

6

6

0


Conclusion





Internal validity


External validity


Empirical example of Thresholds for Replacement: Effect of Open Court (Borman et al)

TR(=.5)= 2r#-rxy|z =2(.29)-.54=.03. Correlation between Open Court and reading achievement

would have to be less than .03 to invalidate inference if half the classrooms in the sample were replaced (e.g., with classrooms with no effect).

TR(rxy =0)= 1-r#/rxy|z =1-(.29/.54)=.47About 47% (abut 23 classrooms) would have to be replaced

with others for which Open Court had no effect (rxy =0) to invalidate the inference in a combined sample.


Conclusion





Internal validity


External validity


Review of Correlational Framework

Statistical control impact of a confound

Internal validity: Impact necessary to change an inference

External validity: unobserved correlations necessary to change an inference


Conclusion





Internal validity


External validity


Exercise 4: Robustness for Sample Representativeness (External Validity)

1)Identify a statistical inference in your own work or in the literature for which there is concern about the external validity

2) Identify possible populations for which the effect may not apply

3) Note the t-ratio and sample size4) Calculate robustness of inference usinghttp://www.msu.edu/~kenfrank/papers/calculating%20indices%203.xls

How do the indices change when you change thesample size?t-ratio?

5) Discuss with a new partner your inference and how robust you think it is. Partner can challenge. Then change roles.




Conclusion





Internal validity


External validity


Thresholds for Sample Replacement: Toy example




Therefore, 2r#-rxy defines the threshold for replacement: TR(=.5):

If threshold = r#=.4 and rxy =.6 then inference invalid

if rxy = 2r#-rxy =2x.4-.6=.2.


Conclusion





Internal validity


External validity


Conclusion

Applications of frameworks

Limitations

There will be debate about inferences: quantify

Assumptions as the bridge between statistics and science


Conclusion





Internal validity


External validity


Applications of Frameworks% bias to invalidate

Any estimate, threshold

CounterfactualGood for experimental settings (treatments)

Think in terms of replacement of cases

CorrelationalGood for continuous predictors

Think in terms of correlations

Both can be applied to Internal and External Validity


Conclusion





Internal validity


External validity


Limitations on Inference

Without random sample and random assignment causal inferences are uncertain. They are inferences.Observation study

Internal validity: omitted variables Randomized Study

External validity: fidelity, attritionParadox of external validity

If results of randomized study influence behavior, then inherent limit on external validity

• Prior to study, subjects sign up because of fit, political pressure, etc

• After study, subjects sign up because they think it generally works

Inferences will be debated based on differences in ExperiencePower Goals


Conclusion





Internal validity


External validity


It’s all in how you talk about it!

Do best method you canInclude relevant controls!

But science is as much in the nature of the discourse as the method

Virtue epistemology • Greco, 2009; Kvanig, 2003; Sosa, 2007

What would it take to invalidate the inference?• Frank, K.A., Duong, M.Q., Maroulis, S., Kelcey, B.

under review. Quantifying Discourse about Causal Inferences from Randomized Experiments and Observational Studies in Social Science Research


Conclusion





Internal validity


External validity


Quantify what it Would Take to Change the Inference

• Objections to moving from statistical to causal inference – No unobserved confounding variables– Treatment has same effect for all

• Robustness indices quantify how much must assumptions must be violated to invalidate inference.

• No new causal inferences!– robustness indices merely quantify terms of debate regarding

causal inferences.• Can be used with any threshold.• Basically, characterizing size of p-value or t-ratio, instead of “barely

significant” versus “highly significant”• Can be used (theoretically) for any t-ratio

– Discuss: Statistical inference as threshold?• Difficult to defend tenuous effects (e.g., p =.049).


Conclusion





Internal validity


External validity


Assumptions are the bridge between statistical and causal inference

Statistical Inference Causal Inference

Assumptions

Cornfield, J., & Tukey, J. W. (1956, Dec.), Average Values of Mean Squares in Factorials. Annals of Mathematical Statistics, 27(No. 4), 907_949.


Conclusion





Internal validity


External validity


In Donald Rubin’s words “Nothing is wrong with making

assumptions; on the contrary, such assumptions are the strands that join the field of statistics to scientific disciplines. The quality of these assumptions and their precise explication, not their existence, is the issue”(Rubin, 2004, page 345).

Ken adds: and we should talk about the inferences in terms of the sensitivity to the assumptions


Conclusion





Internal validity


External validity


Paul Holland and Don Rubin

This [the use of randomization to alleviate the problems of untestable assumptions] should not be interpreted as meaning that randomization is necessary for drawing causal inferences. In many cases, appropriate untestable, assumptions will be well supported by intuition, theory or past evidence. In such cases we should not avoid drawing causal inferences and hide behind the cover of uninteresting descriptive statements. Rather we should make causal statements that explicate the underlying assumptions and justify them as well as possible.”


Conclusion





Internal validity


External validity


Reflection

What part if most confusing to you?Why?

More than one interpretation?

Talk with one other, share

Find new partner and problems and solutions


Conclusion





Internal validity


External validity


Emails of Users'Crosnoe, Robert' <[email protected]>; [email protected]; [email protected]; [email protected] Muller, Chandra L <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Spiro Maroulis <[email protected]>; 'Ben Kelcey' <[email protected]>; 'Minh Duong' <[email protected]>; [email protected]; [email protected]; 'James Moody' <[email protected]>; 'William Carbonaro' <[email protected]>; 'Mark Berends' <[email protected]>; Min Sun <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; '[email protected]; [email protected]; [email protected]; [email protected]; David Williamson Shaffer <[email protected]>; [email protected]; Eric Camburn <[email protected]>; Geoffrey Borman <[email protected]>; [email protected]; [email protected]; Cavanagh, Shannon E <[email protected]>; Dedrick, Robert <[email protected]>; John Lockwood <[email protected]>; Lou Mariano <[email protected]>; Joshua Cowen <[email protected]>; [email protected]; T Gmail <[email protected]>; Yu Xie <[email protected]>; [email protected]; Konstantopoulos, Spyros <[email protected]>; Kimberly S. Maier <[email protected]>; William Schmidt <[email protected]>; Houang <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]

Last sent april 6 2015, 9:15 a.m.

Documents

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal