Meta-Analysis and Strategy Research
Dan R. Dalton
Kelley School of Business
Indiana University
2
A [Very] Brief History of Research Synthesis
• Averaging Correlations?• Combining Significance Levels?• The Narrative Review (aka “Counting”
Review)• Gene Glass (1976) “Invents” Meta-Analysis• Early Critics – “An Exercise in
Mega-Silliness”
3
An Example of Meta-Analysis(Data Are Simulated)
• Research Question: The Extent to which Equity Holdings by CEOs Are Related to Firms’ Financial Performance
• Proposed Moderator: Expected that this Relationship Will be Moderated by the “Maturity of the Firm” (i.e., Firms that Are Five or Less Years Post-IPO vs. Other)
• Studies Available for Meta-Analysis = 30 (10 are not significant, 10 are positive and
significant, 10 are negative and significant)
4
An Example of Meta-Analysis(Data Are Simulated)
• R = Reliability• RR = Range Restriction• M = Moderator (1 = ≤ 5 yrs. Post-IPO; 2 = > 5 yrs. Post-IPO)
r n Ry Rx RRy RRx M
.26
.39
.37
.29
.23
.11
56
225
192
146
70
325
.8
.8
.8
.8
.8
.8
.8
.8
.8
.8
.8
.8
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
2
5
“r” - A Bivariate Correlation
• “r” vs. “d”• R-square• Deriving “r” from “d,” “t,” “F-score,” “Z,”
“Chi-Square” …• “r” from Incomplete Information
r = Z/sqrt n
if “n” = 120 and Z = 1.96 with “r” unknown
then r = +/ - .179 (i.e., 1.96/10.95)
6
“r” – A Bivariate Correlation, cntd.
• -17 to +17 and Enter What?
• Discard the Study?
• “r” and the Z-transformation?
• “r” and Statistical Significance
• And, a “Surprise” About Multiple Non-Significant Results
7
“r” – A Bivariate Correlationand “n”
• “r” As an Independent Variable, a Dependent Variable, a Control Variable, a Moderating Variable, a Mediating Variable…
• “n” – The Sample Size from which the “r” Was Calculated
• To Weight the Observed Correlation in Order to Calculate the Mean Weighted Correlation Across All of the Studies
• “n” and the Correlation Matrix
8
Ry (Reliability of y); Rx (Reliability of x)
• Constructs vs. Observed Variables
• Strategic Management Meta-Analyses with Ry = 1 and Rx = 1
• Strategic Management Variables Are Not That Good
• The Choice of Ry and Rx Levels Is Counterintuitive – Lower Ry’s and Rx’s Will Improve the “Corrected r”
• Ry and Rx at .8
9
RRy & RRx (Range Restriction of y and x)
• Analytical Issues of Range Restriction Have Become Increasingly Complex
• In Strategic Management – RRy and RRx as Deliberate Selectivity in the Sample
• Strategic Management and “Survival” Issues
10
Moderation in Meta-Analysis
• In Meta-Analysis a “Moderator” Is a Subgroup
• Profligate Testing for ModeratorsCapitalization on ChanceLoss of Statistical Power
• Moderators Need Not Always Be Operationalized as a Dichotomy
11
Meta-Analytic Procedures and Results
PART 1: # of
Correlations
CombinedSample
Size
MeanTrue ScoreCorrelation
Std Dev:Mean True
ScoreCorrelation
EntireSample
30 9,685 -.026 .283
Moderation: ≤ 5 Yrs. from IPO
16 2,032 .417 .048
Moderation: > 5 Yrs.
from IPO14 7,653 -.144 .188
12
Meta-Analytic Procedures and Results
PART 2: MeanTrue Score
Correlation
80%Credibility
Interval
90%Confidence
Interval
% VarianceAttributableTo Artifacts
EntireSample
-.026 - .389 : .336 - .112 : .059 5.74
Moderation:≤ 5 Yrs. from IPO
.417 .354 : .479 .396 : .437 80.49
Moderation:> 5 Yrs.from IPO
-.144 - .386 : .098 -.061 : -.227 7.26
13
Meta-Analytic Results:Some Diagnostics
• The Magnitude of the Mean True Score Correlation• Does the 90% Confidence Interval Include Zero?
Suggests that the Mean True Score Is Not Significant• Does the 80% Credibility Interval (Difference
between Low and High Estimates) Exceed .11? Suggests the Existence of a Moderator
• Does the % Variance Attributable to Artifacts Exceed 75%? Suggests that a Moderator Is Unlikely
• And, If the Tests Had Relied on Different Rx and Ry Values? [ .7 = .48; .8 = .417; .9 = .37 ]
14
Results Summary
• There is no simple relationship ( -.026, ns) between CEO equity holdings and firm financial performance. There is, however, some evidence of the existence of a moderating variable.
• There is evidence of a moderating effect for time since IPO. The relationship between CEO equity holdings and firm financial performance for firms 5 years or less from IPO is .417, a significant relationship. The diagnostics suggest that a further moderating effect of this result is unlikely.
15
Results Summary, cntd.
• The relationship between CEO equity holdings and firm financial performance for firms more than 5 years from the IPO is -.144, a significant relationship. The diagnostics suggest that a further moderating effect of this result is likely.
16
Other Issues in Meta-Analysis
• Fixed vs. Random Effects ModelsRandom Effects Models – Population Parameters
May Vary Across StudiesFixed Effects Models – Population Parameters Are
Invariant• “File Drawer” Problem• Unreported Null Results• “Fail Safe” Approach• The Issue Is Less a Matter of Fail Safe
Algorithms than of Reliance on Too Few Studies
17
Other Issues in Meta-Analysis, cntd.
• Quality of Data
• Outliers Statistical OutliersEntry Error Outliers
• Sensitivity to Outliers
• The General Question of Discarding Data
• Disclosure and Replicability
18
Other Issues in Meta-Analysis, cntd.
• The Independence of Data• Entering Data that Are Clearly Not
Independent• A Random Selection, Pooling, a Weighted
“r”, a Weighted “n”• An Interesting Catch-22• “Clearly Reflect the Same Construct”• Independence of Samples
Constructive Replication
19
General Guidelines for Meta-Analysis
• There is no need to transform the input values of “r”s.
• When it is necessary to impute the value of “r,” set “r” = 0.
• For observed variables, rely on .8 for the reliability of the dependent and independent variables.
• With observed variables, it will rarely be necessary to assign a range restriction score.
20
General Guidelines for Meta-Analysis, cntd.
• Use a conservative 90% confidence interval for the meta-analysis diagnostics (for these data, 95% would be an interval of -.128 to .075, much wider than the -.112 to .059 reported).
• Use a conservative 80% credibility interval for the meta-analysis diagnostics (for these data, the 90% would have been an interval of -.493 to .448, much wider than the -.389 to .336 reported).
• Where the meta-analysis software provides an option, rely on a “Random Effects Model.”
21
General Guidelines for Meta-Analysis, cntd.
• Assuming every effort has been made for an exhaustive search for meta-analysis input data, you need not be concerned about “file drawer” issues
• Neither weight nor exclude data on the basis of the quality of the study. Instead, run two meta-analyses and compare the results for the entire data set and a reduced data set without the troublesome data
22
General Guidelines for Meta-Analysis, cntd.
• Only under extremely rare conditions would there be any concerns about the independence of the data; accordingly, there is no need to combine data from separate “r”s in any manner.
• No need to exclude outliers. Instead, run two meta-analyses and compare the results for the entire data set and a data set without the outliers.