Current Issues in Individual, Group, and Organizational Level

Measurement :Strategic Management

Prof. Brian BoydW. P. Carey School

Arizona State Universityhttp://www.briankboyd.com

RoadmapRoadmap

• Background

• State of the art

• Consequences of measurement problems

• Application

BackgroundBackground

• Most of my papers have a strong measurement component, e.g.,– SMJ 90: Environmental characteristics– JMS 91: Implications for planning performance– SMJ 94: Board control– AMJ 98: Discretion and executive pay– SMJ 98: Planning systems– SMJ 01: CEO social capital

• Reviewers often profoundly uninterested in measurement

• What is the ‘state of the art’ for construct measurement in strategy?

• What are the implications of this?

Strategy and Strategy and Paradigm Paradigm

DevelopmentDevelopment• In a Kuhnian sense, management is a

young discipline

• Strategy is one of the youngest subsets of management

• Empirics play a key role in advancing paradigm development

How well does strategy research How well does strategy research contribute to the goal of advancing contribute to the goal of advancing

the paradigm?the paradigm?

Where Were We Two Decades Where Were We Two Decades Ago?Ago?

• Frequent use of nominal and single-item scales

• Inadequate assessment of validity

• Inexplicit treatment of strategy as an organization-level construct

Source: Venkatraman & Grant (1986) AMR

Strategy & Methods:Strategy & Methods:An OverviewAn Overview

• Inattention to methodology and design (Bowen & Wiersma, 1999)

• Few replication studies (Hubbard et al., 1997)

• Insufficient power (Mone et al., 1996)

• Little emphasis on construct measurement (Hitt, Boyd & Li, 2004)

Overall, the rigor of much strategy Overall, the rigor of much strategy research is weak.research is weak.

We’ll focus on these issues

A Power PrimerA Power Primer

Type I errorType I error

Type II errorType II error

Example: Population correlation = .30, a = .05Example: Population correlation = .30, a = .05

N = 30, 50% chance of successful detection

N = 50, 70% chance

N = 100, 90+% chance

Power in Strategy ResearchPower in Strategy Research• Published strategy papers had only 4 in 10 chance

of successful detection – half of recommended level (Mazen, Hemmasi & Lewis, 1987)

• Review of one research stream – only 8% of studies had sufficient power (Ferguson & Ketchen, 1999)

• SMJ has lower power than other journals (Mone, Mueller & Mauland, 1996)

• Few studies report power statistics (Mone et al, 1996)

This is the This is the goodgood news… news…

Blalock’s (1979) Elements of Social Blalock’s (1979) Elements of Social ProcessesProcesses

1. A theoretical language explaining causal relations between constructs

2. An operational language for examining relationships between constructs using indicators

3. An integrative theory describing causal relationships between constructs and indicators

Quality of Strategy Measures?Quality of Strategy Measures?

Lack of consensus on Lack of consensus on measurement of basic measurement of basic strategy constructsstrategy constructs

• Strategic planning• Performance• Diversification• Board oversight• Environment• Discretion

Low emphasis on Low emphasis on measurementmeasurement

• Prevalence of single measures

• Sporadic reporting of reliability and validity

The Role of Measurement The Role of Measurement QualityQuality

ConstructConstructxx ConstructConstructyy Hypotheses Hypotheses framedframed

IndicatorIndicatorxx IndicatorIndicatoryy Hypotheses Hypotheses testedtested

Weak indicators lead to measurement Weak indicators lead to measurement error, and attenuation. Power error, and attenuation. Power calculations are based on fully accurate calculations are based on fully accurate constructs.constructs.

In other words, the statistics presented In other words, the statistics presented earlier earlier overestimateoverestimate the power of the power of strategy researchstrategy research

An ExampleAn Example

ConstructConstructxx ConstructConstructyy N = 150N = 150power = .80power = .80

0.300.30

IndicatorIndicatorxx IndicatorIndicatoryy N = 150N = 150power = .33power = .33

0.100.10

.60.60 .60.60

State of the Art & Consequences of State of the Art & Consequences of Measurement Problems in Strategy Measurement Problems in Strategy

ResearchResearch

Study 1: AssessmentStudy 1: Assessment• Content analysis of

several hundred published articles

• Multiple journals• Multiple years

Study 2: ConsequencesStudy 2: Consequences• Replication of a

prominent research question

• Large sample analysis of US firms

• Manually adjust levels of measurement precision

State State of the Artof the Art

AssessmentAssessment• Identified all empirical strategy papers in key

outlets: AMJ, ASQ, MS, and SMJ• Sampled articles published 1998 -- 2002• Content coded articles for:

– Sample size– Measurement schemes– Reliability

• Statistical test is unit of analysis: 625 tests in 196 articles

• High inter-rater reliability on selection & coding of studies

DescriptivesDescriptives

MeanMean Std. DevnStd. Devn

Sample size 2,559 12,908

IVs 5.66 5.09

Controls 8.62 30.26

Obs/Inds 236.12 1,333.82

Sample size limitations do not Sample size limitations do not appear to constrain measurementappear to constrain measurement

Good News…Bad NewsGood News…Bad News

0.7

0.72

0.74

0.76

0.78

0.8

0.82

0.84

Reliability

IV

DV

Control

50%

55%

60%

65%

70%

75%

80%

85%

90%

95%

No Data

IV

DV

ControlWhile reported reliabilities are While reported reliabilities are strong, this information is typically strong, this information is typically

notnot available in the article. available in the article.

Summary of Content AnalysisSummary of Content Analysis

• Most strategy studies rely on single indicators

• Limited efforts made to report reliability data, when multiple indicators are used

• Many studies report either partial or no correlation matrices, making post hoc asessment impossible

Is this a salient problem,Is this a salient problem, or a non-issue?or a non-issue?

ConsequencesConsequences

Why Do Firms Diversify?Why Do Firms Diversify?

• Amihud & Lev (1981): Diversification is a result of agency conditions.

• Lane, Cannella & Lubatkin (1998): No support for A&L agency model.

• Denis , Denis & Sarin (1999): Both studies contaminated by methodological flaws. Substituting improved measures led to stronger results.

““Though both sets of authors conduct Though both sets of authors conduct similar empirical tests on virtually similar empirical tests on virtually identical data, they arrive at completely identical data, they arrive at completely different conclusions.” Denis et al., 1999different conclusions.” Denis et al., 1999

Methods Issues in the Agency Methods Issues in the Agency ArgumentArgument

MeasurementMeasurement• Both A&L and LCL

studies used broad ownership categories

• Denis et al found better results when using ratio level indicators

• Only single indicator measures used

PowerPower• 3 IV regression, w/ a=.05

power = .80

N = 34 for large effect76 for moderate547 for small

• LCL N=309Denis et al N=933

• Bulk of these studies had insufficient power

Insiders

StockOwnership

BoardControl

Diversification

Firm Size

Duality

Owner Reps

Director Pay

Related

Unrelated

-.16 [2.2]

.11 [2.3]

(Palepu, 1985)

(Boyd, 1994)

Sales

Assets

Equity

Sample:Sample:640 640 Fortune Fortune firms, representing 50 firms, representing 50 2-digit SIC s, and 200 4-digit SICs2-digit SIC s, and 200 4-digit SICs

BoardControl

Diversification

Firm Size

-.15 [2.1]-.12 [1.7]

.11 [2.3]

Related

Unrelated

(Palepu, 1985)

(Boyd, 1994)

Sales

Assets

Equity

Now, Let’s Introduce Now, Let’s Introduce Some Shadows…Some Shadows…

Duality

Insiders

StockOwnership

Owner Reps

Director Pay

BoardControl

Diversification

Firm Size

.09 [2.2] .13 [2.9] -.07 [1.6]

-.05 [1.3] .05 [1.2] .06 [1.5]

.12 [2.7] -.06 [1.3] -.03 [0.7]

-.02 [0.5]

.11 [2.3]

Related

Unrelated

(Palepu, 1985)

(Boyd, 1994)

Sales

Assets

Equity

……And Some More…And Some More…Duality

Insiders

StockOwnership

Owner Reps

Director Pay

Quantifying Shadows…Quantifying Shadows…

(a) Single indicator DV(a) Single indicator DV(2 models)(2 models)

• Failure to detect effect in 1 of 2 models

• R2 drops by a 10X factor

• Path coefficient drops slightly

(b) Single indicator DV (b) Single indicator DV and IV (10 models)and IV (10 models)

• Failure to detect effect in 7 of 10 models

• R2 drops by a 10X factor

• Path coefficient cut in half

Bottom Line:Bottom Line:The A&L and LCL debate is The A&L and LCL debate is

entirely an artifact of measurement entirely an artifact of measurement error and attenuation.error and attenuation.

Do We Know What We Know?Do We Know What We Know?• Strategy research has placed

little emphasis on statistical power

• Construct measurement is not a priority of empirical research

• Many strategy constructs are broad

• Weak consensus regarding measurement of key concepts

How many valid research How many valid research questions have been cast questions have been cast aside due to these method aside due to these method limitations?limitations?

Sales Assets

Employees Other

Measuring Firm Size

What’s Next?What’s Next?

A Comparison: Marketing

Two decades agoTwo decades ago

• The field of marketing was criticized for failing to pay attention to construct validity and associated measurement issues

TodayToday

• Studies routinely provide reliabilities, factor models, and evidence of convergent and discriminant validity

Source: Jarvis, Mackenzie & Podsakoff (2005) JCR

Some Easy StepsSome Easy Steps

• Sneak measurement into substantive topics

• Less attention to knee-jerk criticism (why ROE?) and more critical review of published measures

• Collect multiple measures

• Pay attention to validity

• Hang out with the OB crowd

More Balance in Research FocusMore Balance in Research Focus

ConstructConstructxx ConstructConstructyy SubstantiveSubstantiveResearchResearch

IndicatorIndicatorxx IndicatorIndicatoryyConstructConstruct

MeasurementMeasurementResearchResearch

Factor Models for Better Factor Models for Better MeasurementMeasurement

X X X

A

X X X

C

X X X

B

(a) Common Approach(a) Common Approach

Factor Models for Better Factor Models for Better MeasurementMeasurement

X X X

A

X X X

C

X X X

B

(b) Convergent (b) Convergent andand discriminant validity discriminant validity

Factor Models for Better Factor Models for Better MeasurementMeasurement

X X X X X XX X X

A

(b) CMV test(b) CMV test

BoardControl

StockOwnership

Insiders

Duality

Owner Reps

Director Pay

Constructs: Cause or Effect?Constructs: Cause or Effect?

Insiders

StockOwnership

Duality

Owner Reps

Director Pay

BoardControl

Constructs: Cause or Effect?Constructs: Cause or Effect?

Cause or Effect? Deciding…Cause or Effect? Deciding…Item Reflective Formative

Indicators needed Representative pool Census

Directionality Construct drives indicator

Indicators drives construct

Indicators imperfect representation of construct

Yes Yes

Measurement error Indicator level Construct level

Reliabilities Good Good or bad

Sensitivity to change in indicators

Minimal Substantial

What About Board Control?What About Board Control?

• Substitution, portfolios, and balancing (Rediker & Seth ’85 Finkelstein & D’Aveni ’84) all suggest reflective model

• CED of CFA is 0.97• All indicators load at .001, with

some moderate loading• Supplementary model using a

sixth indicator (CEO tenure)– Main model unaffected– 8 of 12 subsets NS: 67% versus

70% NS in 5-factor tests

BoardControl

StockOwnership

Insiders

Duality

Owner Reps

Director Pay

EnvironmentDimensions

Objective Perceptual

Industry Industry

Firm Outsiders Simulation

Insiders

TMT

Middle ManagersSource: Boyd, Dess & Rasheed (1993) AMR

Slope of SalesMean of Sales

Munificence

Discretion

Levels of Levels of MeasurementMeasurement

Organizational Research MethodsOrganizational Research MethodsSpecial issue on

Strategic Management

Sample topics:Sample topics:• Pitfalls of using archival proxies• Evaluation of construct validity of key metrics• Providing insights into studying ‘unobservables’• Articles/notes and empirical/conceptual papers

welcome• Click here for Call for Papers• Proposals due March 1, 2006

Web Resources

• Boyd, Gove & Hitt (2005). Construct measurement in strategic management research, SMJ

• Boyd, Gove & Hitt (2005). Consequences of measurement problems in strategic management research, SMJ

• Manuscript draft, reviews, and reviewer replies to BGH 2005a/b

• Hitt, Boyd & Li (2004). The state of strategic management research, RMSM