-
WELCOME!
-
Critical Integrative Metatheory: New Methods to Evaluate
Psychological Theories & Models for Review Steven E. Wallis,
PhD [email protected]
Director, Foundation for the Advancement of Social TheoryAdjunct
Faculty, Capella UniversityFellow, Institute for Social Innovation,
Fielding Graduate UniversityEditorial Board, Integral Review
International Congress of Psychology 2012July 22-27 Cape Town,
South Africa
-
We want psychology to be of greater benefit to
humanityLearningPsychoanalysisMotivationCommunicationBehaviorismCognitionStressEmotions
-
Emergence of PsychologyPsychology is NOT much appreciated by
other sciences or the general public. For GOOD reasons!We are not
advancing as a science there is no proof that psychology is better
now than 50 years ago. Why have we failed to improve?
-
One Path to Improving Psychology is the Process of Submission
& ReviewReviewSubmissionTheory Creation
-
Many Rigorous Standards for Evaluating MethodsParticipant
CharacteristicsSampling ProceduresSampling size, power,
precisionMeasures and CovariatesResearch DesignRecruitment
StatisticsData AnalysisEtc., etc., etc., etc., etc.,
Methods Review
-
Standards for Evaluating Theory? TheoryReview
-
What is theory?LensMapMetaphorStoryDiagramNarrativeSet of
axiomsModelEthicsPolicyMental modelSchemaMind mapAssumptionsThese
are all names for a conceptual construct that may be useful for
engaging the world.
-
TheoryA theory is a conceptual construct, a set of abstract
causal relationships.Empowers (to take effective action)Restricts
(limits our sense of truth)Is developed by education, experience,
and social construction
Is (apparently) INNESCAPABLE(so it is apparently IMPORTANT)
-
MetaphoricallyMetatheory is a theoretical lens that may be
directed toward the investigation of theory.
-
Or, more formallyMetatheory is primarily the study of theory,
including the development of overarching combinations of theory, as
well as the development and application of theorems for analyses
that reveal underlying assumptions about theory and theorizing.
-
Three Ways to Evaluate TheoryCreation
Critical AnalysisCategorization
-
Common Methods for Evaluating the Creation of Theory (and
issues)Voracious reading (what? How much?)Be brave, Creative, etc
(how to measure?)Research (empirical data what is data?)Synthesize
existing theory (recursive problem)Reflexivity (how to
measure?)Abstraction (How much is best?)Grounded Theory (more
structured approach)Reflexive Dimensional Analysis (more structured
approach)
-
Common Methods for Categorization of
TheoryHistoricalGeographicalUnit level / Middle Range / GrandStyle
(Literary, Academic, Eristic, etc.)Purpose (Analyzing, Explaining,
Predicting)(ISSUE: Within a category, What Theories are
Better?)
-
Common Methods for Critical Analysis of TheoryPlausibility
ParsimonyCorrespondence to observed factsCoherence to existing
theoryApplicationPropositional AnalysisFalsification(ISSUE: Rarely
Done, Rarely Quantified)
-
CRITICAL Metatheory
Reflects The Need For RIGORBetter to have a specific
metatheoretical lensThe results must be quantified
If our lens of metatheory is cloudy, we wont see the lens of
theory with clarity
-
EXAMPLE: Propositional AnalysisIdentify the logical
propositions.Diagram the causal relationships.Integrate
diagrams.Identify and count the Concatenated aspects (two or more
causal influences).Count the total number of aspects (Complexity of
the theory).Calculate the Robustness(divide Concatenated aspects by
total aspects).
-
Abstract ExampleADCBACCEPropositions within a TheoryCarefully
IntegratedABCDEC = 5R = 0.20
-
Previous ResearchEvolution of Theory & Changes in
ComplexityComplex theories some usefulness
Chart1
7
6
14
11
3
Total number of Aspects
Number of Aspects
Changes in the Complexity of Theory as it Evolves Through A
Scientific Revolution Toward Greater Usefulness
Sheet1
YearTotal Number of AspectsNumber of Concatenated
AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
-
Previous ResearchEvolution of Theory & Changes in
RobustnessRobust theories Very Useful
Chart2
0.140.14
0.170.17
0.210.21
0.360.36
11
Robustness
Year
Robustness
Changes in Robustness of Theory as it Evolves Through A
Scientific Revolution Toward Greater Usefulness
Sheet1
YearTotal Number of AspectsNumber of Concatenated
AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
-
Theory of motivation and competence (White, 1959) 1. More new
stimulation4. More effect on environment3. More action2. More
cognition6. More feelings of efficacy (motivation)C = 6R = 0.17
-
Theory of planned behavior (Perkins et al., 2007) 1. More
expected value2. More subjective norms3. More perceived behavioral
control4. More Behavioral intentions5. More behaviorC = 5R =
0.20
-
Combined TheoriesC = 8R = 0.25
-
Comparing Psychological TheoriesMEASURABLE IMPROVEMENT
-
Approach Provides Editors and Reviewers With a Standard for
Evaluating TheoryPropositional Analysis to determine Complexity and
Robustness Theories of higher Complexity are privileged Theories of
higher Robustness are privilegedTheoryReview
-
To Improve, We Must Push Theories of Psychology to Higher Levels
of Complexity and Robustness
-
Without Rigorous Standards We Become Biased and Our Theories
Become SimpleTheory of motivation
Change in Complexity
90 years of (de) evolution
150 years until it vanishes?!?
Chart1
10
8
13
6
6
7
10
6
7
Complexity
Sheet1
AuthorYearComplexityRobustnessNumber of citationsDecadeNumber of
articles within DecadeList of all AspectsNOTESClarity of
WritingOrigins of theoryYearAtomistic LogicsLinear LogicsCircular
LogicsBranching LogicsConcatenated LogicsNumber of
DiagramsComplexityRobustness multiplied by 10Use of Diagrams?
Perrin1923100111921-19303Fuzzy languagesynthesizes three extant
theories1923130100100No
Mowrer1938801061931-19404Fuzzy languagesome experimental results
have led to rethinking of theory193803010080No
Maslow194313070181941-19505Maslow may be highly cited because
the high number of atomistic statements in his theory allow/invite
others to interpret and link the atomistic logics into more complex
ones.Extant theory, some experimental influence, mostly from
clinical experience1943420200130No
White195960.1752031951-19607Discontent with extant theories of
motivation based on drives - striving to find new insights through
synthesis of multiple trends of thought1959011010617No
Valenstein, Cox, & Kakolewski1970602471961-19709Based on
experiment197005000060No
Solomon & Corbit197470.299991971-19805Diagram is
"illustrative" different from textEmprical and observational of
daily events1974010221729Yes
Weiner1985130.0863371981-19905Detail LevelAbove averageSynthesis
of multiple theories1985030221014.5Yes
Weiner198570.2163371981-19905High LevelAbove averageSynthesis of
multiple theories199100010060Yes
Markus & Kitayama19916086391991-20002Markus & Kitayama -
similar to maslow have a model that is more open to interpretation
- so many cites (BAD for advancing the science, good for advancing
personal careers - the selfish scientistAbove
averageInterdisciplinary - connecting psychology with (cultural)
anthropology2004030012721No
Baker, et al.200460.174792000-20103Figure 1Quite
clearGeneralizations of previous studies and theoriesYes
Baker, et al.200480.254792000-20103Figure 2Quite
clearGeneralizations of previous studies and theoriesYes
Worst clarity is from complex, exception-ridden, run-on
discussions
Best clarity is from clear hypotheses/propositions
ROBUSTNESS TIMES 10
YearComplexityRobustness
1923100
193880
1943130
195960.17
197060
197470.29
1985100.145
199160
200470.21
WITH AVERAGED VALUESWITH AVERAGED VALUES
YearComplexityRobustnessYearComplexityRobustness
19231001923100
193880193880
19431301943130
195960.17195960.17
197060197060
197470.29197470.29
average1985100.1451985100.145
199160199160
average200470.021200470.021
10070.1410070.14
155060.17155060.17
1600140.211600140.21
1750110.361750110.36
178531178531
Because I'm looking for the C&R of each paper's contribution
to the theoretical literature, I averaged the C&R when two
models were presented (Baker, et al) and when two interpretations
of a model seemed required ( Weiner's model could be viewed at high
lBecause I'm looking for the C&R of each paper's contribution
to the theoretical literature, I averaged the C&R when two
models were presented (Baker, et al) and when two interpretations
of a model seemed required ( Weiner's model could be viewed at high
l
For Robustness, there does not appear to be a meaningful
trend.For Robustness, there does not appear to be a meaningful
trend.
For Complexity, there appears to be a trend toward
simplicity.For Complexity, there appears to be a trend toward
simplicity.
We are not merely wandering, but appear to be following the
guide of parsimony - a guide whose treustworthiness is gravely in
doubtWe are not merely wandering, but appear to be following the
guide of parsimony - a guide whose treustworthiness is gravely in
doubt
Meehl, 2002 p. 345 "they only one [criteria of theory appraisal]
they all mention (except Poperians) is parsimony"Meehl, 2002 p. 345
"they only one [criteria of theory appraisal] they all mention
(except Poperians) is parsimony"
This is a mere 100 years compared with the thousands of the
previous studyThis is a mere 100 years compared with the thousands
of the previous study
KEY - Meehl makes suggestions to determine which theories are
best based (in part) on application. But HOW do we choose which
ones to apply?KEY - Meehl makes suggestions to determine which
theories are best based (in part) on application. But HOW do we
choose which ones to apply?
AND - that only tells us which ones are best AFTER we already
know it works. SO what I'm doing is saying which direction shoud we
go to find those theories that work!AND - that only tells us which
ones are best AFTER we already know it works. SO what I'm doing is
saying which direction shoud we go to find those theories that
work!
You must avoid the tank-trap of parsimony and climb the
difficult mountain of complexity before you reach the fertile vale
of RobustnessYou must avoid the tank-trap of parsimony and climb
the difficult mountain of complexity before you reach the fertile
vale of Robustness
Sheet1
00
00
00
00
00
00
00
00
00
Complexity
Robustness
Year
Complexity and Robustness of Psychological Theories Over
Time
Sheet2
00
00
00
00
00
00
00
00
00
Complexity
Robustness
Sheet3
000000000
000000000
000000000
000000000
000000000
000000000
000000000
000000000
000000000
Atomistic Logics
Linear Logics
Circular Logics
Branching Logics
Concatenated Logics
Number of Diagrams
Complexity
Robustness multiplied by 10
0
0
0
0
0
0
0
0
0
Complexity
MBD009182BC.xls
Chart2
0.140.14
0.170.17
0.210.21
0.360.36
11
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet1
YearTotal Number of AspectsNumber of Concatenated
AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
MBD0091876A.xls
Chart1
71
61
143
114
33
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet1
YearTotal Number of AspectsNumber of Concatenated
AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
-
Without Rigorous Standards We Become Biased and Our Theories
Become WeakTheory of motivation
Change in Robustness
90 years of (weak) evolution
900 years to success?!?
Chart2
0
0
0
0.17
0
0.29
0.145
0
0.021
Robustness
Year
Sheet1
AuthorYearComplexityRobustnessNumber of citationsDecadeNumber of
articles within DecadeList of all AspectsNOTESClarity of
WritingOrigins of theoryYearAtomistic LogicsLinear LogicsCircular
LogicsBranching LogicsConcatenated LogicsNumber of
DiagramsComplexityRobustness multiplied by 10Use of Diagrams?
Perrin1923100111921-19303Fuzzy languagesynthesizes three extant
theories1923130100100No
Mowrer1938801061931-19404Fuzzy languagesome experimental results
have led to rethinking of theory193803010080No
Maslow194313070181941-19505Maslow may be highly cited because
the high number of atomistic statements in his theory allow/invite
others to interpret and link the atomistic logics into more complex
ones.Extant theory, some experimental influence, mostly from
clinical experience1943420200130No
White195960.1752031951-19607Discontent with extant theories of
motivation based on drives - striving to find new insights through
synthesis of multiple trends of thought1959011010617No
Valenstein, Cox, & Kakolewski1970602471961-19709Based on
experiment197005000060No
Solomon & Corbit197470.299991971-19805Diagram is
"illustrative" different from textEmprical and observational of
daily events1974010221729Yes
Weiner1985130.0863371981-19905Detail LevelAbove averageSynthesis
of multiple theories1985030221014.5Yes
Weiner198570.2163371981-19905High LevelAbove averageSynthesis of
multiple theories199100010060Yes
Markus & Kitayama19916086391991-20002Markus & Kitayama -
similar to maslow have a model that is more open to interpretation
- so many cites (BAD for advancing the science, good for advancing
personal careers - the selfish scientistAbove
averageInterdisciplinary - connecting psychology with (cultural)
anthropology2004030012721No
Baker, et al.200460.174792000-20103Figure 1Quite
clearGeneralizations of previous studies and theoriesYes
Baker, et al.200480.254792000-20103Figure 2Quite
clearGeneralizations of previous studies and theoriesYes
Worst clarity is from complex, exception-ridden, run-on
discussions
Best clarity is from clear hypotheses/propositions
ROBUSTNESS TIMES 10
YearComplexityRobustness
1923100
193880
1943130
195960.17
197060
197470.29
1985100.145
199160
200470.21
WITH AVERAGED VALUESWITH AVERAGED VALUES
YearComplexityRobustnessYearComplexityRobustness
19231001923100
193880193880
19431301943130
195960.17195960.17
197060197060
197470.29197470.29
average1985100.1451985100.145
199160199160
average200470.021200470.021
10070.1410070.14
155060.17155060.17
1600140.211600140.21
1750110.361750110.36
178531178531
Because I'm looking for the C&R of each paper's contribution
to the theoretical literature, I averaged the C&R when two
models were presented (Baker, et al) and when two interpretations
of a model seemed required ( Weiner's model could be viewed at high
lBecause I'm looking for the C&R of each paper's contribution
to the theoretical literature, I averaged the C&R when two
models were presented (Baker, et al) and when two interpretations
of a model seemed required ( Weiner's model could be viewed at high
l
For Robustness, there does not appear to be a meaningful
trend.For Robustness, there does not appear to be a meaningful
trend.
For Complexity, there appears to be a trend toward
simplicity.For Complexity, there appears to be a trend toward
simplicity.
We are not merely wandering, but appear to be following the
guide of parsimony - a guide whose treustworthiness is gravely in
doubtWe are not merely wandering, but appear to be following the
guide of parsimony - a guide whose treustworthiness is gravely in
doubt
Meehl, 2002 p. 345 "they only one [criteria of theory appraisal]
they all mention (except Poperians) is parsimony"Meehl, 2002 p. 345
"they only one [criteria of theory appraisal] they all mention
(except Poperians) is parsimony"
This is a mere 100 years compared with the thousands of the
previous studyThis is a mere 100 years compared with the thousands
of the previous study
KEY - Meehl makes suggestions to determine which theories are
best based (in part) on application. But HOW do we choose which
ones to apply?KEY - Meehl makes suggestions to determine which
theories are best based (in part) on application. But HOW do we
choose which ones to apply?
AND - that only tells us which ones are best AFTER we already
know it works. SO what I'm doing is saying which direction shoud we
go to find those theories that work!AND - that only tells us which
ones are best AFTER we already know it works. SO what I'm doing is
saying which direction shoud we go to find those theories that
work!
You must avoid the tank-trap of parsimony and climb the
difficult mountain of complexity before you reach the fertile vale
of RobustnessYou must avoid the tank-trap of parsimony and climb
the difficult mountain of complexity before you reach the fertile
vale of Robustness
Sheet1
0
0
0
0
0
0
0
0
0
Robustness
Year
Complexity and Robustness of Psychological Theories Over
Time
Sheet2
00
00
00
00
00
00
00
00
00
Complexity
Robustness
Sheet3
000000000
000000000
000000000
000000000
000000000
000000000
000000000
000000000
000000000
Atomistic Logics
Linear Logics
Circular Logics
Branching Logics
Concatenated Logics
Number of Diagrams
Complexity
Robustness multiplied by 10
0
0
0
0
0
0
0
0
0
Complexity
MBD009182BC.xls
Chart2
0.140.14
0.170.17
0.210.21
0.360.36
11
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet1
YearTotal Number of AspectsNumber of Concatenated
AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
MBD0091876A.xls
Chart1
71
61
143
114
33
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet1
YearTotal Number of AspectsNumber of Concatenated
AspectsRobustness
100710.14Plutarch
1550610.17Cardan
16001430.21Gilbert
17501140.36Two fluid theory
1785331Coulomb
YearRobustness
1000.14
15500.17
16000.21
17500.36
17851
Sheet1
00
00
00
00
00
Robustness
Year
Robustness
Theories of Electrostatic Attraction
Sheet2
00
00
00
00
00
Total number of Aspects
Number of Concatenated Aspects
Year
Number of Aspects
Change in Aspects Over Time
Sheet3
-
To ConcludeEmpirical data is theory laden we cannot understand
data without understanding theory.If we are make psychology more
beneficial to humanity we must directly address theory.If we are to
have effective theory, that theory must be addressed in a rigorous
scientific way (Critical Integrative Metatheory)Those methods must
be applied to review submissions to journals
-
Many Thanks!Steven E. Wallis, Ph.D.
Director, Foundation for the Advancement of Social Theory
http://projectfast.org Fellow, Institute for Social Innovation,
Fielding Graduate UniversityAdjunct Faculty, Capella University
Editorial Board, Integral Review
[email protected] Graduate University is a 501 c
(3) non-profit university of higher learning.
Source: Wallis (building on work of Dubin, Kaplan,
Stinchcombe)Explanation: Based on causal relationships and
concatenation. Here are four components, one is concatenated
(resultant from two or more causal components) So, Robustness
(measure of structure) is one divided by four = 0.25. Strength:
Repeatable, provides objective path for advancing theory.
Robustness has been related to efficacy in practical
application.Limit: Based on ideas of integral thinking and
complexity theory which have not themselves been proved
