Methodology: a real problem

Methodology: a real problemProfessor David (Dai) GriffithsThe Institute for Educational CyberneticsThe University of Bolton

[email protected]

A methodological journey● I am from the humanities, but working with

technology● I will talk about

○ the methodological approaches that have influenced me

○ some examples of problems that worry me● I will not attempt to give a complete history of

methodology in 30 minutes● I don’t want to upset anyone, but no doubt I will!

Caught between art and science

The arts focus on personal experience. I wanted to understand how that related to an external worldoriginating in or based on observation or experience <empirical data>capable of being verified or disproved by observation or experiment <empirical laws>But I wanted to avoid naive realism...“Empirical” in the Merriam-Webster Dictionary

Some answers (or at least questions) from the 1700’s

1748. David Hume, An Enquiry Concerning Human Understanding

● “What is the nature of all our reasonings concerning matter of fact?”

– “the relation of cause and effect”

● “What is the foundation of all our reasonings and conclusions concerning that relation?”

– “Experience“

● “What is the foundation of all conclusions from experience?”

– “priority in time, proximity in space, and necessary connection”

● How does necessary connexion arise

– “from a number of similar instances which occur of the constant conjunction of these events”

Empiricist methodology● In the past 200 years methodologies based on

empiricism have triumphed– Observe regularities (conceived in different

ways)– In the data are facts, or refutations, terlinked

facts about the world emerge from the data (or disproven hypotheses)

● Empirical methodologies demonstrated their success by manipulating the world

Empiricism demonstrated its success with powerful technologies

Social and political questions...● In empiricism (and its close cousin positivism),

there is no room for social or political understanding

● Kuhn’s Structure of Scientific Revolutions – Showed that scientific understanding does not proceed

by gradual uncovering of truth

– Large scale shifts of paradigm change the way that scientists see the world

Critiques from a (broadly) Marxist perspective

● If science is a social activity, how far do power and influence determine what is true?

● The tobacco industry undermined my confidence (echoed by climate science, the claims for genetics)

● Two writers who impressed me● Gould (1981) The Mismeasure of Man

(Excellent on statistics)● Rose (2013) Cells and Brains: The Promethean

Promises of the New Biology

Paradox undermined my confidence further

Russell: Does the baber shave himself?Answer: it is nonsense.

But Goedl showed...● if you have a self-consistent recursive axiomatic

system powerful enough to describe the arithmetic of the natural numbers

● Then there are true propositions about the naturals that cannot be proved from the axioms.

This was deeply shocking.It showed there were limits to what could be achieved by empirical logic.

Recursion is a deep problem● Hume

The mind has never anything present to it but the perceptions, and cannot possibly reach any experience of their connexion with objects. Hume (119)

● Empiricism has triumphed because as humans we have very similar experiences of the physical world

● As social beings we have different experiences of the world

● When we study ourselves, we do not find that shared facts emerge from our observations

Cybernetics: an alternative set of theories with a focus on recursion

● Alternative ways of conceptualising information. (Bateson, Shannon, McCulloch)

● Modelling and conceptualising self reference (Maturana, Luhmann, Leydesdorff, Beer)

● A performative (Pickering) method of experimentation, modelling (e.g. Pask, Beer, Checkland) and abduction (e.g. Bateson)

● Very helpful in understanding empiricism in the context of lived experience and professional practice

● I chose ‘Professor of Educational Cybernetics’ as my title

I was also drawn to...

● Phenomenology: Denying the relevance of the empirical to the human. Phenomenology “has, as its exclusive concern, experiences intuitively seizable and analysable in the pure generality of their essence” (Hussrl 1913, in Moran 1990)

● Grounded theory: (Glaser & Strass) Accepting the empirical, but ‘coding’ to emerging local theoretical framework from the data

● Action Research (Lewin, Argyris, Heron). Research applied to achieve a desired social change, with a spiral of progressive improvement.

I have been involved in all three, but...

● None of phenomenology, grounded theory or action research have a strong method for cumulation of results

● This makes it hard, or impossible, to make sense of the technological or social interventions

● So maybe empirical, positivist approaches are the right way to go in social science...

The Empirical strikes back

US Dept. of Education, project requirements, 2003“… to determine whether the project produces meaningful effects on student achievement or teacher performance.”“Evaluation methods using an experimental design are best for determining project effectiveness. Thus, the project should use an experimental design under which participants--e.g., students, teachers, classrooms, or schools--are randomly assigned to participate in the project activities being evaluated or to a control group that does not participate in the project activities being evaluated.”

Evidence based policy● Quantitative experimental methodologies with

control groups ○ confirm that managerial control is possible○ provide defensible answers for managers

● But do they always provide the right answers?● I suggest you read Seddon on this

Seddon (2008) Systems Thinking in the Public Sector: The Failure of the Reform Regime.... and a Manifesto for a Better Way

Big money is at stake. Is education like the climate, and tobacco?

● McGraw Hill: “Using LearnSmart to study has proven to lead to improved learning efficiency, greater engagement, and better career readiness” http://chronicle.com/items/biz/pdf/McGraw-Hill_LearnSmart-Effectiveness-Study.pdf

● Pearson: “More than 6 million students around the world are now using Pearson MyLab & Mastering products. … Integrated usage of these programs has shown to provide measurable gains in student retention, subsequent success, and overall achievement”

● http://www.pearson.com.au/why-pearson/technology-learning/mylabsmastering/proven-results/

Gregory Bateson on the dormitive principle

“ A common form of empty explanation is the appeal to what I have called ‘dormitive principles’, borrowing the word dormitive from Molière. There is a coda in dog Latin to Molière’s Le Malade Imaginaire, and in this coda, we see on the stage a medieval oral doctoral examination. The examiners ask the candidate why opium puts people to sleep. The candidate triumphantly answers, ‘Because, learned doctors, it contains a dormitive principle’.”Gregory Bateson, 2002. Mind and Nature: A Necessary Unity. Hampton Press, Cresskill NJ. p 80.

The dormitive principle in the classroom

Mea culpaWhere is the pedagogic goodness of a good lesson located in a lesson? If we can capture it, we can share it!

I looked for the teachers activity and tried to capture that with IMS LD. Others look for it in content or curricula.

We can test for it, assess it, publish it, and average it. But it is an idealisation, with no criteria for causal efficacy.

More generallyWhy does that child learn more than the other?

Because, learned doctors, they contain more or less capacity to learn. Now let’s go for our tribunal lunch.

Three domains of the real, according to Bhaskar & Critical Realism

The REAL: Mechanisms and structures with enduring properties

The ACTUAL: events (and non-events) that are generated by the mechanisms

The EMPIRICAL: events that are actually observed and experienced

Adapted from Mingers, J. & Brocklesby, J., 1997. Multimethodology: Towards a framework for mixing methodologies. Omega, 25(5), pp.489–509.

Informed by Critical Realism...● Take an unexplained phenomenon ● Propose hypothetical mechanisms that would, if

they existed, generate or cause the phenomenon which we are trying to explain

● Not a traditional empirical approach because ○ The explanation does not emerge from the data○ The data is not the phenomenon to be

explained● Go from experiences in the empirical domain to

possible structures in the real domain. ● Competing explanations supported / eliminated● Compatible with the performative cybernetic

approach

Pawson and Tilley apply this to social science evaluation

From Pawson, R. & Tilley, N., 1997. Realistic Evaluation, Sage Publications Inc.Page 72

By what method do we observe outcomes?

● When trying to confirm or deny a mechanism, use as many different methods as you like.

● Statistics are valuable, especially with big data and analytics. They serve to confirm or rule out mechanisms, but do not give direct access to them

● Multiple methodologies create epistemological contradictions which will need to be handled with care

● The mechanism provides a means for cumulation● I recommend Mingers Realising Systems Thinking:

Knowledge and Action in Management Science

Market equilibrium theory: Goldman Sachs report, 2004

“The development of the capital markets has provided significant benefits to the average citizen. Most importantly, it has led to more jobs and higher wages… The capital markets have also acted to reduce the volatility of the economy. Recessions are less frequent and milder when they occur. As a result upward spikes in the unemployment rate have occurred less frequently and have become less severe.”W. Dudley, US Chief Economist Goldman Sachs.

R.G Hubbard, Dean, Columbia Business School

An epidemic

An epidemic

http://guardianlv.com/2013/11/adhd-becomes-an-epidemic/



Visser et al. Nov. 2013provide figures for 2011

● Approx 2 million more U.S. 4 to 17 year olds diagnosed with ADHD in 2011 than 2003.

● Taking medication for ADHD○ 69% of children with current ADHD○ 6.1%, of all children: 3.5 million children

● Medicated ADHD increased by 28% from 2007 to 2011.

Response to this data...

● Dr. John Walkup, Director Child & Adolescent Psychiatry at Weill Cornell Medical College

● ... the data suggests the increasing diagnosis rate of ADHD is getting closer to the true prevalence of ADHD, which is even higher.○ "We've been working so hard for so long to

improve treatment… If the prevalence rate is 9 to 11% and we're getting 8% currently diagnosed, it suggests that the public advocacy for treatment is paying off."

http://thechart.blogs.cnn.com/2013/11/22/adhd-diagnoses-rise-to-11-of-kids/comment-page-2/




● Dr. John Walkup, Director Child & Adolescent Psychiatry at Weill Cornell Medical College

● ... the data suggests the increasing diagnosis rate of ADHD is getting closer to the true prevalence of ADHD, which is even higher.○ "We've been working so hard for so long to

improve treatment… If the prevalence rate is 9 to 11% and we're getting 8% currently diagnosed, it suggests that the public advocacy for treatment is paying off."





● Dr. Allen Frances, former chair of Psychiatry Department at Duke University○ Psychiatry... is a history of fads, and we are

now suffering from a fad of ADHD○ the rates have tripled over the past 15 years

because of sales pressure from pharmaceutical companies selling stimulants to treat ADHD.

○ "We are medicalizing immaturity and turning childhood into a disease," Frances said.http://thechart.blogs.cnn.com/2013/11/22/adhd-diagnoses-rise-to-11-of-kids/comment-page-2/



Visser et al. conclude...

● “Efforts to further understand ADHD diagnostic and treatment patterns are warranted”

● How could we resolve this debate?● As researchers you need to be able to take a

position on this kind of debate, within your field and beyond

● The implications cut deep into your work. I suggest that you think hard about this, not just apply a recipe.