Fisher2012 Jiaxing China Keynote PROMS

Scientific and Economic Value of the Metrological Point of View

William P. Fisher, Jr.University of California, Berkeley

Pacific Rim Objective Measurement Symposium

6-9 August 2012Jiaxing, China

Overview

• Some basic economic principles shared by science and commerce

• Three points of view on measurement in education

• The kinds of markets created by the three approaches to measurement

• A plan for the future

Economic Principles Shared ByScience and Commerce

• Separate local economies– Different currencies– Different weights and measures– Higher costs of exchange– Less efficient, harder to compare values

• Unified regional and global economies– Same currency– Same weights and measures– Lower costs of exchange– More efficient, easier to compare values

Example 1 of Scientific Market

• Biochemistry– Equipment calibrated in universal reference

standard metrics– Test results always reported in common units– Measures available on the spot– Easy to coordinate research across labs– Result: SARS virus sequenced in weeks by network

of labs, vaccine successfully synthesized

Example 2 of a Scientific Market

• Custom tailored suits– Tape measures calibrated in universal reference

standard metric– Results always reported in common units– Measures available on the spot– Easy to coordinate across tailors– Result: measures can be sent around the world

and a well fitting suit obtained with little trouble

Example 3 of Scientific Market

• Education– Tests typically not calibrated at all– If they are calibrated, they are in local units– Test results are usually reported in unique units– Measures available only after costly data analysis– Very difficult to compare outcomes outside of

special contexts– Result: Improvement efforts repeatedly fail,

quality uncontrolled, costs spiral higher

The Ideal Efficient Market

• Cost of estimating value is very low• Cost of comparing value for price is very low• Supply and demand easily match up• Low value for price: cannot compete• High value for price: rewarded• Improved value easy to recognize• Improved value pushes out old value

Readily available high quality information on product or service

High Cost Low Cost

Hard to match supply and demand

Easy to match supply and demand

Market Efficiency

Basic Economics

Hard to know how toimprove quality

Easy to know how toimprove quality

QualityImprovement

Hard for customers to find quality

Easy for customersto find quality

CustomerQuality-Seeking

Three Points of View on How to Present Information

on Educational Outcomes

• True Score Theory• Measurement Theory• Metrological Traceability

True Score TheoryDisconnected Scores and Tests

• School 1– Student A has a score of 22 on a reading test.– This classroom averages a score of 24.

• School 2– Student Z has a score of 18 on a reading test.– This classroom averages a score of 26.


• Who has more reading ability, A or Z? ??• What can one student read that the other

cannot? ??• Which classroom reads better on average? ??• Which student is more on track for college

readiness? ??


• School 1– Student A’s reading scores on 2 tests are 22 & 32.– The classroom average score goes from 24 to 30.

• School 2– Student Z’s reading scores on 2 tests are 18 & 32.– The classroom average score goes from 26 to 40.


• Who gained more in reading ability, A or Z? ??• What new texts can A and Z read? ??• Which classroom improves more? ??• Are both students on track for college

readiness? ??

• Result:– Very high cost, almost useless information

Disorganized, uncontrolled, decaying

Measurement TheoryConnected Measures and Tests

• School 1– Student A has a measure of 22 (+/- 2) on a reading

test.– This classroom averages a measure of 24 (+/- 1).

• School 2– Student Z has a measure of 18 (+/- 2) on a reading

test.– This classroom averages a measure of 26 (+/- 1).


• Who has more reading ability, A or Z? A• What can one student read that the other

cannot?– Text with measures between 18 and 22.

• Which classroom reads better on average? 2• Which student is more on track for college

readiness? ??

• School 1– Student A’s measures on 2 tests are 22 & 32 (+/- 2).– The classroom average goes from 24 to 30 (+/- 1).

• School 2– Student Z’s measures on 2 tests are 18 & 32 (+/- 2).– The classroom average goes from 26 to 40 (+/- 1).


• Who gained more in reading ability, A or Z? Z• What new texts can Z read?– Those with measures between 18 and 32.

• Which classroom improves more? 2• Are both students on track for college readiness?

??

• Result:– Very high cost, incomplete, but useful information


Organized, expressive, preserved

Metrologically Traceable Measures

• School 1– Student A’s measure (22, +/- 2) is inferred when 73%

of the items built into a reading assignment targeted at 22 are answered correctly.

– This classroom averages a measure of 24 (+/- 1).• School 2– Student Z’s measure (18, +/- 2) is inferred when 76%

of the items built into a reading assignment targeted at 18 are answered correctly.

– This classroom averages a measure of 26 (+/- 1).

Metrologically Traceable Measures

• Who has more reading ability, A or Z? A• What can one student read that the other

cannot?– Text with measures between 18 and 22.

• Which classroom reads better on average? 2• Is one student more on track for college

readiness? Yes, A

• School 1– Student A’s measures on 2 tests are 22 & 32 (+/- 2).– The classroom average goes from 24 to 30 (+/- 1).

• School 2– Student Z’s measures on 2 tests are 18 & 32 (+/- 2).– The classroom average goes from 26 to 40 (+/- 1).

Metrologically TraceableConnected Measures and Tests

• Who gained more in reading ability, A or Z? Z• What new texts can Z read?– Those with measures between 18 and 32.

• Which classroom improves more? 2• Are both students on track for college

readiness? No, but A is

• Result:– Very low cost, complete and useful information

Metrologically TraceableConnected Measures and Tests

Coordinated, harmonized, growing

What to choose?True Score Theory Economics

School 1Average Grade 7 End of Year Teacher’ QuizReading Score = 89%

Average Gain in 7th Grade Readingas measured by in-classquizzes and tests: ??

Annual tuition = US$5,000 Cost of average gain in reading scores = US$??

School 2Average Grade 7 End of Year Teacher’ QuizReading Score = 94%

Average Gain in 7th Grade Readingas measured by in-classquizzes and tests: ??

Annual tuition = US$1,000 Cost of average gain in reading scores = US$??

Not enough information to decide!

Simulated data

What to choose?Measurement Theory Economics

School 1Average Grade 7 End of Year StatewideReading Measure = 32 (+/- 6)

Adjusted average gain in7th Grade ReadingMeasures = 10 (+/- 4)

Cost of adjusted average gain in reading measures = US$5,000.00




Best buy

But do you really want to buy the average gain?

Simulated data

• My 7th grader’s gain – US$1,000 for 6 units– US$166.67 per unit gain

• Your 7th grader’s gain– US$1,000 for 9 units– US$111.11 per unit gain


50% greater cost!


ReadingAbilityScale

What to choose?Metrology Economics

Best buy

We might repeat the Measurement Theory outcomes…







Simulated data

• My 7th grader’s gain – US$833.40 for 6 units– US$138.90 per unit gain

• Your 7th grader’s gain– US$1,250.10 for 9 units– US$138.90 per unit gain

What’s a parent to choose?Metrology Economics

Same per unit cost!

Simulated data

Readily available high quality information on product or service

High Cost Low Cost

Hard for customers to find quality

Easy for customersto find quality

CustomerQuality-Seeking

Basic Economics

High stakes measurement theory

cost per test item:> US$3,000.00

Routine theory-informedmetrologically traceable

cost per test item:< US$0.01

What’s a teacher to choose?Metrology Economics

Cost per unit gain:US$180

Simulated data

Cost per unit gain:US$620

What’s a principal to choose?Metrology Economics

Three schoolsTwelve months each

A | B | C

Cost per unit gained

US$458 US$208 US$116

Simulated data

Bett

er R

eadi

ng O

utco

mes

Basic Shop Floor Questions

• What is variation trying to tell us? (Deming)• Which variations are due to common causes,

and which are due to special causes? (Shewhart)

• How far can educational outcomes be maximized, and unwanted variation reduced?

• Can variation in outcomes be reduced by bringing all students to the highest levels?

What’s needed?

• System of distributed units• Instruments measuring in uniform metrics• Predictive construct theories to bring down costs• Low cost items and administration• Immediate results• Continuous Quality Improvement (CQI) training

and tools• A culture that rewards innovation

What’s needed?

• We need commitment to a long range vision of quality education.

• But vision is not enough; we also need:– Skills– Incentives– Resources– Plans

What’s needed?

Vision + Skills + Incentives + Resources + Plan = Sustainable Change

+ Skills + Incentives + Resources + Plan = Confusion

Vision + + Incentives + Resources + Plan = Anxiety

Vision + Skills + + Resources + Plan = Resistance

Vision + Skills + Incentives + + Plan = Frustration

Vision + Skills + Incentives + Resources + = Treadmill

Adapted from Knoster, T. P., Villa, R. A., & Thousand, J. S. (2000). A framework for thinking about systems change. In R. A. Villa & J. S. Thousand (Eds.), Restructuring for caring and effective education: Piecing the puzzle together, 2nd Ed (pp. 93-128). Baltimore: Paul H. Brookes.

Disorganized, uncontrolled, decaying

Organized, expressive, preserved

Coordinated, harmonized, growing

Thank you

Documents

Fisher2012 Jiaxing China Keynote PROMS