TC 1

The Chimera of Validity

Bringing the Validity Conversations Home: When

Educational Measures Go Public

Eva L. Baker CRESST National Center for Research on Evaluation, Standards, & Student Testing UCLA

Educational Assessment, Accountability, and Equity: Conversations on Validity Around the World

Teachers College, Columbia University 29 March 2012, New York

© Regents of the University of California

TC 2

The Chimera of Validity

© Regents of the University of California

TC 3 © Regents of the University of California

TC 4

Quality Validity Zen and the Art of Motorcycle Maintenance

R.M. Pirsig, 1974

“Real Quality must be the source of the subjects and objects, the cone from which the tree must start.”

© Regents of the University of California

TC 5

Common Language Validity

1. Well grounded, sound, supportable

2. Producing the desired results; efficacious

3. Legally sound and effective; incontestable; binding

4. a. Containing premises from which the conclusion may be logically derived

b. Correctly inferred or deduced from a premise as a conclusion

© Regents of the University of California

TC 6

Validate!

• To declare or make legally valid;

• To mark (like a transfer or passport);

• To substantiate and verify (decisive, conclusive, capacity to resist attack, capable of ending a debate)

American Heritage Dictionary of the

English Language, 1981, p. 1414)

© Regents of the University of California

TC 7

Conflicting Interpretations

Public & Policy Makers

Testing Professionals

Users

© Regents of the University of California

TC 8 © Regents of the University of California

TC 9

Reasons for Early Scholarly Studies

• Thorndike

• Exploration of human capacities

• Uses for selection

• Normal distributions as foundation of design and interpretation

• Largely assumed stability (fundamentally not subject to change by schools or programs)

“A test is valid for anything it correlates with” Guilford, 1946

© Regents of the University of California

TC 10

Faces of Key Testing Innovators • Ralph Tyler • Art Lumsdaine • B.F. Skinner • E.F. Lindquist • Ben Bloom • Chester Harris • Lee Cronbach • Sam Messick • Bob Glaser • Ed Gordon

© Regents of the University of California

TC 11

Standards for Educational and Psychological Testing

© Regents of the University of California

TC 12

“Standards”

• Mid 20th century

• 1999 APA-AERA-NCME sponsored

• Synthesis of accepted state of knowledge and best practice

• Not “aspirational”

• Serve all areas in education and psychology using testing—neurology, clinical, etc.

© Regents of the University of California

TC 13

“Standards” Cont’d

• Revised every decade or so

• Documentary

• Guidance–not enforceable by organizations

• Power through legal precedent, prophylactic

© Regents of the University of California

TC 14

Emerging Notions of Validity

• Unitary concept • Not types of validity, like criterion, predictive

• Purpose-driven

• Validity argument–theory & evidence Vs. claims

• Assessments designed to measure learning rather than traits

© Regents of the University of California

TC 15

Key Elements of the 1999 Standards

• Enormous effort to involve the field

• Validity as unitary concept, argued with a range of theory, designs and data

• Purpose and claims

• Guidance for assessment designers involving cognitive demands and other test attributes

© Regents of the University of California

TC 16

1999 Standards

• More standards • Longer chapters to clarify concepts • Fairness chapter and standards • Weak technology chapter • Thin policy chapter • No rating scheme for Standards, choices

depended on professional judgment • Not universally acclaimed

© Regents of the University of California

TC 17

“Going Public” What Happened?

• Given my hopes, not much

• Swamped by NCLB requirements

• Ambitious schedules

for developers

• Restricted testing time

• BOTA

• Cost, cost, cost

© Regents of the University of California

TC 18

Making a Validity Argument • Consequences

• Meaningfulness

• Fairness

• Cognitive complexity

• Content quality

• Transfer and generalizability

• Instructional sensitivity

• Cost and efficiency

© Regents of the University of California

Linn, Baker, & Dunbar, 1991

TC 19

Key Practical Barriers

• Clarity of standards

• Technical quality of complex tasks

• Evidence of transfer and generalized learning

• Time to conduct studies before use

• Scalable evidence of classroom or other learning experiences

• Confounded by accountability

© Regents of the University of California

TC 20

Challenge of Multiple Purposes

• Assessments for certification, progress and improvement

• Assessments of certification and admissions

• Assessments of children used to determine adult performance (teacher evaluation)

• Research use as dependent measures

© Regents of the University of California

TC 21

Looking to New CCSSs

• Carefully designed–many standards

• Consortia specifications of useful assessments

• Graphics to clarify meaning and relationships to avoid narrowness and trivial alignment • Maps, blueprints, ontologies

• Strategies for data management and validity studies

© Regents of the University of California

TC 22

Ontology Design

Specifications

Relevant texts

Best Practices

Expert judgment

Principle

Procedure

Fundamental principles

Includes components

Concept

requires requires

definition

related-to

DO

MA

INS

© Regents of the University of California

TC 23

My Picks • Unitary validity argument

• Process measures of learning

• Evidence that assessment scores will rise because of high quality instruction/learning

• Translational graphics

• Interrupting predictions

• Choice by students, as in badges and qualifications

© Regents of the University of California

TC 24

Going Public in the Future

• Short term questions • Who is taking the test?

• Hacking

• Favorite solution: Zacharias–Give up security • Publish everything in advance, menu to guide

choice

© Regents of the University of California

TC 25

Technology Now and Soon • Technology affects behaviors, brain function,

preferences

• Personalized, point-to-point communication

• Crowdsourcing Vs. expertise

• Desire to be noticed–metric: millions over weeks

• Constrained choices

• Weakened formal institutions

© Regents of the University of California

TC 26

Everyone Can Know Everything About You

• Data capture from multiple sources (sensors, cameras, phone)

• Integrated profiles of individuals

• Achievement, tacit preferences or psychological states

• Reports

• Privacy, personal choice, new institutions

• Generational problem

© Regents of the University of California

TC 27

To Educate for the Unpredictable

• Focus on challenging skill learning in content: Adaptive problem solving, situation awareness, risk, decisions, metacognitive skills, social and collaborative skills

• Transfer and generalization tasks–learning to learn

• Emphasize choice

© Regents of the University of California

TC 28

Validity? • Pertaining to skills

• Managed by different institutions, corporations (who are just like people)

• Quality and mobility issues

• Competition

• Adaptability rather than prediction

© Regents of the University of California

TC 29

Why the Chimera?

• Imagine options

• Imagine validity as a hybrid

• Chimeras are scary

© Regents of the University of California

TC 30

http://www.cse.ucla.edu

Eva L. Baker

eva@ucla.edu 310-206-1530

email

© Regents of the University of California

TC 31

Credits for Figures • FACE:

http://www.google.com/imgres?q=face+child+black&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvns&tbnid=CXdt2WKPZy5gDM:&imgrefurl=http://newsone.com/the-education-zone/associatedpress3/education-secretary-says-minority-students-face-harsher-discipline/&docid=urHjqFA9_HP0VM&imgurl=http://newsone.com/files/2012/03/black-kids-face-harsher-treatment-in-school-discipline-.jpg&w=640&h=380&ei=rWtqT43vAoSTiAKN6NjIBQ&zoom=1&iact=hc&vpx=878&vpy=496&dur=189&hovh=173&hovw=291&tx=187&ty=109&sig=115080220995703348838&page=2&tbnh=144&tbnw=209&start=36&ndsp=47&ved=1t:429,r:4,s:36

• http://www.google.com/imgres?q=face&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsa&tbnid=W3X2sZNZGUwm0M:&imgrefurl=http://freeestuffs.com/megan-fox/megan-fox-face-758/&docid=BJzuVhbicsGP6M&imgurl=http://freeestuffs.com/wp-content/uploads/2011/12/Megan-Fox-face-758.jpg&w=1600&h=1200&ei=ZmhqT5LsJYfYiQKQlPD4BA&zoom=1

• http://www.google.com/imgres?q=face+child+asian+american&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsb&tbnid=W2zGjs11BeRUcM:&imgrefurl=http://blog.world-mysteries.com/science/how-many-major-races-are-there-in-the-world/&docid=WiG9D5oVBXB5SM&imgurl=http://blog.world-mysteries.com/wp-content/uploads/2011/02/asian-child.jpg&w=425&h=282&ei=bG1qT_CBKsHciAKk2MXSBQ&zoom=1

• http://www.google.com/imgres?q=face+latina+girl&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvns&tbnid=emyPNGe6zfReFM:&imgrefurl=http://stockshop.wordpress.com/&docid=czgeS27bjgGMqM&imgurl=http://stockshop.files.wordpress.com/2010/03/hispanic-girl-smile.gif&w=433&h=288&ei=5G9qT9zBA6mqiQLz3vS7BQ&zoom=1&iact=hc&vpx=1279&vpy=645&dur=1833&hovh=183&hovw=275&tx=201&ty=130&sig=115080220995703348838&page=1&tbnh=151&tbnw=207&start=0&ndsp=43&ved=1t:429,r:15,s:0

© Regents of the University of California

TC 32

Credits for Pictures • Glaser: http://www.fabbs.org/index.php?cID=351

• Cronbach: http://oaks.nvg.org/lee-cronbach.html

• Gordon: http://www.google.com/imgres?q=edmund+gordon&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnso&tbnid=qPa0aELm65205M:&imgrefurl=http://www.tc.columbia.edu/news.htm%3FarticleID%3D7866&docid=QIHvISuu5URgSM&imgurl=http://www.tc.columbia.edu/i/media/7866_EdmundGordon.jpg&w=250&h=364&ei=qHVqT8GjCeeWiAL-0IiBBQ&zoom=1&iact=hc&vpx=185&vpy=121&dur=6186&hovh=271&hovw=186&tx=79&ty=154&sig=115080220995703348838&page=1&tbnh=154&tbnw=116&start=0&ndsp=39&ved=1t:429,r:0,s:0

• Bloom: http://www.google.com/imgres?q=benjamin+bloom&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsbol&tbnid=4KSwaC6jMQtOZM:&imgrefurl=http://www.smartkpis.com/performance_management_resources/Education_and_Learning/Bloom-Benjamin-1913-1999-&docid=R1czzBRamLn5eM&imgurl=http://www.smartkpis.com/photos/learning/Bloom-Benjamin-smartkpis-photo-12.com.jpg&w=200&h=300&ei=JHRqT-akLcWjiALChfGCBQ&zoom=1&iact=hc&vpx=522&vpy=622&dur=361&hovh=240&hovw=160&tx=59&ty=150&sig=115080220995703348838&page=1&tbnh=164&tbnw=109&start=0&ndsp=37&ved=1t:429,r:9,s:0

• Tyler: http://www.google.com/imgres?q=ralph+tyler&hl=en&client=safari&sa=X&rls=en&biw=1593&bih=986&tbm=isch&prmd=imvnsbo&tbnid=u8pZGDhDp4h8GM:&imgrefurl=http://www.ecs.org/html/aboutECS/AwardsWinners.htm&docid=R7w7sBgPzz-qPM&imgurl=http://www.ecs.org/html/aboutECS/Photos/RalphTyler.jpg&w=115&h=156&ei=i3RqT-GgF8GfiALd_ozQBQ&zoom=1&iact=hc&vpx=987&vpy=228&dur=5253&hovh=124&hovw=92&tx=64&ty=137&sig=115080220995703348838&page=1&tbnh=124&tbnw=92&start=0&ndsp=45&ved=1t:429,r:6,s:0

© Regents of the University of California

TC 33

Credits for Figures • http://gpaper139.112.2O7.net/b/ss/gpaper139,gntbcstglobal/1/H.3--NS/0" height="1" width="1"

border="0" alt="" />

• http://www.education.com/reference/article/thorndike-edward-lee-

• Persig: http://reelwamps.com/wp-content/uploads/2011/08/ChimeraWeb.jpg

• Test standards: teststandards.org

• Early test standards Amazon

• http://c2.ac-images.myspacecdn.com/images02/129/l_0157a4bec6d648a482fafb71d28befa9.jpg

• http://reelwamps.com/wp-content/uploads/2011/08/ChimeraWeb.jpg

• http://www.bates-communications.com

• http://www.frontdoor.com

• http://www.bates-communications.com

• http://library.thinkquest.org/04oct/00492/Quiz_pencil.jpg

• http://crooksandliars.com/files/vfs/2010/11/gavel.jpg

• http://zeusiswatching.files.wordpress.com/2010/10/100_2816.jpg?w=510

© Regents of the University of California